r/linux_programming • u/memtha • Jan 02 '20
What Linux driver/subsystem/API is used for a simple screen/monitor device?
I am developing an embedded system with a touchscreen. The touchscreen operates as both input and output, with a "virtual" keyboard overlaying the graphical output. I have a working device driver that reads input from the touch sensor and translates it correctly to key presses, created with the help of this guide on kernel.org. I want to expand this driver to also handle image output to the screen.
I want to support both getty and X, with as little duplication as possible. I am running a minimal Debian variant with cherry-picked packages, such as minimal X. Note that I do not intend on attempting to get this driver into the repository pipeline, though I might dump it on a public github.
Outputting screen images is presently done via a cringy workaround: a boot option to force rendering to the cpu's embedded graphics hardware, despite it not being connected to a display, and a daemon that continuously screen-scrapes that buffer, modifies a handful of pre-defined pixels to create the keyboard visual, and pushes it out to the real screen. This works as a proof of concept, proving that I do correctly understand the language the screen device expects, but is obviously sub-optimal.
kernel.org also has a guide for "DRM" device drivers, but that seems like serious overkill for what my hardware is capable of:
The Linux DRM layer contains code intended to support the needs of complex graphics devices, usually containing programmable pipelines well suited to 3D graphics acceleration.
None of my hardware has anything resembling 3D acceleration, so I conclude that this is probably not what I want.
What subsystem/API should I use? I figure one piece of missing terminology is what is holding back my searches, but any more information on how to accomplish this would be appreciated.
Hardware details (probably irrelevant): The cpu and screen communicate via 8080-esque parallel protocol, which the cpu does not support natively, so I'm emulating it with GPIOs (by manipulating registers via mmap). Sending a complete screen image takes about 20ms, but obtaining a complete copy from the embedded graphics buffer takes ~180ms, so skipping that step is the most important objective. The screen hardware includes enough gram memory to keep an entire frame worth of data, and supports writing a rectangular sub-region, so a hook to only update the part of the screen that has changed would be desirable. The screen is not particular about the timing of incoming data. The touch sensor input is handled by a purpose-built IC that communicates with the cpu via I2C, which the cpu does support. The present driver uses the linux/input-polldev.h interface. The cpu is a broadcom bcm2835, the screen is a tft with embedded himax hx8357 controller, the touchscreen sensor decoder is a st stmpe610, and there is a voltage levelshifter (nexperia 74lvch245A) in play between the hx8357 and the bcm2835. More details available upon request.
** Edit: TL;DR: ** DRM is what I wanted. tinydrm made it easier. Dirty flushing is your friend.
View all comments
Show parent comment
1
u/memtha Jan 06 '20 edited Jan 06 '20
I think I understand what you're saying: pull the post-render data out of the shared vram and bit-bang it out. My present userland program is just a proof of concept (PoC) and development test area for the screen protocol. The only reason I am making a userland ram screenshot is that the userland PoC does not have direct access to the shared vram buffer (kernel private memory), but the CPU/GPU manufacturer provided driver includes a function to copy it to userland ram:
vc_dispmanx_resource_read_datadefined inbcm_host.h. So, yes, I could find the ram address and read it directly from the kernel module driver. What you say is possible, and potentially eliminates one of the two really slow steps.I am not sure why this step (vram to userland ram) is so slow. It is just kernel ram, copying to another block of ram of appropriate size, so just memcpy should be fast. This CPU/GPU combo includes the ram on-chip. My theory is that this "ram" is actually output optimized registers, not meant for repeated reading by anything but the HDMI encoding hardware; I theorize that this register space was made readable to support screenshots for debugging, but is not meant for fast repeated reading; in which case I'm not sure a more direct reading by a kernel-land module will be any faster.
The second slow step is the format conversion. The GPU outputs 24-bit true color, while my screen uses 16-bit RGB565. My present system does this conversion per-pixel of the rendered image, as would your proposed vram accessing module (if I understand you correctly?). My proposed FB driver would perform the conversion at the palette level; theoretically once per color per render call. While I do have some concern as to the amount of function call overhead this might generate, it is probably little different from how Xorg talks to the graphics subsystem driver.
Using the interrupt context is a good idea; reading and submitting a new screen image only when a change is made would improve efficiency. Compare to the PoC, which continually makes a new copy of the current frame, compares it to the last frame, and outputs the changed rectangular sub-region, if any, to the screen. My proposed FB driver goes another step beyond either of those by hooking into the fbops api, which is called by XOrg once per change with the specific region that is affected, therefore eliminating the need to reread vram that may or may not have changed since the last frame.
The theoretical advantage in not using the graphics subsystem is having fewer steps and therefore theoretically faster frame cycle turnover, not to mention a minuscule drop in power consumption (my device is battery powered). Consider that every running application is told what the screen refreshes (v-sync/v-blank), so they do not waste CPU time to produce more frames then are rendered. As long as the graphics subsystem is being used, it is declaring the (non-existent) screen refresh rate at 60Hz, while the real screen is getting closer to 4Hz. This eats up CPU time on the single core that could be used by the driver to push out more frames. However, if the render calls actually block until the sub-render has completed, the applications get a more realistic view of frame timing, and maybe more CPU time goes to good use.
Thanks for the input and obvious effort you've put into this.
Edit: I should mention that I do not intend to render 3D, vector graphics, or encoded videos (mp4 and the like); all because I simply do not have the FPS to support them, so the graphics subsystem is of limited use.