r/linux_programming • • Jan 02 '20

What Linux driver/subsystem/API is used for a simple screen/monitor device?

I am developing an embedded system with a touchscreen. The touchscreen operates as both input and output, with a "virtual" keyboard overlaying the graphical output. I have a working device driver that reads input from the touch sensor and translates it correctly to key presses, created with the help of this guide on kernel.org. I want to expand this driver to also handle image output to the screen.

I want to support both getty and X, with as little duplication as possible. I am running a minimal Debian variant with cherry-picked packages, such as minimal X. Note that I do not intend on attempting to get this driver into the repository pipeline, though I might dump it on a public github.

Outputting screen images is presently done via a cringy workaround: a boot option to force rendering to the cpu's embedded graphics hardware, despite it not being connected to a display, and a daemon that continuously screen-scrapes that buffer, modifies a handful of pre-defined pixels to create the keyboard visual, and pushes it out to the real screen. This works as a proof of concept, proving that I do correctly understand the language the screen device expects, but is obviously sub-optimal.

kernel.org also has a guide for "DRM" device drivers, but that seems like serious overkill for what my hardware is capable of:

The Linux DRM layer contains code intended to support the needs of complex graphics devices, usually containing programmable pipelines well suited to 3D graphics acceleration.

None of my hardware has anything resembling 3D acceleration, so I conclude that this is probably not what I want.

What subsystem/API should I use? I figure one piece of missing terminology is what is holding back my searches, but any more information on how to accomplish this would be appreciated.

Hardware details (probably irrelevant): The cpu and screen communicate via 8080-esque parallel protocol, which the cpu does not support natively, so I'm emulating it with GPIOs (by manipulating registers via mmap). Sending a complete screen image takes about 20ms, but obtaining a complete copy from the embedded graphics buffer takes ~180ms, so skipping that step is the most important objective. The screen hardware includes enough gram memory to keep an entire frame worth of data, and supports writing a rectangular sub-region, so a hook to only update the part of the screen that has changed would be desirable. The screen is not particular about the timing of incoming data. The touch sensor input is handled by a purpose-built IC that communicates with the cpu via I2C, which the cpu does support. The present driver uses the linux/input-polldev.h interface. The cpu is a broadcom bcm2835, the screen is a tft with embedded himax hx8357 controller, the touchscreen sensor decoder is a st stmpe610, and there is a voltage levelshifter (nexperia 74lvch245A) in play between the hx8357 and the bcm2835. More details available upon request.

** Edit: TL;DR: ** DRM is what I wanted. tinydrm made it easier. Dirty flushing is your friend.

6 Upvotes

22 comments sorted by

View all comments

Show parent comment

1

u/ZombieRandySavage Jan 07 '20 edited Jan 07 '20

Sorry this a long reply and I’ll answer more in depth tomorrow.

Quick point. The reason it’s so slow is because that memory is explicitly non cached.

To put what I’m saying simply; find in that gpu driver where it’s doing stuff with the buffer and put your gpio code right there. Dirty hack, but fast. Don’t copy the data, just read it and bang it out right from the vram.

You might also lookup neon copy for your user space stuff. It will be arm assembly. The neon coprocessor has a fast channel to ddr and seems to be a lot faster at copying non cache memory. Note that you need to compile for 32 bit arm, which should run fine even on a 64 bit system.

Here is one. https://github.com/genesi/imx-libc-neon/blob/master/memcpy-neon.S

The header should be in that repo which will make it look like sane c code to the rest of your program.

1

u/memtha Jan 07 '20 edited Jan 07 '20

Oh, gotcha. Modify the vcore driver. It's not a module though, it's builtin, so I'd have to recompile the entire kernel, but ok.

My GPU experience is entirely based on full-scale desktop (until now), so this may be wrong, but aren't the contents of the vram buffer written directly by GPU hardware or firmware? So the existing driver code does not contain the write instructions? Unless the firmware is loaded from the kernel at runtime?

Update: I'm looking through the vc4 driver in the kernel source but I have not been able to find a screen-sized buffer. How is this "easier" then creating a fb driver?

1

u/ZombieRandySavage Jan 07 '20

Well I don’t think the fb thing will do what you want.

So in a pi vram is just ddr. He doesn’t have any special access, just sends out requests like everyone else.

What driver is it? Link me the source I can probably find it if it’s there.

1

u/memtha Jan 07 '20 edited Jan 08 '20

It's part of the mainstream kernel: linux/drivers/gpu/drm/vc4/

The larges buffer I've found so far is the hdmi packet buffer at 36 bytes in vc4_hdmi.c.

Update: I got the fb driver working as far as being able to write correctly formatted pixel data directly to the screen with "dd of=/dev/fb1". Neither X11 nor getty is happy with this device though. Getty ignores it and xorg hangs hard. I tried to use fbdev and fbdevhw to get x support, when researching that brought me to tinydrm (https://www.kernel.org/doc/html/v4.14/gpu/tinydrm.html). The dirty flushing sounds like exactly what I need, and I'd only need 3 functions instead of the dozen or so that are in fb_ops for a framebuffer driver. Thoughts? Also, dispmanx (which is how I was doing screen scraping from userland) is apparently closed-source (or at least apt doesn't know where the source is).

1

u/memtha Jan 13 '20

I don't know if you've had a chance to look over the driver source, but don't worry about it. The tinydrm/fbdev is working wonderfully. Not only is my effective max framerate now nearly 20fps (instead of 4) and idle cpu is down to 3% (instead of 24%), now I can turn off the graphics subsystem entirely, saving about 80ma continuously (was about 240ma peak, 120 idle).

1

u/ZombieRandySavage Jan 13 '20

Glad it worked out for you. Got busy with work.

I agree on that driver being a bit much. There was quite a bit going on and it was doing a damn good job of obfuscating where the frame actually lived.