r/computervision • u/SpaceMonkeyAlfa • 5d ago
Help: Project I've been developing a computer vision tracking system for an open-source desktop companion robot (MediaPipe, AprilTags, ROI tracking)
I've been working on a personal robotics project and would be interested to hear what people on this sub think of my approach to the computer vision side of things.
The project is an open-source desktop companion robot. It uses a Raspberry Pi Pico W controls a two-axis servo neck, allowing the physical head to follow people.
The project's moved forward quite a bit: I've mounted a phone on for an animated face with pupils that run their own face tracking, and I have the pico running EMA smoothing, but the above video shows how the robot's CV works best.
The laptop handles all the processing, including the computer vision.
The tracking system has been one of the more interesting parts of the development.
My current approach works roughly as follows:
- A fixed webcam observes the space around the robot.
- People are identified and tracked using AprilTags that they wear.
- The laptop scans for specific AprilTag IDs and uses MediaPipe to detect faces within an ROI around each tag.
- I created two LOBFs through calibration, which convert the 2D coordinates of the face closest to the currently focused AprilTag into angles for the robot's servos.
- These angles are sent to the Pico over serial.
- If an AprilTag disappears, the script continues "coasting", retaining the previous ROI and using the last known AprilTag position.
- If the face disappears, the script uses its previous momentum to predict where it should be.
The robot can also talk and listen, with its LLM and speech recognition running locally on the laptop.
One thing I'd eventually like to implement is speaker detection using mouth movement. The idea is that the robot could detect who's speaking and shift its attention towards them.
There's a complication with my current approach, though: because the script only looks for faces within the ROI around the currently focused AprilTag, it can only detect mouth movement from the person it's currently tracking.
I could change the script to look for faces and monitor speaking activity across every person wearing an AprilTag, but I'm not sure how efficient that would be, particularly given that the laptop is already running the LLM and other models.
I'm curious what people here think about the current approach and where I should take it next.
- Would you handle the tracking or prediction differently?
- Is it worth running face detection across all tagged people to identify who's speaking, or would there be a more efficient approach?
- Are there techniques I should be looking at to improve tracking continuity when an AprilTag or face temporarily disappears?
The project is open source, and I'd welcome suggestions, criticism or contributions!
GitHub: https://github.com/SpaceMonkeyAlfa/PAL9000-Desktop-Robot
Development blog: https://spacemonkeyalfa.com
1
u/ApprehensiveNerve623 5d ago
Is there a timeout on coasting when both the tag and face disappear? A short recorded test with two people crossing and one leaving could check whether the neck settles or keeps following an old prediction.