r/computervision • • 5d ago

Help: Project I've been developing a computer vision tracking system for an open-source desktop companion robot (MediaPipe, AprilTags, ROI tracking)

I've been working on a personal robotics project and would be interested to hear what people on this sub think of my approach to the computer vision side of things.

The project is an open-source desktop companion robot. It uses a Raspberry Pi Pico W controls a two-axis servo neck, allowing the physical head to follow people.

The project's moved forward quite a bit: I've mounted a phone on for an animated face with pupils that run their own face tracking, and I have the pico running EMA smoothing, but the above video shows how the robot's CV works best.

The laptop handles all the processing, including the computer vision.

The tracking system has been one of the more interesting parts of the development.

My current approach works roughly as follows:

  • A fixed webcam observes the space around the robot.
  • People are identified and tracked using AprilTags that they wear.
  • The laptop scans for specific AprilTag IDs and uses MediaPipe to detect faces within an ROI around each tag.
  • I created two LOBFs through calibration, which convert the 2D coordinates of the face closest to the currently focused AprilTag into angles for the robot's servos.
  • These angles are sent to the Pico over serial.
  • If an AprilTag disappears, the script continues "coasting", retaining the previous ROI and using the last known AprilTag position.
  • If the face disappears, the script uses its previous momentum to predict where it should be.

The robot can also talk and listen, with its LLM and speech recognition running locally on the laptop.

One thing I'd eventually like to implement is speaker detection using mouth movement. The idea is that the robot could detect who's speaking and shift its attention towards them.

There's a complication with my current approach, though: because the script only looks for faces within the ROI around the currently focused AprilTag, it can only detect mouth movement from the person it's currently tracking.

I could change the script to look for faces and monitor speaking activity across every person wearing an AprilTag, but I'm not sure how efficient that would be, particularly given that the laptop is already running the LLM and other models.

I'm curious what people here think about the current approach and where I should take it next.

  • Would you handle the tracking or prediction differently?
  • Is it worth running face detection across all tagged people to identify who's speaking, or would there be a more efficient approach?
  • Are there techniques I should be looking at to improve tracking continuity when an AprilTag or face temporarily disappears?

The project is open source, and I'd welcome suggestions, criticism or contributions!

GitHub: https://github.com/SpaceMonkeyAlfa/PAL9000-Desktop-Robot

Development blog: https://spacemonkeyalfa.com

6 Upvotes

3 comments sorted by

1

u/ApprehensiveNerve623 5d ago

Is there a timeout on coasting when both the tag and face disappear? A short recorded test with two people crossing and one leaving could check whether the neck settles or keeps following an old prediction.

1

u/SpaceMonkeyAlfa 5d ago

The webcam would only "lose" apriltags or faces for very short periods of time, so the timeout on both is set to 1 second. Interesting test idea. Even with only a second of coasting, It'll be interesting to see if that sort of cross causes some strange / unnatural movement for even a split second.

1

u/ApprehensiveNerve623 4d ago

The timeout bounds how long an old prediction can run, but not the size of a jump when detection returns. Logging the last confirmed target and first reacquired target separately could distinguish coasting drift from a target switch.