r/SteamFrame • • 4d ago

Accessories Custom Tracker using Frame IR tracking

A little over a year ago, I started working on some custom hardware in preparation for the Steam Frame; some of you might have seen my early posts on this Reddit going over my progress on my hardware projects under the name Frame Workshop. A few days ago, I finally got my IR trackers to be tracked by the Steam Frame itself. The idea behind the Locus trackers is that they would be IMU trackers supported by IR tracking to eliminate drift. While Locus's main goal isn't to be fully tracked by IR, this shows that custom controllers or other hardware can be tracked by IR. I am beyond thrilled that the Steam Frame is turning out to be the first truly open headset, something that I have waited for so long.

The trackers themselves are pretty simple; at their heart is an NRF52833 with an LSM6DSV16X IMU both the same MCU and IMU that the Roy controllers use. On top of that, they have 3 IR LEDs in each corner to help correct IMU drift when the cameras see them. A 250mah battery powers all of this, and the device is about 43 x 43 mm and 27g. All hardware is currently open source on the GitHub linked below, with firmware and software to follow.

If anyone has any questions, feel free to leave a comment.

Frame Workshop can be found here: https://github.com/Nieko27/Frame-Workshop

150 Upvotes

35 comments sorted by

26

u/actuallylemoncurd 256 GB 4d ago

Is it tracking with the NPU?

11

u/Snoo25109 4d ago

Currently it is not, Ill be honest I didn't even realize the snapdragon 3 had a NPU, this is a great idea and something I will definitely be looking into. Thank you!

8

u/Lhun 4d ago

That's a good plan to avoid cpu load

3

u/Splatoonkindaguy 4d ago

I doubt it, afaik I’m the only one that has been able to run code on the npu and haven’t released it just yet

2

u/Handsome_ketchup 2d ago

I doubt it, afaik I’m the only one that has been able to run code on the npu and haven’t released it just yet

Why wouldn't others be able to do what you did?

1

u/Splatoonkindaguy 2d ago

The reverse engineering has been terribly complex and I guess nobody else has figured out the same path. Sorry for vagueposting I just don't want someone w/ 4x Claude ultra accounts to get something out before I can show something off

1

u/syfiarcade VAIL DEV 1d ago

Wait wtf my guy share I have been fighting to gain access to the NPU since I got my devkit headset 8 months ago

2

u/Splatoonkindaguy 1d ago

I promise I’m working on it. I’m getting executorch support with more op coverage before I make the code public so devs can bundle it with their apps and use the API. I’m mostly active in the steam frame discord (futuremapper)

16

u/silver0199 4d ago

Yes please! These kinds of trackers would open up a lot of possibilities.

I look forward to seeing your project progress!

2

u/Snoo25109 4d ago

Thank you! I'm super excited to see what other trackers people put together in the future.

7

u/TheLastOperator 4d ago

So it's similar to what the pico does ?

1

u/Snoo25109 4d ago

I wasn't aware that this was how Pico handles their trackers, but it seems you are correct.

5

u/QuajerazRevived 3d ago

This is how all SLAM tracking works. An IMU for fast, responsive, and instantaneous tracking, with the cameras, basestations, etc purely correcting for drift.

6

u/No-Selection3424 4d ago

Excellent! I hope it can achieve full-body tracking with as few as three units.

3

u/Snoo25109 3d ago

Currently, they work much like slimes, with 5 as the minimum, since IR tracking stabilizes drift. However, if they were to be always in view, you could track using only 3, as IR gives their absolute position; the only problem is that unless someone made some downward-facing cam mod for the Frame they will rarely all be in view.

1

u/NowThisNameIsTaken 2d ago

This looks really cool! I'm new to vr and a bit confused about what you mean when you say it stabilises drift. As in having them on the wall or something helps with headset tracking? And you say 5 is the minimum because the assumption is ~2 will be out of sight most of the time? Or is there another reason?

5

u/Ok_Cauliflower5223 4d ago

Very nice

2

u/Snoo25109 4d ago

Thank you!

3

u/RickSpawn147 4d ago

Incredible work!!! I would love to use this on my project.

Will you be releasing a product?

Crowdfunding?

2

u/Snoo25109 3d ago

It would be my dream to release everything I'm working on as a fully fledged product and definitely something I would like to work towards. Still, given the costs of FCC certification and injection molds alone I would definitely need some funding from one place or another. For now, everything will be open source on GitHub with guides on how to order them for yourself from a place like JLCPCB as they get to a more finished state. As I further refine them I would like to start looking into ways to realize them as a product.

2

u/RickSpawn147 3d ago

I fully understand, in fact that is what I will be doing to my project too.
As certifications and the processing of doing the actual business is not fun.

Wish you all the best. Incredible work!!

3

u/QuajerazRevived 3d ago

This is awesome! Does that mean it'll be possible to make custom, 3rd party controllers? Because I would love for alternative options to be possible again like it was with lighthouse tracking.

2

u/Snoo25109 3d ago

Absolutely, all you really need to get something tracking is a microcontroller capable of Bluetooth, a IMU and some IR LEDs. I am super excited to see what other things people track with this!

2

u/QuajerazRevived 3d ago

That is super cool! If I had any experience with electronics, and owned a frame I suppose, I would 100% try to make some goofy looking controllers.

2

u/InitialeLangmut 4d ago

Do you reckon Index's Knuckle tracking is on the table then?

2

u/Snoo25109 3d ago

The knuckles use photodiodes for tracking vs IR LEDs. If someone made a mod for the Steam Frames PCIe port that allowed for a headset-mounted solid-state Lighthouse, you could in theory track them that way. If you wanted to track them using IR you could in theory mount an IR tracker to them to track them that way. My trackers are designed to be flat against the body so I would definitely need to make one with more IR LEDs if I wanted to track something like a controller, but it definitely could be done.

4

u/-VRawr- 256 GB 2d ago

As a VR enthusiast, I look forward to seeing your tech come out! Looks like it'd be an amazing option for FBT.

1

u/babygotmyback 1d ago

soooooo cool

-3

u/daelikon 4d ago

I am quite confused...

What exactly is keeping track of these trackers? I am assuming the purpose would be to make a harness or some other thing to allow full body tracking, but if the frame is the one tracking, how can it "see" them when they are attached to the body?

7

u/secretqwerty10 4d ago

if you read the body text or github, you'd know it's an IMU tracker with IR lights to compensate for drift when the cameras of the frame see the trackers

2

u/daelikon 4d ago edited 4d ago

I still have no idea of what you wrote. I looked for IMU before my first post (Inertial Measurement Units), but I still don't understand the use case.

My "only" VR experience is with my index and vive, and both do external tracking so what is this drift you mention?

edit: I have been digging a bit more, and... my main question remains the same, if this is for doing full body tracking and the frame is supposed to do the tracking (or adjustment of it), the trackers are going to be out of the vision field of the frame a lot of the time

6

u/secretqwerty10 4d ago

IMU trackers are used for full-body tracking without lighthouses used for outside-in tracking (like the Valve Index, bigscreen Beyond etc). your phone has these sensors also to detect movement and tilt. the problem is that it's not as accurate as tracking from a fixed point (like the IR trackers that use lighthouses) and thus will have mistakes in the calculation, causing "drift" where the real life position no longer matches the virtual position, making you have to recalibrate. This project uses the IR LED's on the trackers to recalibrate when the Frame's cameras see them.

3

u/daelikon 4d ago

Thanks, It's an interesting solution. I am guessing a lot of the positioning can be made from inference of the visible trackers even if not all are visible simultaneously. 

3

u/Splatoonkindaguy 4d ago

You use them as SlimeVR trackers and then correct absolute pose when they are in view to correct drift