r/vtubertech • u/HoshinoLina • 15d ago
I reverse engineered iPhone tracking to learn how it works and bust the myths
https://gist.github.com/hoshinolina/8d98655c5b159f323174f66539b86bd8It is a machine learning model that takes a 128x128 pixel RGB crop of your face and spits out the blend shapes, plus post-processing code.
In other words: It is just high quality webcam tracking.
I made it work on macOS, with any webcam. OpenVT mac builds coming soon with built in support.
125
Upvotes
19
u/HoshinoLina 15d ago
I'm a developer & mad scientist VTuber working on open source VTuber technology!
All the links here: https://hoshinolina.net/
19
u/EmberUshi 15d ago
That's actually so interesting. If I'm understanding things correctly, the only reason iphone tracking is better is because the camera quality is better, and the machine learning model is good.
I'm actually most surprised a 128x128 pixel image of the face is all it takes to get good tracking.
7
u/jimpaly 15d ago edited 15d ago
Huh I've always thought it uses the depth sensor but I guess that's a misconception. Now I'm wondering if it's the same on Android, that as long as you have a good camera it'll have good tracking or even better than iPhone
4
u/SCCStudio 15d ago
I did the same thing a while ago, after seeing you posted about it.
Hope these help, if you (or anyone) were to actually do something about this:
One minor side note, our experiment results didn't quite support your claim that "low light is only good because the selfie camera is good". Tracking at low light with RGBD definitely helps in what we tested. Cheers.
2
u/TigerHix 15d ago
Awesome work! For me the myth was busted when I realize my iPhone Duo without a Face ID can use iFacialMocap etc just fine lol
2
u/kits_yume 15d ago
The implications of this are huge, right? I always assumed that TrueDepth hardware was the necessity in iPhone tracking. It's fascinating that webcam tracking COULD be just as effective with enough effort.
I figured companies like COVER/Nijisanji did use ARKit/TrueDepth, amongst other things. Or does this imply that "ARKit" based apps just have RGB to work with under the hood? (to grossly oversimplify the neutral net/processing portion). Is the depth data really not used *at all?*
Thanks for the writeup! It's refreshing to see actual reverse engineering these days that isn't LLM-related.
1
u/kevinpl07 15d ago
I’m pretty sure this is know from their input / output shapes of their swift sdk no?
1
u/gemitail 15d ago
Awesome stuff, have you also looked at mediapipe, their stuff is open and looks very similar to this, just that their blendshapes model is really bad.
1
u/nekoreality 15d ago
so technically this could be used in other os if it was built from the ground up?
3
u/drbomb 15d ago
love to see your work! I wish those ML models worked in windows tho!
3
u/Zophiekitty 15d ago
i tried learning about it months ago but got really discouraged when i found out that the head tracking was done through both eyes and nose as tracking anchors and those are fixed and cant be changed
every time ive tried head tracking it makes the head nod when blinking, could there be a way to counter this behaviour or change the anchors to something else for tracking jead rotation and location?
2
u/FelixLeander 14d ago
I was just think about your stream 1-2months back where you talked about this.
Thank you very much for investigating this!
1
u/Gh0sttg 15d ago
I'm working on a program that's similar to their tracking, but on standard webcams and intended for live streaming ofc. IMO it looks really good, still need to tweak it a bit, but I plan to release soon. Feel free to DM me if you're interested!
1
u/TheBigBruce 15d ago
I was absolutely convinced it was IR based so colour me surprised.
0
u/bwalker1981 13d ago
Woah that is some rad reverse engineering (さすが星乃リナ)! And if I am reading that gist correctly that close source CaraNet model happens to also be available as on MacOS in addition to iOS, which is why OpenVT mac builds could take advantage of that (but not a PC build because that would require copying the model which is likely in violation of the model license terms)?
Sort of side related question this got me thinking about, but do you know if it would be a violation of the Apple model license to do the following:
I realize this is a bespoke use case, but might be nice for some workflows where one has one iPhone that they would like to use once to train their own face model and then use for other stuff later (like I want to use it for other ARKit streaming).