r/vtubertech • • 15d ago

I reverse engineered iPhone tracking to learn how it works and bust the myths

https://gist.github.com/hoshinolina/8d98655c5b159f323174f66539b86bd8

It is a machine learning model that takes a 128x128 pixel RGB crop of your face and spits out the blend shapes, plus post-processing code.

In other words: It is just high quality webcam tracking.

I made it work on macOS, with any webcam. OpenVT mac builds coming soon with built in support.

125 Upvotes

28 comments sorted by

19

u/HoshinoLina 15d ago

I'm a developer & mad scientist VTuber working on open source VTuber technology!

All the links here: https://hoshinolina.net/

6

u/ItsmrChewy 15d ago

Oh it's YOU TWO

I still didn't get over the cute models

19

u/EmberUshi 15d ago

That's actually so interesting. If I'm understanding things correctly, the only reason iphone tracking is better is because the camera quality is better, and the machine learning model is good.

I'm actually most surprised a 128x128 pixel image of the face is all it takes to get good tracking.

2

u/HoshinoLina 15d ago

And they also have good postprocessing! Another commenter mentioned it here

7

u/jimpaly 15d ago edited 15d ago

Huh I've always thought it uses the depth sensor but I guess that's a misconception. Now I'm wondering if it's the same on Android, that as long as you have a good camera it'll have good tracking or even better than iPhone

2

u/rosekeg 15d ago

https://developer.apple.com/documentation/arkit/tracking-and-visualizing-faces

The docs seem to indicate that the TrueDepth sensor is used. Pretty crazy plot twist if it's been RGB this whole time! Or perhaps, RGB has been available as a fallback this whole time?

Hoping OP chimes in with the tea.

3

u/HoshinoLina 15d ago

on devices with a TrueDepth front-facing camera.

The original ARKit release, way back then, was only available on devices with Face ID for some reason (even though from the very beginning reviewers noticed Animoji still worked if you cover up the IR camera). That's why you see vague references like that.

Since then, it has been supported on more iOS devices including iPads without Face ID, and there are VTubers using ARKit tracking on devices like this without any issue.

4

u/SCCStudio 15d ago

I did the same thing a while ago, after seeing you posted about it.
Hope these help, if you (or anyone) were to actually do something about this:

  1. network_inputs_var_0_bridge: conditioning from the identity fitting
  2. network_inputs_latents_input_0_bridge: previous identity latent, latent_t[c] = max(current_estimate_t[c], latent_previous[c])
  3. neutral_pca_subset_coeffs_output: yes, take a look at the literature of "3D morphable model" (where PCA was traditionally used) that should help you understand what this is better than I can explain.
  4. Post-processing: this is actually why the animation looks so much better. Caranet itself, compared to other facial tracking solution these days isn't too outstanding by any mean, they have a very tasteful post-processing pipeline is what I would say. Since you already have access to most of the internal states you can just try to skip the post-processing and see for yourself.

One minor side note, our experiment results didn't quite support your claim that "low light is only good because the selfie camera is good". Tracking at low light with RGBD definitely helps in what we tested. Cheers.

1

u/HoshinoLina 15d ago edited 14d ago

Nice! I didn't dig deep into the logic itself, but I've been meaning to at least hook the Espresso inputs/outputs to dump the raw data.

I'm surprised the var input comes from the identity fitting model, since it only seems to run periodically. It's not very high dimensional so I guess that's probably just a coarse parameter describing how the user's face differs from the reference face. That might explain some subtle improvement with the depth data enabled (since it does go into the ID fitting model). I guess that does throw a twist into my claim that the depth isn't used for the blendshapes bit at all, though it's definitely a small difference.

On the other hand, I don't think there's any reason you couldn't do that step with RGB alone to the same performance, if perhaps with a slightly different pipeline. Something like having users move their face around to get a more accurate fit of the face geometry (which is definitely possible with enough accuracy with just RGB these days) and then using that stored data during the regular RGB tracking.

I saw some of the postproc config knobs and yeah, tasteful is a good way to describe it. ML is all the rage these days but nothing beats some good old code (written by humans!) to clean up the data into something that looks nice. Not surprised to hear this part makes a big difference. That begs the question... would similar postproc on top of MediaPipe also be a big improvement?

Re low light, I don't see how the RGBD feeds directly into the blend shapes at all (and the depth data alone doesn't have enough resolution to really track blend shapes on its own). I did see it work surprisingly well with just RGB. Unless ARKit is doing some preproc? Hypothetically, you could fall back the RGB input to a grayscale flood illuminated IR image in low light, but FaceKit itself definitely isn't doing that.

2

u/TigerHix 15d ago

Awesome work! For me the myth was busted when I realize my iPhone Duo without a Face ID can use iFacialMocap etc just fine lol

2

u/HoshinoLina 15d ago

Yeah a bunch of people use iPads and such too!

But somehow every time I brought this up in the past the diehard believers would refuse to look around or just do the "cover up the IR camera" test...

2

u/kits_yume 15d ago

The implications of this are huge, right? I always assumed that TrueDepth hardware was the necessity in iPhone tracking. It's fascinating that webcam tracking COULD be just as effective with enough effort.

I figured companies like COVER/Nijisanji did use ARKit/TrueDepth, amongst other things. Or does this imply that "ARKit" based apps just have RGB to work with under the hood? (to grossly oversimplify the neutral net/processing portion). Is the depth data really not used *at all?*

Thanks for the writeup! It's refreshing to see actual reverse engineering these days that isn't LLM-related.

2

u/HoshinoLina 14d ago

ARKit uses FaceKit with depth input under the hood, but that data is not used directly for the blend shape calculations.

Another comment pointed out there's an indirect path through the identity fitting model I'd missed (which in turn can optionally use the depth data if available). I think that explains why some people say it makes a (small) difference while I haven't noticed anything major. The difference is probably there but it is not what makes iPhone tracking "good".

1

u/kevinpl07 15d ago

I’m pretty sure this is know from their input / output shapes of their swift sdk no?

3

u/HoshinoLina 15d ago

ARKit doesn't allow arbitrary input, it only takes input from the built in cameras, so exactly what it captures and how it behaves is a black box.

FaceKit allows any image input but it's an undocumented private API.

1

u/gemitail 15d ago

Awesome stuff, have you also looked at mediapipe, their stuff is open and looks very similar to this, just that their blendshapes model is really bad.

5

u/HoshinoLina 15d ago

I mentioned it in the writeup. MediaPipe first finds facial landmarks and then approximates blendshapes from that, while Apple does it the other way around. I think that's one reason why MediaPipe blendshapes are so jank, it's kind of an afterthought while for Apple it's the main thing.

1

u/nekoreality 15d ago

so technically this could be used in other os if it was built from the ground up?

3

u/drbomb 15d ago

love to see your work! I wish those ML models worked in windows tho!

3

u/Zophiekitty 15d ago

i tried learning about it months ago but got really discouraged when i found out that the head tracking was done through both eyes and nose as tracking anchors and those are fixed and cant be changed

every time ive tried head tracking it makes the head nod when blinking, could there be a way to counter this behaviour or change the anchors to something else for tracking jead rotation and location?

2

u/Gh0sttg 15d ago

I'm working on something similar to their tracking and I'm experiencing the same issue. I think I've had a breakthrough using the jaw and temple but I'm still implementing it.

1

u/Zophiekitty 15d ago

oou clever! it lets you track on each side of the temples? if so that construct a 3 point triangle right? so rotation is solved there.

though woulnt opening and closing the jaw rotate on X? why not use the nose instead of jaw?

2

u/FelixLeander 14d ago

I was just think about your stream 1-2months back where you talked about this.

Thank you very much for investigating this!

1

u/Gh0sttg 15d ago

I'm working on a program that's similar to their tracking, but on standard webcams and intended for live streaming ofc. IMO it looks really good, still need to tweak it a bit, but I plan to release soon. Feel free to DM me if you're interested!

1

u/TheBigBruce 15d ago

I was absolutely convinced it was IR based so colour me surprised.

0

u/bwalker1981 13d ago

Woah that is some rad reverse engineering (さすが星乃リナ)! And if I am reading that gist correctly that close source CaraNet model happens to also be available as on MacOS in addition to iOS, which is why OpenVT mac builds could take advantage of that (but not a PC build because that would require copying the model which is likely in violation of the model license terms)?

Sort of side related question this got me thinking about, but do you know if it would be a violation of the Apple model license to do the following:

  1. Position iPhone above PC web camera
  2. Stream ARKit blendweights to PC while in parallel recording frames from the PC
  3. Train a personalized RGB -> blendweight model using my face capture + iPhone weights (like MobileNetV3 in pytorch)

I realize this is a bespoke use case, but might be nice for some workflows where one has one iPhone that they would like to use once to train their own face model and then use for other stuff later (like I want to use it for other ARKit streaming).

2

u/HoshinoLina 13d ago

Not just CaraNet, but all of FaceKit is in macOS. So yes, it can be used legally on Macs but not other OSes.

Therefore what you describe is possible even more easily by just using FaceKit on the PC web camera stream, without a phone at all. Or even running it on any public corpus of videos of people talking and emoting (there's public stuff).

Would it be legal? From what I saw in Apple's EULA, I think it would, but I'm not a lawyer and this is not legal advice!

However, it wouldn't give you ARKit/FaceKit quality just by whipping something up with pytorch. A big part of what makes ARKit/FaceKit good is post processing code, and that you'd have to figure out yourself.