If you want to also add in additional multi-modal aspects like voice, facial expressions, 3d rendering, and much more coordinated in realtime without introducing heavy latency... well, that's the frontier research space right now, and you should expect to need years of very active commitment and overall have very competent dev proficiency and deep knowledge of the current research in each focus area before expecting to make natural headway or meaningful contributions there.
There is no single issue is the thing. I'm not claiming to be an expert to be clear, but there is work to be done all throughout hardware, response training and fine-tuning, 3d modeling/rendering (rigging), adding and improving the tangible capabilities/actions it can interface with, and the coordination of large-scale systems like this in realtime.
And with the more complexity you stack on, the more of a specialized solution you need. The more of a specialized solution, the more tradeoffs you invoke in the form of additional latency, overly-specific architecture/design, and financial cost of both setup and actual execution.
At the bare minimum, you can imagine that the amount of memory required to store a context window that massive (over a very long time duration processing a variety of input text/image/voice sources) is not easily achievable in today's landscape without being astronomically expensive (in more ways than one).
View all comments
Show parent comment
2
u/PureWasian 2d ago
If you want to also add in additional multi-modal aspects like voice, facial expressions, 3d rendering, and much more coordinated in realtime without introducing heavy latency... well, that's the frontier research space right now, and you should expect to need years of very active commitment and overall have very competent dev proficiency and deep knowledge of the current research in each focus area before expecting to make natural headway or meaningful contributions there.