Meta researchers demonstrated that the Quest 2 can have full body tracking with no additional sensors, with the help of AI.
Meta researchers developed a system for full body tracking with Quest 2 that uses only the existing headset and controllers, and no additional sensors of any kind. The system is called QuestSim and uses AI that was trained on hours of motion captured subjects. After the AI system was trained, it could analyze the position of the headset and controllers, and factor in human physical limitations to make a reasonably accurate prediction of the body’s entire position:
The researchers tested QuestSim’s accuracy on two subjects of different heights, both of whom had not been analyzed previously. The result was that QuestSim was very accurate in conventional movements such as walking or running. It was less accurate in more unpredictable movements such as dancing, but still wasn’t wildly off in my opinion.
One issue is that the test subjects appeared to have average age and body compositions. It was not shown whether QuestSim could accurately predict the body movement when the subject was much heavier or lighter, or much older or younger. Nonetheless, the approach is promising and with more training data, QuestSim might become a practical method of full body tracking.
it’s the problem of prediction, and somewhere behind the AI training stuff : it can only create limitation – that limitation coming from repetition.
I explain : if you train the AI to avoid an obstacle 100 times the same way, on the 101th time, the AI will consider that you will act the same than the 100 others. But it’s wrong : if you try to avoid 100 times an enemy bullet (and fail), there will be no chance that on the 101th time the AI would consider that you try to innovate with another strategy, similar to the 100th others but with a variance – a variance that can’t be detected by any of the 4 cameras of the quest.
I got the same limitation with the handtracking : I have a physical problem on one of my fingers, and of course, the handtracking algorithm can’t interprete.