Machine learning technology is literally judging your skeleton while you play
Author:
Publish date:
Just Dance 2024 is the most technologically advanced game in Ubisoft’s long-running dance series. That’s because it uses cutting-edge machine learning to monitor your performance and compare it with thousands of other players in order to give you a more accurate score.
Launched October 14 on Nintendo Switch, PlayStation 5, and Xbox Series X|S, Just Dance 2024 tasks you with replicating the dance moves of on-screen characters. The game monitors you through either the Nintendo Joy-Con or the Just Dance Controller App on your smartphone to assess how well you’re doing. But now, Ubisoft has beefed up its use of AI in helping to better track your body.
Camera-only mode has been in Just Dance before, of course, but not like this. So resource-heavy is the mode, says creative producer Matthew Tomkinson, only the latest iOS and Android-powered smartphones are able to run it. “The system will learn out of plenty of different videos,” he explains. “We feed the system with plenty of videos of our own performance that represent what is the good move to do. And if I dance to it or if you dance to it we will have a slightly different interpretation, but if we take plenty of examples the system will learn by itself what is the good move to do and what moves shouldn’t be done, and then we'll compare it with your performance.”
The more data Ubisoft gets, the more accurate Just Dance will be. It works by visualizing your skeleton and placing it in 3D space. From there, the game can understand where and how exactly you’re moving - and whether or not you’re any good. “We have a great machine learning team in Ubisoft that works by using the videos from the game and extracting skeletons,” says Tomkinson. “But you know, like just small stick figures out of those bodies. And they manage to guess even depth…Through machine learning they manage to understand that.”
The AI-powered camera mode not only promises better accuracy, but frees players up from having to awkwardly wield a controller while trying to perfect the latest track from Bad Bunny. It even lets more people join in on the fun, supporting up to six players simultaneously. To use it, you just need to download the Just Dance mobile app, which is free on iOS and Android.
According to the official press release, “the Just Dance dedicated app turns players’ smartphones into a controller with its phone-scoring technology, which allows up to six players to dance without any additional accessories.” Essentially, you’re transforming your smartphone into a motion sensor that both examines the accuracy of your moves and lets you navigate through the game.
Could Ubisoft leverage the same machine learning technology to improve accuracy in other areas of the experience, such as calorie tracking? “Well, it could be an opportunity,” says Tomkinson. “We didn't use AI to do it. The way we did it is that we compared it with different tracking devices, the main tracking devices that people use when they do fitness. So we have all the accelerators, we have the same information as you have on a connected phone or a watch…But it’s quite wide how different the different tracking systems can give you some calories, so in the end I would say we are at an average of what these devices usually give.”
If you don’t like the idea of an AI silently judging your bad dance moves before uploading it into a big data bank as an example of how NOT to move, however, at least on Switch you have the option of going dark and using the Joy-Con. Click here for our full Just Dance 2024 interview with Matthew Tomkinson, in which we talk machine learning, hands-free mode, and the magic of Whitney Houston.
