Projects
PumpScore
Automatic Score Detection and Recognition by Computer Vision using OpenCV and Tesseract
Introduction
Pump It Up is a long running dance game similar to the legendary Dance Dance Revolution (DDR). The key difference is that Pump It Up (PIU) utilizes a 5-panel layout that ellows for more varied and complex patterns or stepcharts. This game is a classic in arcades around the world, and has a cult following that also have their own cabinets. For those who have their own Pump cabinets, a popular hobby is streaming. Such players such as FEFEMZ from Korea, and Seth "F333tz" Sanchez have built a career on streaming this game on online livestream sites. PumpScore is a novel idea whereby we are able to create a basic program to automatically extract scores, grades, and information from the score screen that displays after each song.
The Score Screen
The score screen is a standardized screen that displays a lot of information about each song's play. It features either one player on either side (right or left), two players at once (both sides), one doubles player (both sides), or 2+ co-op players (one score for the group). This score screen shows static information such as the alias of each player (if they have a card registered), a 'title' for the player that can be earned by completing objectives or challenges, cabinet location (arcade name). The score screen also displays dynamic information including the song title of the song that was just played, the number of Perfects, Greats, Goods, Bads, Misses, Max Combo, Score and Upscore, as well as the song's difficulty level and personal / machine best scores. The score screen is displayed for a few moments before returning back to the song select screen, or the game over screen depending on where you are with your set. The screen can also be skipped by tapping on the middle panel after all information is displayed. Because this screen displays a lot of pertinent information, for those who play often and have access to recording gameplay, PumpScore may be valuable in gathering play data, displaying novel visualizations, and tracking tournament data.
How can we automatically detect this information?
OpenCV: Detecting Bounding Boxes
The Pump It Up Score Screen is standard for each song (besides missions), and will display the information in a structured pattern each time. Because of this, we can identify regions of interest (ROI) for each value of information that we want to extract. To do this we need to extract bounding boxes or each ROI. Thankfully, a user on Stack Overflow answered a question by providing a script to output bounding box coordinates when drawing the region of interest.
Code Snippet and Gif Preview adapted from @nathancy on Stack Overflow.
When applying this to our test score screen, this is the result:
