It was such a pleasure working with world famous pianist Jon Schmidt and the Piano Guys on this breakthrough AR app. This futuristic tool brings us one step closer a world where AR is used to help make education fun and rewarding.
Sheet music highlighting notes as they are played also scrolls in the app to aid learning, and there’s a view which lights up and color codes the keys onscreen to allow students to “jam with Jon” on their own piano. The app can use the mobile device’s microphone to listen out for pitch discrepancies and can provide real-time feedback.
The Piano Guys, Massive Technologies and The Scan Truck have generated a realistic avatar of Jon Schmidt to perform “with total visual and musical accuracy.” Smartphone or tablet users just need to fire up the AR Pianist app, point the device’s rear camera at any piano and the app will put Schmidt at the keyboard.
The app uses machine learning to place the virtual virtuoso at the piano within an accuracy of a centimeter, and the device holder can move around the space with six degrees of freedom to view his playing from any angle. If you want to get closer to the action, you can zoom in, and should you want to use the experience to learn from a master, playback can be slowed down by up to 100 times.