Explainer: How augmented reality lets us overlay a digital world

When Pokémon GO was launched in mid-2016, an insular world was suddenly turned inside out. For the very first time in the history of video gaming, no one was complaining about screen addicts being cooped up alone in their bedrooms twiddling their joysticks. Suddenly the problem was the opposite: they were out and about in force, wandering real streets and public spaces in search of virtual monsters.

Downloaded more than 500 million times in the months after its release, the frenzy around Pokémon GO was for many an introduction to augmented reality, combining reality and virtual reality by superimposing a digital fantasy land on to a real-world environment.

Originally conceived as a combination of the Pokémon universe and Google Maps, the game used mobile phones’ GPS, built-in cameras and internet connectivity to map virtual monsters onto real geography. The object was to ‘catch’ those monsters, and to do so required getting up close, through actual travel through physical space. Thousands of accidents and more than a dozen deaths were attributed to players looking too much at their phone screens rather than their actual surrounds.

While the novelty of Pokémon GO wore off quickly, Snapchat filters that change your face in real time or drop dancing hotdogs in your photos have never been more popular. Augmented reality is very much a technology that is here to stay.  

Here are the main ways you can find it being used already, and the areas in which it shows greatest promise.

Now you see it

One common application of AR is similar to what Pokémon GO does – adding digital elements to real-world settings, but for useful reasons rather than entertainment. For instance, perhaps you want to see how a couch might look in your living room before you buy it. You point a camera towards the area where the couch will go.  Software analyses the space using information from the camera’s range-finding sensors and then digitally displays the couch on the phone’s screen as if it were there in real life.

Context is everything

Another application AR is already being used for is to provide contextual information about  things you are looking at. In combination with GPS and an internet connection, you can find information, for example, about a restaurant just by pointing your phone lens at it.

Bogies on your six

AR technology has been used in military aircraft to improve pilot capability for more than a decade. Incorporated into the helmet, information on a thin piece of glass in front of the eyes allows a pilot to look over readouts without having to glance down and lose situational awareness.

The latest helmets even allow a pilot to ‘see through’ the plane. The helmet for pilots of the new F-35 Joint Strike Fighter, for example, is connected to six cameras on the fuselage of the plane that allow a pilot to see in all directions; if a missile were approaching from behind, you’d be able to turn around and see it.

It’s educational

AR has uses beyond the battlefield. Students can also use it to ‘see’ what they are studying. Monash University in Melbourne, Australia, has incorporated AR technology into its Earth Sciences Garden, which it considers the most comprehensive in the world. This ‘living’ geological map of the state of Victoria includes close to 500 rock specimens that reflect the regional patterns of rock outcrops, with plants native to each geographical region. Garden visitors can use a phone app to view a 3D display that ‘expands’ the physical environment through time and space to tell a story going back millions of years and supply extra information to the ‘real life’ walk through the garden.

Talking your language

Ever read a sign while overseas and wondered what it meant? Or found yourself trying to find a word you know on a menu so you can order something? Thanks to Google’s translation tool and AR technology the answer can be had by simply pointing your phone camera. The app ‘reads’ the text before overlaying it with the translation, enabling you to understand road signs, menus or a ticket machine.