What it is and the value it drives
Over the past 20 years, Human-Computer Interaction, the way humans engage with digital devices, has radically evolved. At home and work, it was common to use desktop computers, with a simple display, keyboard, and mouse, to retrieve information and manually input data and instructions. Digital experiences were constrained to one room and one mode of interaction. Since then, mobile devices, such as smartphones and tablets, have created new entry points to our digital worlds, as they allow access to snippets of information and enable connectivity throughout the day. In recent years, strides in technology and user research have resulted in the creation of two complementary but related modes of digital experiences: on one hand, personal connected devices infuse the surrounding world with computing power as they proliferate; on the other hand, "mediated reality" technologies enable immersion into computer-generated worlds.
Personal connected devices encompass smartphones, tablets, laptops, and wearables such as smartwatches and fitness trackers. They allow users to consume information on the go, collect granular information about their health and habits, and work from anywhere. As more devices become "smart", users can improve the efficiency of interactions and minimize their duration.
Mediated reality refers to virtual spaces that are related to the physical world in varying degrees, from overlaying information to it (Augmented Reality or AR), to replacing it entirely (Virtual Reality or VR), as shown in the chart below. Cameras and sensors project images onto a screen: in VR, the screen is part of a head-mounted helmet that isolates the wearer, while in AR, the display combines a feed of the real world with computer-generated images.
What value do these new interfaces bring?
First, more immediate access to relevant information. Instead of having to log in to a computer, navigating menus, and typing options, a smartwatch can instantly tell us the weather, show our latest email, or directions to the next destination; in AR, contextually relevant information, such as real-time translations, place names, or items for purchase, is presented to us in the real world, right where it matters; VR immerses us into model environments, live events, or training demonstrations, so that knowledge is vividly conveyed.
Second, the capability to input data and controls to computers in even more natural and efficient ways. Wearables and mobiles can track fitness data without requiring manual effort, while software "widgets" allow us to set alarms, transfer money, and send messages at the quick press of a button or by voice; AR/VR makes it possible to use hand gestures and controllers to manipulate the virtual world, allowing for smoother interactions.
Where it is today
Thanks to the explosion of the "app ecosystem", nearly every consumer service is available on a smartphone: online banking, e-commerce, mobility services, and entertainment. In the enterprise world, they provide ways to authenticate users, and deliver trainings. They can even fulfill use cases where dedicated devices would have been previously required, such as point-of-sale functionality, or restaurant orders, or connecting to industrial systems.
Wearable devices like smartwatches and wristbands have enjoyed widespread success in the consumer sectors, with value propositions centered on fitness tracking, productivity tools, and fashion: in the US, 21% of the population regularly wears a smartwatch or fitness tracker, while sales in Japan amounted to 1.2 million devices in 2019, growing at a steady 8% year on year in the second quarter of 2020.
Industrial wearables are starting to appear, with the promise of providing workers with more contextual information in a distraction- and hands-free manner so as to maximize safety. For example, RealWear is a head-mounted, Android tablet-class wearable computer that industrial workers can attach to their helmets and control by voice to do video calls with mentors, access documents, and visualize and transfer data. In late 2019, RealWear entered the Japanese market and began establishing partnerships with various system integrators and OEMs.
Most Japanese consumer electronics manufacturers have hesitated to enter the consumer-wearable space, facing players with a software advantage, and ones with an advantage in manufacturing commoditized hardware. Instead, many are focusing on electronics for the B2B sector, with even fewer betting on business wearables. Sony Europe is launching mSafety, a B2B wearable device and platform that can be leveraged by healthcare providers to monitor patients, or by employers to monitor worker safety in hazardous workplaces. In another example of custom B2B wearables, TECCO is a wrist-mounted navigator, which can vibrate to guide factory and logistics workers to the correct components to mount or pick up, reducing time spent to search and identify them.
Initially developed for video games and home entertainment, AR and VR are progressively finding more use cases in the consumer space. In retail, furniture multinational IKEA offers its "IKEA Place" AR application, which lets buyers visualize how items would look in their home directly on their phones. AR is being exploited by marketers worldwide for its surprise factor: in an award-winning marketing stunt, Burger King Brazil, for example, let its customers virtually "burn" competitors’ banners and billboards in their application, revealing coupons underneath.
VR is being used as a tool to create immersive experiences for education and meetings. Microsoft has developed several learning applications for its HoloLens mixed reality headset: its findings report benefits in retention, test scores, and engagement. Startup “VR Education” developed Engage, a platform for virtual reality events, which, partly due to the 2020 COVID-19 pandemic, saw a revenue boost from an increased interest in virtual events.
AR/VR is being implemented to improve training across industries: in retail, Walmart's VR-based training used 360° video from stores to train employees in interpersonal skills, reporting a 10-15% increase in the test score of attendees; in aerospace, AR/VR developer Inlusion collaborated with aviation service companies to create training and testing experiences for mechanics, who could receive demonstrations of maintenance situations in a completely safe setting. Several players in the energy and materials sector, from BMA to Rio Tinto, have developed VR systems to navigate mines and plants for training and demonstration purposes.
In the medical field, VR is being used for training and pre-surgery simulations; taking it one step further and into the operating room, Israeli startup Augmedics developed an AR headset which allows surgeons to view a patient's spinal anatomy "as if with X-ray vision", and accurately position surgical tools as they are controlled remotely.
At the moment, obstacles to widespread adoption of AR/VR are the cost of headsets relative to the number of available use cases, the relatively poor quality of AR experiences on smartphones due to high data requirements, and the high degree of specialized knowledge required to develop such applications. The last obstacle is partially mitigated for phone-based Augmented Reality by the efforts of both Google and Apple to develop Software Development Kits (SDKs), which provide programmers with useful building blocks to build performant applications.
Privacy and security are also crucial areas that AR technology makers will need to address. Since the release the first smart glasses, consumers and privacy guarantors have voiced concerns about devices with “always-on” cameras and microphones. While such features are necessary to provide the augmentation of reality, they affect the privacy of not only the wearer, but also bystanders, who in many jurisdictions are protected by wiretapping and paparazzi laws that require explicit consent for recording. In order to truly unlock widespread adoption, it will be key to build devices with “privacy-by-design” engineering (e.g., clear indication of recording status, possibility to easily disable recording, etc.), as well as pursue transparency on data capture and develop strong policies to protect users and third parties.
How the technology will continue to evolve
Looking at smartphone technology, there are trends towards simplification of design and hybridization of form factors.
In terms of simplification, devices are coming out with fewer ports and buttons, and smaller bezels. Thanks to wireless charging, Bluetooth connectivity, and on-screen fingerprint sensors, there may soon be fully port- and bezel-free phones with displays that wrap around the edges. Vibration-sensing technology, such as that being developed by UltraSense, will make it possible to produce devices with no physical buttons, where the whole surface can be a tactile interface.
Hybridization is the trend of existing form factors fusing with new ones: "phablets", phone-tablet hybrids, are an example of the attempt to create phones with larger screens and better processors that enable more work to be done on smartphones. Since phablets present the drawback of being less pocket-friendly while not substantially increasing screen size, new form factors have been emerging. Several makers have been experimenting with reintroducing flip phones and phones with folding screens, which would allow for a smartphone-sized device that can be expanded into a tablet as required.
Supported by the success of smartwatches and fitness bands, wearables can be expected to take even more diverse forms, such as smart jewelry like brooches or rings. "Hearables", including smart earphones and hearing aids, will likely find more use cases as audio-based interfaces become more sophisticated thanks to AI assistants, and as noise-cancelling and voice-enhancing technology is further miniaturized and optimized.
Even personal items have the potential of becoming "smart", providing granular insights on our lifestyle. At recent consumer electronics forums, there have been displays of connected toothbrushes for dental health monitoring, water canteens for hydration tracking, and belts for fall prevention in elderly people. Such devices currently require linking with apps and often involve cumbersome Bluetooth pairing processes, but as 5G and IoT platforms make connectivity simpler and more pervasive, it will be possible to design more user-friendly software and hardware.
Technology for mediated realities is much younger, and a few key technology developments are expected over the next decade to significantly improve the experience.
First, headsets are currently quite large and heavy, and in order to deliver good performance, they need to be tethered by a cable to a computer, gaming console, or dedicated processing unit. As the technology is miniaturized, devices will become smaller and lighter, and untethered headsets will be more common, such as Facebook's Oculus Quest: this will make for more mobile solutions, which will particularly increase the appeal of Augmented Reality.
Second, on the theme of increased mobility, the adoption of 5G, especially for smartphone-based AR, will make it possible to stream experiences in real time when outdoors, thus improving on the current long latencies and limited image quality.
Third, there will be improvements in graphics hardware as well as advances in graphics processing, such as "foveated rendering", which optimizes processing power for human vision by rendering only the focal point to the maximum level of detail. Such optimizations will lead to smoother and more realistic graphics, which will make the experience more enjoyable, and help address the common phenomenon of VR motion sickness (also known as "cybersickness").
The key future applications
When it comes to wearable devices, there is a trend towards creating shorter, more continuous, and more relevant interactions throughout the day, rather than longer sessions of digital consumption as is the case for computers and smartphones.
From the viewpoint of AR/VR, adoption will be driven by decreasing prices, increased portability, and further improvements on "development toolkits", which allow engineers to create virtual experiences on top of well-established platforms.
Below are some notable use cases across industries which are likely to gain adoption.
Retail. Still in experimental stages of adoption, AR solutions such as Perfect will allow shoppers to try on makeup and beauty products with "magic mirrors" in the shop, creating a faster and more fun experience, or even on their smartphones at home, thus augmenting the comfort of e-commerce.
Healthcare. From the patient side, VR will make it possible to receive visits or see newborns in isolated environments in a safe and realistic way for doctors, there will be more VR and AR applications for learning and training, as well as solutions to let patients explain and understand their conditions, even at a distance. Preventative care and telemedicine will be made even more convenient with dedicated wearables that help monitor health markers for chronic patients: for example, smart clothes by Skiin make it possible to monitor vital signs, sleep habits, and symptoms of individuals requiring more attentive care, and noninvasive wearables to monitor diabetes are under development.
Industrial. Building on the cost savings, effectiveness, and safety gains of AR/VR training and handbooks, more factories will leverage such solutions to build capabilities and make maintenance simpler and more intuitive for on-site workers: for example, when repairing a car or a piece of machinery, an AR visor can clearly label components for the operator and provide step-by-step instructions directly overlaid on the physical machine. Wearables will also help in monitoring the health of employees as they work, drive, and operate machinery, and will help ensure their safety and fitness to work.
Real estate. Remote room viewings powered by VR are becoming a reality, thanks to solutions developed by companies such as ROOV in Japan: prospective buyers and renters can view rooms at the real estate office, reducing the time required for direct visits, and requiring the owner to accommodate only one viewing to record the model.
Media and entertainment. Despite a devoted following, AR/VR entertainment is still relatively niche, with cumbersome headsets, high prices, and a lack of mainstream offerings. If these issues are resolved, VR has the potential to create immersive experiences in games and also sporting events and concerts, where attendees can join remotely while experiencing the sensations of live performance.