With their HoloLens 2 project, Microsoft and Volkswagen are collaborating to put augmented reality glasses in motion

Imagine putting on a pair of augmented reality glasses when leaving in the morning and getting into your vehicle. As your self-driving car takes you to your destination, you get holographic displays of traffic information, weather conditions, shopping recommendations, and architectural highlights along the way. And at dawn, you can use holographic controls in front of you to adjust the interior temperature to your liking.

This is the future of mobility envisioned by researchers at German automaker Volkswagen, who see augmented reality as one of the key components of future mobility concepts. To get one step closer to that vision, Volkswagen has collaborated with Microsoft to allow the HoloLens 2 mixed reality headset to be used in moving vehicles for the first time.

The new “mobile platform” mode for Holo Lens 2 overcomes a major limitation of mixed reality headsets and creates potential for the technology to be used in new ways – by training drivers to handle harsh road conditions, for example, or creating new user experiences for self-driving vehicles. And while mobility is Volkswagen’s focus, capacity could in future be shared between other industries.

Black and white portrait of Dr. Andro Kleen, Data Science Team Leader at Volkswagen Group Innovation.
Dr Andro Klein.

“We believe that mixed reality insights are the most intuitive insights we can provide to improve our customers’ user experience,” says Dr. Andro Kleen, Data Science Team Leader at Volkswagen Group Innovation. “Because what you see there, and what you have to process, is very close to what humans normally see and process. It’s not that abstract.

An early adopter of augmented reality technology, Volkswagen introduced augmented reality head-up display in its identifier. family of electric cars in 2020 that projects navigation arrows, lane markings and other environmental information in front of the cars.

But Volkswagen was thinking about the potential of augmented reality even earlier, Kleen says. In 2015, for example, Volkswagen embarked on a research project using autonomous vehicles and augmented reality to teach driving on a racetrack. Tested at Volkswagen’s track factory in Ehra-Lessien, Germany, the Race Coach The system used a head-up display that overlaid lines of arrows on the track for drivers to follow, and provided steering and braking cues to guide them through a graduated set of lessons.

Volkswagen hoped to use HoloLens for the research project, but soon ran into a problem. When the device was placed in a moving vehicle, its sensors lost tracking and the holograms it normally displays gone. Kleen’s team reached out to Microsoft for help and contacted Marc Pollefeys, Microsoft chief science officer and expert in 3D computer vision and machine learning. The Volkswagen team traveled to Microsoft headquarters in Redmond, Washington to show Pollefeys and his team their work with HoloLens and their vision for augmented reality technology.

“We had long discussions,” says Pollefeys, now director of Microsoft’s Mixed Reality and AI Lab in Zurich, Switzerland. “They presented their use cases and what they hoped to enable. They were eager to work with us to find a solution and be able to use HoloLens in these situations.

The two teams began collaborating around 2018 to develop mobile platform functionality for HoloLens 2, which required solving a fundamental problem. HoloLens uses two main types of sensors that measure its movement: visible light cameras and an inertial measurement unit, or IMU, which measures acceleration and rotational speed. Together, the sensors mimic the way humans see and move around the world.

But similar to how being in a car or boat can cause motion sickness when what appears to be a stable environment is actually in motion, when tightly coupled HoloLens sensors begin to disagree in a moving environment – with the inertial measurement unit recognizing movement and the cameras unable to – things fall apart.

Photo taken inside a car showing navigation images overlaid on the windshield through the use of HoloLens.
Connection inside and outside the vehicle: A navigation element positioned outside the vehicle completes the 3D map integrated into the dashboard.

To solve this problem, Pollefeys’ team developed an algorithm that models the gaps between sensors and allows HoloLens to continue tracking. But testing the capacity posed another challenge. Since Covid-19 restrictions in Switzerland made it impossible to try the technology in a vehicle with another person at the time, Joshua Elsdon, a senior Microsoft software engineer who worked on the project, had to find solutions from his apartment in Zurich.

Elsdon designed a fictional rig using a plastic box, sticking pieces of tape inside to add visual texture and give HoloLens cameras something to follow. He rode trams and buses around Zurich with a HoloLens headset, making sure his holograms held up as the vehicles moved. At night, Elsdon would go up and down the elevators in his building, testing the technology.

“We had to do a lot of tests in my apartment. These are not ideal development conditions,” says Elsdon, who is now based in Redmond. “It was all done remotely and distributed to different countries, which was interesting.”

After initial prototyping was completed, the team also conducted testing on Puget Sound, near Microsoft’s Redmond campus. They rented recreational boats, took them out on the water, and used external measurement equipment to evaluate the performance of the HoloLens head tracking system on a mobile platform.

Microsoft then tested the feature with Volkswagen. Volkswagen researchers established a two-way data connection between the vehicle and the HoloLens to display and control real-time information from the car. Finally, the team implemented several demonstration use cases to study how virtual interfaces could improve the interior of future vehicles.

Photo taken inside a car showing navigation images overlaid on the windshield through the use of HoloLens.
Mobile Platform Mode, combined with vehicle position data, allows HoloLens 2 to be used in new ways.

“We connected a positioning system that tracks the location of the vehicle. This way we were also able to place 3D elements such as POI information outside the car. This opens up completely new possibilities not only for displaying holograms in the forward-facing driver’s field of vision, but also wherever the user wearing the glasses is looking,” says Michael Wittkämper, augmented reality expert at Volkswagen.

Microsoft rolled out the mobile platform feature a few months ago and it is already attracting interest from maritime businesses and organizations, who are using HoloLens to remotely connect maritime workers with mechanical experts via Microsoft Dynamics 365 Remote Support. The app allows an expert in another location to look through the other person’s HoloLens 2, share their field of view, diagnose a problem, and provide information.

Previously usable only when a ship is in port, this ability is even more necessary when ships are at sea and equipment fails.

“The further away the equipment or machine is, the more difficult it is to have the expert on site,” says Pollefeys, who is also a professor of computer science at ETH Zurich, a public research university. “This feature has proven critical to unlocking HoloLens 2 for Seaspace.”

HoloLens’ mobile platform functionality is currently supported for use on large ships, and Microsoft plans to further refine it for use in elevators, trains, cars, and other mobile environments. Kleen envisions multiple ways to use technology to connect the interior of vehicles to the outside world, such as helping truck or bus drivers navigate narrow streets, identifying points of interest along road routes or providing entertainment to passengers.

A hand adjusts the air conditioning using VR in a Volkswagen automobile.
The direct communication of the AR headset with vehicle data makes it possible to adjust the temperature or control the intensity and direction of the airflow by gesture.

Kleen and Pollefeys believe augmented reality will become an increasingly important aspect of mobility in the future as smaller, more compact versions of smart glasses become available. Pollefeys characterizes mixed reality glasses as the third generation of personal computing devices, after personal computers and cell phones.

“Microsoft is not just interested in commercial devices like HoloLens, but also in the longer term, devices that would make sense for consumers to use in everyday life,” Pollefeys says. “With augmented reality glasses, you can walk around the world and information can appear in context, where it is relevant. You could communicate with other people and also in 3D, as opposed to a small screen.

Kleen imagines augmented reality enabling seamless and connected mobility experiences, with people wearing smart glasses when they leave the house and receiving information through them, from navigation assistance to entertainment, throughout their journey. daytime.

“We see this as a move towards a mobility system where different mobility products and solutions will be connected,” says Kleen. “The basic assumption is that this technology will become lighter and smaller, and we believe that as this happens more people will get their hands on it and incorporate it into their daily lives – and so in their way of getting from A to B.”

Top photo: Volkswagen demonstrated the future of fully autonomous driving in its Gran Turismo concept car, the ID.VIZZION. (Images courtesy of Volkswagen)

Leave a Reply

Your email address will not be published.