The voice whispering in your ear: Microsoft HoloLens 2.0 gets an AI upgrade

It is not an exaggeration to say that deep learning has taken the world of computer vision by storm.

Microsoft's HoloLens is the first self-contained, holographic computer, enabling you to engage with your digital content and interact with holograms in the world around you.

It contains a custom multiprocessor called the holographic processing unit (HPU).

The HPU is responsible for processing the information coming from all of the onboard sensors, including Microsoft's custom time-of-flight depth sensor, head-tracking cameras, the inertial measurement unit (IMU), and the infrared camera.

This processor is part of what makes HoloLens the world's first, and still only, fully self-contained holographic computer.

The issue is that a device like the HPU consumes a lot of power because of the complex tasks it has to complete.

That is where the HPU 2.0 comes in, re-designed with efficiency in mind while taking on the same tasks.

Marc Pollefeys, HoloLens director of science says, "We put the battery on your head, in addition to the computer, the sensors, and the display.

"Any computations we want to run locally for low-latency, which you need for things like hand-tracking, have to run off the same battery that powers everything else, so what do you do?

"You create custom silicon to do it.

The second version of the HPU, currently under development, will incorporate an AI co-processor to natively and flexibly implement DNNs.

The chip supports a wide variety of layer types, fully programmable by developers.

Harry Shum showed an early spin of the second version of the HPU running live code implementing hand segmentation at a press conference.

The AI coprocessor is designed to work in the next version of HoloLens, running continuously off the HoloLens' battery.

Pollefeys continues, "This is just one example of the new capabilities we are developing for HoloLens, and is the kind of thing you can do when you have the willingness and capacity to invest for the long term, as Microsoft has done throughout its history.

"This is the kind of thinking you need if you're going to develop mixed reality devices that are themselves intelligent.

"Mixed reality and artificial intelligence represent the future of computing, and we're excited to be advancing this frontier.

Microsoft has a comprehensive partner program for the HoloLens, including several companies in New Zealand and Australia.

These partners are involved in creating apps and developing the HoloLens.