HoloLens Co-inventor Avi Bar-Zeev Shares: The Road To All-weather AR Glasses

The road to all-weather XR glasses

For over 30 years, I've worked on XR, the metaverse, and spatial computing, including assisting or advising on 10 different XR headset projects. I've had the great fortune to contribute to a range of projects early on, mostly by proving or denying requirements and defining key user experiences before teams spend $1 billion building them. Along the way, I've learned lessons about what works and what doesn't. Sometimes, the correct answer is "not yet mature."

I will not disclose any information that my past employer still considers proprietary. I'll provide links to published patents that can give us some insight. That's the actual purpose of patents, believe it or not. Of course, I would avoid patents that I think would invite speculation.

It's important to be careful not to take anything I say or write as evidence of any company's product plans, or even any criticism of anyone's efforts. This was not my intention at all.

For background, the first real XR experience I built was a CAVE. At the time I borrowed a $250,000 computer and giant projector, and invested another $30,000 in raw materials. Disney's $100,000 VR headset, which was made in the '90s, never made it to the commercial market. It requires cables hanging from the ceiling to carry the weight, but hundreds of thousands have tried it.

By January 2010, I hope we'll be ready to start developing ready-to-use consumer XR glasses. It's time to make a big splash in this slow-moving field. Plus, Google Glass and Magic Leap emerged around the same time. Fortunately, one of the tasks we received at Microsoft's incubation team was to find new ideas for the next generation of XBox.

"Be aggressive, even making us say 'this is crazy,'" an Xbox executive said at the time.

I certainly have "radical" ideas. Our small team soon started working on a new product concept "Screen Zero": replacing all screens with one screen. I was in charge of technical exploration and helped define the experience in the first formative year. My reasons for leaving are not worth the space in this article. But after more than a thousand people, HoloLens launched in 2016.

This is groundbreaking equipment. But it's still not an all-weather consumer wearable. None of today's Magic Leap 2, Snap Spectacles, Varjo or Quest.

So, what exactly does it take to achieve AR glasses that can be worn all day?

1. Maximalism and Minimalism

The maximalist approach, like the one HoloLens eventually took, refers to the consolidation of numerous sensors, algorithms, and power into a high-end system. Once we have the engineering and user experience settled, we can theoretically narrow it down. But that takes more time, and it can take up to a decade for certain electronics to optimize power alone.

The Cambria and similar devices are similarly maximalist, so much so that they simulate AR with a giant opaque VR display and multiple cameras, with precise pixel-by-pixel control over the mix of reality and simulation. Maximalism is best for high-end applications and core R&D. Some argue that even if the applications are more industrial, they are the only tools that work at the moment.

However, even the most expensive devices on the market today cannot be worn all day, and cannot be used for ordinary social interactions or even to walk safely on the streets. Maximalist features such as holographic or light field displays are not yet mature.

On the other hand, devices with a minimalist approach include Amazon Echo Frames, Snap Spectacles, and Ray-Ban Stories, to name a few. They only pack technology that fits within the current constraints of the wearable glasses form factor, often ditching displays altogether.

Still, can glasses without a display count as XR?

If it enhances one's situational perspective, I'd say yes. A podcast or music mix isn't XR because it still doesn't sense you or your current environment. GPS navigation apps are a bit of a match, but not enough. Adding spatial audio and cameras for AI and pinpoint positioning definitely counts as XR, even without a display.

Minimalist approaches can sell more products in the short term, and they usually focus on a specific aspect and make it a best-in-class solution. If you can do it right, like the Walkman and the iPhone, you can sell for billions of dollars. But don't think minimalism is easy, because in many ways, it's hard to do well.

2. Best method

I had hoped that the XBox's Screen Zero was a hybrid between minimalist glasses and a maximalist console. AR glasses should be similar in size to Oakley glasses or smaller. The next host will do most of the heavy lifting for up to four pairs of glasses in the same room.

3. Ergonomics, energy consumption and heat

Why separate? It comes down to energy consumption, or more accurately: heat.

All the work of a computer ends up in the form of heat, and others may include photons and/or mechanical drives, such as sound. How much heat? A pair of lightweight AR glasses can generate about a watt of energy.

A typical smartphone can generate about 10 watts and then heat up. A host or PC can consume 10-100 times more energy than a smartphone, and therefore 100-1000 times more energy than glasses. Think tiny LED flashlights and giant clothes dryers. This is a huge difference in energy consumption.

To balance energy consumption, we need solutions similar to what I have been working on since 2010, such as data fusion, split rendering, optimized rendering streams. For lower bandwidths, eye-tracking-based time warping rendering works well and covers communication delays.

The CPU, camera, display, and RAM consume a lot of energy and generate a lot of heat. So, the secret is to use them sparingly in most cases. Think low-power custom hardware: new contextual sensors, ultra-low-power displays, and algorithms that cleverly "wake up."

This will take time to resolve. For example, if your optical mouse and host controller are not in standby, their batteries will drain, so they start off with a tethered solution. Today, an optical mouse can last several months on a small battery.

4. Outside the living room

To support a wide variety of use cases, optically transparent AR systems need to ideally block light from bright environments (even window or living room lighting), better personalize and mix spatial audio with reality, adjust optical focus, capture and reproduce Virtual holograms of others and more.

Even if the device is divided into the main unit and the headset, there is still too much "stuff" in our head. A huge ergonomic challenge is getting rid of the giant straps that most XR devices still use that make headsets look like face-hugging bugs.

Any kind of straps that require tensioning will limit the diversity of the user base (considering head size variations and susceptibility to messing up hair), and it will reduce the likelihood of wearing glasses. This means that all-weather equipment must be super light and generally fit just as well as regular glasses.

Getting to the size of Oakley glasses or smaller often means moving more of the system into a clip or remote. Magic Leap offers such a clip. I've noticed XR devices using a neckband setup, which means hanging the computing components around the consumer's neck. With this split design, the more separate work is done, the lighter the eye/head assembly can be.

In 2010, I was personally more interested in the biceps because it was an anchor point and a short line if needed. This keeps heavier and hotter components away from the head and neck and has a lot of surface area to dissipate heat. Biosensors located on the arm can also detect gestures, similar to how the Meta uses the Control+Labs device to read gestures from the wrist. In this way, product designers can still say "no cables".

5. Focus

Because a large number of adults need to correct near and far vision, an all-day wearable form factor often needs to magnify and focus the real world for us. At a minimum, this means that the lenses need to support custom vision parameter optics. A waveguide company that Snap acquired earlier announced plans to embed the related optics into functional vision lenses, but that's been difficult.

But is one vision parameter enough? Many adults only need glasses for reading or driving (farsightedness), which means they need glasses to be able to switch to different states. So, do we need two or three pairs of expensive glasses? Or bifocals, trifocals, that bend light accordingly depending on where you look? (Practical, but not ideal)

One solution I'm pushing is dynamic optical adjustment, allowing the same pair of glasses to support reading or driving, and even magnify fine print and distant signs. This also allows more friends to try it out easily. Imagine if your XBox only supports single-player games and you have two or three friends in the room? Kind of a bummer.

The current best dynamic focusing methods include Alvarez (mechanical slide) and liquid-filled adjustable lenses. Mechanical solutions tend to reduce reliability. There's a fair amount of R&D going into stacking special LCDs and changing focus electronically. Mega acquired a company to take on the task.

Then there is the problem of focusing the virtual image based on the current line of sight. Avegant and Magic Leap showed us how to quickly switch between two focal lengths, simulating a simple light field display, which is important for navigating "virtual objects" at the proper focal length within the wingspan. I've looked into several ways to continuously scan the focal length before, but commercial monitors aren't fast enough in practice.

Tracking your eyes can aid processing, reduce computational load, and provide more natural user input. I am very familiar with the eye-tracking problem and alert decision makers to the risks early on.

Finally, on the business side, Luxottica makes a lot of money by selling low-cost, high-priced eyewear. It dominates the market today, and most eyewear brands you know. XR glasses companies have to work with them or against them, neither of which are easy choices. Meta chose to work with them to develop the Ray-Ban glasses. The company's competitors include Warby Parke and other small players. You can't sell good new products without good distribution channels and partners.

6. Contrast

Magic Leap 2 offers a way to selectively blur the natural world. I've been working on this problem since 2010, but there is still no perfect solution. A series of optical engineers did not consider it necessary. The reasons are as follows:

It is generally understood why transparent "additive" displays cannot render "black". RGB=0,0,0 for black, which doesn't actually add anything and is not visible in the existing light. However, we can easily trick you into perceiving blacks and shadows by approaching brighter areas.

The really difficult problem arises when you take your XR glasses outdoors and look at a wall lit by bright sunlight, probably near a dark or shaded area. Certain areas may be 1000 to 10000 times brighter than others. The contrast is so pronounced indoors that AR visuals look downright scary. Optical engineers often argue that to overcome this, you need to output more light. Their optics are typically only 1%-10% efficient, which means most of the light doesn't even get into your eyes and just adds more heat. Recall that you can't just design a system around optics because heat is one of the biggest limiting factors.

The reality is that any pair of transparent AR or video see-through glasses needs to take the real scene into account when making visual enhancements. In the case of transparency, glasses often need to be subtracted from real lighting to obtain the desired final color. In the case of video perspective, the display can replace the entire pixel, but any transparency in the virtual 3D scene still needs to be mixed with the background color read from the camera. So what you're basically looking at is a high-energy camera and circuitry that's either transparent or opaque. This is a huge design constraint as it adds energy and weight while blocking the eyes.

On the surface, selective shading with clear glasses is less expensive than adding power to the display or adding a camera. In 2010, I placed a simple monochrome LCD in front of the waveguide. It works as expected, rendering 3D solid objects with soft black outlines. But it has drawbacks, including the need for dynamic calibration, the LCD distorts real light (mainly the refraction of the control wires). It has poor dynamic range on its own. Outdoors, sometimes you need near 100% opacity. Indoors, especially in social situations and telepresence, you want to be more transparent to see people directly in the eyes.

The main objection to this approach is that LCDs or other spatial light modulators are often out of focus, an inch from the eye. But the distortion is such that with the right plus and minus transparent AR displays and some fast, low-power sensors, your sunglasses can block sunlight, glare or headlights without dimming your vision elsewhere. You can subtly darken the world and make recommended books look sparkling. With more advanced subtraction (filtering), the glasses can even recolor the world, enhance night vision, and even provide biometric feedback when you're feeling restless or distracted.

I made various demo stuff and spent a long time looking for a better way. But they all have certain drawbacks. However, the implementation of Magic Leap 2 gives me hope that the core issues will be resolved.

7. Network

Radios also require energy, so there is always a tradeoff in a split system. The most promising future lies in the use of higher-end radio frequencies to achieve lower power and higher bandwidth than today. But the main challenge is that this frequency cannot pass through skin or walls (for better or worse). So, the solution needs to be very clever, as radio waves bounce and beam around rooms and people, so more transmitters may need to be used than there are today. Obviously, this adds cost and complexity.

For all-day wearables, it also requires the network to be available before selling products that rely on it. This limitation is the biggest reason why businesses never release the split rendering solution I advocate. 5G is closer to what we need, but at least in the US, this mostly solves the problem of lower latency and more people using the network at the same time. We need more than 5G, but it's a good start.

To get rid of the original "host" (or similar) in the room, while maintaining a small and lightweight form factor, we need a way to "edge" computing, combined in a way that doesn't invade our privacy. Sending their biometric sensor data to any edge or cloud solution is very concerning for anyone because it can be easily abused.

8. Camera

Putting the camera to the glasses is a tricky one. Google Glass has made many mistakes in social acceptance and has come under widespread attack. But Snap seems to have few problems in that regard. At the same time, Facebook has been working on capturing the details of everyone's life, presumably in order to deliver more personalized ads, whether we like it or not.

Certain cameras are energy-intensive, such as 3D scene digitization, and digital occlusion of people or objects. To properly place 3D graphics, you need to continuously track the head in space, and cameras are still the leading solution. By leveraging IMU sensors, we are increasing energy consumption.

Taking photos or videos is a fairly popular use case, especially if it's more natural and convenient than other devices. However, due to size and power constraints, the quality of the photos will be lower than that of a typical smartphone. Providing a small white light in the frame is not enough to solve the complex problem of social approval.

Scene understanding is easier to imagine as a major feature of camera glasses, in part because it doesn't have to take pictures of other people, but more importantly it opens up the most important new use case for wearing glasses all day: situational understanding.

9. Experience

In 2010, the R&D I most hope to develop is AR glasses that support powerful eye-tracking and body-tracking capabilities, and explore a more natural spatial computing user interface beyond the traditional "rectangle within a rectangle" of PCs and mice. While the hardware certainly has its limitations, widespread adoption of XR requires someone to address the experiential question of "how" to interact in the future? "3D box in a box" is clearly not. So we still have a lot of work to do.

While Meta is focusing their VR efforts on passing the so-called "Visual Turing Test," XR glasses that can be worn throughout the day need to be more useful than others. Many imagine AR layers or channels permeating our reality, marking everything we see, adding information, telling spatial stories in 3D, and redrawing the world. While this is likely to be on-demand, that's not the day-to-day experience I'd expect.

Most of the time, people want to improve what they do often: communicate, navigate, discover the world around us, understand and even change a place, shop, experience content, and make money from work. To be successful, XR glasses need to do better than we do with smartphones or other devices.

Here's what a smartphone can't do. Imagine a pair of normal-looking glasses that dynamically adjust focus and selectively block light. They can talk to you proactively and privately, without you having to type text or ask verbally. That alone would be a billion-dollar product. Such glasses can help you remember things or provide trustworthy recommendations as part of your daily experience (vs. push ads).

The most important research I've done in this area is asynchronous communication using non-visual XR glasses. Smartphones’ voice and text capabilities are good enough today, but do they know when you’re struggling to focus? Can they help you switch contexts at the right time to keep work or play flowing? Here’s how Where the glasses can shine (assuming we can trust the manufacturer).

Everything I've described is very hard, and the relevant technology is almost non-existent. It's not yet on a minimalist track, but that's because we haven't prioritized it over miniaturization of optics and maximization of the field of view. But if you ask the question: What kind of XR glasses can succeed where others haven't? I still think everything I've listed above will help you build a great product.