The Vision Pro's scary side effect

The reviews are in, and the tech press is lauding the Apple Vision Pro headset for delivering on the company's promises. It's well-designed, the video and sound are startlingly precise, the "Minority Report"-style gestural interface is future-tastic. Nobody's exactly sure what it's for, or whether even the Readiest Players One will spend $3,500 on it, but hey — that's gadgets for you.

Still, this is a new gadget frontier. The Vision Pro, like the similarly kitted-out Quest 3 and Quest Pro headsets from Meta, uses what's known as "passthrough" video — cameras and other sensors that capture imagery of the outside world and reproduce it inside the device. They feed you a synthetic environment made to look like the real one, with Apple apps and other non-real elements floating in front of it. Apple and Meta are hoping that this virtual world will be so compelling that you won't just visit. They're hoping you'll live there.

That, unfortunately, could have some very weird and very messy consequences for the human brain. Researchers have found that widespread, long-term immersion in VR headsets could literally change the way we perceive the world — and each other. "We now have companies who are advocating that you spend many hours each day in them," says Jeremy Bailenson, director of the Virtual Human Interaction Lab at Stanford. "You've got many, many people, and they're wearing it for many, many hours. And everything magnifies at scale."

Meaning: Our brains are about to undergo a massive, society-wide experiment that could rewire our sense of the world around us, and make it even harder to agree on what constitutes reality.

The short-term side effects of virtual reality are well established. People in synthetic environments tend to misjudge distance, both at a distance and close up. That's no surprise: Even in the real, three-dimensional universe, our ability to determine how close or far away something is is subject to all kinds of external factors. Virtual environments, with their lower resolution and synthetic 3D, make all that worse — which is especially bad if you're one of those users posting videos of yourself doing things like skateboarding and driving while wearing a mixed-reality headset. You think your hands are in one place, they're actually in another, and pretty soon you're driving your Honda Civic through a supermarket.

Objects in a headset can also get funhoused. That's called object distortion — things get warped, and change size or shape or color, especially when you move your head. A video render can't compete with the processing speed and fidelity of your eyes and brain.

These are all, as the IT people say, known issues. For a few minutes or an hour, long enough to play a game or watch a movie, they're minor annoyances. But wear perception-shifting glasses for days at a time — as Bailenson's team of researchers did — and the problems get worse. Way worse.

The team wore Vision Pros and Quests around college campuses for a couple of weeks, trying to do all the things they would have done without them (with a minder nearby in case they tripped or walked into a wall). They experienced "simulator sickness" — nausea, headaches, dizziness. That was weird, given how experienced they all were with headsets of all kinds. And they felt all the distance and distortion effects: thinking elevator buttons were farther from their fingers, or experiencing difficulty bringing food to their mouths. But as any of us would, they adapted — their brains and muscles learned to compensate for their new view of the world.

That seems like a solution, but it ain't. When people adapt to a perceptual change for long enough, the real world starts to look wrong in the opposite direction. If you wore glasses that turned your vision upside down, let's say, you'd have to adapt again when the glasses came off. The longer you're inside a funhouse world, the longer the weird perceptual aftereffects last. So people who spend their workday inside a Vision Pro might go home at night with a miscalibrated targeting system and what feels like a shroom hangover.

Here's where the passthrough video gets uniquely important. Old-school cyberpunk envisioned virtual reality as an all-encompassing synthetic environment. New-school techies, meanwhile, proposed an augmented reality of digital pop-ups floating on see-through lenses, Google Glass-style. But both of those approaches have limits. Full, sense-isolating VR hasn't progressed much further than niche entertainment, while AR tends to make both its apps and the real world look bad. From a visual standpoint, passthrough is the least-worst solution — but its social consequences are scarier.

Because passthrough captures and then re-renders reality, it can have an unnerving, distancing effect over time. When Bailenson's colleagues actually tried to talk to people, the world turned into a giant, confusing Zoom. Video chats, as we've all experienced, are plagued by delays in responses and missed social cues. Conversations lose subtlety, but it's good enough for a meeting. But passthrough magnifies the effect — the people you talk to start to seem unreal. Up close, they look like avatars. Farther away, they become just part of the background.

Bailenson describes the feeling as one of social absence. Other people just aren't fully there. He doesn't put it this way, but I'll wave the warning flag: Long-term use of passthrough headsets could make it easier to think of other people as unhumans — non-player characters in a gamified, uncanny valley.

We all live in our own perceptual bubbles. Every person has slightly different sensory thresholds — we see colors a little bit differently, hear at different levels of acuity, are more or less sensitive to different odors. And we process all that with brains uniquely tuned first by our genes, and then by a lifetime of neural changes, of thinking and doing.

But in general, we agree on some common ground. Even if your blue looks a little different than mine, we can agree on what color the sky is. Maybe my tolerance for chili peppers is higher than yours, but we both know when we're eating them.

Headsets make the walls of those sensory bubbles even thicker, and harder to bridge. We already lack for common ground politically. Now, as millions of Americans wear VR headsets for hours at a time, we may find ourselves unable to agree on our physical reality. The headsets will put things into our visual world that aren't there for anyone else. The objects aren't objective.

And that's not all. "These headsets can not only add things to the real world, they can also delete them," Bailenson says. He first realized VR's strange editing function while he was playing a game on the Quest 3 that "knocked out" portions of the real walls around him and replaced them with a virtual scene. "I've been doing VR and AR for a while," he says, "and I had never in my life seen deletion work so well."

At first that seems pretty great. Stuck on a crowded bus? Delete everyone and replace them with the first-class cabin of a jumbo jet. Hate intrusive billboards? Replace all the commercial images with soothing vistas of your choosing.

But what happens when the technology gets good enough to delete, say, homeless people? Or Pride flags? You can see where I'm going here — literal erasure. When the sci-fi writer William Gibson came up with the concept of cyberspace, he described it as a "consensual hallucination." This is the exact opposite — billions of discrete, unshared hallucinations, each one snowflake-special.

"What we're about to experience is, using these headsets in public, common ground disappears," Bailenson says. "People will be in the same physical place, experiencing simultaneous, visually different versions of the world. We're going to lose common ground."

Stipulated: Everyone always freaks out about new consumer technology, and the freak-out is almost always the same. The new form of sensory input will harm kids! It's a dangerous distraction! It's socially alienating! They said it about the iPhone, about the Walkman … hell, half a millennium ago they said it about the book. New technologies arrive, and we adapt.

And I don't need to lean hard on my nerdiness to imagine fun sci-fi uses for passthrough. The real potential here is the ability to see the invisible informational metastructure of the world — translation overlays; pop-up tags that display people's names and pronouns and where you know them from; walking directions; user-manual-linked X-ray vision for assembling an Ikea coffee table. Link my shopping list to the aisles I need to visit in the supermarket. Maybe even expand my vision beyond what my meatbag eyes can do, and let me see into the ultraviolet, or perceive electrical fields. Passthrough has limits, but it might also have superpowers.

Like me, Bailenson isn't a freaker-outer; he loves VR, and thinks the new headsets are keen. He knows that over time, screens will get better resolution and faster rendering. New algorithms will minimize distortion. It's not the technology he's worried about. It's how much we're going to become immersed in it.

"The world's going to be just fine," he says. "People adapt to media. These headsets are incredible. But philosophically, I do not believe we need to be wearing these headsets for hours every day."

We've been here before — and very recently, at that. A decade or so ago, no one paused to consider the unintended consequences of thrusting millions of people into impossible-to-moderate social networks. And we all know how that turned out. Now, we're on the verge of sticking millions of people into helmets that give us all our own editable realities. That's why the kind of research that Bailenson is conducting on passthrough headsets is so important. "I'm encouraging all scholars to act with some urgency to understand them," he says.

In the meantime, while he's doing his work, maybe don't forget to take that Vision Pro off once in a while. The longer you keep it on, the more you're turning yourself into a human guinea pig — one with really, really bad depth perception.

Adam Rogers is a senior correspondent at Business Insider.