As most of you inside of the mixed reality (MR) industry know, MR was first defined in 1994 by Paul Milgram and Fumio Kishino in their much cited paper called “A Taxonomy of Mixed Reality Visual Displays”. According to this, MR is everything between the real and virtual environment, essentially any experience where both elements of the real and virtual co-exist. Please note, that it does not include the 100% real and 100% virtual environment endpoints on the far left and right of the spectrum. My long-time friend and sparring partner Mark Billinghurst actually summarized the paper nicely in his Medium post “What is Mixed Reality?”.
As much as this has helped me to explain what Mixed Reality, Augmented Reality and Virtual Reality are, I have always struggled with this depiction, because of mainly two reasons:
1. Mixed Reality is more than just what we see
As the title suggests, Milgram and Kishino focus on everything visual in their “A Taxanomy of Mixed Reality Displays” paper. In a later and second paper, Milgram even lists seven types of MR displays to explain further the various combinations possible between real and virtual environments. He describes elements like videos, TVs, optics, monitors, displays, images, 3D graphics, see-through HMDs etc. As legit as it is to focus on the visual, it is also very limiting as any experience we have goes way beyond what we see only. We also feel/touch, hear, taste and smell. Sometimes, if not most of the time, these senses contribute significantly as to how we perceive reality and increasingly also Mixed Reality experiences. There are countless of devices out there on the market to stimulate our various senses. For example, they stimulate our sense of smell ( https://feelreal.com/ ), our sense of touch ( https://haptx.com/ ) to trick our brains into thinking that what we experience is real rather than virtual.
2. The Mixed Reality continuum should be circular
Before getting into the circular rather than linear argument, let me fill the continuum with some examples that we can draw from technology available or at least almost available to us today.
The far left point of the spectrum is Reality. Without any co-existence of Virtuality, standing in a tropical forest can serve as a good example of experiencing Reality only. Billboards and digital screens at Times Square in New York city are already a form of Mixed Reality, even without any smartphone displays or AR glasses at play – ok, understood.
With simple smartphone based PokemonGo-like experiences and by wearing head-mounted displays like a Hololens or Magic Leap, we are moving along to the right on the continuum, mixing more virtuality with reality. VR headsets with see-through AR modes are somewhere in the middle of the continuum as both reality and virtuality can be rendered in various combinations of dominance in front of your eyes.
Now, let’s add more senses. The Void takes this to another level as not only your vision and hearing are manipulated, but also your sense of touch as you are exploring a physical environment with real haptic elements corresponding to your virtual experience inside of the headset. This is perhaps as far right on the continuum as we have come so far with today’s technology, at least in a commercially available way.
The next step towards the far right on the continuum would be something like the OASIS as shown in Steven Spielberg’s movie Ready Player One, where the characters wear more advanced future versions of today’s VR goggles, suits and gloves to stimulate the body’s senses from the outside. The characters in the movie know that they are in an artificial world to play and they know they can come back to reality at a push of a button.
Let’s move even further to the right, actually as far right as we can. In the original Matrix movie, Neo lived in a completely virtual world indistinguishable from reality, ingested directly into his brain, no goggles required. (If this sounds like a thing of the future, have a look at Elon Musk’s latest venture called Neuralink, a company focussing on brain implants to connect computers directly to the human brain.) The virtual world created for Neo is so perfect that he is not aware of the fact that everything he experiences is a simulation only. This all encompassing simulation is what I would define as 100% virtuality, therefore the end point of the continuum on the far right.
When a simulation is 100% virtual and you are not aware of being in it, it is no longer virtual but subjectively reality. That’s why the two points on each end of the linear continuum are actually the same, and therefore should be connected and depicted as a circle. As you have heard I am sure, many scientists out there argue that we could all be in a simulation right now, in a 100% virtuality, our reality. If Elon Musk’s tweets on this theory don’t convince you, perhaps the article “Do We Live in a Simulation? Chances Are about 50–50” recently published in the Scientific American journal will.
So. Who cares?
The purpose of this article is of course not to degrade Milgram’s work, quite the opposite. I think he is one of the leading pioneers to define the meaning of Mixed Reality. However, by incorporating all of our senses to this thought experiment and by bending Milgram’s linear continuum to a connected circle, we may better illustrate the destination we are aiming for. The more our technologies advance, the more senses we stimulate and the more the quality of what we feel, see, touch, smell increases, the closer we get to a perfect simulation. A perfect simulation is 100% virtuality, which in turn will become our perceived reality and connects the two ends of the spectrum. Considering the speed of advancements in the last five years alone, it is only a matter of time until we reach that point. As they like to say, it is not a matter of if but a matter of when. I don’t know if it will be another 10, 20 or perhaps even more than 100 years. What I do know is that we should take the time we have to think about the trajectory we are on. We need everyone, and I mean everyone outside of our industry “bubble” to know about it. The circular Mixed Reality continuum can perhaps help illustrate the relationship between 100% virtuality, a perfect simulation and subjective reality so that we can better discuss and prepare for the phenomenal opportunities but also the significant risks that come along with it.
