Virtual
adjective
~ almost or nearly as described, but not completely or according to strict definition.
Synonyms: effective, in effect, near, near enough, essential, practical, to all intents and purposes
For Computing:
~ not physically existing as such but made by software to appear to do so.
Reality
noun
~ the world or the state of things as they actually exist, as opposed to an idealistic or notional idea of them.
~ a thing that exists in fact, having previously only existed in one’s mind.
the quality of being lifelike or resembling an original.
Synonyms: the real world, real life, actuality
Put those two together, and maybe you will begin to understand the concept behind today’s current rage in technology. To be able to put yourself in an artificial reality – a world that appears to exist, but has been designed or manufactured by your peers, or maybe even yourself. VR in this era spans across many genres: games, experiences, film, training in the science and medical fields, education, digital painting and sculpture, and so on.
This session takes a look at this fantastic addition to our digital interactive experiences and explores where it got its start, how it works and where we think it might lead us in the next few years ahead of us.
Today’s virtual reality technologies stem from inventions that date back to the 19th century, almost to the very beginning of practical photography. In 1838, the first stereoscope was invented, using twin mirrors to project left-eye and right-eye views of the same scene, as a single three-dimensional image.This technology eventually developed into 3D slide viewers like the View-Master, patented in 1939 and still produced today.
The use of the term “virtual reality,” however, was first used in the mid-1980s when Jaron Lanier, a former ATARI employee began to develop the gear, including goggles he named the Eye Phone and gloves known as Data Gloves, needed to experience what he called “virtual reality.” He licensed this to the toy company Mattel who produced a product called the Power Glove, but it really didn’t take off.
Even before that, however, tech engineers were developing simulated environments. One milestone was the Sensorama in 1956 by Morton Heilig’s. Morton’s background was in the Hollywood film, and he wanted to see how people could feel like they were “in” the movie. The Sensorama experience simulated a real city environment, which you “rode” through on a motorcycle. Multisensory stimulation let you see the road, hear the engine, feel the vibration, and smell the motor’s exhaust in the designed world.
Heilig also patented a head-mounted display device, called the Telesphere Mask, in 1960. He described it as as “a telescopic television apparatus for individual use…The spectator is given a complete sensation of reality, i.e. moving three dimensional images which may be in colour, with 100% peripheral vision, binaural sound, scents and air breezes”. Many inventors would build upon his foundational work.
By 1965, another inventor, Ivan Sutherland, offered the Ultimate Display, the first head-mounted device that he suggested would serve as a window into a virtual world. The device was used in immersive simulators. This thing was absurdly heavy, though, and had to be suspended from above the user.
The 70’s and 80’s were an exciting time in the field. Optical advances ran parallel to projects that worked on haptic devices and other instruments that would allow you to move around in the virtual space. At NASA Ames Research Center in the mid-1980s, for example, the Virtual Interface Environment Workstation (VIEW) system combined a head-mounted device with gloves to enable the haptic interaction.
Through the 1990’s, the gaming industry took a stab at incorporating virtual reality into their games. The first publicly available VR games came in the form of specially made arcade machines. These provided stereoscopic visuals at a low resolution and the ability to navigate around in the virtual world and interact with controls. But success in gaming’s early VR was rare. Sega introduced the Sega VR for the Genesis at CES in 1993 but the product never made it to production for store shelves. Even Nintendo jumped into the VR realm with its Virtual Boy – a 32-bit 3D video game table-top console complete with a headset. Unfortunately this unit was a flop due to its high price and graphics that were limited to the colors red and black – not very appealing… or real.
After this, VR pretty much vanished until 2012. This is when the Oculus Rift was introduced to the world as a breakthrough Virtual Reality device during a crowd-source fundraiser on Kickstarter. The project far exceeded its financial goal and was able to go into actual production and saw great success. So much so that the company was purchased by Facebook in 2014 for $2 billion.
Today’s current virtual reality gear owes a lot to the pioneering inventors of the past six decades who paved the way for the low-cost, high-quality devices which are now easily accessible.
Please watch this 20 minute documentary by VR Roundtable on the history of virtual reality. It goes into a bit more depth and has some great visual accompaniments.
So what is actually happening to our brains when we engage in a VR experience? In a way, it’s kind of magic, or a trick. We exist today with a lifetime of worldly interactions – a majority of which comes from our visual observations. We have an immense library of rules and understanding of how our physical world works. The sky reminds us which way is up. Shadows tell us where the light is coming from. The relative scale of things helps us grasp distance. These are all things that VR developers have to acknowledge and figure out how to apply them correctly to the artificial environments they are designing. If they don’t, a user’s experience could prove to be quite unpleasant. In an effective virtual environment, moving objects should follow your expectations of the laws of physics. Shading, texture and focus should allow you to determine depth and distance. Sometimes, when the virtual cues don’t quite match your brain’s expectations, you can feel disoriented or nauseated. This is known as cybersickness – related to motion sickness, but with perceived motion instead of physical. Because the human brain is much more complex than even the most sophisticated computer, developers are still trying to understand which cues are most important to focus on with VR.
Today’s science involved in virtual reality is not complete rocket science. More a perfect timing of surfacing technologies coming together for a unique collaborative experience. Most of the success can be attributed to the mobile phone industry – this is where a lot of the breakthrough innovations have been emerging. Let’s have a look inside the mechanisms involved with VR so we can better understand how it works and what is actually at play.
The typical headset relies on either two separate screens – one for each eye, or one elongated screen that can display dual images. Lenses are used to help magnify and focus the visual content directly to your eyes without crossover or interference. A dedicated headset will power its own displays via a series of power and AV cables that are connected to a gaming console or PC. The more accessible and affordable units are merely a housing for an individual’s mobile device to be inserted and used as the content delivery, viewed through similar magnified lenses and a goggle-like wearable headset. These VR systems rely on the mobile phone or tablet hardware and downloadable apps to provide the interactive experience. This can be quite engaging, depending on the processor, graphics display, size and app programming. Many users find that this approach comes with some distracting deficiencies – latency issues, trouble with the lack of precision in device placement and often the mobile phone is just not wide enough to saturate the visual scope.
Hard-wired headsets are capable of much more performance, relying on the PC or console’s hefty processors, fast and memory-packed graphics cards and industry-leading content developers with bigger budgets. The issue here is that they are tethered. The amount of wiring that is involved here can be very overwhelming and burdensome for the user. One can get tangled or yank the cables out with too much movement, or even run into a more distressing situation: not enough ports to support the necessary peripherals. The Oculus Rift needs at least three USB 3 ports and an HDMI output. It also requires a beefy video card, speedy processor, ample RAM and a recent OS. This can get spendy real quick for someone ready to jump in to the higher end VR experience, as the Oculus itself costs $400.
The technology within the headset (or mobile phone) that plays a leading role in the responsiveness of the user is the accelerometer. Most mobile phones and tablets are equipped with this device, and the wired dedicated headsets come with them built in to the package. An accelerometer is an electromechanical device that will measure acceleration forces. Liken it to a carpenter’s level, only it’s miniaturized and electronic – allowing it to communicate with a computer or microcomputer. These provide orientation sensing feedback that tells the computer what the user’s head is doing, essentially. Looking up, down, turning to the left, etc. This information is processed and interpreted by the software in order to deliver a lifelike sensation of looking around in one’s virtual environment. For headsets that require a phone or tablet, the user-feedback experience stops here.
For dedicated headsets, another useful technology is available – motion sensors. These devices rely on infrared and/or standard video feedback of either infrared lights that the headsets emit in several locations on the front of the housing, or the actual human form that is front of the camera – interpreted by body detection programming written into the code of the programs that are running the VR experience. This input gives the computer the ability to process the distance of the user, what kind of movements are being executed and even differentiate and locate parts of the body, like arms, hands, torso and head. They can even detect multiple persons and their movements as well. It’s quite impressive.
The higher end VR sets also have built in to themselves microphones that upon vocal expression will process the sound files and incorporate them into the virtual reality scene, such as moving a 3D character’s mouth real-time as the VR user speaks. And of course, audio dedicated to the user is an important factor, with many games and experiences/movies providing full surround sound to help connect the person to the world in as natural a way possible.
All of these technologies coming together in this one interactive package is a wonderful example of how emerging science and research can deliver groundbreaking products and connect people to new experiences and growing moments.
Let’s have a look at some of the applications of VR and the impact it’s having in our society. As stated earlier, virtual reality in this era spans across many genres: games, experiences, film, training in the science and medical fields, education, digital painting and sculpture, and so on. This technology also allows us to travel to places we never have and maybe never will travel to, exploring historic monuments, museums, locations that are nearly impossible for humans to be at (underwater, in a volcano, the International Space Station, Mars, etc.). We will show some examples of a few of these areas in order to provide some insight as to how it might play a role in your life, or how you might see yourself as a contributor to an experience or interactive expression.
TRAINING
Let’s say you are enrolled in a biochemistry program at the university and just starting your first year of studies. Your access to labs and resources is quite limited, and for a good reason: you are too green and could make a critical mistake and/or possibly expose yourself or fellow students to danger. With VR, we are seeing new controlled and safer ways to introduce students and researchers to laboratory experiences that have no impact on physical spaces, resources or other people, yet still provide the necessary instructions on procedures and tests that can have a tremendous impact on the educational moments for these fields of study. The sciences are just starting to see the tremendous impact on training and education VR can have. Imagine students having tried and failed again and again until they get something right before ever stepping foot into a real lab, when then they can do their work and research in a much safer and competent fashion.
The same goes for the medical field. Doctors are getting superior training on anything from simple daily procedures to intense organ transplants, all without the possibility of making unwanted if not fatal mistakes. They can even practice a procedure several times before executing the real thing days or hours in advance. The wildest thing to have happened in medicine regarding VR might just be this next story.
In 2016, Italian neuroscientist Dr. Sergio Canavero shocked the medical world when he announced that he would be able to execute the first transplant of a human head onto a new body. The patient is a Russian man who suffers from a fatal genetic disorder that wastes away his muscles and neurons that control his body. A major concern in the medical community is that transplanting a head onto a new body could create serious confusion and madness. That’s a huge switch for anybody, and who knows if the patient would be psychologically prepped for it. So Canavero found an engineering company to develop a VR experience to help the patient prepare for the dramatic body-swap. The VR therapy would be used for 3-4 months before the surgery and would (in theory) help cover not just the sensation of looking down and seeing someone else’s body, but also some of the distress and symptoms of chemical imbalance and neurological fatigue and reestablishment. CRAZY!
ART MAKING
Chances are you have already had some thought about what it might be like to create art and content for VR games or experiences. But have you thought about creating art with VR? This is a relatively new form of artistic expression, and there’s plenty of development that is still going on as we speak, but creative minds have been hard at work to see what it is that VR can allow us to do to make marks, build things and share ideas and stories. One popular program that got its start early in the game is the 3D painting software by Google, Tilt Brush. This allows the user to paint in 3D space – giving you the freedom to more or less sculpt your strokes as opposed to laying them down on a two-dimensional canvas or paper. The interactivity is quite responsive and provides a wide range of tools, brushes and palettes, not to mention the great digital bonus of the Undo and copy/paste. It also supports mirroring for painting in symmetry and the ability to export in many formats for various interfaces and deliveries, i.e. color printing, 3D printing, export to image editing and 3D modeling software.
But what is even more interesting is the ability to share your work with other VR users. Now you can experience a painter’s elaborate piece by walking or moving into or through it – seeing it from any angle or distance you desire. Sculpture is seeing a breakthrough with the use of VR, giving the artist access to any virtual medium and at any scale. A ceramic artist can throw a pot traditionally in the virtual space, or start a piece by cutting away at a large block and creating a work via reduction. Then send the file to the library or local makerspace and have it printed in a matter or hours.
In 2015, futurist Keram Malicki-Sánchez created the FIVARS Festival of International Virtual & Augmented Reality Stories, the first Virtual and Augmented Reality showcase dedicated wholly to narrative forms specific to spatialized media. In 2016, the first VR project in Poland called The Abakanowicz Art Room was realized – it was a 360-degree documentation of the art office Magdalena Abakanowicz, made by Jarosław Pijarowski and Paweł Komorowski.
In reaction or acceptance to the VR movement across the globe, many art museums have begun making their content accessible via VR including the British Museum and the Guggenheim. This allows access to major historical and critically acclaimed works for essentially anyone in any place at any time – and that is wonderful.
CINEMA
Being a visual medium, it is not surprising that the film industry looked at the new and improved emergence of VR and said, “Ooooh. I want.” But it was not an easy adoption of the technology needed to make 3D movies truly work and be accepted by the masses. If you think about the first challenge, the lens, it was obvious that one was not enough. So filmmakers and engineers started to experiment with multiple lenses and varying lens types in order to begin to capture what was close to full 360 degree images. What helped was the access to smaller and smaller cameras that could record digital pictures and advanced software and image recognition algorithms that could stitch together the immense amount of visual information and output one cohesive movie file. After this hurdle was tackled, even more important creative questions arose. What about the frame? How do we direct the viewer’s eye? How do we cut? And it was also obvious that when creating a VR film, it will almost always be perceived as a first-person experience.
Developers of these VR or 360 films have found some useful options for delivery thanks to new advances in playback coding and online services. You don’t necessarily need a VR headset to watch. Youtube and smartphone developers have made it easy to watch this content within the confines of the standard screen. The viewer just needs to swipe a finger or move the mouse in the direction they desire to explore what is going on around them.
Have a look at this video to see how VR is being implemented in today’s medical industry.
Watch this video from Google, showcasing an amazing experiment of artists in residence – traditional artists using Tilt Brush and how their work was transformed.
Take a look at this creepy 360 VR film that the Lionsgate studio made to help promote the remake of Stephen King’s IT. You will be able to look around in full 360 view while Pennywise the clown takes you on a short journey.
And now please watch this presentation by Google’s Principal Filmmaker, Jessica Brillhart on the challenges that VR movie makers face in this new era.
This technology having seen quite an extended history, with many failures and falls from fashion, faces some interesting scrutiny. Is it here to stay? Will it really provide the solutions and entertainment and experiences that it currently promises? Chances are that it will last, and will see some major advances – not just technically, but socially and ethically.
Virtual reality could eventually impact all of the senses. Right now, VR is limited to a user’s visual and auditory senses, but in the future this will likely be enhanced. The general manager of Dell’s gaming PC manufacturer Alienware, Frank Azor, said in an interview with TIME last year, “Once you begin catering to the rest of the senses, like what we feel body-wise, temperature-wise, and smell, the reality factor of virtual reality becomes stronger and the virtual piece begins to fade.”
Breaking the barrier between the virtual world and the physical world with high-end touch sensors could be the next step in virtual reality, and it won’t be here for a while, but VR hardware and software makers are already thinking about how this technology can go from immersive to fully interactive.
Wires will surely go away, providing easier setup and more mobility. The price will have to come down if it is to catch on and stay around. It is predicted that virtual reality will find a way to unite with augmented reality (AR, the topic of our next session). This marriage will undoubtedly manifest in new lightweight and portable wearable gear, if not end up as bio transplants that directly tap into our optic systems. We will dive into this topic in greater detail in a future session for this class.
Jesse Schell, an American video game designer, delivered an interesting presentation at the 2016 Virtual Reality Developers Conference (VRDC) titled, “40 Predictions for VR/AR Through 2025”. Watch it here and see what ideas you might have about VR’s future after processing his thoughts.