Augmented Reality

VR vs AR
Many of the technologies are similar for AR and VR systems. For example, HMD may be used in both systems; fast real-time rendering processes are necessary to achieve sufficient performance; tracking of the user is required in both environments; both systems need immersive environments. However the differences are quite significant.

The very visible difference between the two systems is the type of immersiveness. Most VR systems require a totally immersive environment. The visual, and in some systems haptic senses, are under control of the computer –  the VR system models the artificial world completely and actually provides immersiveness with the virtual world. In contrast, an AR system augments the real world scene and attempts to maintain the user’s sense of being in the real world. The user sees their actual surroundings but with the addition of computer generated images which are overlaid on various objects within the real world. They are still aware that they are in the real world as compared to the full immersion in a virtual world.

VR systems operate with large data sets (especially when the real world is focused) and often requires high realism and needs textures, real photographs, or special illumination approaches to achieve the required level of realism. In contrast, the number of objects to be rendered in AR systems are relatively few (one car, one building, etc.). Also the requirements for realism (textures, colours, illumination and shading models) are significantly low compared to VR models. Many objects such as trees, traffic lights, guiding boards that have to be introduced in VR models, are not of interest in AR systems.

Key Components to Augmented Reality Devices

Sensors and Cameras
Sensors are usually on the outside of the augmented reality device, and gather a user’s real world interactions and communicate them to be processed and interpreted. Cameras are also located on the outside of the device, and visually scan to collect data about the surrounding area. The devices take this information, which often determines where surrounding physical objects are located, and then formulates a digital model to determine appropriate output. In the case of Microsoft Hololens, specific cameras perform specific duties, such as depth sensing. Depth sensing cameras work in tandem with two “environment understanding cameras” on each side of the device. Another common type of camera is a standard several megapixel camera (similar to the ones used in smartphones) to record pictures, videos, and sometimes information to assist with augmentation.

Projection
While “Projection Based Augmented Reality” is a category in-itself, we are specifically referring to a miniature projector often found in a forward and outward-facing position on wearable augmented reality headsets. The projector can essentially turn any surface into an interactive environment. As mentioned above, the information taken in by the cameras used to examine the surrounding world, is processed and then projected onto a surface in front of the user; which could be a wrist, a wall, or even another person. The use of projection in augmented reality devices means that screen real estate will eventually become a lesser important component. In the future, you may not need an iPad to play an online game of chess because you will be able to play it on the tabletop in front of you.

Processing
Augmented reality devices are basically mini-supercomputers packed into tiny wearable devices. These devices require significant computer processing power and utilize many of the same components that our smartphones do. These components include a CPU, a GPU, flash memory, RAM, Bluetooth/Wifi microchip, global positioning system (GPS) microchip, and more. Advanced augmented reality devices, such as the Microsoft Hololens utilize an accelerometer (to measure the speed in which your head is moving), a gyroscope (to measure the tilt and orientation of your head), and a magnetometer (to function as a compass and figure out which direction your head is pointing) to provide for truly immersive experience.

Reflection
Mirrors are used in augmented reality devices to assist with the way your eye views the virtual image. Some augmented reality devices may have “an array of many small curved mirrors” (as with the Magic Leap augmented reality device) and others may have a simple double-sided mirror with one surface reflecting incoming light to a side-mounted camera and the other surface reflecting light from a side-mounted display to the user’s eye. In the Microsoft Hololens, the use of “mirrors” involves see-through holographic lenses (Microsoft refers to them as waveguides) that use an optical projection system to beam holograms into your eyes. A so-called light engine, emits the light towards two separate lenses (one for each eye), which consists of three layers of glass of three different primary colors (blue, green, red). The light hits those layers and then enters the eye at specific angles, intensities and colors, producing a final holistic image on the eye’s retina. Regardless of method, all of these reflection paths have the same objective, which is to assist with image alignment to the user’s eye.

Control
Augmented reality devices are often controlled either by touch a pad or voice commands. The touch pads are often somewhere on the device that is easily reachable. They work by sensing the pressure changes that occur when a user taps or swipes a specific spot. Voice commands work very similar to the way they do on our smartphones. A tiny microphone on the device will pick up your voice and then a microprocessor will interpret the commands. Voice commands, such as those on the Google Glass augmented reality device, are preprogrammed from a list of commands that you can use. On the Google Glass, nearly all of them start with “OK, Glass,” which alerts your glasses that a command is soon to follow. For example, “OK, Glass, take a picture” will send a command to the microprocessor to snap a photo of whatever you’re looking at.

Many of the top augmented reality companies are seeing great success by helping seasoned industries adopt and apply this new technology for their unique business needs. A strong example of augmented reality in use is in the field of healthcare. From a routine checkup, to a complex surgical procedure, augmented reality can provide immense benefits and efficiencies to both patient and healthcare professional.

Physical Exams
Imagine that you walk into your scheduled doctor (or dentist) appointment, only to find your doctor (or dentist) wearing an augmented reality headset (e.g. Google Glass). Although it may look strange, this technology allows him or her to access past records, pictures, and other historical data in real-time to discuss with you. Instantly accessing this digital information without have to log into a computer or check a records room, proves to be a major benefit to healthcare professionals.

Integration of augmented reality assisted systems with patient record management technologies is already highly desirable utility. Data integrity and accessibility is a major benefit to this type of system, where record access becomes instantaneous and consistent across all professionals to the most current records, instructions, and policies.

Surgical Procedures
Let’s take this example one step further and imagine that we are going in for a surgical procedure. Before the anesthesia takes effect, we notice that the doctor is wearing an augmented reality headset. The doctor will use this throughout the procedure for things such as display of surgical checklists and display of patient vital signs in a dashboard fashion. Augmented reality assisted surgical technologies assist professionals by providing things such as interfaces to operating room medical devices, graphical overlay-based guidance, recording & archiving of procedures, live feeds to remote users, and instant access to patient records. They can also allow for computer generated images to be projected onto any part of the body for treatment or can be combined with scanned real time images. The benefits of using augmented reality include a reduced risk of delays in surgery due to lack of familiarity with new or old conditions, reduced risk of errors in performing surgical procedures, and reduced risk for contamination if the device allows surgeons to access information without having to remove gloves (i.e. hands-free) to check instruments and data.

We already know the big three tech giants who always lead the way in IT innovations – Google, Apple and Microsoft. This time in the AR space there is one more competitor – Neuralink, an Elon Musk’s initiative in the artificial intelligence and augmented reality space. While there are countless startups trying to come with innovative solutions these companies will likely remain the enablers of technology. A big advantage these companies enjoy is a wide community of fans and techies associated with their brand. This helps in easy adoption of new products and deeper market penetration.

Apple
In the WWDC 2017, Apple launched ARKit – a framework that will help developers create augmented reality apps for Apple devices. In the same WWDC event Alasdair Coull, Creative Director at Peter Jackson’s ‘Wingnut AR’ (of Lord of the Rings fame) showcased a live demo with a tablet, where spaceships engaged in a battle. The entire visualization was created using ARKit and Unreal Engine. This framework will run on all devices running iOS11. While Apple has not yet made major announcements about wearable AR, the recent patent for a headset and rumours of a collaboration with Carl Zeiss to create trendy glasses indicate that they are up for something interesting.

>> ARKit
Revealed during WWDC, ARKit is Apple’s developer toolset to help facilitate the creation of AR apps, which was then released as part of iOS 11. The technology solves a number of issues surrounding the technology in a number of different ways, with the platform working with existing iOS devices, including iPhones and iPads, without any extra hardware.

ARKit uses its Visual Inertial Odometry (VIO) to accurately track the environment and the device’s position using a combination of sensor data and CoreMotion data. These two inputs alone can allow an iPhone to report accurately its motion without additional calibration. It is also possible for ARKit to analyze a scene in a camera’s view to determine horizontal planes, such as floors and tables, which can be used to place virtual objects. Once detected, ARKit can also retain the location of the detected planes, even if they fall out of the camera’s field of view temporarily. The lighting situation of an environment is also monitored by ARKit, with the data able to be used to accurately light virtual objects, making the illusion of them appearing in the user’s view more believable.

Developers were quick to take to ARKit, creating apps before its public availability to test out the framework. Notable examples include an app for measuring the dimensions of a room using the camera, and a remake of 80’s band A-Ha’s “Take on Me” music video.

Google
Google the leader in mobile innovations has yet again excelled in augmented reality. Googlers have already shown their skills with Google Glass, a one of a kind accessory which was way ahead of its time. At the time of launch, it couldn’t succeed because of privacy concerns and pricing. Even Tango, a mobile initiative from Google didn’t play well. Nevertheless, products fail but the experience and knowledge base stay forever. The expenses made by Google in RnD of these projects will definitely help it solidify a space in glasses AR. This year Google also launched ARCore an augmented reality kit that will empower developers to easily create augmented reality apps for Android devices.

Microsoft
Microsoft has also made serious initiatives in the augmented reality market. In fact, they are the ones to combine AR and VR in complete sense thus making the strongest move in the mixed reality space. Microsoft’s HoloLens headset and the upcoming Windows Mixed Reality operating system indicate their strong presence in the AR  and VR space. Microsoft’s initiatives are expected to pick up once these bite-sized apps and fleshed out AR experiences begin to look for a cohesive, unifying user interface to define the platform.

NASA is using Microsoft’s HoloLens to “travel” to Mars. They hope to be able to use the technology to control future rovers and virtually explore the land around the rovers.

Cirque du Soleil has also been using Microsoft’s HoloLens. The circus empire recently showed off how it has used the technology to design its sets and stages. The technology even lets them see how the show will look by using holographic performers.

The Visionary: Neuralink
Elon Musk (Paypal, Tesla, SpaceX) considers artificial intelligence a threat to human existence and to prepare humans to match with AI he founded Neuralink, a neurotech company aiming to decipher the secrets behind communications in the human brain. Neuralink will provide medical aid to people with degenerative brain diseases (like Alzheimer’s) by providing them with an extended artificial memory. This technology has the potential to enable brain typing and skin hearing (like the Facebook endeavour). Once mainstream, it can even allow sharing memories and moments using AR without the need for any special equipment.

To wrap up, augmented reality is a potentially disruptive technology which if leveraged properly can help us succeed dramatically in multiple spheres from marketing to healthcare to entertainment. It is up to us media artists as to how we think of freeing ourselves from the box to animate the world around us.