The history of virtual reality

As is often the case when recounting the story of technology that’s sprung from Silicon Valley, the history ofvirtual realitythrows up a lot of the same names and places. And when you’re dealing with such high-tech ideas, sometimes the truth can be blurred as much by military interest as it can be by late nights and too much caffeine.

What we do know is that virtual reality, or something very close to it, was first mentioned in a 1936 short story by Stanley G Weinbaum. ‘Pygmalion’s Spectacles’, published in the June 1936 edition ofWonder Storiesmagazine, features ‘a device vaguely reminiscent of a gas mask’ that has ‘goggles and a rubber mouthpiece’ and which plays back holographic recordings held in a strange liquid, bringing sights, sounds, smells and even a sense of touch to its wearer.

CinematographerMorton Heiligthen took up the baton, writing in the 1950s of cinema technology that could speak to all our senses. He would make his vision a reality in 1962 thanks to theSensorama, a mechanical cabinet that the viewer sat in which was capable of displaying stereoscopic 3D movies with stereo sound. It could also tilt its occupant and blow air into his or her face to simulate wind or convey smells. Only five short films were made for it, as Heilig was unable to secure financial backing, but he would also patent a head-mounted display device.

Flight simulators

The struggle for funding is another thread that runs through this story. If your idea needs equipment that may cost many tens of thousands of dollars, where can you find that sort of money?Thomas A Furness III, now a professor at the University of Washington and often called the ‘grandfather of virtual reality’, solved this problem in 1966 when he was commissioned by the US Air Force to build the first flight simulator.

The military’s interest in VR as a technology for training continues to this day, as Matthieu Poyade, a Research Fellow in 3D Programming at the Digital Design Studio of The Glasgow School of Art, explains: “In VR you can train people for tasks that would be too complex to train in the real world. Flight simulators are usually very expensive, but if you’ve got a much more affordable device, like a haptic device for example, that can allow you to train particular abilities or specific skills.

“In the US there’s been quite a long relationship [between military equipment manufacturers and VR companies],” Poyade continues. “In the UK and in Europe the relationship is slightly less than 10 years old, but it’s being built. It’s comfortable for the military companies because they can save a huge amount of money by using VR rather than mocking up a specific tank or boat or aircraft carrier. With VR they can assess several situations. For instance, they can carry out human factor assessments, like the accessibility of specific pieces of equipment, taking into consideration the physical characteristics of users.”

The Sword of Damacles

Away from the military contracts, progress was being made in other labs, too, and in 1968 Dr Ivan Sutherland of MIT (Massachusetts Institute of Technology) createdThe Sword of Damocles, a ceiling-mounted mechanical arm that held a head-mounted display using two CRT (Cathode Ray Tube) screens that communicated simple wireframe graphics to the wearer. The arm tracked their head movements as they looked around, but was difficult to use as the viewer’s head was clamped into its helmet. Its ‘unique’ appearance is behind its name, but it’s recognisable as the forerunner of the VR systems we see today.

Another ancestor of today’s technology was developed in 1978, again at MIT, with funding from military technology agency DARPA (Defense Advanced Research Projects Agency). TheAspen Movie Mapwas a little like Google Street View, with footage of the streets of Aspen, Colorado, recorded on Laserdiscs, linked to a database of the city’s layout, and played back using a touchscreen interface that could bring up information about buildings that were touched, such as restaurant menus or historical details. It even used a vehicle with cameras on top that, unlikeGoogle’sversion, drove slowly in the middle of the road and towed a bicycle wheel behind it to help measure distance and trigger the cameras.

“It was the first time we used the term ‘virtual environment,’” says Professor Scott Fisher, now an Associate Dean of Research at the University of Southern California, who worked on the map project at MIT. “We built a special camera rig to go on top of a truck that we had, then drove it up and down every street and around every corner in Aspen. Our rig had four cameras, pointing left, right, front and back, and we tried lots of different lens systems to achieve a wide angle. We’d stitch the footage together on the discs, and Street View is very much a derivative of that early work.

“We didn’t have it running in a head-mount, but we did do it stereoscopically,” Fisher continues. “We had a room, one whole wall of which was a back-projected video display, and you could point at the screen. So we projected the Aspen footage onto that while sitting in an instrumented Eames chair, and you could actually drive around Aspen while you were sitting in that room. We also filmed on the ski slopes with helmet-mounted cameras by renting cranes to shoot from high above. We also did handheld stuff where we’d go into all the restaurants and some commercial buildings on the main street in Aspen. Basically, we tried to build as comprehensive a visual database as possible.”

The Department of Defense saw the map as a way for its soldiers to quickly familiarise themselves with new areas they may be operating in. Although driving slowly through a warzone while dragging a bicycle wheel might still put some lives at risk, it would be quicker and cheaper than building replicas of operational areas, a technique pioneered by Israeli commandos for hostage rescue operations.

The Sayre gloves

At around the same time, scientists from the Electronic Visualization Laboratory at the University of Illinois in Chicago were creating the first wired glove. Known as the Sayre glove, it was created by Daniel J Sandin and Thomas Defanti from an idea by Richard Sayre, and used light emitters and photocells in the fingers. As they flexed, the amount of light hitting the photocell varied, allowing finger movement to be translated into electrical signals.

In 1982, Thomas G Zimmerman would file a patent for such an optical flex sensor, and would go on to work with Dr Jaron Lanier – the man who coined the term ‘virtual reality’ – to add ultrasonic and magnetic hand position tracking technology to a glove. This led to what would become theNintendo Power Glove, sold alongside a small number – two – of NES games in 1987.

“Virtual reality originally meant an extended version of virtual worlds,” says Lanier, who these days is to be found working for Microsoft Research as well as writing books and music. “Ivan [Sutherland] had talked about the virtual world that you would see through a headset like that. He didn’t make up that term; it actually comes from an art historian called Susanne Langer, who was using it as a way to think about modernist painting. To me, what virtual reality originally meant was moving beyond the headset experience to include some other elements, which would include your own body being present, so to have an avatar where you could pick up things, and also where there could be multiple people, where it could be social.”

The Cyberglove

Datagloves continue to be a part of the VR world today, and in 1990 Virtual Technologies Inc released the CyberGlove. It’s a product that exists to this day in the form of theCyberGlove III, which is mounted with up to 22 sensors and can detect movements of less than one degree.

1982 also saw the videogame world become interested in VR. Atari opened a VR research lab that year, and the company would employ Zimmerman, Fisher, Lanier and many other VR pioneers. The videogame crash of 1983, sometimes known as the ‘Atari Shock’, saw 97% wiped off revenues from gaming in North America. Atari was sold and split up, its VR lab closed, but many of its researchers were able to continue their work, leading to an explosion of new companies.

For Lanier this was VPL Research, which had its first HQ in a corner of his San Francisco home. Zimmerman joined him, and the company produced datagloves and a head-mounted display, becoming one of the first in the world to sell such items. LSD enthusiast and early transhuman thinker Timothy Leary, searching for legal means to explore consciousness after his release from jail, became associated with the company through his friend Brenda Laurel, a VPL employee who had also worked for Atari.

“We sold the first general-purpose simulator,” explains Lanier. “I made the first surgical simulator collaborating with a doctor at Stanford Medical; the first vehicle prototyping simulator; the first architecture simulators. We made the first versions of all those things, and we sold the kits that allowed a lot of other people to make a lot of their prototypes.”

NASA project

Fisher, meanwhile, was aiming for the stars. In 1985, he founded the Virtual Environment Workstation Project at NASA’s Ames Research Center in Mountain View, California. The lab’s purpose was to produce a virtual reality system for astronauts, so they could control robots outside a space station instead of taking part in risky EVAs. “It usually took almost four hours to do all the pre-breathing and the prep to do an EVA for a satellite inspection or a space-station inspection,” says Fisher.

“And it’s obviously dangerous to do that. So one of the big applications of what we were doing was to be able to control robots in teleprescence mode from inside the space station. The head-mounted display had super-wide-angle optics – about a 180° field of view – so we built a camera system that had matching optics. The servos on it were matched to specs that we found for fighter pilots in combat who are moving their heads pretty fast to figure out where a threat is coming from. We really over-specced the robots we built, but that was the configuration we passed on to Lockheed and Grumman who were designing, and eventually building, the systems for use outside the space station.

“We did a lot of work with instrumented gloves to actually control the remote robot arms, trying to figure out how we map human hand gestures onto the dexterous end effector that some of the contractors were building,” Fisher continues. “At one point we even instrumented the gloves with tactile feedback so the astronauts would have a sense of what they were touching. We started looking at temperature feedback, too, although that’s super complicated because the nerves that respond to temperature respond at a different rate than for haptic feedback.”

At Ames, Fisher would meet Mark Bolas, now Associate Professor of Interactive Media at the University of Southern California. Bolas would go on to found VR hardware company Fakespace in 1988.

Gaming developments

Then, in 1990 Dr Jonathan Waldern, who’d been researching VR since 1985 at Loughborough University with the support of IBM’s labs in Hursley, Hampshire, demonstrated his Virtuality gaming system as part of the Computer Graphics 90 exhibition at London’s Alexandra Palace. That’s when the world of videogaming started to take another look at VR.

Virtuality comprised a stereoscopic visor, handheld joysticks and network capability to enable multiplayer gaming. Datagloves and 3D tracking systems were also available, and British Telecom bought two units for its research labs, to explore the possibilities of telepresence communication systems. There were talks about bringing Virtuality to Atari’s Jaguar games console, launched in 1993, but they fell through.

Japanese gaming companySegaannounced its Sega VR system for coin-op arcade games and its Mega Drive console in 1991, although the home version would eventually be cancelled. Sega was rumoured to be working on something similar for its 1994 Saturn console, but if the headset existed it was never announced.

Fierce competitorsNintendowould get in on the act with its Virtual Boy in 1995, but the commercial failures of such systems led to a retreat from virtual reality by gaming companies. Virtuality would keep going until 1997, but a lack of demand for the expensive machines eventually led to the company’s breakup.

VR slump

“I left the scene in around 1992,” explains Lanier, “and for most of that time it wasn’t very well known. There was a burst of publicity toward the end of the 80s and the beginning of the 90s - I was in a similar position to Palmer Luckey in recent years, where I was on the cover of a lot of magazines – and then it kind of went away again.”

The years after 1997 represent a slump for VR. Google would launch Street View in 2007, and Palmer Luckey would design the first prototype of theOculus Riftin 2010, rekindling public interest and bringing us to where we are in 2016 and the promise of a defining year in the evolution of VR.

“When I started to work in VR, around the year 2000, we had head-mounted displays that weren’t too different to what you see today,” says Poyade. “They were a bit more bulky because of the cables, but the difference was the price. They were several thousands of pounds for one device. Today it’s lighter and cheaper and more accessible for customers.

During the last 10 years, the software we use to develop VR content has improved. When I started, it was hardcore coding, quite complicated, and now it’s more like a graphical language with a very tiny amount of coding required. Users with less expertise in programming and coding are more able, nowadays, to develop VR content.”

It’s only now that technology has become cheap and accessible enough that virtual reality andVR headsetscan have another chance to shine. Built on the thinking and ingenuity of over sixty years, this cutting-edge blend of technology and art is about to hit the mainstream, and peering into virtual worlds could become as normal as watching TV.