Lecture 1

3D Graphics and Virtual Reality

1.1. Introduction

1.1.2. About this course

This short course is a series of six lectures which will be introducing the area of 3D graphics and virtual reality (VR). Six lectures are not enough to delve deeply into any one area of this subject so we will concentrate more on providing an overview of the field and brief introductions into the mathematical background, and the practicalities of model or virtual world creation.

The format of the course will be as follows:

Lecture 1

We will introduce the concept of VR and discuss the characteristics a VR system and the various components which make up a VR system. We will also look at some of the many applications for VR.

Lectures 2 and 3

Lecture 2 and 3 will look more formerly at the subject of 3D graphics and introduce some of the mathematical operations which are used in 3D graphics engines. The main goal of this lecture is to familiarise you with the concept of 3D space and provide a mathematical grounding in basic 3D operations such as translation, rotation and scaling.

Lecture 4, 5 and 6

These last lectures will provide a more practical grounding in VR principals. We will quickly cover various techniques used in creating a 3D environment, using VRML (Virtual Reality Modelling Language) as a real example. It would be very hard to teach a new language such as VRML in the space of three, 50 minute lectures, so the lectures are aimed more at the principles and techniques involved rather than the language itself. The use of VRML here is to provide a realistic example of how these techniques can be used.

1.1.2. Books

You shouldn’t need any books for this course, however if you are interested and want to find out more about the topics we will be covering then look at the following two books. This course was based on these books, with "Virtual Reality Systems" covering the first three lectures and "VRML 2.0 Sourcebook" being used mostly for the remaining three lectures.

This book covers Virtual Reality in general, including the history, applications and software systems. It also goes into the maths behind 3D graphics to some extent and also for real-time simulation. It has some lovely 3D images and a free pair of 3D glasses - cor!

This book is concerned purely with VRML (version 2.0) and is a great one to learn how to create virtual worlds and objects. It covers all of the modelling techniques supported by VRML2 and gives excellent explanations as to how to use them. It only briefly covers more advanced topics such as prototyping and scripting. The book does repeat things a lot which makes it a bit tedious to read cover to cover, but means its an excellent reference for the language.

http://www.cms.dmu.ac.uk/~cph/VR/whatisvr.html

1.1.3. Other resources

There are numerous pages on the web which cover computer graphics, virtual reality and especially VRML. Two excellent electronic courses in computer graphics are:

This is a well-prepared course covering a wide range of 3D and 2D graphic techniques. These course notes take the form of lecture slides with very brief notes below each one. A lot of knowledge and background information is taken for granted so it may prove difficult to understand for people without that information. A very good reference for finding out more nonetheless.

This course takes the form of extensive lecture notes. Most of the course content is concerned with the implementation of 3D graphics and algorithms but is useful for anyone interested in learning more.

VRML has a very large network of support in the WWW community and there are hundreds of pages devoted to tutorials, examples and resources. A few good starting points are:

http://www.sdsc.edu/vrml/

http://vrml.sgi.com/

http://vwww.com/

1.2. Virtual environments

The term Virtual Reality has been (mis)used widely to refer to almost any form of computer graphics and interaction. There are numerous definitions of the term VR and contention as to what a VR system actually is. Many people view VR as an attempt at modelling the real world as believably as possible, others see it merely as an advanced form of human-computer interface. Certain views suggest that a true VR system must include paraphernalia such as head mounted displays (HMDs), tactile feedback gloves, and full body-suits. These suggest that the interface to the world is what characterises a VR system, whereas other views deem the actual 3D environment and the level of interaction to be of more importance. Regardless of these views, everyone generally accepts VR for what it is; a system for providing an interactive exploration of a three dimensional virtual environment. A large number of professionals in the field have opted for the term Virtual Environment as it carries less of the hype and expectations which VR has been associated with.

In the early days of VR research the technology was rare and expensive meaning that only the dedicated researchers or companies who had a realistic practical application of the technology were developing systems. Things have improved greatly over the past few years with computer hardware becoming increasingly more powerful while steadily becoming cheaper and cheaper. Systems capable of running a VR application are now commonly available within people’s homes and in the everyday workplace. Increasingly companies are investigating what VR has to offer them and how they can put the technology into practice, research into VR is now commonplace.

1.2.1. Computer graphics

Computer graphics is now a well established field and is used in countless applications and situations. This course is concentrating on 3D computer graphics which is a more specialised yet still extremely widespread area. Computer aided design (CAD) systems led the way with 3D graphics. These systems allowed 3D models of real-world objects to be constructed as geometric descriptions of their structure. By supplying other information such as the position, size, colour and orientation of these objects it is possible to render realistic images of a product before any real-world prototype has been built. The implications of this are incredible savings during product development as ideas and designs can be constructed and evaluated in a fraction of the time and cost of building an actual prototype. CAD is now a vital tool in a large number of research and engineering applications.

Virtual environments are based on similar principals to CAD systems. A database is used to store the geometric representation of the objects within the VE. A graphics engine constructs an image of the scene, which can be viewed from any angle or position, and displays the image in a 2D form as if looking through a camera. The term used for creating this image is rendering. Rendering a 3D scene is a complex and time consuming process. In order to produce a moderately detailed scene it was often necessary to wait for several minutes or even hours for a single image. Computers have progressed a great deal and it is now possible to produce realistic images in a fraction of a second. This is what made VR possible, being able to generate several images a second meant that a user could control the viewpoint or camera and explore the virtual world in real-time.

There will always be a trade-off between realism and interactivity. The more realistic a scene must appear then the longer it will take to render and the slower the virtual environment will update. This again produces two different opinions on what a virtual reality system’s goals should be. Some people believe that a virtual reality system must look real, requiring the most detailed images possible. Other people believe that it is the fluidity and responsiveness of the 3D environment that provides the key to a VR system. Both these approaches are valid, but in terms of computer graphics they are mutually exclusive. Incredibly detailed images will make the VE appear more realistic, but movement through the environment will be slow and cumbersome, detracting from the experience. On the other hand, lesser-detailed scenes will appear false and artificial, but movement through the environment will be smooth and responsive giving a heightened sense of immersion.

1.3. Virtual reality systems

A typical VR system consists of six main components: the virtual world; graphics engine; simulation engine; user interface; user inputs; and finally, user outputs. Figure 1 shows these components. The virtual world, simulation engine, graphics engine and user interface are all internal components to a VR software package, the inputs and outputs (whatever they may be) are external.

The virtual world is the scene database which contains the geometric representations and attributes for all objects within the environment. The format of this representation is dependant on the graphics and simulation engines used. The graphics engine is responsible for actually generating the image which a viewer will see. This is done by taking into account the scene database and the viewers current position and orientation. It also includes combining information from the scene database with textures, sounds, special effects, etc. to produce an impression that you are looking into the scene from a particular point. The simulation engine actually does most of the work required to maintain a virtual environment. It is concerned purely with the dynamics of the environment - how it changes over time and how it responds to the user’s actions. This includes handling any interactions, programmed object actions, physical simulation (e.g. gravity or inertia) or user actions. Finally, the user interface controls how the user navigates and interacts with this virtual environment. It acts as a buffer between the virtual world software and the myriad of input and output devices which may be used. Inputs and outputs are mostly independent of the VR software except in specialist applications.

Figure 1. The six components of a typical VR system.

1.3.1. Virtual reality software

There are hundreds of different software packages which allow users to either experience virtual worlds, or even to create and edit them. The majority of professional VR packages offer the same basic functionality, allowing a world to be created from any number of 3D objects which can be arbitrarily defined using either graphic primitives or specific face sets. These packages also offer total freedom in viewing the virtual world from any conceivable position and orientation. Different systems merely offer additional features, perform operations better, give better performance or image quality, etc. A more interesting development of 3D graphics engines and immersive environments has occurred in the computer games industry. Regardless of the stigma of computer games within the serious research community, it is undeniable that they offer some of the most immersive, usable and engrossing virtual environments - how would they sell otherwise?

Most professional VR packages are very expensive and often require high specification workstations to run properly. The benefits of such systems are their flexibility and generic nature. Anyone who needed such powerful packages, and could afford them, could also afford the computing power needed to run them. Computer games, on the other hand, have had to evolve in a much more restrictive environment. In order to be successful they have to sell as many units as possible at a price which ordinary computer owners can afford. In order to do this they must be developed to run on as many "normal" computers as possible. The tricks needed here are to fit as many features as possible into the product, while using as little computing power as possible. Clearly not an easy task. For this reason 3D computer games have evolved in more pronounced steps than professional VR systems. Professional VR systems at the outset have tried to create a flexible, true-3d world almost irrespective of the hardware requirements, whereas 3D games have tried to provide as much as possible in the commonly available hardware at the time.

1.3.2. Important factors in VR systems

There are many factors which can attribute to a realistic and believable virtual environment, several of the more important ones are listed below.

Visual realism

The level of realism in a scene aids considerably in making a believable environment. Ray tracer and professional animation systems produce incredibly realistic images such as those used in special effects for movie production. Some of the best applications of this technology result in the viewer not noticing any transition or discrepancies between real footage and computer generated effects. Providing this level of detail and sophistication is extremely complex and requires a great deal of rendering time. More and more advanced features are slowly appearing in virtual environments though it will still be a long time before we reach the same quality which current computer animation can provide.

Image resolution

Image resolution is another factor which is closely linked with visual realism. Computer generated images consist of discrete picture elements or pixels, the size and number of these being dependent on the display size and resolution. At higher resolutions the discrete nature of the display becomes less apparent, however, the number of pixels in the image becomes vastly greater. As the colour and intensity of each pixel must be generated individually, this puts a heavier load on the graphics system.

Frame rate

Frame rate is another affect of the discrete nature of computer graphics and animation. To give the impression of a dynamic picture, the system simply updates the display very frequently with a new image. This system relies on the human phenomenon of persistence of vision, our ability to integrate a rapid succession of discrete images into a visual continuum. This occurs at frequencies above the Critical Fusion Frequency (CFF) which can be as low as 20Hz. Normal television broadcasts update at a frequency of 50Hz (in the UK - 60 Hz in the US). This means that in order for a virtual environment to appear flicker free, the system must update the image greater than 20 times each second - again a heavy load on the graphics system.

Latency

Latency is probably one of the most important aspects of a virtual reality system which must be addressed to make the environment not only more realistic, but simply tolerable. Latency or lag is the delay induced by the various components of a VR system between a user’s inputs and the corresponding response from the system in the form of a change in the display. As latency increases, a user’s senses become increasingly confused as their actions become more and more delayed. Chronic cases can even result in simulator sickness, a recognised medical problem associated with virtual environments. Latency must be kept to a minimum in order to create a usable VR system.

1.3.3. Types of VR systems

As previously mentioned, the term VR has almost limitless interpretations. Even restricting this term to the realm of computer generated virtual environments still leaves many possibilities. There are a number of different classifications of VR system which are based mainly on the interface methods used. For more information on this topic see "What is VR" by Jerry Isdale.

Window on World (WoW) or Desktop VR

One of the most common and accessible forms of VR is Desktop VR or Window on World (WoW) systems. These systems do not rely on any specialist input or output devices in order to use them. Typically a normal computer mouse and monitor is sufficient. A quote by Ivan Sutherland in his 1965 research paper highlights the goals of this approach:

"One must look at a display screen as a window through which one beholds a virtual world. The challenge to computer graphics is to make the picture in the window look real, sound real and the objects act real." [Computer Graphics, Volume 26, Number 3]

Video mapping

Monitoring the user with a video camera provides another form of interactive environment. The computer identifies the user’s body and overlays it upon a computer generated scene. The user can watch a monitor which shows the combined image. By gesturing and moving around in front of the camera the user can interact with the virtual environment.

Immersive VR

The goal of VR is to completely immerse the user within a synthetic environment, to make them feel a part of that environment. This means they are effectively cut off from the real world and instead have their own presence and viewpoint within the virtual world. This usually involves a head mounted display (HMD) which provides visual and auditory feedback. Another such immersive system uses a ‘cave’ environment. This is a room which uses multiple, large projectors to display the appropriate viewpoints on each wall of the room. One important difference between using an HMD and using a cave system is that the latter can be a directly shared experience, whereas using HMDs each user would only be aware of other users via a graphical representation of them.

Telepresence

Telepresence links remote sensors and cameras in the real world with an interface to a human operator. For example, the remote robots used in bomb disposal operations are a form of telepresence. The operator can see the environment which the robot is in and can control its position and actions from a safe distance. Such systems are used widely in any applications which must be performed in hostile or dangerous environments.

Augmented reality

Augmented or mixed reality provide a half way point between a non-immersive and fully immersive VR system. AR systems overlay computer generated information over the user’s view of the real world, without completely occluding it. Examples of such applications are head up displays (HUD) used widely in modern military aircraft. These superimpose flight data such as altitude, air speed, artificial horizons or targeting information upon the pilots field of view. This can be on a cockpit mounted display, or even upon the pilot’s helmet visor. There are many, possibly very useful, applications of AR systems which will probably be more acceptable and desirable than fully immersive or desktop VR systems.

Fish tank VR

This phrase was used to describe a hybrid system which incorporated a standard desktop VR system with a stereoscopic viewing and head tracking mechanism. The system used LCD shutter glasses to provide the stereoscopic images and a head tracker which monitored the user’s point of view on the screen. As the user moved their head, the screen display updated to show the new perspective. This provided a far superior viewing experience than normal desktop VR systems by providing motion parallax as the user moved their head.

1.4. Applications of virtual reality

The scope for the useful application of VR technology is so vast that it would be futile to try and cover every aspect of it. The possibilities afforded by VR are limitless and this is quickly being recognised by industry as more and more research programs are created. With the growing availability of high specification computers and VR software, it will not be long before nearly everyone will own a computer capable of supporting virtual environments. Being able to use and create virtual environments will one day be as natural a computing task as to be able to use a word processor to create and read documents.

We will now take a look at some aspects of VR applications. This list is only a very small sample of the full potential of this technology.

1.4.1. Flight simulation

One of the main contributors to VR research is the work that came from developing immersive simulations. In both civil and military aviation, pilot training is an incredibly costly and time consuming business. Pilots must spend hundreds of hours of flight time during their training and even still this cannot prepare them for all the possible emergencies or problems that may arise in a flight. Flight simulators were developed to provide a safe and realistic addition to pilot training. Pilots could use the simulators to enact almost any conceivable emergency scenario which would not usually be possible during a real flight.

Aircraft simulators typically consist of a cockpit reconstruction, mounted upon a system of hydraulic jacks. These jacks allow the entire cockpit to be moved in various directions to simulate the forces and motion experienced during real flight. Before the advent of high powered computer graphics systems, flight simulators were based upon realistic scale models of a landscape. A miniature camera was mounted upon a overhead trolley system which allowed the camera to be positioned anywhere within the model landscape and orientated to face almost any direction. The image from this camera was relayed into the simulator cockpit onto monitors fixed in position of the windows of the cockpit. The simulator would move and orientate the camera in response to the pilot’s inputs using the flight controls.

High powered graphics systems have allowed flight simulators to become increasingly more realistic and flexible. Instead of the scale model landscape, modern flight simulators have vast databases of geometrically modelled terrain. The simulators have advanced flight models and can reproduce any flying conditions required, including weather effects, equipment failure or emergency situations which ordinarily could prove fatal.

1.4.2. Engineering and design

Virtual reality has always been a part of engineering and design in some form or other, ever since computers became powerful enough to support sophisticated graphics. Computer aided design (CAD) and computer aided manufacturing (CAM) tools have long since been used to great effect in creating and evaluating designs and even in manufacturing a product. Engineers can use such software to create a full set of technical schematics for a product and then view it as it would be seen when manufactured. This can be taken further to even simulate how the product would operate and to evaluate any design limitations such as operational loads and material stress.

One other aspect of engineering in which VR can play a useful role is in industrial prototyping. The design process usually involves creating a number of, often fully working, scale model or real sized prototypes of a product. These prototypes are created to evaluate the product before the design is finalised and it goes into full production. Often these prototypes are very expensive and time consuming to construct, especially in large projects such as car design. VR and CAD tools can be used to quickly prototype and evaluate a product. The benefits of this approach allow a far greater flexibility than a model. Often the virtual prototype can be created automatically from the schematics and can be quickly revised to demonstrate design alternatives.

1.4.3. Visualisation

The fields of data visualisation, information visualisation and software visualisation play important roles in understanding large information structures. Data visualisation deals with quantifiable information such as experimental results, which generate large data sets. Information visualisation is a more complex field that involves visualising data base systems which often contain large amounts of abstract information or knowledge. Finally, software visualisation is used to create a visual image or representation of a software system, typically in an attempt to aid a software engineer in understanding its construction and operation. All three can be characterised as typically very large information systems, which can contain abstract information or data.

These fields have all been serviced relatively well by normal 2D tools, but more and more research is looking into the applications of 3D graphics and VR technology to aid the understanding of these information systems. Data visualisation has used 3D to highlight trends and anomalies in large, multidimensional data sets and also for displaying demographic data superimposed on meaningful image maps. VR has been used to visualise and query large databases by generating a ‘landscape’ from the structure and contents of the databases. Users can then intuitively explore the database by using their natural abilities and perception. Some research has even combined theories from town planning and information visualisation to create more legible environments. Finally, software visualisation has seen limited success with standard 2D representations. The size and complexity of modern software make visualising or understanding their structure an increasingly difficult problem. Investigations are being made into how VR can help to create more understandable and information rich software visualisations.

1.4.4. Architecture

Architects have always used CAD software for planning and designing buildings. Even when such software was limited to simple 2D elevations it proved extremely beneficial. Now, CAD systems play an even greater role in architectural design by providing plans, sections, elevations, line perspectives and fully rendered visualisations of the interior and exterior of a building. VR systems are increasingly being used for large projects to provide clients with a virtual walk through of their proposed building. This allows the architects to interact and communicate with the client on specific points and enable the client to gain a better idea of what the final result will be. It also means that if any changes are to be made in the design, they can be made quickly and cost effectively at an early stage in the construction process.

Another useful and widely used application of VR is store or office planning. If a company is interested in rearranging their work environment or replacing their office furniture or equipment then they need to plan the new arrangement. It is often hard to mentally visualise the appearance and ergonomics of a particular office environment so VR is used to help provide a more concrete representation. The office building is modelled in the virtual environment as with the furniture. A client can then experiment with different configurations and arrangements and assess each by walking through the virtual office. Factors such as light levels, clutter, access problems or even fire regulations can be easily assessed and the environment instantly changed to suit.

1.4.5. Human factors modelling

VR is being used increasingly to model human behaviour and human considerations in the design of new products or buildings. For example, one such system can simulate a fire in a building and a user can view how the virtual occupants react to the emergency. This allows a demonstration of important factors such as escape strategies, fire modelling, human behaviour and spatial awareness of complex buildings. Other models incorporated can simulate human behavioural response under different emergency conditions. Factors such as individuals physical size, speed of movement, level of aggression, people’s reaction to the fire and other people, the size and location of the fire itself and hence any effects from it such as smoke, toxic fumes, etc.

Another project, by the Centre for Computer Graphics research at the University of Pennsylvania, has developed "Jack" who is a virtual human which can be used to evaluate many aspects of human factors in new designs or situations. Jack is a fully articulated human 3D model which incorporates 68 joints. All articulation points are restricted to moving within human limits. Also, the limbs and torso possess attributes such as mass, centre of gravity and moment of inertia allowing Jack to be subjected to dynamic simulations in order to observe his movement and reaction, for example in a crash simulation. Jack can also be customised to the physical attributes of any individual.

Jack possesses a sophisticated animation system which endows him with various motor reflexes such as grabbing, stepping or walking, and reaching without losing balance. Jack can be used in other virtual environments to evaluate their effectiveness and explore human factors such as reach space, field of view, joint torque load and collision.

1.4.6. Physical simulations

VR systems have been used greatly for visualising simulation results allowing the user to see the invisible. One application provided a simulation of a wind tunnel experiment with a model aircraft as the subject. The simulation modelled the flow of air over the surface of the virtual model using accurate physical equations to provide realistic results. A user could enter the virtual experiment and inspect any aspect of it freely without interrupting the simulation. The user could also introduce smoke trails at any point and view how the smoke behaved in the air currents. Another early example is molecular modelling. One such system used a large boom arm control which a chemist could use to manually ‘dock’ compounds into the appropriate receptors. The simulation modelled the atomic forces at work in the molecules and provided tactile feedback to the user.

1.4.7. And the rest…

There are so many other real applications and possible applications for VR that it is futile to try and cover them here. Some others not mentioned above are:

 


This page is maintained by Peter Young, please send any comments to (peter.young@durham.ac.uk).

Last updated: Monday 19 January, 1998.