Adapting the OpenRelativity toolkit to multi-perspective dome projection

(1)

Adapting the OpenRelativity Toolkit to Multi-Perspective Dome Projection By Zachary W. Sherin

S.B., EECS M.I.T., 2015

Submitted to the

Department of Electrical Engineering and Computer Science in Partial Fulfillment of the Requirements for the Degree of Master of Engineering in Electrical Engineering and Computer Science

at the

Massachusetts Institute of Technology

January 2016

Z

0 >J

The author hereby grants to M.I.T. permission to reproduce and to distribute publicly paper and electronic copies of this thesis document in whole and in part in any medium now known or

hereafter created.

C

Author:

Signature redacted

Depyartment of Electrical Engineering and Computer Science

January 29th, 2016

Signature redacted

ertified by:____

Philip Tan, MIT Game Lab Research Scientist, Thesis Supervisor January 2Ato 2Q.4 (- 1

ccepted by: Signature redacted

Dr. Christopher TarmL. Cairman, Masters of Engineering Thesis Committee

MASSACHUSES INSTITUTE OF TECHNOLOGY

JUL 19 2016

LIBRARIES

Figures:

1. A screenshot of Real Time Relativity

10

2. Synchronized fireworks at high speed of light (top) and low (bottom). Note the apparent desynchronization

11 3. A moving duck boat at a high speed of light and contracting at a low speed of

light

12 4. A moving duck boat under the effects of Lorentz Transformation and Doppler

Shift

13 5. The bright colors caused by moving towards an object (left), and the

darkened colors caused by moving away (right)

13 6. Fireworks under no time dilation (left) compared to fireworks traveling at 80%

of the speed of light

14

7. Short diagram of the relativity shader. Full code found in Appendix A

16 8. Layout of the Hayden Planetarium. The seats in the smaller section are

blocked off for Einstein's Playground, creating a direction of focus with the remaining seats

23

(6)

1. Teaching Special Relativity

1.1. Introduction

Special Relativity is considered one of the foundations of modern physics. Despite its discovery over a hundred years ago, the topic is rarely taught before college and university classes. Even then, special relativity is often relegated to higher level courses rather than introductory physics. In MIT's freshman electricity and magnetism course, it is not taught except in the advanced course. In the advanced course, special relativity is considered largely optional and too difficult to understand. The mathematics of the traditional way to learn special relativity are beyond most high school classrooms, so college is often the first contact point for special relativity. Special relativity is currently taught most often as building from the theory itself and leading to the most interesting results. While a theoretical background is necessary to understand the results and implications of special relativity, the end result is that there is an enormous barrier to accessing this material. In addition, there are no easy experiments to display, unlike most other topics in physics. For both of MIT's freshman physics courses, there are experimental displays for almost every topic. Given the constraints necessary to see special relativity in action, lecturers often have to make do with thought experiments or written problems. This leads to the current state of most physics teaching - very few courses cover special relativity, and many treat it as an esoteric, unnecessary piece of understanding [1].

1.2. Challenges with Teaching

To quote the current MIT OpenCourseWare version of advanced electricity and magnetism [2] (a course taught for first or second year students) "[Special Relativity] is, in my experience, by

(7)

far the most confusing subject that we cover all term. Further, it is not strictly necessary to learn this material inside and out in order to understand and excel in [this class]." It is immediately obvious with this opening line that special relativity is considered both difficult and not terribly useful, even though it is interesting.

The reasons for this are varied, but research into student understandings [3] suggests that students come to the subject with ingrained misconceptions. The research by R.E Scherr et al. suggests a fundamental set of misconceptions on the part of most students because special relativity introduces problems that are so far removed from the reality of human experience; the simple concept of "simultaneous events" is thrown out the window.

1.3. Improving Understanding with Games

The works cited above also speak to a way to target the most fundamental misunderstanding of special relativity. However, these works focus on producing an understanding of special relativity from a mathematical perspective, rather than an intuitive one. While a deep, mechanical understanding of the topic is necessary for students to use special relativity in further study, an intuitive, lighter understanding of special relativity can both encourage deeper study and help recognition of relativistic problems.

A Slower Speed of Light and the Einstein's Playground experience are both built with the

intention of imparting intuitive grasps of special relativity concepts, without delving too deeply into the difficult mathematics behind special relativity. They are meant not as an academic aid to produce better students, but an interest aid to help students understand the effects presented to

(8)

them. These programs are a step towards making special relativity accessible to students who may not yet have the math or physics preparation to conceptualize relativity..

2. OpenRelativity and A Slower Speed of Light

2.1. Introduction

In the summer of 2012, a team of students at the MIT Game Lab were assigned the task of making a game that could help introduce people to the topic of special relativity. Gerd Kortemeyer, who produced a prototype of a real-time relativistic simulation, handed it to our team with the requirements that the final product be as accurate as possible, exist on a human scale, and be accessible to a wide range of players. His intent was to create a teaching tool that would not necessarily impart players with a college level understanding of special relativity, but rather inspire them and leave them knowing enough to seek out more. It was meant to provide an intuitive understanding of what traveling near the speed of light has been imagined by those already familiar with the theory.

2.2. Past Work

The code base Gerd Kortemeyer delivered to GAMBIT in 2012 was originally a class project for an Advanced Directed Studies course for seniors at MSU's Lyman Briggs College in 2009. The prototype was created by MSU undergraduates Jordan Fish, Jesse Hacker, Justin Kienle, Alexander Kobylarek, Michael Sigler, and Bert Wieringa. Written for Microsoft's XNA framework

in C#, the end result was a game about moving around a racetrack at different speeds of light.

(9)

The game was simple, but effectively introduced players to the geometric and time dilation effects of travelling near the speed of light. The player had to avoid many obstacles on the course, as well as deal with the difficulty of navigating a world under the effects of special relativity. Most notably, the codebase allowed for third-party moving objects, as long as they moved at a constant velocity for all time. The game required you to beat your old time (in the world frame, see section on time dilation below) at completing the race track.

Real Time Relativity (RTR) [4] is a tool very similar to OpenRelativity in that it allows the user to play around with different speeds and effects, but differs from OpenRelativity and A Slower Speed of Light in three important ways. The player in Real Time Relativity is a vehicle, one that does not really have a human scale. This, along with the environments (an empty city or interplanetary space) makes RTR a more investigative experience, rather than A Slower Speed of Light, which tries to create a sense of wonder. Additionally, there are no third-party moving objects in RTR. The third difference is that the toolset of OpenRelativity is available online, where anyone can add it to their own games or experiments. OpenRelativity is a toolkit, rather than a complete experience like A Slower Speed of Light or RTR. (See the section on Einstein's Playground below for more details.)

(10)

Figure 1: A screenshot of Real Time Relativity

2.3. Effects Simulated

OpenRelativity is capable of simulating many effects to present a detailed representation of traveling near the speed of light. The different effects can operateat all times, although the Doppler Shift and Searchlight Effect can be turned on and off as desired. The effects are described in the following sections.

(11)

2.3.1. Runtime Effect

Light takes a certain amount of time to reach your eye. Normally this is unnoticeable as the speed of light is incredibly high. However, when the speed of light is lowered, the time difference between when something happens and when it is observed becomes apparent. If we were looking at ten synchronized pendulums, arranged in a row at different distances, you might see them as all out of sync.

-~~~F - T -- T

--mewm

Figure 2: Synchronized fireworks at high speed of light (top) and low (bottom). Note the apparent desynchronization

(12)

2.3.2. Lorentz Transformation

As a consequence of the speed of light being finite in all reference frames, an object moving near the speed of light contracts with respect to a stationary observer. This contraction forces the speed of light to be the same in reference frames where Galilean mechanics would suggest a moving observer would be going faster than light. The visual effect of this contraction in the game is enhanced or diminished by the Runtime Effect, leading to objects appearing increasingly spherical.

fl

Tj

_I

[

;rJr

T~

Figure 3: A moving duck boat at a high speed of light and contracting at a low speed of light

2.3.3. Doppler Shift

Light, being a wave, is subject to Doppler Shift. Traveling towards or away from a light source will change its apparent wavelength, perceived as color. Wavelengths get shorter (more blue) if

I

12

I

(13)

you move towards a light source, and longer (more red) as you move away. This color change is very apparent whenever the player moves in OpenRelativity, as everything in the world around the player changes color depending on relative velocity. When third-party objects move towards or away from the player, they will also change color.

"~

I

-

j -~

Figure 4: A moving duck boat under the effects of Lorentz Transformation and Doppler Shift

2.3.4. Searchlight Effect (Relativistic Aberration)

When traveling near the speed of light, there is an aberration of light that causes you to perceive more light in the direction of your movement, and less in the opposite direction. When moving in OpenRelativity and A Slower Speed of Light, colors in front of you get more intense and those behind you get less intense, manifesting as over-saturation in front of the player and a dark cone behind the player.

Figure 5: The bright colors caused by moving towards an object (left), and the darkened colors

caused by moving away (right)

13

(14)

2.3.5. Time Dilation

As you travel near the speed of light, your time passes more slowly than a stationary observer would. For example, if I moved around the world near the speed of light, and I was carrying a stopwatch, that stopwatch would show less time had passed when compared with one that I left on the ground near my starting point. This is all relative; the Earth is moving very fast, but as long as that is our "rest frame," then everything is relative to Earth time.

Figure 6: Time-delayed fireworks featuring no time dilation (left) compared to fireworks traveling at 80% of the speed of light (right)

(15)

2.4. Implementation Details

These effects can be simulated on every object in the game. As the number of objects increases, so does the load on the machine. Initially, these effects were run on the CPU, but that proved to be too much load for even a small number of objects in a scene. Thus, the responsibility of computing these effects was given to the graphics card through the use of shaders. These are written in a high-level shader language in the Unity game engine, which can be cross-compiled to most systems. The shader code is capable of taking several global variables about the object, such as its velocity and position relative to the player, and then calculates the Searchlight Effect, the Lorentz Transform, Runtime Effect, and Doppler Shift together. Since these operations are performed in parallel, the system is much faster after

off-loading the computation to the graphics card.

However, many operations are still performed on the CPU side, such as keeping track of time dilation for each object, which is simply converting the "global" game clock into "local" time for each object according to the formula above. Keeping this information in the shader would have been inefficient, as interactive gameplay requires access to this information and pulling data from the graphics card is difficult. A diagram explaining the effects and their order on the graphics card is shown below. As this thesis follows the dome experience more than the original OpenRelativity project, I will not go deeper into the workings of the original system.

(16)

Rotate Velocity to be undrec: oral (easier con-putatcn)

Azply Lorentz tra-sform to the ob ect M

VERI EX SH

Get colors from texiuro sets id ffse IR, UV: Get ap)prcx mate wavelength d stribution from XYZ color inforration

OVI 0

AUE

N love object according to runtime effect

tbject back to local soace

Transforri from RGB to XYZ

Shift wiviengjths according to Dopp er

Trarsform waveleng-h d:stributicn mnck o XYZ XY7->RGR and roturn

PRAGMENT SHADER

Figure 7: Short diagram of the relativity shader. Full code found in Appendix A

3. Dome Experiences and Learning

3.1. Introduction

Planetarium domes have come far from their original use of simply showing the night sky above. Using custom light projectors, the earliest planetarium domes were darkened concrete theaters that could show the arrangements of the stars over earth. Over the course of decades, these shows were used to entertain and teach about the motions of heavenly bodies and the location of constellations. Recently, many planetariums have moved to a digital projection system that allows for a greater range of experiences. The museums show different types of entertainment, from full-dome videos of NASA rocket assembly, to animations of underwater and outer space experiences. These different experiences are designed to take advantage of the immersive qualities of a full dome. Domes/curved screens can be used to display an image larger than

(17)

your field of view, creating the illusion of looking at the actual space instead of an image of the space. They are a plausible alternative to virtual reality that doesn't require stereoscopy.

3.2. Dome Experiences and Previous Work

Executed well, a full dome experience can be more memorable than a standard video on a computer screen. By utilizing the full space around a viewer, they are brought deeper into the experience and remember more of the provided information [5]. These attractions are designed to be both entertaining and educational, and are mostly non-interactive. However, some planetarium shows have a lecturer answering questions from the audience as they are guided through a presentation with visual aids projected onto the dome. The Boston Museum of Science puts on astronomy lectures [6] where one speaker informs the audience about space while another museum staff member choreographs images of the universe on the dome. This way, the participants not only hear about interesting celestial bodies, they experience travelling around them inside the dome.

3.3. Interactive or Passive

Though full-dome experiences can aid in an individual's recollection and comprehension of material, there are many reasons to move beyond passive experiences into interactive. For a long time, the processing power necessary to produce real-time interactivity at full dome resolutions was simply out of reach. Now, however, even a standard HDMI output can be used to push the necessary images. A suite of software exists to go from game output to correctly-projected dome image. These systems are enabling a whole new set of experiences within the dome.

(18)

For education, a system that responds to your inputs and is not exactly the same each time can draw a user further into the experience, allowing the audience to put effort into the experience and remember its outcomes. If a user can be immersed just by being in the dome, then a logical continuation of that path is to allow the world of the exhibit to respond to them. Additionally, the larger setting and crowd suggest different possibilities for interaction from a standard single-user experience.

4. Einstein's Playground

4.1. Introduction

The finished product is an hour long presentation, including time for questions and deeper explorations of any given experiment if the crowd wishes. Much like the museum's existing astronomy experiences, a presenter speaks to the audience while an operator runs the experience from a workstation. The operator guides the experience by changing the speed of light and cueing up different experiments. Meanwhile, the presenter helps the audience to understand what they're looking at, and encourages them to draw their own conclusions about how each experiment will play out before signalling the operator to demonstrate.

(19)

4.2. Technical explanations

4.2.1. Pipeline for rendering to the dome

The system for rendering from the Unity engine to the final output dome projectors can be explained in three parts. These three are the actual game engine, Unity, the output software, Syphon, and the slicing software for projection, Blendy Dome VJ.

The OpenRelativity codebase is written in Unity using the C# language. Unity is a game engine available for free download, offering additional features and support with a paid version. Einstein's Playground was written on top of OpenRelativity and modified to allow the presenter more control over elements that were not pertinent to OpenRelativity, such as acceleration and field of view, as well as running the choreographed experimental sequences.

The Unity engine uses the Syphon plugin [7] to export the final 2d rendertexture out to the dome system. Syphon uses a client and server model, then shares graphical memory between them rather than streaming data. This saves time and memory since the texture simply lives on the graphics card where can be accessed by any project using Syphon's client example.

The final piece is Blendy Dome VJ, where the host computer uses Syphon to retrieve the output of the Unity engine. Blendy Dome VJ then takes that texture and slices it according to a pre-made 3d model of the physical dome. It feeds these sliced images to the projector system such that the image is rendered correctly across the entire dome.The software must be primed with a pre-made model of the dome, or else it will not know how to slice the images. Since the image transforms are pre-calculated for the dome, it can run relatively quickly.

(20)

4.2.2. Code modifications

The most obvious modification made to the code is to move from a single perspective to the multiple perspectives necessary to generate the dome image. Rather than use a single camera and stretch its viewing angle to approximate a full dome view, the image is rendered from multiple cameras stitched together. There are five cameras arranged in five faces of a cube. Each has a ninety degree viewing angle, which combine to create a hemispherical image. The edge of the camera hemisphere is actually about 1300 off from the dome's center, but on average this is constrained to only display 110". The dome view is set to situate the sky at the top of the dome and the horizon around its edge. These camera views are mapped as textures on a 3D polygonal dome surface, and that projected image is fed into the Syphon system as mentioned above. A description of the setup, along with the math behind its transformations, is available in [8].

Simple modifications were made to the system to allow control over parameters like dome perspective angle, player acceleration and the various preset speeds of light. Rather than using the default OpenRelativity speed of light operation, where the player holds a button to adjust the speed of light, Einstein's Playground uses a group of presets that switch the speed of light instantly. All controls are on a keyboard, with optional use of a wireless game controller if the presenter wants to have control over the system as well. Most of the variables are locked during the actual experience to ensure a smooth experience and minimize motion sickness in the viewers.

(21)

4.3. Overview of the experience

4.3.1. Advantages of a Game

Special relativity, as shown in the quote from MIT's own course notes, is difficult and unintuitive material. Teaching special relativity requires a large amount of time and effort from both the teacher and the students, and is not easily translated to a single experience. Stimulating the interest of a student while introducing a new topic can be an enormous barrier to cross. Special relativity is often presented with obtuse experiments or in galactic terms. While some students may respond to these approaches with enthusiasm, others may require more than equations on a chalkboard or thought experiments to stoke their interest.

The dome experience is not unlike a well-prepared university lecture and demonstration. The main difference between Einstein's Playground and an in-class experiment is that the former is virtual. However, it is a high level, practical exploration of special relativity that may garner more interest than the standard methods. Student engagement is paramount to effective teaching, so providing a high-level lecture with a relevant and interesting visual experience can keep new ideas fresh in the minds of students. The experience will certainly not leave participants with a mathematically grounded understanding of the effects of special relativity. However, by engaging the participants with the almost magical ramifications of special relativity, Einstein's Playground is meant to provide a jumping off point. It can be seen as a promise of interesting reward for putting in further work of understanding special relativity.

(22)

4.3.2. Presentation of the Experience

Attendees are led into the dome and sit facing a single direction. The Museum of Science's dome is set up for full 360-degree experiences, but to focus attention on the presenter and on a single experiment at any given time, we decided to not utilise 360-degree seating. However, the image itself still wraps around the entire dome, and the attendees could, if they desire, look behind them and still perceive relativistic effects. The presenter begins with a discussion of the history of special relativity and the ideas behind what they're about to see. As the projectors start, the viewers see the Museum of Science from the middle of the Charles River. With a brief explanation of where they are and what they are about to see, the presenter moves into the

beginning of the show.

Accompanying the presenter is someone running the actual experience, hereafter referred to as the driver. Anywhere that this document suggests that something happens, it is by instruction of the presenter to the driver, or the driver's own interaction with the experience. Since it is an interactive system, the presenter may be overwhelmed by having to run the experience and lecture at the same time. Instead, the driver does this work for the presenter, using a set of events linked to keys on the keyboard.

(23)

_ll

4:

Figure 8: Layout of the Hayden Planetarium. The seats in the smaller section are blocked off for Einstein's Playground, creating a direction of focus with the remaining seats

4.3.3. Experience Description and Discrete Experiments

Einstein's Playground is advertised as a single experience, but can be thought of as discrete segments or experiments that each explain a different effect of traveling near the speed of light. The exact dialogue for each of these segments is not set in stone, and it is up to the presenter how exactly to describe the visuals. However, a general plan for the order of experiments and what they entail is described in the following sections. The experiments were set up to ease the audience into the effects of special relativity. We start with almost none of the effects visible and work towards showing them all at once. This way the audience is able to build an understanding of special relativity without being overwhelmed.

(24)

4.3.4. Runtime Effect and Time Dilation

At the start, the speed of light is set to its nominally real value, which is too high to perceive the effects of traveling near the speed of light. Additionally, the Doppler Shift and Searchlight Effect are turned off to allow a more gradual acclimation to the effects. The first experiment is to launch a set of fireworks behind the Museum of Science. At high speeds of light, the fireworks go off and produce a bang after a few seconds. They appear to all be synchronized, even though sounds of the explosions come at different times due to the difference in the speed of light and the speed of sound. By lowering the speed of light, the previously synced fireworks now all appear to go off at different times. The presenter can use the example of the firework sounds going off at different times to explain the seemingly different launch times of the other fireworks.

After explaining the runtime effect to the audience, the presenter asks them to notice the height difference between the first fireworks launch and those launched at a lower speed of light. Due to the timer on each of the fireworks, the effects of time dilation result in them exploding at a further point in the air than at higher speeds of light. The firework particles also disappear at a much further point from the original explosion than they do at lower speeds of light, resulting in bigger explosions. These effects combine to create a clear picture of how the runtime effect and time dilation manifest themselves. At this point, the presenter may answer questions or demonstrate these effects at different speeds of light before moving onto the next demonstration

(25)

4.3.5. Doppler Shift

The presenter then directs the audience's attention away from the sky above the museum and towards the water in front of them. The presenter may speak about Doppler Shift with sound, and how most people have experienced the change in pitch as a fire truck drives by blaring its siren. This effect can be demonstrated by releasing a duck boat (a Bostonian favorite) to drive

by the user. Its engine produces a low drone that shifts in pitch as it moves by the user. The

presenter then describes how Doppler Shift affects light as well and repeats the experience with a lower speed of light, causing the duck boat to change color as it moves past the front of the dome. From there, the presenter is free to describe how these color changes manifest themselves, or perhaps launch a few fireworks under the effects of Doppler Shift before moving on to the next experiment.

4.3.6. Lorentz Transformation and Relativistic Aberration

Now the presenter explains that the next part of the presentation is to move the audience along the river. Up to this point, the audience has remained more or less static within the experience, while other objects have moved and changed under special relativity. The presenter describes what changes will manifest, particularly the effects of Relativistic Aberration (the Searchlight Effect), which have not yet been apparent. Then, the driver moves the audience towards and away from the 3D model of the museum to demonstrate Doppler Shift, Relativistic Aberration, and Lorentz Transformation on the environment. As described in the layout section below, the indicators for the speed of light are laid out in an alley around the player to display the Lorentz Transformation, which is more difficult to see in the presence of the clearly visible color effects.

(26)

4.3.7. Finale

After a short discussion of these effects, the presenter then informs the audience that they will get a short demonstration of what a firework show might look like with all the effects at once. On clicking the final movement button, the final scripted sequence of the experience happens. The audience is flown around the buildings of Boston and finally lift off into the sky with various fireworks going off around them. This final sequence is to serve as both a segue out of the experience and to demonstrate the full power of the OpenRelativity toolkit.

5. Discussion and Future Work

5.1. Einstein's Playground as a Teaching Aid

Einstein's Playground is set up as an aid for someone teaching the material rather than a standalone experience to a person interested in special relativity. It is not a complete standalone experience like A Slower Speed of Light, but instead augments the teaching experience and allows for a more memorable introduction to the topic. The system of Einstein's Playground allowsnot only for a presenter to give a prepared lecture on special relativity, but also to set up demonstrations that were not prepared ahead of time. It is a flexible toolkit that allows the presenter to give a visual aid to the difficult subject of special relativity. These aspects make it an excellent teaching tool for those instructors interested in adding a more practical portion to their discussions of special relativity.

(27)

5.2. Dome Experiences and Interactivity

Einstein's Playground serves as a powerful example for the medium of interactive full-dome experiences. Most interactive experiences, particularly video games, are designed with the intent of each player being able to interact with the world. Einstein's Playground instead plays to the strengths of a full-dome theater, allowing the audience to focus on the events unfolding and the lecture rather than on completing tasks within the system. It is still a real-time simulation, not just a prerendered movie, allowing the presenter to change tactics or repeat experiments from different angles as necessary. Rather than forcing a single-player experience (such as A Slower Speed of Light) into a dome, the goal of Einstein's Playground was to enable a new form of interactivity to engage a large audience all at once.

5.3. Future Work

While Einstein's Playground in its current state is a coherent experience, future work towards making it more extensible would enable any planetarium to craft their own special relativity lectures. In its current state, only someone with good knowledge of Unity could effectively create a new experience. Additionally, other planetariums may not find the location of Boston as compelling as the Museum of Science, so the demonstration may be limited in its appeal. Future collaborations with the Charles Hayden Planetarium in the Museum of Science at Boston could involve working towards a different style of interactive experiences that are more applicable to other planetariums.

(28)

Bibliography:

[1] K. Gibson, Special Relativity in the Classroom, Doctoral Dissertation (Physics), Arizona State

University.

[2] Scott Hughes, Spacetime Introduction to Special Relativity (2005, MIT OCW)

htto. //web. mit.ed L /S-a h uphes/wwwIv8.022!Iecl -i j c31

[3] R. E. Scherr, An Investigation of Student Understanding of Basic Concepts in Special

Relativity, Doctoral Dissertation (Physics), University of Washington (2001)

[4] C. M. Savage, A. Searle, L. McCalman, Real Time Relativity: exploration learning of special relativity, American Journal of Physics 75, 791 (2007)

[5] L. Zimmerman, Stacia Spillane, Patricia Reiff, Carolyn Summers: Comparison of Student

Learning About Space in Immersive and Computer Environments (2014)

[6] Explore the Universe, Boston Museum of Science Hayden Planetarium

btp//www.imos. org/planetariun/exiolore-the-universe-live

[7] Syphon plugin website and introduction, //svchon vOO2.infe/

[8] Creating fisheye views with the Unity engine, Paul Bourke (2011)

http.//paulIbourke.rnet/dome/unitv3d/

[9] Gerd Kortemeyer, Philip Tan, and Steven Schirra, (2013): A Slower Speed of Light:

Developing intuition about special relativity with games FDG 2013, FDG '13 Proceedings of the International Conference on the Foundations of Diaital Games, ACM New York, NY, USA p. 400-402

[10] Gerd Kortemeyer, Jordan Fish, Jesse Hacker, Justin Kienle, Alexander Kobylarek, Michael

Sigler, Bert Wierenga, Ryan Cheu, Ebae Kim, Zach Sherin, Sonny Sidhu, and Philip Tan,

(2013): Seeing and Experiencing Relativity - A New Tool for Teaching? The Physics Teacher [11] Richard Conn Henry Teaching Special Relativity (Third International Conference on the

Nature and Ontology of Spacetime, June 13-15, 2008, Montreal): Minkowski trumps Einstein ittrp://wwvvw rpantaneto.co.uk/issue33/henr .htrn

(29)

Appendix A

(Relativity Shader Code)

owarning Upgrade NOTE: unityScale shader variable was removed, replaced with I

Shader t iv i t

{

Properties {

MainTex ('Base (RGB)", 2D) = {} /" //Visible Spectrum Texture ( RGB )

UVTex("UV",2D) = {} //UV texture IRTex("TR",2D) = {} //IR texture

viw ( Vector) (0,0,0,0) //Vector that represents object's velocity in variable is

_strtTime ( , float)-O //For mioving objects, when they created, this

set to current world time

_Cutoff ( '6ase AI, Range (0, 9)) - 0.1 //Used to determine when not to render alpha materials

}

CGINCLUDE

// Upgrade NOTE: ex( luded shader from DXII and Xbox360; has st ruct s wil hout sencanti c .

(struct v2f members pos2,uvI,svc,vrdraw)

#pragma excluderenderers d3d1 xbox360

// Upgrade NOTE: excluded shader from Xbox3O, nas structs without semanti s (strct

v2' cmmbers pos2,uvl,svc,vr)

// Not sure ;hen this^ got added, seems like unity did it automatically some update?

#pragma excluderenderers xbox360

#pragma glsl

#include

shift variables, used to

xla 0.39952807612909519 xlb 444.63156780935032 xlc 20.095464678736523 xha 1.1305579611401821 xhb 593.23109262398259 xhc 34.446036241271742 ya 1.0098874822455657 yb 556.03724875218927 yc 46.184868454550838 za zb zc

make guassians for XYZ curves

2.0648400466720593 448.45126344558236 22.357297606503543 u( ttr Ine e he I e 1( IRRANGE 400 IRSTART 700 UV_RANGE 380 //Color #define #define #define #defi ne #def ine #define tdefi n e #define 4def i ne #define #define -de fi n e //used #def i re #define ldefi ne 29

(30)

#dlefine UVSTART 0

/T hi s i s the dat a rent 4rrm h x T rer Ic the 6

struct v2f

{

float4 pos : POSITION; //internal, used for display

float4 pos2 TEXCOORDO; //Position in world, relative to player position in

wo r -1d

float2 uvi TEXCOORDI; //Used to specify what part of the texture to grab in the fragment shader(not relativity specific, general shader variable)

float svc: TEXCOORD2; //sqrt( 1 - (v-c)A2), calculated in vertex shader !-o save operations in fragment, It's a term used often in lorenz and dcppler shi ft

calculations, so we need to keep it cached to save computing

float4 vr TEXCOORD3; //Relative velocity of obje-t vpc viw

float draw TEXCOORD4; //Draw the vertex? Used to not draw objects that are calculated to be seen before they were created. Object's start time is used to determine this. If something comes out of a building, it should not draw behind the building.

//Varicbles that we use to access texture data sampler2D _MainTex;

sampler2D IRTex; sampler2D UVTex;

sampler2D CameraDepthTexture;

float4 _viw - float4(0,0,0,0); //velocity of object in world float4 _vpc float4(0,0,0,0); //velocity of player

float4 _playerOffset - float4(0,0,0,0); //player position in world float _spdOfLight = 100; //current speed of light

float _wrldTime 0; //current time in world float _strtTime - 0; //starting time in world

float _colorShift = 1; //actually a boolean, should use color effects or not ( doppler + spotligit).

float xyr - 1; // xy ratio

float xs 1; // x scale

uniform float4

uniform float4

MlainTexTexelSize; CameraDepthTexture ST; //Per vertex operations

v2f vert( appdataimg v { v2f o; the player is o0pos o pos at 0,0,6

mull Object 2Wtorl , v. vertex, //get position in world frane

_playerOffset; //Shift s'ich that we use a coordinate system where

0.uv1.xy v texcoord; //'ge the UV oorcinate foi the current vertex, will

he p~ar d t -ragment :hare

float speed sqrt; pow ivpc x ,2 pow (_vpc y ,2) + pow((_vpc.z),2),

/ w p/(i+v vp'2

float vuDot _{_vpc.x _viw.x}+ _vpc y _viw y vpcz'_viw z:

30 a

(31)

velocity dotted with velocity of tre cjer:.

float4 uparra;

//IF our spd

rave a ;he:k here just to be sa

if ( speed 0 )

{

uparra (vuDot (speed

ek2lul-ty Coi9m90 iii 00 N, s- 0

speed) _vpc; / /ct Ihe paralC c ocpoent or

the object's velocity I

//If our speld is "Oro, Sed c a. ' <eocity to zero

else {

uparra 0;

I

//Get the perpendicular compodeit oF our velocity,

float4 uperp _viw - uparra;

//relative velocity calculatior

float4 vr -( _vpc uparra (sqrt(1 speed'speed))

//set our relative velocity

o.vr - vr;

vr = -1;

//relative speed

float speedr - sqrt( pow((vr.x),2) + pow((vr.y),2) o svc = sqrt( 1 speedr ' speedr); // To decrease

fragment shader, we're storing this value

ju-t hy suhtir-ct ion

uperp)/(1vuDot);

+ pow((vr.z),2));

numher of operations in

///You need this otherwise the screen flips and weird stuff happens 4ifdef SHADERAPID3D9

if (_MainTexTexelSize.y < 0)

O.uv1.y - 1.0- o.uvl.y;

#Iendif

//riw = location in world, for re erence

float4 riw o pos , /Ps itio that ''11 be used in the outpu

if (speedr 0) // If speed is zero, rotation fails {

float a, //angle

float ux; float uy;

float ca; //cosine of a

float sa; /// sine of a

if ( speed = 0)

{ an( the world's Z axis

//we're getting the angle between our z direction of movement

a -acos(-_vpc.z/speed); if ( _vpc.x !0 vpc.y K 0 ) { I else f ux uy _vpc.y/sqrt(_vpc.x'_vpc.x _vpc y-vpc.y), _vpcx/sqrt(_vpc.x,_vpc x _vpc y _vpcy,, ux uy 0, 0; I ca cos(a)> sa sin(a); 31

I

(32)

'K ii

,Iayer s movemlent i l n

'ovemen t in one diretion.

magnitud i of velo(ity is in

o.pos.z'(uy sa),

uy-uy'iI ca)% o pos z (ux

}

tre Z direction

' 1"T h'Ii l rr ca s n IC -I 1 ;.!

/A n ! we rotIatLe urI r r. '" mucL to makc i t t e Z direction

riw x o pos.x (ca ux ux ,I-ca)) o pos y ux uy

riw y o pos x

sa)

riw z o pos.x uy-sa

uy'ux ( cal)) o pos y ca

o~pos.y lux sa , o pos z cai.

// here beginas i e ioignal cate, mve "y the ouys behn rd rIhe

Relativity game

/IRotate our vIicity

float4 rotateViw float4rid., '

if ( speed !- 0 )

I/r've ne rotate our object's velocity so that we krlep

cern)isterl with the etatin OUr Pl1yer's velc (ity

rotateViw x viw x * ca , uxiuxIl ca))

-_viw y ux uy'(I ca) _viw z uy sa) _spdOfLight;

rotateViw.y _viw x (uyux(ca)) _viw.y ca

uy;uy 1 Ca)) _viw. z (ux sa) spdOfLight;

rotateViw z viw x , uy sat _viw y (ux-sa;j+

_viw z'(ca))' -spdOfLight;

} else { rotateViw x rotateViw z rotateViw.y (_viw. x) Kviw. z _viw.Y) _spdOfLight; spdOfLight-: _spdOfLight;

float c - riwxiw x + riw.y

posi tion squared (position doted it h posi t ion)

riw.z'rotateViw z));

direction

riw.y - riw.z'riw z),, S//f nn'er

Float b - 2 ( riw.x rotateViwx + riw.y rotateViw.y

//next get posit ion doted with ve Ioc it y should be only in

float d (_spdOfLight7_spdOfLight) (rotateViw x rotateViw x

rotateViw y rotateViw y rotateViw. z'rotateViw. z);

float tisw float b (sqrt ( ( b b',

c))) '( floatL'floatr2)) d)

Sfloat )float (4; ) d

''Check to make sure that objects that have veloc iy c d t p before they were created (Moving Person objects behind Sender objects)

if _wrldTime tisw ,_strtTime 11 _strtTime= 0)

{ o draw I; } else 32 1 ca)

(33)

o. draw ;

}

//get the new position offset, based or the rew tie we jst curJ

//Should only be in the Z direction

riw.x += rotateViw.x tisw;

riw.y rotateViw.y tisw;

riw.z rotateViw.z tisw;

/Apply Lorentz iansfor

// float newz =(riw.z + state.PlayerVelocity * tisw) /

state Sqrtone jisVSquaredCWDividedByCSquared;

//I had to break it up into steps, unity was getting order of operations wrong.

float newz (((float)speed-_spdOfLight, , tisw);

newz = riw.z + newz;

newz /7= (float)sqrt~l speed-speed));

riw.z : newz;

if (speed

float trx - riw.x;

float trry = riw.y;

riw.x - riw.x - (ca + ux ux (1-ca)) + riw~y (ux uy (I-ca))

-riw. z (uy sa);

riw y trx uy'uxi(-ca)) riw.y ( ca + uyuy: (1-ca))

riwz(ux saY

riw.z trx (uy sa) trry (ux'sa) + riw.zi(ca);

} } else { o.draw 1; I riw _= playerOffset;

//Transform the vertex back into local space for the mesh to use it

o pos mul(_World20bject41.0,riw);

o pos2 = mul(Object2World, opos

o pos2 playerOffset;

o pos mul(UNITYMATRIXMVP, o pos);

return 0;

}

//Color functions, there's no check for divinion by 0 which may cause issues on

//some graphics cards.

float3 RGBToXYZC( float r. float g, float b)

{ float3 xyz; xyz x C.13514 r - 0.120432 g /.057128 b; xyz.y 0.0668999 r , 0.232706 g + 0.0293946 b; xyz z 0.0r - 0.0000218959'g 3.358278 b; 33

(34)

I float3 { float3 { return xyz; bloai x F-C y z) float3 rgb; rgb x - 9.445 x 5.1445 y - 1.16389 z; rgb y 2.86007 x , 5.77745 y - 0.179627 z; rgb z 0.000174791 x 6.666352664 y 2.79113 z; retin rgb; weightFromXYZCurvesfloat3 xyz) float3 returnVal;

returnVal.x 0.0735606 xyz.x 0.623R7693 xyz.y 0. 3 6 8J7 xyz z;

returnVal.y -0.0665378 xyz x 0.i344M * xyz.y 0.000417865 xyz z;

returnVal z 0.0600299624 xyz x 6.00666665249 xyzy 66464424

ret;rn returnVal;

FJclai getXFromCurve(float3 param, f124t shift)

{

float top- param. x xla exp floar ((pow( ( param y-shift) - xlb,

2'(pow(param.z shift 2 pow xlc.2)))%)-;sqrt(

float( 2) (float 3 1413926 52979223) ;

float bottom: sqrt((fIat)I powparam.z shift,2))+j pow(x1c,2)).

float top2 param x xha exp ( float( (pow (param.y shift) - xhb,2) 9(2 (pow(param.z shift,2) pow(xhc 23))))) sqrt(

(foat(2)(float)3.14159265358979323))

float bottom2 - sqrt((float)( pow param z shift,2))(1 pow xhc 2 i:

return topi, bottomi - top2 bottom2

float getYFromCurve(float3 param, float shift)

{

float top param x ya exp. float powi param.y shift) yb 2)

/(2 (pow(param.z shift ,2pow'vc sqrt

12 oloa t ) 3, 141592C535 79323)

float bottom sortt 'float L pow param. z shift 2') .'pow(yc 2 return top, bottom;

2)

float getZFromCurve(float3 param, float shift)

float top param x za exp( float pow, param y shift) zb 2)

(2 pow(param. z shift 2)-pow(zc 2 sqrt( oat 2 float )3 141359265350279 25

Float bottom - sqrt tot 1 pow param zshift 2 1 powizc 2

-eturn top bottom;

float3 constrainRGBi float r, float g F1' b)

{ - oa W; w . r r; 34 xyz z; I (float Ifloat1 } { tflA1 ft

(35)

w w w w w w; g w g; b) w b; if (w , 0) { r w } w r; w (W g)' w w b) if w 1 ) { r w; g w; b w; float3 rgb; rgb.x r; rgb.y g; rgb.z b; return rgb;

//Per pixel shader

float4 frag(v2f i) { //Used to float x1 float yl float z1 g + w; b -- w; g w; b w; ifes color COLOR modifications

maintian a square scale ( adjust for screen aspect ratio

i pos2 x 24xs;

i.pos2.y 2,xs/xyr;

i. pos2.z;

/7 ( 1 -(v/c)co,(rieta) ) / sqrt ( I (v/c) 2

float shift ;1 ((xlivr.x yli.vr.y a zli.vr z)/sqrt( x1 x1 yl yi i.svc;

if ( _colorShift a)

{ }

shift /- shift;

//Get initial color

float4 data = tex2D (_MainTex, i uvl) rgba; float UV - tex2D( _UVTex, i.uvl) r;

float IR - tex2D( IRTex, i.uvl) r;

//Set alpha of drawing pixel to 0 if vertex shader has determined it should not be drawn.

data.a = i.draw ? 1 : 0;// data.a:d;

float3 rgb - data.xyz;

//C o i- shift due -o doppler, go r GB XYZ, shift, then back to RGB.

float3 xyz RGBToXYZC(float rgb x float rgb y),float(rgb z)-float3 weights - weightFromXYZCurves'xyz);

float3 rParam gParambParamUVParam, IRParam;

rParam x weights.x; rParam y float 615: rParam. z float 8;

35 + zI zl i)

(36)

gParam x weights y; gParam y ( float) 550; gParam z - float)4;

bParam x weights z; bParam y (-loatl 463; bParam z ( float'5;

UVParam.x 0.02; LVPara;.y UV_START UVRANGE UV; UVParam z (float)5; IRParam.x 0.02; IRPara'i.y IR_START IRRANGE IR. lRParam'.z float)5, float xf pow (1/shift),3)-getXFromCurve rParam, shift',

getXFromCurveigParamshift) getXFromCurve(bParam. shift) i getXFromCurve:I[RParam. shift)

getXFromCurve(UVParam, shift);

float yf = pow((i shift),3) getYFromCurve(rParam, shift)

getYFromCurve(gParamshift) - getYFromCurve bParam shift) getYFromCurve(IRParamshift getYFromCurve(UVPa raim,shift);

float zf pow((1 shift),3) getZFromCurve(rParam, shift)

getZFromCurve(gParamshift) getZFromCurve(bParamshift) getZFromCurve(TRParamshift)

getZFromCurve(UVParam shift.)

float3 rgbFinal XYZToPGBC(xfyfzf);

rgbFinal = constrainRGB(rgbFinal.x,rgbFinal-y, rgbFinal.z); //might not he needed

//Test if unityScale is correct, unity occasionally does not give us the

correct scale and you will see strange things in vertices, this is just easy way to test //float4x4 temp = mul(unityScale.w* _Object2World, World20bject);

//float4 temp2 = mul( tempfloat4(

(lFat )gi na . x, (float)rgbF inal y, (float) r-gFii . ,/return temp2;

'/float4 temp2 -float41(

float )rgbFina] .x,(float)rgbFinal .y,(iloat)rg.DFina.4 z data.a );

return float4((floatirgbFinal.x,(float'rgbFinaly,(fioatrgbFinal.z,data a); //u ie ie For any real bui ld

}

ENDCG

Subshader {

Pass {

/ 'h our proertie' , 7ur tihings' sRuc F, s Ir rF/'paency

Cull Back ZWrite On

ZTest LEqual

Fog { Mode off //Fog does not shift properly and there is no way to do so with

AlphaTest Greater 0

Blend SrcAlpha OneMinusSrcAlpha

CGPROGRAM

pr'WAga fragmentoption ARBprecision hint nicest

,pragm vertex vert '.pragma fragment frag

Ipragma target 3.0

ENDCG

} }

Adapting the OpenRelativity toolkit to multi-perspective dome projection

Z

Signature redacted

Signature redacted

JUL 19 2016

LIBRARIES

ARCHIVES

Contents

Figures:

1. Teaching Special Relativity

1.1.

Introduction

1.2.

Challenges with Teaching

1.3.

Improving Understanding with Games

2. OpenRelativity and A Slower Speed of Light

2.1.

Introduction

2.2.

Past Work

2.3.

Effects Simulated

2.3.1.

Runtime Effect

--mewm

2.3.2.

Lorentz Transformation

fl

Tj

I

[

;rJr

T~

2.3.3.

Doppler Shift

I

"~

I

-

2.3.4.

Searchlight Effect (Relativistic Aberration)

2.3.5.

Time Dilation

2.4.

Implementation Details

3.

Dome Experiences and Learning

3.1.

Introduction

3.2.

Dome Experiences and Previous Work

3.3.

Interactive or Passive

4. Einstein's Playground

4.1.

Introduction

4.2.

Technical explanations

4.2.1.

Pipeline for rendering to the dome

4.2.2.

Code modifications

4.3.

Overview of the experience

4.3.1.

Advantages of a Game

4.3.2.

Presentation of the Experience

4.3.3.

Experience Description and Discrete Experiments

4.3.4.

Runtime Effect and Time Dilation

4.3.5.

Doppler Shift

4.3.6.

Lorentz Transformation and Relativistic Aberration

4.3.7.

Finale

5. Discussion and Future Work

_I