Jack on the Holodeck 1 (Before You Begin)

Project Introduction

This project seeks to explore the blurry boundary between the feelings of self and others in VR, which can be brought to the users through transitions between first and third person perspectives. 

Literature Reviews and Thoughts

1. Dualism and Animism can be perfectly applied to virtual spaces.

David J. Chalmers in his book “Reality+: Virtual Worlds and the Problems of Philosophy” talks about how though Descartes’ dualism is widely rejected in the real world, it works perfectly in a virtual space.

To prove this, here is the scenario: there are a group of people who were born in a virtual world. And “In the ‘outside world,’ which is the world we are living, they’re strapped into immersive VR headsets.” Everything they ever see and hear comes from a headset and ultimately from the virtual world.

In this situation, the minds of these people are non-physical in the virtual world, but physically based on their brains in an “outside world” that they will never know. Their minds are separate from their physical bodies, and their minds and bodies interact through a mechanism that can never be understood in the virtual world, which fulfills the concept of Dualism.

Animism is the belief that objects, places, and creatures all possess a distinct spiritual essence. It is widely used for ancient human to explain some natural phenomenon they couldn’t understand.

In the virtual spaces, all the NPCs and physical objects behave and perform certain events based on some psychological conventions in the real world, which again cannot be interpreted by the physics in the virtual world. From the perspective of the virtual world, these NPCs and physical objects seem to have their own minds/souls which is a perfect demonstration of animism.

Based on these interesting points, I started to think about how minds in a virtual space can interact with each other.  If Dualism and Animism can be applied to virtual spaces, it is then proved that everything in a virtual space, including player characters, non-player characters and physical objects, all have their distinct minds. If players can perceive the existence of these minds and interact with them, the whole game can be a large web of interactions among minds.

Thus, while playing games, players’ minds can keep interacting with and be influenced by all the minds in the virtual spaces.

2. Non-VR games usually have emphasis on the characters, while VR games focus more on the feelings of embodiment and immersion.

A study was done by Johanna Roettl and Ralf Terlutter about “how an identical video game that is either played in a 2D, stereoscopic 3D or Head-Mounted-Display (HMD) VR version is experienced by the players, and how brands that are placed in the video game are affected”. The result shows that “3D and VR lead to higher presence, i.e., to a pronounced feeling of ‘being in the game’, but game evaluation did not differ between the 2D, 3D, and VR versions”. Also, players’ memory for the brands placed is negatively affected by enhancement in technology, which means players are more impressed with the characters and stories behind the scenes in 2D games.

The result of the research verifies my personal experience with all kinds of VR and 2D games. When I play 2D games, I always feel that I am separate from the main characters in the games, even though I am able to control them. While in VR games, the feeling is much emphasized that I am the characters. It would be interesting if these two features could be combined and realized in one game. So that, the players can experience the feeling of embodiment(self) and the existence of the character(others) at the same time, which may blur the boundary between self and others.

3. Transitions between 3PP and 1PP in VR are beneficial.

Two studies done by Patrick Salamin and Henrique Galvan Debarba propose and prove the benefits of switching between first and third-person perspectives in VR from different perspectives.

It’s shown that the transitions could help to make the whole experience more immersive. Also, seeing the character from third person perspectives is a direct way to show the distance between players and characters, which provides space and possibilities for players feel the existence of the characters and then the personalities of the characters.

4. Soma Design: Treat users as united whole of mind and body is better to design for users’ feelings

Kristina Höök in his book Designing with the Body, Somaesthetic Interaction Design advocates a new approach “soma design”, which is “a process of creating harmony between the soma and the products”.

“Soma” is defined as “the self that is a united whole of mind and body, in which our physical being produces and affects our thinking, and our mental and emotional experiences influence physical outcomes”.

Soma design is the methodology I should follow to create harmony between new physical interactions between players’ bodies and physical UIs (user interfaces) and interactions between players’ minds and the minds of player characters, NPCs, and physical objects in virtual worlds.

For the physical interactions, I want to let players’ whole bodies to get involved to control and play the game, instead of using traditional peripheral devices such as mouse and keyboards. I plan to use whole body motion capture with VR so that my players could see their own bodies moving around in the virtual worlds. This also leads to the potential to have new ways of physical interactions in this project because the body movements of the players can be used to control things/events in the experience.

For the interactions between minds, I will achieve it using ways including transitions between 3PP and 1PP as mentioned.

Conclusion

This project derives from my interest in integrating the features I like from other games into my own game. I was impressed by how players’ minds can interact with the minds of the player characters, non-player characters, and even physical objects in a virtual game world, which is featured in 2D/non-VR games. I was also amazed by the sense of embodiment VR games can bring to the players. After this, the combination of the two features from 2D games and VR games is proved to be possible by the two studies about the benefits of usage and switch of first and third-person perspectives in VR. Then, following the method of soma design, I will be able to regard my players as whole unities of body and mind and create a user-centered experience, in which players can find their minds influencing and influenced by other minds in the virtual worlds, but also amazed by the new physical ways of interaction.

Methodology

According to the literature reviews above, I have two objectives I want to achieve in this project:

1. I want my users to feel the personalities of the characters in the VR experience I am going to create.

2. I want my users to physically get involved into interesting ways of interactions.

The first objective focuses more on the interactions between minds, while the second objective focuses more on the involvement of physical body movements. This leads to the methodology I mentioned before called soma design. Soma is the self that is a united whole of mind and body, and soma design is the design methodology to treat users as somas, which requires designers to focus more on users’ sensations, feelings, emotions, and subjective experiences. To achieve my two goals and the concept of soma design, I will use the methods as following:

1. I will use soma trajectories to record the users' feelings.

2. To carry out the soma design process, I will do user testing based on a prototype.

Soma Trajectory

Soma trajectory is a possible and desirable way to consider and record how a user feels through an interaction, both in body and mind.

According to research did by Kristina Höök and his fellows, soma cannot readily be quantified in and of itself. Instead, it can be described as dimensions in which I can move between opposing values. And those values should be set based on the overall goals of the project to indicate some characteristics of the experience. Also, to show the changes of the feelings through the whole process of the experience, the dimensions is designed to be placed on the y axis of a coordinate system, while the values on the x axis represent the interaction time. And this is called soma trajectory.

Dimensions I am going to use:

1. Interactive and non-interactive

As I mentioned above, the whole experience should be the combination of interactions about minds and body movements. Thus, “interactive” should be a criterion for this project.

2. The feeling of self and the feeling of others

I want my users to feel the personalities of the characters in an immersive VR experience. More specifically, what I expect is the combination of the feeling of self/embodiment and the feeling of others, which is expected to provoke users to think about their self-identification in virtual spaces.

3. Comfortable and uncomfortable

Feeling comfortable is a common but important goal for every user-centered experience. Especially for this project, my users are expected to wear different kinds of devices which may cause users to feel uncomfortable and lead to bad effects to their experience.

As shown below, the trajectories will be presented as diagrams with the soma esthetics dimensions on y axis and interaction time on x axis.

The interaction time on the x axis is different from the “clock time”. It’s more like the representation of progression through an experience. To enhance this representation, I include a set of “important points” to represent some important events through the experience.

To better analyze and record the feelings of my users, I will use Canonical Trajectories to visualize the expected experience of our users, then Participant Trajectories to record the real users’ feelings during the tests, and finally Historical Trajectories to combine all the previous records into one diagram, which represents the accounts of multiple users and will be used to reflect on the experience as a whole.

Prototype

For the prototype I mentioned above to support the user tests, I made a VR experience with special story and transitions between first person perspectives and third person perspectives.

The story is a Cthulhu style horror. The player will play both the roles of an old Cthulhu god and a human being character living in a floating island. The human being living on the island has to offer up sacrifice to the old god to allow temporary spirit possession from the old god to help the character fight against the enemy.  This is the time when the player can transfer from third person perspective to first person perspective, which is the character’s perspective. Once the character offers up sacrifice to the player, the connection between them goes deeper, so that the time of the spirit possession becomes longer. While the side effect is that the character’s mind will be gradually replaced by the player’s mind and finally the player will be able to keep the first-person perspective and explore the world. This experience is called “Before You Begin” because it can serve as a Prequel of another adventure where players explore the world as the main character. The story is designed to let players feel the personality of the character and also the feelings of embodiment of their own existence in the virtual space.

Based on the background story, I created a 3D environment using UE5 as shown below:

I also made several scenes about offering up sacrifice and waving the sword to fight against the enemy:

I tried to seek ways to let VR and motion capture work together, but the VR headset and mocap equipment I used were not designed to work together. Thus, for this prototype, I didn’t make it to combine VR and motion capture together, which means my users didn’t get chance to see their bodies moving in the virtual spaces. The possible solution is to use ROKOKO motion capture system, which supports stationary mocap, so that VR HMD will control the position of players’ avatars in virtual spaces and the motion capture system will control the body movements of the avatars.

Technical Showcase

Blueprints for Game Mechanics

Sword VFX

User Testing and Analysis

As I mentioned above, I made the first prototype of the project and tested it with several users. I asked them to draw the soma trajectories based on their feelings during the whole experience. There was also some general feedback to the whole experience.

Before the user tests, I drew several Canonical Trajectories to demonstrate my expectation to the feelings of my users/ players during the whole experience:

Then I let my users to draw their Participant Trajectories to show their real feelings to the experience:

I only had one prototype and user testing, so there is no Historical Trajectories.

Based on the trajectories I got, I made some analyses and proposals to improve the overall experience.

Firstly, for the diagram with comfort on the y-axis, I expected my users to become more and more used to the device and the style of the experience, so that they will feel more and more comfortable as time passes. However, the feelings of comfort turned out to be subjective, which means different users had different feelings and definitions of comfort. As you can see in the diagram, one user felt more and more comfortable during the process, while one felt less comfortable as time passes. To keep the whole experience at a relatively comfortable level, I may need to take more user tests.

The same is true for the feelings of self. This is also a kind of subjective feeling that people happened to feel in contrary ways. The good news is that what I want to achieve in this project is letting users acknowledge the existence of the personalities of the characters, which is the transitions between the feelings of self and others. The diagram shows that all the users felt the way I wanted them to feel.

The last diagram about interactive was quite similar to what I expected my users to feel. The whole experience was not pretty interactive at the beginning, but when the transitions between third-person perspectives to first-person perspectives happened, the experience became more interactive.

The general overall feedback on the prototype was good. People were all impressed by the transitions between perspectives and the interactions to fight against the smoke from first person perspective. However, all the users mentioned that they wished to see more personalities and randomness of the character because the stories about the character in this prototype were all shown in pre-recorded animations. Also, it would be more interesting to have multiple characters and different kinds of ways of interaction.

Conclusion and Future Work

This project derives from my interests in both non-VR games, which feature personalities of characters and background stories, and VR games, which feature a sense of embodiment and immersiveness. My readings about philosophical thoughts about virtual reality inspire me to the interactions between minds in virtual spaces and the research about the third person perspective in VR provide me with ways to combine features in both non-VR and VR games in one experience. Based on my pursuit to have my users both mentally and physically get involved in the interactions, I decided to follow the design process of soma design and use soma trajectories to record the feelings of my users during the whole process of the experience.

For future work, according to the feedback I got from user testing, the background stories and the ways to demonstrate the personalities of the main character should be more enriched and polished. More user tests should be done. Also, I am thinking about a multi-player experience where transitions between perspectives happen among several players and characters so that the boundary between self and others will get blurred.