Issue 2. Virtual Fictions. Storytelling in VR

Notes on Blindness. Experiencing a new perception

By Fabien

English subtitles available.

Notes on Blindness is a rare and sensitive VR experience on the perceptions of blind people, and the farsightedness they experience when losing eyesight.

Arnaud Colinart, director and producer of this VR was our guest at the Printemps du Virtuel in Paris. A chance to discuss VR, altered perceptions and immersive storytelling.

Making the invisible visible

Arnaud Colinart, you are the producer, director, and really the handyman of Notes on Blindness, or at least of its VR component. So can you tell us a bit more about the project for those who have not seen it yet?

Arnaud Colinart: Notes on Blindness is the story of John Hull, who is, or actually was, as he passed away last July, a British theology professor who turned blind at the age of 45 and recorded an audio diary of what was happening as he was losing his sight. Both the film and the VR experience are based on those recordings. He recorded 16 hours of tapes, which were used by Peter and James for the film’s audio, so all you see is actually a reenactment – we could say “visual dubbing” – with actors. Characters are only pretending to speak: all you hear is actually taken from the tapes. For the VR aspect of it, it’s pretty much the same idea, except we recreated in real time 3D – following gaming standards – what John Hull called “a world beyond vision”.

So what steps did you take with VR, to make the invisible visible?

Arnaud Colinart: Well at the start of the project we actually didn’t want to use VR at all. We originally wanted to do a full audio project, but we quickly realized it was impossible to really involve a spectator “sighted” spectator in a solely audio experience, as sight and outlook are key elements in how we perceive our environment. So the question we had to ask was how to visually recreate the experience of blindness. What we chose to do was to use everything related to auditory activity: we move through a dark space, and we use various visual elements depending on the sounds linked to the current set piece. We really came to VR in three phases: it was an audio project at first, then we integrated it in a flat, 360° view – what we call the magic window: what you see directly from your iPad or your phone – and finally we added this more immersive dimension through the headset and 3D. That’s really how we arrived to VR: as soon as we started prototyping, it was clear that the immersion in the world we were building was much stronger. That’s how it became fully part of the project.

Beyond sound, the sense of touch is often alluded to in Notes of Blindness: you talk about the wind a lot.

AC: We’ll approach the sense of touch in one of the 360° version of the project, as there’ll be a version that does not require a VR headset. It’s a project that is partly funded by Arte. The question of accessibility is important to us, so this 360° version will be available to anyone with a smartphone. The sense of touch is less present in the narration of the VR version – but some of it transpires through John Hull’s tale: the relationship to the body, and how it feels. That is very much part of the VR version.

It is clear now, at least for those that already tried the experience, that the main idea is to step into the shoes of someone who is progressively losing his sight. Instead of just being a story of loss, John Hull tells us how he perceives, or how he is starting to re-perceive the world around him – which truly is a brand new world. We won’t label it as good or bad, but it is definitely different, and that’s why we speak of altered perception. Did you show this to people currently in this situation?

AC: Very early on we started working with two persons that are visually-impaired, but they’ve had their handicap from birth. This is very different from someone that is slowly losing his sight. As of today, we haven’t done that sort of test, as it was difficult to organize, but it is something we look forward to. What we did realize, to my great surprise, is that they were few visually impaired people that had access to the experience of spatialized sound. I imagined there were opportunities for this public, but apparently not. One of the problem encountered is that if we place a sound of wind in front of you and show you images of leaves moving, you’ll be positive it is the sound of the wind in the trees. But if we play the same sound without a visual, well you won’t be so sure: it might be the sea, or something else. So for a public of visually impaired people, we have to ask ourselves how they will perceive the recording: it will depend of the nature of the auditory assets we have, but I’m not sure we have such a firm grip on that. The experience will therefore rely more on a form of audio-description – which is why we will probably favor a 360 non-VR version for this particular public.

Hearing you, it feels like you’re talking about the combination of senses, and I think it’s interesting to rediscover how our senses work. This was made a little abstract by the proper definition of the five senses. But today, in science, or literature, or any subject that deals with perception, we realize that sensory combination is important: a sense often inform another, and its through these combinations that we built mental images. Is that something you guys worked on as well, these sensory combinations.

AC: Actually we didn’t want to really get too close to synaesthesia, since it is…

Sure, synaesthesia is somewhat the extreme example of this.

AC: Exactly. Actually we are currently working on the last scene of the experience, which is somewhat linked to this, at least in a visual sense. We’ll see if it makes sense with the project, and we’ll see if we end up using it, but we prototyped it. It is definitely something we looked into: the work of Oliver Sacks, everything related to synaesthesia, not in comparison with John Hull’s story, but in how we could use it for the representation and transmission of John Hull’s story. We studied what we could learn from synaesthesia, and the ways in which it is explained. Are there anything there we could use, and would that makes sense in how we tell this story?

Stepping in the shoes of a farsighted blind man

We talked about altered perception, and I think that’s something we encounter in several works of VR, something in which VR can be a valuable medium. It is not only the capacity to offer us another point of view, even the point of view of someone else, but also to give us various perceptions, to show us that there are many possible states of being to our world. Is this altered perception something that could inform us beyond the realm of blindness, on say people who are entering old age, or any other type of fundamental change?

AC: I’ll have to answer that in several parts. First, there’s an element that is talked about a lot in VR, and that is the idea of “presence” – that is, the fact that one is transported and able to interact with a world that was built for a specific medium. This presence allows us to see the world through the eyes of someone that, from a narrative perspective, is us, but who is not physically us. It is paradoxical, but at the same time it definitely reinforces empathy.

Then you have the fact that we’re not a 3D, 360° video. we’re an interactive experience. The way the story unfolds is directly linked to the behavior of the user, which we’ll monitor in order to start specific scenes at specific moments, which I think plays strongly in the immersive factor. Via interaction, we’re really going head on with the degree of immersion the user is going to feel.

Moreover, I think what really makes this an empathic experience isn’t really the VR medium, but the narrative process and John Hull’s story. I feel like in Notes on Blindness, we really used the VR medium to strengthen that story, which remains the heart of the experience. There are projects that will focus solely on the technical possibilities of VR, but they will not necessarily have an engaging narrative. Since we used the story as our starting point, we received offers from producers or authors, on themes related to handicaps. I do believe there’s a lot to do in VR on subject such as diseases, internment…

But in all this the question remains: what’s the story? We did use this technology as much as we could, while keeping all the possible technical limitations in mind, but it is first and foremost the story of John Hull.

Interaction and involvement in Virtual Reality

You were discussing user interaction and involvement. It’s something I’d like you to develop a bit. As opposed to other experiences we’re showcasing today, Notes on Blindness has a true interactive narrative. I believe it’s one of the real challenges of VR, to find the right balance between offering interaction and offering a set structure. Is that something on which you worked particularly hard?

AC: Well what we used is what Amaury La Burthe of AudioGaming calls elastic storytelling. That is, we know point A and point B, we know where the story starts and where it ends, and depending on how the user behaves we are going to stretch or to compress the storyline, to tailor it as perfectly as possible to the user.

I think once we’re out of this first phase of VR, which is great but has a “great, I’m in a rollercoaster, or I’m in Africa or whatever” side to it, the question we’re going to ask is: I am in this world, but how much can I interact with it? Clearly one of the first shortcoming we’ll encounter is that even in a video we interact, if the director places an important scene in a particular direction, the user can look elsewhere.

In an interactive project like Notes on Blindness, we can only start a scene if the viewer look at a specific direction. In my opinion, we’ll see a lot of projects that will be 360° videos at first, but I think we’ll move on to interactive experiences. The concept of direction will shift to scenes activated by user behavior. That’s why I believe a number of projects that are 360° videos are really 180°, because it is amazingly hard to take in consideration or to use the remaining 180° efficiently in a pure video experience.

Sure, and it’s not necessarily that interesting to be constantly moving your head to see what’s happening behind you.

AC: The scenes we selected for Notes on Blindness really are focused on exploring a world, so in that context, it does make a lot of sense to look around you. It is the audio narrative itself that says “a car starts behind me, or my son is sitting next to me and now he’s gone”.

One of the main lesson I took out of this experience is that VR allowed us to gain in immersion, but the narrative itself did work perfectly in a 360° setting. As soon as we prototyped and put on the headset, we were transported inside this universe, and everything around us was gone. In a way, we’re not too far from the experience of going to the movie theater.

It’s not the same experience, watching a movie in a screening room or at home: you enter the room and you leave the world behind. At home, you never know when the phone’s going to ring, the neighbors are going to show up, or you kid’s going to wake up. VR allows us to get back this truly great immersive strength.

At Fabula we talk about experience enhancers. We feel like VR really has the potential to intensify any experience – good or bad. It can make things more pleasant or unpleasant. In any case, we definitely get something more out of it, especially when its coupled with spatial exploration.