Hacking_the_senses_with_VR-Marshmallow_Laser_Feast-Fabbula11

What is Virtual Reality?

VR, the world beyond our senses, with Marshmallow Laser Feast

By Fabien

Marshmallow Laser Feast’s output counts some brilliant exploration of Virtual Reality : the music video “Stor Eiglass” for Squarepusher, a 2012 VR experience sans headset, and more recently, a VR work in which you are “In The Eyes of The Animal”.
Fabbula visited the London studio on a sunny afternoon and sat down with Barnaby Steel and Robin McNicholas.

From Projection Mapping to Virtual Reality

Fabbula: How did you guys start MLF?

Barnaby: It was a conscious reaction to the repetition of normal commercial work. It makes sense to repeat the formula of what you do from a commercial perspective, but creatively the ground hog day is a killer. We set out to break the routine and create a space where we have the freedom to follow inspiration, explore, experiment and discover.

Did Bill and Ted say, ‘Make it and they will come!’? Either way, investing our corporate gold coins back into our personal work somehow created a feedback loop where we started to get commissioned to do the passion projects.

Hang glider © Rob Pybus

Robin: When we started the company, it was also a reaction on what can happen with small companies. As you expand and put bums on seats you start to become tied to the skills of that team which restricts the breadth of your creative output and ultimately becomes boring.

Instead, we are keeping our overheads low and keeping the core team in coding and generalists. From that core team we can direct highly complex projects and build teams of specialists to fill in the gaps.

That’s changing now as we are getting more involved in VR and its technologies. For the first time in our careers we are so inspired the possibilities of realtime tools for mixed reality experiences that we are hiring game engine gurus specifically

Fabbula: Do you remember what first caught your interest in VR? How did you get to do your first project?

Robin: We used Unity to make a projection mapping project for Sony. We were creating a virtual world just like you would for VR, but instead of seeing it with goggles, we projected this virtual world onto a physical room. By tracking the camera and “the mechanical eye”, the virtual world perspective stays true to the way we see the real world, with all lines converging on our eyeballs.

“For us that was a direct encounter with mixed reality, years before VR had its come-back”

Barnaby: It was addressing what’s frustrating about projection mapping, which can only be seen from one point of view, like a photo. With this work we freed the viewing angle, turning walls into windows to a virtual world. For us that was a direct encounter with mixed reality, a few years before VR had its come-back. It’s worth noting that we were inspired by “Caves” which perform this same trick within the walls of well-funded institutions.

Robin: It was super lo-fi, but it articulated a lot of our creative intentions. The real hook to the piece was in the way the homemade lo-fi contrasted with the climax, the illusion of a living room transformed into a virtual world. It was a real passion project and we camped in the room downstairs for days.

In a project so all consuming we got carried away! Sony had asked for three thirty second films but we delivered three one and a half minute single-take films. In retrospect, whenever a passion project aligns with a commercial brief, our habit is to go crazy and invest all the budget and more into trying to realise its potential!

Expanding the concept of Reality

Fabbula: From there to VR seems to be a tiny step. How did you make the jump to the VR medium ?

“Science and new technology shine a light into the abyss, revealing a new understanding of the world and questioning our concept of reality”

Barnaby: When the Oculus Rift came onto the scene we jumped on the early Dev Kit. Our first project was exploring VR as a means of revealing the invisible forces acting on an F1 car for Lotus and Burn energy drinks. We took this project to the first Oculus Connect which is where we got picked up by Chris Milk and VRSE. That project was our first experience of a new and exciting workflow, a bit like the dream architecture in “Inception”. Once the main backbone of the virtual world was built, we adjusted the final looks by tweaking sliders from within the virtual world. This process was a revelation, in contrast to procedural animation where you imagine, build and render, we were able to play the potential looks like a synth, reacting to happy accidents and saving the golden nuggets.

Since then, we have been completely sold on VR. Our interest for the moment has crystallised on VR as a window to a view of the world beyond the limits of human senses. Science and new technology shine a light into the abyss revealing a new understanding of the world and questioning our concept of reality.

“Nature is not our enemy, to be raped and conquered. Nature is ourselves, to be cherished and explored.”― Terence McKenna.Photo ©Jon Hanna

Barnaby: In the Eyes of the Animal” uses virtual reality as a way of expanding our concept of reality. What does the world look like from the sensory perspective of a midge? How can it sense the increase in C02 coming from the breath of a human 20 meters away? How does the lifetime of a mayfly compare to that of a 3000 year-old tree?

Alan Watts gave a wonderful lecture that compares geological time to human time by comparing an apple and a tree. From an apple’s perspective, other apples are seen being born and dying, but the tree barely changes in a apple’s lifetime. We think of our planet earth as a dead rock, but on a geological timeframe our rock peoples like an apple tree apples.

By scanning a real forest and situating the installation in that same space, we were able to align the virtual world with the real world so that following the experience you would see the forest in a new light. In some ways photography does this just through unusual camera positions, with VR you can really be something else.

In The Eyes Of The Animal. ©Luca Marziale

Fabbula: This has deep ramifications that we are very keen to explore here at Fabbula.  Am I right to assume that In the Eyes conveys a certain message on our connection with nature ?

Barnaby: Coming from a photographic background, I’m tuned into the way that new perspectives on the everyday world can reconnect viewers to the wonder we experience as a child with fresh eyes. The repetition of day-to-day sensory bombardment can make us numb to the world around us, and VR can work like an unblocker, it shakes our snow globe.

“In the Eyes of the Animal” puts the viewer’s neck deep into a swamp, shrunk down to the perspective of a dragonfly on a log, gliding above the trees like a bird of prey. What we find more interesting than these pure perspectives on the forest is the question, what does this same forest look like beyond the limits of our senses, through the eyes of the animals?  Science gives us a window into this world, but the reality is that a human can never truly understand what the world looks like from, for example, a bat’s perspective. It’s like explaining to a blind person what the world looks like: although you can both reach out and touch the same object, no words can truly convey the richness and complexity of vision.

After a deep dive into the thesaurus we were very inspired by the paper by American philosopher Thomas Nagel, ‘What is it like to be a bat?

“Broadening what we think of reality”

Connecting with rocks and beasts

Fabbula: Feeling as a fly gives a new perspective on the ecosystem of the forest. If we humans start to have experiences as animals or even as what is perceived as inanimate objects  (like a stone… could we feel as a stone ?) What do you think the impact can be  ?

Barnaby: We are living in an age of such disconnection that inadvertently through daily consumption we are destroying our planet. Growing up in the city it’s easy to become an adult having had no connection to nature and other living things and why would we? We live in a man-made constructed reality; the language, the rules, the things we take for granted are constructed by man in advance of our arrival.

Virtual reality, this cutting edge technology that flowers from the earth through the minds and hands of some very smart people, might prove to be a powerful means of reconnection to the rock that makes this whole show possible. Using VR to connect with nature sounds ridiculous, but the experience can be profound.

As Allan Watts puts it, ‘The rocks are alive.’

Fabbula: A VR experience sometimes feels like living a moment, and not just a moment of entertainment. As creators, do you feel a responsibility for these experiences? Like you’re passing on a message?

Robin:

We have certain motivations about carrying messages about nature and reconnection to the planet, but it’s not at the forefront of our minds, nor is the concept of providing a sanctuary for people.

It’s more about a conversation within our extended family where things surface that align with the group passion. The Evernote (great software) ideas’ bank is overflowing but when we get a chance to realise our projects, it becomes about how we can make those as wonderful as possible. It’s our responsibility to make them amazing and we assume that if we can be amazed ourselves, then it will work for others too.

Making the mind drift

Robin: There is an interesting place where VR stores itself in your brain. It’s different from film and games. We used to joke that it slots right where your dreams are kept (laughs).  For me, the Crescent Bay demo was more dreamy than the memories of any film I saw.

For one of our projects at the moment we are considering the long understood connection between breathing and mental focus to achieve a meditative state. Imagine seeing your breath (all smokers and workers of frozen food section of the super market know what this is like) but instead of realism we create a particle cloud that moves like air but changes colour based on particle velocity and the time taken between breaths, essentially visualising the relationship between breath and relaxation. It sounds simple but with the right level of detail and finesse, it becomes a powerful meditative tool. Finding escapism and finding the place where your mind can drift is something that VR is amazing at.

In the tube or elsewhere in the city we are submerged with adverts. For us it’s quite nice to make a place with none of that, a place where you just float around and go somewhere that feels like the God hole. Years go by and as a culture you’ve had your dose. You know like when you find yourself in an empty church and it’s a place where you can experience release and escape from the outside world.

“A place where you just float around and go somewhere that feels like the god hole”

Ancona, Basilica Cattedrale di San Ciriaco, Italy, 2014. ©Markus Brunetti

Emergent narratives in VR

Talking about storytelling and narratives, how do you see them working in the VR medium?

Robin: We deliver alternative narratives from the normal ones. The great thing about this area of work is that it doesn’t need to be rooted in the traditional sense of a beginning, middle and end.

It can be angulous. It can be endless. A place you revisit, or joining a game that was started before you were born.

 

DMT trip in Enter The Void, a movie directed by Gaspar Noé.

Barnaby:

We view the world from between 5-6ft high, we touch, sniff, see and hear, we skateboard, skydive and eat tuna. We tickle our senses in all manner of ways but ultimately we are locked in our bodies looking out. VR frees us from our bodies and offers the potential of experiences outside our normal reality.

This isn’t the 1st time the doors of perception have been swung open. Humans have been exploring hallucinogenics, ayahuasca, mushrooms etc, since the dawn of time and it interesting to consider how this altered states translate into VR. VR offers escapism and as Huxley once put it “The urge to escape from selfhood and the environment is in almost everyone almost all the time.”

Robin: Because VR is a new medium, there are all these things that came before it, and are intrinsically referenced. When Flash on Internet came to be, people would present a book and turn the pages of a book on internet!

And that’s interesting to see with VR, people start to make films, start to make games until Virtual Reality takes root and forms its own conventions, its own norms and vocabulary. The audiences will also be very different. You’ll get gamers, films goers, theatre goers, they’ll get their share. But you’ll get others as well….

Fabbula: Are there other fields where you think VR is well adapted?

Barnaby: I think VR will find a great partner in music. People still pay a lot of money and travel great distances to see a gig. By not being limited by the costs of LED, health and safety restrictions of lasers or the laws of physics, we’re excited about VR worlds generated by music. Imagine landscapes of sound that ripple like water, where stems can be manipulated and sound is malleable like putty.

It’s interesting that you are getting into game technology. Games often use their world as the story itself, as a kind of spatial narration rather than being time-based. A great example is GTA, where the city is the story. With immersive VR, it could be even more so. Do you think so? Do you think that there are other narrative elements in VR beyond the sense of place?

Barnaby: There are really so many possibilities, ultimately VR will be able to replicate not only reality but anything we can dream of. We agree that immersive VR enhances the sense of presence and that this opens new types of game play that wouldn’t work when looking at a screen in your living room.

For example, in the real world I can sit on a mountain and daydream for hours, and this can’t be fully translated to a TV, but in VR it just might work.

We found that by scanning nature and simulating natural phenomenon like fluid dynamics, we were able to tap into and in some ways enhance, the inherently relaxing qualities of being in nature.

What’s next for MLF?

Robin: In the Eyes of the Animal” feels like a trailer for the main event! We want to explore the rich ecosystem around a single rainforest tree. The relationships between animals is fascinating, but to experience the sheer physical scale and biodiversity will make for an amazing experience. If we can see the world through the eyes of endangered species maybe we might reconsider the impact of our actions.

 

Thanks for reading through. You liked this story ? Don’t hesitate to write a response (we read them all), follow us (no spam, promise!) or share (we like that too!). 

Fabbula is a magazine about virtual reality, exploring new practices of storytelling, design, music and visual arts in the VR medium.