Sara Rafsky interviewed VR and “immersive journalism” pioneer Nonny de la Peña about her latest piece, “Out of Exile: Daniel’s Story,” included in the New Frontier section of the 2017 Sundance Film Festival. Using the real audio recorded by teenager Daniel Ashley Pierce in 2014, this virtual reality project creates an animated interactive experience that puts viewers in the middle of the emotionally charged and violent encounter that ensued after Pierce came out as gay to his family. Pierce’s family subsequently kicked him out of their home. The experience ends with testimony from him and other LGBTQ homeless youth, rendered in holograms created by using the technique of videogrammetry, sharing their experiences.
The conversation has been edited and condensed for space and clarity
Sara: Could you briefly describe the project, and why you chose to make this story?
Nonny: Forty percent of homeless youth in America come from the LGBTQ community. After I did a TEDWomen Talk, Sara Ramirez, who is the actress from Grey’s Anatomy, came up to me, and said that she was on the board of this organization the True Colors Fund, which was trying to fight homelessness among the LGBTQ community. We started talking about it and she told me about Daniel Pierce’s story and the audio that he recorded, and it seemed to me that it was the perfect project to tell the story.
Given that audio, how do we not just tell kind of a tragic – this incredible, difficult, private thing that’s been happening across America- but also try to give a message of hope? I thought that that was equally important in this piece, and that’s how we ended up doing the hologram that lets Daniel talk about how he got his life together. But we also wanted to show other people and how diverse the group actually is. Of the forty percent of homeless youth that are LGBTQ, the majority are people of color. So it was really important to make sure that they were included in this.
Sara: How did you select those other people? Where did they come from?
Nonny: They are four people who had been thrown out by their families and had been introduced to me by the True Colors Fund. These were just extraordinary individuals. They’ve been through these terrible things and the point of the piece is to try and convey their unbelievable resilience and not just the “Holy shit how can families do this?” Could other people, by seeing what these kids go through, reconsider their positions? Could the piece itself offer some hope for kids going through this?
It’s a journalistic story, right, it’s a real story. The statistical, investigative reporting about these kids that are homeless in America and are part of the LGBTQ community, that’s an amazing piece of journalism and an amazing documentary all by itself. But then when you go in and you try to put people on the scene and make them really understand what’s going on, you have a much better possibility of creating a bridge.
Sara: Can you talk a little bit about both the design process – how you wanted the experience to look and feel, and then also the technical process you used to realize that vision?
Nonny: We always try to use as much of the source material as we can. So of course the audio was all from the real thing. We brought Daniel in and he gave us photographs of his home at the time so that we could reproduce it as exactly as possible. And we also brought him in to really work with us and the actors to try to remember beat by beat what happened. And that was a big deal because we had to be very careful. We had to work with his memory of what happened and that memory is very much confirmed and substantiated by what you’re hearing on the audio.
So then I had to put the actors in motion-capture suits and direct them in the performance and make videos from that performance. And then from there, the motion-capture has to be cleaned up and we had to make the models in the studio.
And then the second half is using AI technology to make the holograms and we brought each one of the kids into the studio and had them tell their stories. Unlike with CGI, the holograms don’t track you. They don’t look at you. If you turn away and walk away they’re just kind of just speaking out into the space. We know that in documentary that talking heads can be the kiss of death, right? Well guess what? So can a talking body and talking bodies. And so you have to think about how do you construct holograms in a way that keeps them engaged and interesting? And so that’s when we began to play with new techniques and placement and inter-cutting between them.
I think that that was kind of the trial and error phase. We had no budget, so I’m just basically financing it in-house. I’m trying to figure out how do we make it more beautiful to illustrate what they’re doing and yet we have no budget. Every documentary maker knows what that’s like. Or pretty much everyone.
Every time we make a VR piece, pretty much it’s never been done before, right? It’s using technologies that we have to write new code for. And push envelopes on. And re-think. And try this or try that. It’s super, super complex. And trying to make sure that there’s time for the creative aspect. Especially at this moment when VR has just exploded and the requests for my time are [overwhelming].
Sara: There was a lot of interesting stuff coming out of Sundance this year and a lot of conversations around new ways in which VR is evolving. Are there new technologies that you’re excited to explore in the future?
Nonny: Well I think if we’re able to get holograms that actually have the ability to interact with the viewer and once we can get into more networked experiences that will be super interesting.
Sara: Lastly, you’ve been a pioneer of VR and “immersive journalism” for years. What do you think virtual reality and immersive journalism mean both conceptually and in practice when the zeitgeist is all about “alternative facts” and fake news”? Do they potentially collide in a beneficial way? Or as a further distraction and source of confusion? Or something more frightening?
Nonny: So PBS FRONTLINE went into a state prison [for the piece 6×9] to recreate a solitary holding cell using photogrammetry and real images. That’s as close to the truth as you can get. There’s nothing ‘alternative’ about that fact. Because it has the same depth and volume in the way we really experience the world. I mean you can’t touch things. But that is as real and contrary to the alternative fact world as you can possibly imagine. And I think that that is going to really change the way that we share and think and move.
If indeed the Apple iPhone 8 is going to have lenses for shooting in depth, which means sharing your world in depth, I mean why would you share your world in a flat way if you can share it in a way that you’re actually experiencing it? And I think that’s going to really change a lot of things. I just don’t think we’ll share things flat. Depth is coming.