Seeing the future at NYFF's Convergence series

New York Film Festival

On top of its strong slate of features and documentaries, this year's New York Film Festival also looked to the future with "Convergence," a program of immersive and interactive works.

The fifth edition of the program ran Oct. 1-4, with two world premieres among the nine works, plus talks with producer Lindsay Doran and others.

Pieces ranged from augmented reality installations like Priya's Mirror (Ram Devineni, Dan Goldman, Paromita Vohra, Shubhra Prakash and Monica Singh), built around a comic book, to Sherlock Holmes & The Internet of Things (Lance Weiler and Nick Fortugno), in which participants solve crimes by using technology to find and decipher clues.

Most of the Convergence pieces employed virtual reality, the holy grail of interactive gaming. VR has become increasingly polished as cameras and software improve. But on a narrative level, the process is still in its infancy, with style and conventions yet to be worked out.

The installation Cardboard City (Kiira Benzing, Stina Hamlin and Alyssa Landry) uses both VR and augmented reality (AR) to show how gentrification is affecting artists and residents in the Gowanus section of Brooklyn. A wall of the Elinor Bunin Munroe Film Center became a cardboard version of the neighborhood skyline, with storefronts and apartment windows.

Pointing an iPad or smartphone at specific artworks triggered a display of photos or brief video clips devoted to housing policy, artists evicted from lofts, and the face of a changing neighborhood seen through time-lapse photography.

The AR wall took two months of post-production, with developers using the Aurasma and Vuforia apps.

AR presents new challenges to storytellers. In a traditional documentary, moviemakers present an argument, back it up with explanations and evidence, and reach a conclusion. Since users determine which way to point their sensors, Cardboard City AR has no set narrative structure.

Instead, director Kiira Benzing of Double Eye Productions explains that there are multiple entry points into the story. However, that means users might see the "conclusion" first, and miss or skip over explanatory material. Benzing and experience designer Katie Edmonds hope to use Cardboard City to educate viewers and increase awareness of real-estate issues.

The centerpiece of the installation is a VR short called Cardboard City or Cardboard City 360. The minute-long piece was shown at this year's Sundance Film Festival in the Samsung Studio. It was co-winner of the Samsung Gear Indie VR Filmmaker Contest and its “There in 60 Seconds” challenge.

Cardboard City 360is a tour of animator Danielle Ash's studio. A previous winner of the Helen Hill Award, Ash constructs detailed streetscapes from cardboard, then uses stop-motion photography to animate them. In shorts like Pickles for Nickels, she evokes a blue-collar Brooklyn that has almost completely disappeared, using techniques that date back to the dawn of cinema.

The VR short lets viewers inspect all four sides of Ash's loft studio from top to bottom, including works in progress, tables covered with supplies, and even a clip from one of her shorts. Remarkably, the VR piece includes a bit of stop-motion magic as a rollercoaster construction suddenly comes to life.

Cardboard City 360 took two days to shoot, using a monoscopic 360-Heros rig and seven GoPro cameras. According to Benzing, the shoot required eight days of post-production at East Coast Digital. The team used Autopano Video Pro and Autopano Giga stitching software; After Effects for cleanup; and Premiere for editing.

Benzing used a Ricoh Theta for pickup shots for Boxed Out, a follow-up to Cardboard City 360. The second VR piece will also employ Unity, a game engine which will let the moviemakers "spatialize" the audio.

At this point, viewers cannot "walk" through Ash's studio, but only turn in place to see different areas of the loft space. That means they might be looking in the wrong direction when Ash's stop-motion animation starts. "Spatializing" the sound may help direct viewers to see specific effects. For Boxed Out, Benzing is working with Evolving Technologies Corporation to use sound and interactivity so viewers will have the ability to make choices and move deeper inside Ash’s world.

Traditional filmmakers have a number of methods to tell viewers where to look, from lighting, framing, camera movement, acting, and especially editing. But VR eliminates many of these. Framing is essentially impossible when VR cameras are capturing an entire room. Directors can't move the camera to a specific area without affecting the illusion of an immersive reality. And editing can completely disrupt the VR experience for viewers.

Cardboard 360 does contain one cut at the end of the stop-motion sequence, a choice that also brings the scale of objects into play. Ash remembers how difficult it was to shoot the 25 still frames for the sequence, as she had to hide behind a shelf after each rollercoaster adjustment.

Although it's easy to see how VR will be used in gaming, as a narrative tool it's still in its early stages. It's not clear how to edit in a VR space, how to make sure viewers see what you want them to see, and how to allow a story to unfold.

One other problem with VR is its lack of a communal experience. Viewers watch VR alone, isolated by a headset and earphones. Solitary, even cocooned, VR viewers can't interact with others—laughing a jokes, gasping at shocks—like traditional moviegoers can.

Judging from the growth of streaming services and handheld devices, younger viewers may not mind watching by themselves. Perhaps the ability to message while watching will resurrect a sense of community. Or VR may evolve into another multiplayer online game feature.

You can view Cardboard City 360 at YouTube, Samsung VR on the GearVR headset and the Littlstar app for Android and IOS.