In this project, I took it as a challenge to capture the viewers’ eye movement patterns, which differ from movie to movie. I proposed that these patterns were directly the result of movie directors’ unique style of shooting scenes. In the first phase, I built eye tracking glasses and a software to capture the motion of one eye of a person watching a movie scene and the looking position data. In the second phase, I built the visualization which draws the eye movement curves on the wall, and the wooden controller lets users to explore these patterns by browsing different movies using an intuitive interface.

Our focus while watching movies shifts continuously from corner to side, actor to actress, dancer to assassin, making our eyes move continuously on the screen from left to right, top to corner, etc.. What we are not aware is that the smooth/complex patterns of these focus shifts of our eyes are different in each scene/movie and posses the signature and style of the director.

“Eyes on You, Tarantino!” tries to capture and visualize these patterns by projecting the patterns on a column, and gives users the opportunity to explore the movies in reference to these patterns using a physical controller surrounding the column.

Project includes two parts:

1. Tracking the eye movements of the audience with custom build eye capturing glasses, and calibration of the captured data.

At the first stage, I created the eye tracking glasses with a camera attached to one lens and recording one eye. Using Processing, I wrote a code to track the position of where the eye is looking, Using this glasses and the code, I recorded the eye movements of 4 people watching the different scenes from Death Proof (Quentin Tarantino), Rear Window (Alfred Hitchcock) and Hyperballad (music video from Bjork, Michel Gondry).

Early sketches of the installation

2. Creating a physical installation to present the captured patterns, in a meaningful data visualization which allows users to browse the movies in reference to these patterns.

At the second stage, I worked with data to see if I can observe any patterns that would be unique to the scenes and directors. The patterns I ended up focusing on:

• Directors’ shooting style defines how jittery or smooth our eyes follows the screen
• In the scenes where actors stand still, Tarantino prefers providing “eye action” moving camera continuously around the actor. Hitchcock prefers positioning actors on viewport separately to provide “eye action”. In Gondry’s example, he uses a unique technique to oscillate the camera that makes the eyes oscillate left and right.

After observing these various patterns, I designed a physical installation in which these patterns can be projected in accordance to the scenes being displayed. After a set of sketches and cardboard prototypes, I made the project ready to be exhibited at ITP’s Winter Show 2007.

“Shall you create something for the world’s largest high resolution video wall!” (120 x 12 feet, to be precise)

and we added:

“Shall there be storytelling! Shall there be live performance! Shall there be physical computing!”

IAC building in Manhattan consists worlds widest high-resolution screen, of which Robert Moon and I were lucky enough to get an opportunity to create ‘something’ for this massive screen, as part of Big Screens class at ITP, NYU. Although there were no limitations as of concept, we took this as our challenge to make sure we have storytelling, live performance and physical computing in the project! Then we produced

JohnTrain: an interactive cinematic performance

A housebreaker’s story as an assassin is reflected on this 120 x 12 feet square video wall, accompanied with the live performance of the housebreaker interacting with the screen simultaneously.

In the performance, a housebraker breaking into a house notices the train toy which makes him remember his last day working as an assassin. His playing with the train toy set interacts with the screen, on which the story of his last day is reflected on the screen. The housebraker tells his story to the audience in sync with the unfolding story being visualized on the screen.

In the design process, Robert Moon and I questioned how such a wide screen would be used at its best. The screen’s wideness make it impossible to see the all big picture at once, and made the audience to look either to the left or to the right, hence allowing them to focus only one side of the screen. We wanted to use this constraint as a complication of our project; we wanted users to follow multiple things happening on different sides of the screen.

Our solution was to display a train car with 7 windows, covering the screen from side to side. Audience could follow a hide’n'seek action inside the train through the windows. We wrote a story wrapping this feature: In the setting of 2050s, an assasin hired by the communist government trying to find and kill an activist girl in the train compartments, while the girl changes compartments for various reasons.

We shot the scenes for 7 compartments separately in front of blue screen, then edited them to create this 120 x 12 feet video. Using Processing, we developed the interaction between the toy train set used in performance and the train on the screen.

Exhibited at:

IAC Gallery, December 7th, 2007.

NYU Tisch Interactive Telecommunications Program Winter Show, December 16-17th, 2007, New York, NY.

Gameplay

Game play is very similar to Dance Dance Revolution: you are supposed to press on the corresponding shapes that you see on the screen with a good timing. However, in this game corresponding shapes are attached to a bra and to a boxer through touch sensitive sensors. The level increases as you accomplish catching a given number of shapes. As levels increase, game gets harder, faster but more intimate as new shapes are introduced to the game, which are on more intimate sections of the undergarments.

Design

In the design process, Jenny was already working on building these new controllers as her thesis project. Since we took the challenge of bringing a new type of experience to the game, I focused on creating the most simple game for users so that they can easily map the controller functionality of the undergarments to the actions in the game.

We got inspired by Dance Dance Revolution, since it was a fairly known game with a simple interface, where dancers tap on the circles on the platform that is corresponding to the shape they see on the screen. We applied the same concept for this project, and used distinctive icons on the undergarment sensor areas that could be easily mapped to the icons they see on the screen.

You can have more information on the project at JennyLC.com.

exhibited at:

SIGGRAPH Unraveled07,
Aug 2007, San Diego, CA

Around the Coyote, Jul 2007, Chicago, IL

NYU Tisch Interactive Telecommunications Program Spring Show,
May 2007, New York, NY

NYU Tisch Interactive Telecommunications Program Thesis Event,
May 2007, New York, NY