Our focus while watching movies shifts continuously from side to side, actor to actress, dancer to assassin, making our eye move continuously on the screen from left to right, top to corner, etc.. Sometimes fast, sometimes very smooth.. What we are not aware is that the smooth/complex patterns of these focus shifts of our eyes are created and defined by the directors and holds the signature and style of the director.
In this project, these patterns are observed by tracking the eye movements of spectators with the custom built eye tracking glasses, which has a camera attached to one of the lenses. Eye movements are recorded while prospectors are watching the scenes, then this recording is processed generating the eye movement data in xml format. This data is plotted on a structural column via projection, and user is invited to explore these patterns with the kiosk style controller.
This project gives user a different opportunity to browse the movies back and forth in reference to these eye movement patterns. Movies themselves are displayed on a horizontal surface attached to the column, allowing users to browse back and forth in the movie. User is allowed to browse through the movie or through the eye movement patterns on which they can refer the past/future of the movements and attached dialogs. The shape emerging on the projection focuses on the latest portion of the pattern but keeps the past visible, allowing users to see the big picture and compare the pattern to the other movies' patterns.
"Eyes on You, Tarantino!" is exhibited at ITP Winter Show 2007.
The Emerging Patterns
EYT includes three different scenes, specifically chosen for their interesting/original shooting technique. As a result, these three scenes generate very unique eye movement patterns.
Michel Gondry - Bjork "Hyperballad" Music Video has an amazing visual in which the camera tilting and panning around the head of Bjork while not moving the focus. This motion generates a sinus wave in the spectators eyes while he/she is focused on Bjork singing.
Alfred Hitchcock - Rear Window presents a scene of a dialog between Cary Grant and Grace Kelly. Hitchcock's use of static camera keeps the eye focus on specific point on the screen, -either left or right- according to the person talking. This creates a 'square wave' like pattern.
Quentin Tarantino - Death Proof includes a scene where camera is following a girl's telling a story while it is turning around the diner table in a single shot. The subject (the girl) doesn't move, camera turns rounds around the table, but the position of the girl on the screen moves from right to left to right very slowly.
Michel Gondry - Bjork "Hyperballad" Music Video Alfred Hitchcock - Rear Window Quentin Tarantino - Death Proof
Eye Tracking
In the eye tracking part of the project, I built simple eye tracking glasses. Simply, woodworker's glasses minus lenses plus a small camera attached to the frame. The camera attached provides high resolution close-up recording of the eye while the spectator is watching the movie.
Calibration
Calibration phase includes two sections: First, the calibration of the system to the eye. In this phase, the position of the eye pupil in the recorded eye video (live stream if it is live tracking) is specified, and the scale of the eye movement is matched on the positions on the screen.
Recording
In real-time eye tracking, first the live camera input is calibrated for the eye, then eye movement data is recorded while the subject is watching the movie.
In pre-recorded eye tracking, the eye is recorded while the person is watching the movie. Then the recorded eye video is put into calibration.
Data in XML
The calibrated video is put into an algorithm, which generates the pixel position of where the subject is looking on the screen. This data is exported as xml file, which is later used in the visualization part of the project.
Source Code
Java and Processing are used to create the eye tracking and visualization. To display videos, Dano's VXP Library is used. All the code below are licensed under Creative Commons Attribution-ShareAlike 2.0.
eye tracking application: eyeCapture3XML.pde
sample eye data: sinanRW.xml
visualization: eytSourceCode.zip
progress
This is a summary of the whole progress documented at my blog, itpedition.
September 2007 - single shot scene, mr. tarantino, inspirations.. I was watching Death Proof, latest film from Tarantino, I was amazed by a scene in which a girl is telling a story to her friends at a diner. In this one shot scene, the camera positions her at the right of the screen, then following her tellings, it moves her to the left and back, in a very slow motion. This was not a very spectacular thing in terms of directing, but it made me realize my slow eye movements while I was watching her. I started wondering about the differences the might emerge out of different scenes.
With the heat of the moment, I sat down, digged in to my undergrad senior project codes, to write a software that can track where I am looking at the screen. I hookep up my webcam to my hat, and built the software that accomplishes the goal in the horizontal axis. Simply put, I calibrate the eye position and eye movement->display scale, and the application looks for the average position of the black pixels (pupil) with respect to white area. I decided to focus on this project in Computers for the Rest of You class at ITP.
The next day - Lucas, the eye tracking master! I asked Lucas Longo to use his equipment to see how well my eye tracking code would perform. He was already working on eye tracking, and he had a great cam-attached-to-glasses equipment, firewire, high res, etc.. I was amazed with the tests I made at his place, and having been there, I recorded Caleb Clark and Lucas Longo's eye movements in a dialog scene from Rear Window, Hitchcock. In this scene, the camera has two positions: one shooting the guy (James Stewart), one shooting the woman (Grace Kelly, my oh my!). Having visualizing the result in my mind, I needed to built a nice way to show these eye movement patterns.
The same day, I built another application that processes the eye recording and calculates where the eye is looking. At the same time, the movie and the point presenting the point the eye is looking is displayed in the back. This way I was able to observe where we have been looking on the screen while watching the scene.
October 2007 - The First Visualization: Looking through some interesting visualization pieces, I was inspired to built something starwarsy, 3Dsy, flyingsy. In this visualization, I wanted to give both the future and the past information of the eye movements. Then came the third application filed under this project, (file: eyeCapture3Visualize.pde). Also, I decided to work on this project for my other class Spatial Design, in which I built a 3D wall structure in Maya using these eye tracking data; I put two different person's eye tracking data on top of each other, and lofted in between. It became and interesting structure, in which the difference of two person's watching movie (one is more focused on the actors, the other is more exploring the whole scene) could be observed.
October - November 2007 - How to visualize? Then, slowly I found myself stuck in answering to question "How to visualize my point?"; Everyone I was talking to about my project were interested in learning who was looking where (which is, I think, a very cheesy analysis in eye tracking issue) (men look at the breasts, oh reaally?), but my point was different. I wanted emphasize that these eye movements were some kind of art-unobserved, which originates the signature of the director, and differents patterns would be observed among directors and their movies. Dan O'Sullivan made a good explanation for my search; I was starting with the movie, then focusing on human watching the movie, then turning back to the movie itself.
I tried to explore couple of more visualizations, as I was getting more inspired with the physical visualization examples of data (especially, I liked these pieces). Trying to think more physical (which is, I guess, my obsession), I created a visualization considering the top of lockers area at ITP. The idea was to print frames and shape a piece of wire as the eye movements. (note to self: explain more here).
November 2007 - I gotcha! Not being satisfied with the last piece, I grabbed a projector, and tried to see what I could do with the regular plotting of the eye movements, then which I created the visualization below.