Touchless Interfaces: A Digital Reality

First: Some Science

Information is an interesting thing. It is the glue that holds our reality together. Right now your mind is interpreting the flow of information that my mind has manifested as this article. The brain to brain link that is active was likely unrealized until just now. That’s reality for ya. Ignore the man behind the curtain.

In the book The Grand Design, Stephen Hawking and Leonard Mlodinow lay out the concept of Model-dependent realism. It defines the way in which the human brain receives, interprets, and processes sensory information. “Our perception-and hence the observation upon which our theories are based - is not direct, but rather is shaped by a kind of lens, the interpretive structure of our human brains.” The purpose of model-dependent realism is to overcome needing to know the meaning of our existence. We store experiences in memory and compare what we observe with what we know. Our interpretation is added to our model and our life goes on.

Now: Marry that Science to Advertising

Advertising’s aim is to integrate an alternate reality into the memory of the observer. It’s the craft of designing a realistic false reality to trigger an intended behavior in the user down the road. The internet has become a valuable tool for marketers to get their message in front of users, but the message is deformed by the user knowing they are looking at a screen.

The screen may not be the largest display medium available, but it’s analytical capabilities are vast. We can capture time on page, how often links are clicked, and where people came from. A successful campaign is largely based on how many users makes their way through the funnel, reach the end, and click on the call to action, or like the page. Metrics can establish the success or failure of a given campaign given the route a user takes.

But how do you measure the success of out-of-home advertising?

In the lab we work on side projects that we think could impact the future of advertising and push the bounds of what it means to work in this industry. We get the opportunity to think outside the box and work toward a more technologically rich future.

We wanted to build a silent interface between information and a participant’s observation of it. That means no mouse and keyboard, voice commands, or any direct knowledge by the user that a computer is involved. Its purpose should be to process body language, user focus, hesitation, and frustration in order to analyze the participant’s engagement with the information presented.

Now we create.

So that’s what we did. Using the Kinect we created a system for tracking an area that is cloned into a virtual replica of itself. We capture and analyse the skeletal position of the users within the area and detect activities that match our predefined list of potential interactions. By evaluating the surface topography around the skeletons, we determine several things such as body type, hips direction, shoulder direction, where the head is looking, and where the user is pointing. This goes into a formula that determines the level of participation and focus given by the user.

Our system also monitors a user’s joint angles over time in order to graph their body language over the course of their interaction. We can then tell you about the trends that have developed over the course of the campaign.

With this system we have created a new type of ad platform for Out of Home advertising. A previously thought “traditional” advertising unit is now taking on a digital identity with clearly defined metrics.