in|situ| Lab - Log In

all in|situ| jobs

Capturing and Retrieving Memories of Physical-World Interactions with Mobile Devices

Internship proposal at in|situ|

Advisor: StÚphane Huot (


Mobile devices are widely used as daily digital companions to keep personal notes and reminders. Current mobile note-taking applications allow users to edit simple textual data and sometimes augment them with audio and pictures (Figure 1-left). Other applications, including maps, navigation and photo geo-tagging systems, use mobile sensors, such as a camera, a GPS and accelerometers, to retrieve and display information about the surrounding environment with augmented reality (Figure 1-right). However, most of these applications only present knowledge and information about the environment but rarely personal information or notes.

Figure 1 : A standard mobile note taking application (left). The Layar augmented reality application (right).

Previous work has already demonstrated interest in location-based notes and reminders (3) and has proposed applications to augment traditional PIM systems with location awareness (1,2). In addition, commercial or free mobile applications like GPSNote allow textual note-taking about specific locations.


The goal of this project is to go further and explore how advanced mobile interactions could allow for taking personal notes about the physical environment as well as improving their contextual presentation (4,5). In particular, instead of adding simple "static textual notes" about an object or a location, we are interested in how augmented-reality techniques can help in capturing and retrieving memories about how to interact with the physical world: how to enter a PIN on a mechanical or a digital keypad lock, how to type a key sequence on a keyboard, how to record and retrieve the position of knobs on a console, etc.

As a first step, the student will familiarize with research on Human-Computer Interaction (HCI), focusing on mobile-devices interaction and mobile augmented reality (5,6). As a second step, the student will work on the design and implementation of augmented-reality interaction techniques for taking and presenting contextual notes about physical-world interaction. During this step, the student will mainly focus on interaction and visualization techniques, rather than technology and algorithms for mobile augmented reality (e.g., image processing, location detection, or real-time tracking). Consequently, due to potential technological limitations, the final implementation may be just a functional mockup. Finally, the student will learn about HCI evaluation methods and, ideally, will conduct an experiment to test the developed techniques with human subjects.

Skills of ideal candidate

Solid programming skills in Java (Android) and/or Objective-C (iOS), training in Human-Computer Interaction (design, implementation and evaluation), interested in pursuing research.


[1] Suresh Chande, Panu Vartiainen, and Kimmo Rńm÷. 2007. Active notes: context-sensitive notes for mobile devices. In Proceedings of the 4th international conference on mobile technology, applications, and systems and the 1st international symposium on Computer human interaction in mobile technology (Mobility '07). ACM, New York, NY, USA, 716-723.

[2] Anind K. Dey and Gregory D. Abowd. 2000. CybreMinder: A Context-Aware System for Supporting Reminders. In Proceedings of the 2nd international symposium on Handheld and Ubiquitous Computing (HUC '00), Peter J. Thomas and Hans-Werner Gellersen (Eds.). Springer-Verlag, London, UK, 172-186.

[3] Pamela J. Ludford, Dan Frankowski, Ken Reily, Kurt Wilms, and Loren Terveen. 2006. Because I carry my cell phone anyway: functional location-based reminder applications. In Proceedings of the SIGCHI conference on Human Factors in computing systems (CHI '06), Rebecca Grinter, Thomas Rodden, Paul Aoki, Ed Cutrell, Robin Jeffries, and Gary Olson (Eds.). ACM, New York, NY, USA, 889-898.

[4] Niels Henze, Gregor Broll, Enrico Rukzio, Michael Rohs, and Andreas Zimmermann. 2008. Mobile interaction with the real world. In Proceedings of the 10th international conference on Human computer interaction with mobile devices and services (MobileHCI '08). ACM, New York, NY, USA, 563-565.

[5] M. Rohs and B. Gfeller, Using Camera-Equipped Mobile Phones for Interacting with Real-World Objects, Advances in Pervasive Computing, 2004, pp. 265-271.