in|situ| Lab - Log In

Interactive Paper

Interaction Techniques

The knotty gestures is a simple yet powerful technique for interacting with paper. Knots are tiny circles that can be added to any gesture. Users can leave subtle marks that permit both immediate interaction in the flow of writing and create rich opportunities for future interaction. We have conducted two experiments to evaluate the design and recognition heuristics and demonstrated that people can successfully execute knotty gestures, even without feedback. Knotty gestures provide users with a subtle, in-the-flow-of-writing technique for tagging information and subsequently interacting with the paper. The current implementation of the technique has been based on Anoto technology and Livescribe pens.

Left: using knotty gestures to make recordings over notes. Right: nesting knotty gestures to perform calculations over tabular data

Our work on interaction techniques for paper has been based on years of experience studying diverse groups of users, such as biologists [2], using paper.


  1. Theophanis Tsandilas and Wendy Mackay (2010) Knotty Gestures: Subtle Traces to Support Interactive Use of Paper. In AVI '10: Proceedings of the Working Conference on Advanced Visual Interfaces. ACM, pages 147-154. (google)
  2. Wendy Mackay, Catherine Letondal, Guillaume Pothier, Kaare Břegh and Hans Sřrensen (2002) The Missing Link: Augmenting Biology Laboratory Notebooks. In Proc. ACM Symposium on User Interface Software and Technology (UIST 2002). ACM, pages 41-50. (google)

Supporting Creativity

We focus on the creative use of paper in the music composition process, particularly the interaction between paper and end-user programming. Working in close collaboration with composers and researchers at IRCAM, we have designed paper-based interfaces to help composers create their own, individualized composition languages and tools and to experiment with music, on-paper and on-line.

Musink: Handwritten gestures over paper music scores become programmable musical parameters


  1. Jérémie Garcia, Theophanis Tsandilas, Carlos Agon and Wendy Mackay (2012) Interactive Paper Substrates to support Musical Creation. In CHI '12: Proceedings of the 30th international conference on Human factors in computing systems. ACM, pages 1825-1828. (google)
  2. Louis Bigo, Jérémie Garcia, Antoine Spicher and Wendy Mackay (2012) Papertonnetz: Music Composition with Interactive Paper. In SMC '12: Proceedings of the 9th Sound and Music Computing Conference. Pages 219-225. (google)
  3. Jérémie Garcia, Theophanis Tsandilas, Wendy Mackay and Carlos Agon (2011) InkSplorer: Exploring Musical Ideas on Paper and Computer. In NIME '11: Proceedings of the 2011 conference on New interfaces for musical expression. Pages 361-366. (google)
  4. Theophanis Tsandilas, Catherine Letondal and Wendy Mackay (2009) Musink: Composing Music through Augmented Drawing. In CHI '09: Proceedings of the SIGCHI conference on Human factors in computing systems. ACM, pages 819-828. (Best Paper Award). (google)
  5. Catherine Letondal and Wendy Mackay (2007) The Paperoles Project: An analysis of paper use by music composer. In Proceedings of CoPADD 2007.. (google)
  6. Catherine Letondal, Wendy Mackay and Nicolas Donin (2007) Paperoles et musique. In Proceedings of IHM 2007. ACM, pages 167-174. (google)

Reflecting on Personal and Collaborative Activity

We studied the evolving work practices of biologists and the role of paper and electronic laboratory notebooks to support their individual and collaborative activity. In a participatory design approach, combined with longitudinal field we tested Prism, a hybrid lab notebook for biologists to capture, visualize and interact with streams of activity.

Integrating multiple streams of activity in Prism

Prism provides an interactive access to cross-linked activity streams: hand-written notes on paper, type-written notes on-line, and key points (web sites, emails, analysis results) from users' computer activity, as well as web based activity.We used Prism as an extensible technology probe that users could adapt to integrate additional streams and share information from other biologists. Our key findings include the notion of master documents, whether paper or electronic, and the importance of redundancy, which biologists use to make sense of their data. Prism provides a flexible, extensible tool that supports individual and collaborative reflection in creative work.


  1. Aurélien Tabard (2009) Tools for forgetting, from interaction traces to reflection. these, Université Paris-Sud, Orsay, France. (google)
  2. Aurélien Tabard, Wendy Mackay and Evelyn Eastmond (2008) From Individual to Collaborative: The Evolution of Prism, a Hybrid Laboratory Notebook. In Proceedings of ACM CSCW'08 Conference on Computer-Supported Cooperative Work. ACM, pages 569-578. (google)
  3. Shengqiong Yuan, Aurélien Tabard and Wendy Mackay (2008) Streamliner: A General-Purpose Interactive Course-Visualization Tool. In Proceedings of KAM'08, International Symposium on Knowledge Acquisition and Modeling. IEEE Press. (google)

Sustainable Design with Reusable Paper

We explored the need for sustainable design with paper: how people really print and how we can take advantage of novel, reusable paper technology. We conducted two studies to investigate user's printing behavior. A key finding of the first study was that users often need an intermediate state between the electronic and physical forms of their documents. The second study examined users' predictions of which types of documents required this intermediate state. We formulated these findings into design guidelines that take into account: examination phase, transitions, cognitive and emotional reasons, and task- and event-relevant documents.


  1. Julie Wagner and Wendy Mackay (2010) Exploring Sustainable Design with Reusable Paper. In CHI '10: Proceedings of the SIGCHI conference on Human factors in computing systems. ACM, pages 1871-1874. (google)