in|situ| Lab - Log In

MIDWAY Inria and FRQNT International Associate Team

MIDWAY (Musical Interaction Design Workbench And technologY) is an Inria international associate team created in 2014 from a collaboration between in|situ| and McGill University/CIRMMT.
funded by the Inria International Associate Teams Program, the Fonds de recherche du Québec – Nature et technologies and the France Canada Research Fund

Coordinators: Stéphane Huot (mjolnir, Inria Lille, formerly @in|situ|, Inria Saclay) and Marcelo M. Wanderley (McGill University/CIRMMT).

Participants:

in|situ|, now exsitu: Wendy Mackay, Michel Beaudouin-Lafon, Theophanis Tsandilas, Joseph Malloch, Jérémie Garcia.
mjolnir: Justin Mathew, Fanny Chevalier.
CIRMMT/IDMIL: Michael McGuffin, Jeronimo Barbosa, Johnty Wang,Philippe Leroux, Ivan Franco, Ian Hattwick.

About MIDWAY

The MIDWAY associated team involves three partners: the Mjolnir group at Inria Lille, the exsitu (forme in|situ|) group at Inria Saclay – Ile-de-France, and the Input Devices and Music Interaction Laboratory (IDMIL) from the Centre for Interdisciplinary Research in Music Media and Technology (CIRMMT) – McGill University. Our goal is to design and implement MIDWAY, a musical interaction design workbench, to facilitate the exploration and design of new interactive technologies for both musical creation and performance.

Each laboratory has extensive experience developing new interactive technologies and studying interactive phenomena from complementary points of view. The two groups share multiple, complementary research interests that the MIDWAY joint team will help them to explore together: In|Situ|’s experience working with composers to develop novel tools, toolkits and interaction models will complement IDMIL/CIRMMT’s knowledge and experience designing new musical instruments and their studies of the musical creative process.

News

  • February 2016 - Marcelo has been awarded an Inria International Chair within Mjolnir @Inria Lille for the period 2016-2020.
  • July 2015 - Fanny Chevalier (Mjolnir, Inria Lille) joined the MIDWAY project to collaborate on the visualization of mapping strategies.
  • May 2015 - Stéphane is now a collaborator member of CIRMMT.
  • May 2015 - MITACS Globalink - Inria Research Award for Justin Mathew.
    Justin has been awarded a MITACS Globalink - Inria research award (an internship funding). From June to August 2015, he will be in Montréal were he will work with Catherine Guastavino and Marcelo on 3D audio perception for spatialisation interfaces.
  • March 2015 - Joint publication: What Does 'Evaluation' Mean for the NIME Community? (see (1)).
    Jeronimo, Joe, Marcelo and Stéphane study on the evaluation of DMI in the NIME community has been accepted for publiocation as a full paper at NIME 15. It will be presented at the conference by Jeronimo in June 2015 (talk & poster).
  • October 2014 - Stéphane Huot and Justin Mathew have moved to Inria Lille (MJOLNIR group).
    Stéphane got promoted Research Director at Inria Lille from where he will continue to coordinate the project with Marcelo. Justin will continue is PhD at Inria Lille and will remain involved in MIDWAY.
  • June 2014 - Jérémie Garcia defended is PhD thesis and will now join IRCAM as a postdoctoral researcher.
    Jérémie will continue to collaborate on MIDWAY (and applying for postdoctoral funding to join Marcelo's group in Montréal).
  • March 2014 - Joseph Malloch joined in|situ| for a 18 months postdoctoral position.
    Joe's postdoc is funded by Digiteo/Digicosme to work on Digipods, a project related to the Digiscope Equipex project. He will also be involved in MIDWAY.
  • End of 2013 - Jeronimo Barbosa and Johnty Wang joined IDMIL as PhD candidates, working on the design and evaluation of musical instruments.

Visits

2016, June 21 - 23, Stéphane @Montréal

  • Baptiste, Marcelo & Stéphane: defining a roadmap for MIDAY’s last project ( machine learning tools in the MIDWAY framework for interaction design).
  • Marcelo & Stéphane: Planning for MIDWAY's closing workshop and follow-up (Marcelo's Inria International Chair).

2016, April 26 - 28, Marcelo @Lille

  • Jeronimo, Marcelo & Stéphane: design of a user study for evaluating the learnability of DMIs.
  • Marcelo & Stéphane: Planning for Marcelo's International Chair and MIDWAY's follow-up.

2016, February 22 - 24, Stéphane & Fanny @Montréal

  • Johnty, Joe, Marcelo, Stéphane & Fanny: continuing work on visualization of mappings.
  • Filipe, Marcelo & Stéphane: continuing work on the design of a UI for the DMIs hardware building blocks developed by Filipe.

2016, January 25 - February 5, Filipe Calegario @Lille

  • Filipe, Marcelo & Stéphane: work on the design of a UI for the DMIs hardware building blocks developed by Filipe.

2015, November 23 - December 11, Jeronimo & Marcelo (Dec. 7-9) @Lille

  • Jeronimo, Marcelo & Stéphane: design of a user study for evaluating the learnability of DMIs.

2015, November 4 - 7, Stéphane, Wendy & Michel @Montréal

  • Jeronimo, Marcelo & Stéphane: continuing work on the design and evaluation of musical instruments, design of a user study.
  • Johnty, Joe, Marcelo, Stéphane & Fanny: continuing work on visualization of mappings.
  • Marcelo, Wendy, Michel & Stéphane: continuing work on models for the MIDWAY workbench/framework.

2015, October 12 - 15, Marcelo & Michel @Lille

  • Marcelo & Stéphane: MIDWAY report for 2nd year and research agenda for 3rd year.
  • Johnty, Joe, Marcelo, Stéphane & Fanny: continuing work on visualization of mappings.
  • Marcelo, Michel & Stéphane: continuing work on models for the MIDWAY workbench/framework.
  • Marcelo & Michel: jury members for Jonathan Aceituino PhD's defense.

2015, July, Johnty @Paris

  • Johnty, Joe & Stéphane: started working on visualization of input mappings (history, versioning, alternatives). Involved Fanny Chevalier, Inria researcher in the Mjolnir group and expert in Information Visualization.

2015, May 12 - 15, Stéphane @Montréal

  • Marcelo and Stéphane: follow up on previous workshop and work on MIDWAY research agenda
  • Marcelo and Stéphane and Jeronimo: continued working on our recent work on evaluation of DMI (see (1))

2015, March 23 - 24, Marcelo, Jeronimo and Johnty @Paris

2015, January 12 - 13, Marcelo @Lille

  • Marcelo and Stéphane: definition of Orsay Workshop agenda - March 23 - 24

2014, December 6 - 12, Stéphane, Joe and Justin @Montréal

Planned:

  • Stéphane, Joe, Marcelo, Jeronimo and Johnty: work on the design and evaluation of musical instruments
  • Joe, Justin, Stéphane and Marcelo: prototyping sessions and workshops with people from IDMIL/CIRMMT (researchers and composer) on tools for rapid and robust prototyping of musical instruments (with both software and hardware components).
  • Joe, Stéphane, Marcelo and Michael: investigate new visualizations/visual langages for complex interaction mappings with libmapper and ICon
  • Justin, Stéphane and Marcelo: plan for joint actions and exchanges in 2015 related to Justin's PhD thesis (tools for 3D audio spatialization) and MIDWAY.

2014, December 1 - 4, Marcelo @Paris and @Lille

  • Marcelo and Stéphane: coordination of the project after the 1st year and planning for the 2nd year
  • Marcelo, Stéphane, Wendy, Michel and Theophanis: prepare the next MIDWAY workshop in Paris (early 2015)

2014, April 23 - 25, Stéphane and Theophanis @Montréal

  • workshop on software tools for interaction and input mapping (ICon and libmapper) with:
    Marcelo, Michael, Stéphane and Theophanis in Montréal
    Wendy, Michel and Joseph in Paris (videoconference)
  • work sessions with Marcelo
  • meetings with IDMIL members

2013, November 4 - 9, Wendy, Michel and Jérémie @Montréal

  • workshop on "designing interaction" (see http://www.cirmmt.org/activities/workshops/research/designing_interaction):
    Marcelo, Michel, Wendy, Jérémie, Philippe and Joseph in Montréal
    Stéphane and Theophanis in Paris (videoconference)
  • Jérémie worked with Philippe on the design of a software library to interpret data from Anoto digital pen in the OpenMusic environment (see (2)).

Results

Models and engineering for creative platforms: The ICon + libmapper Tools

We have created a bridge between our two main tools for interaction design: ICon (Saclay) and libmapper (McGill). We have developed a plugin in ICon that allows to connect with libmapper distributed architecture, to compose logical interaction devices from multiple distributed input sources in real-time, and to use them as processing devices in the ICon visual data-flow editor. This plugin will be included in the future releases of both ICon and libmapper by the end of the year 2015 and will be intensively used in our future prototyping sessions.

Reflective tools for musical creative processes: The visualization of mappings

Mapping, or the connection between output variables from an input device and the inputs of an algorithm to generate sounds (synthesizer) in a DMI, is an essential part of novel instruments, as it defines the very essence of a DMI, i.e. how performer actions generate/modify sounds. Although several tools have been developed in recent years to facilitate the design of mapping strategies for DMIs, most of these do not provide clear ways to visualize the mapping strategies developed. We are currently implementing tools in our ICon/libmapper platform to track mappings over time, as well as providing novel visualization schemes to understand the various connections between variables, based on previous developments in Information Visualization at the Mjolnir team. Such developments will allow for more effective retrieval of existing mappings, as well as clearer strategies to chose which mapping characteristics to act upon.

Interactive tools for creativity: Evaluation of musical instruments

In order to develop interactive tools for creativity, we have explored the evaluation of existing tools as a first step into defining the strengths and weaknesses of current designs. Indeed, to our knowledge, there are no general design and evaluation methodologies available to guide the development of novel DMI/NIME, though such methodologies could be very useful at least to help designers conceive and evaluate DMI prototypes. One reason is the large diversity of design and evaluation contexts possible in musical interaction, i.e. when one evaluates a novel DMI, is this evaluation done from the perspective of the DMI designer/manufacturer, the musician playing it, or the audience watching it be performed? Before aiming at finding universal or one-fits-all methodologies, one needs to clearly define the contexts of use of novel DMIs, as well as the perspective from which they are to be evaluated. The work in the second year of MIDWAY has led to a joint paper between the partners (Mjolnir and CIRMMT) published at the 2015 International Conference on New Interfaces for Musical Expression (NIME2015), held in Louisiana, USA, in May 2015 (see (1)). This work allowed MIDWAY researchers to bridge between the fields of music technology/NIME and human-computer interaction (HCI) and meta-evaluate the last 3 years of the NIME proceedings in order to better understand the use of evaluation methodologies in this field.

During his 3-month stay in Montréal in the summer 2015, Justin has conducted an experiment on the perception of 3D trajectories with audio and/or visual feedback in collaboration with Catherine Guastavino. This preliminary work has revealed some discrepancies in the perception of trajectories given the feedback condition, which we have to consider for the design of new interaction techniques for 3D audio mixing. At a longer term, we will investigate how such perceptual knowledge can be included in our interactive tools for mapping (e.g. the system could be able to help the designer or to notify her of potential perceptual issues for a given mapping).

Workshops

2015, March 23 - 24, workshop on "the MIDWAY framework" in Paris

2014, April 23 - 25, workshop on "software tools for interaction and input mapping" in Montréal

2013, November 4 - 9, workshop on "designing interaction" in Montréal

Publications

Joint publications related to the project or involving authors from both teams.

  1. Jeronimo Barbosa, Joseph Malloch, Marcelo Wanderley and Stéphane Huot (2015) What Does 'Evaluation' Mean for the NIME Community? In NIME '15: Proceedings of the 15th International Conference on New Interfaces for Musical Expression. 6p, to appear. (google)
  2. Jérémie Garcia, Philippe Leroux and Jean Bression (2014) pOM: Linking Pen Gestures to Computer-Aided Composition Processes. In Proceedings of the International Computer Music Conference ICMC-SMC '14. Pages 384-390. (google)

Publications of the partners related to the project.

  1. Jérémie Garcia, Theophanis Tsandilas, Carlos Agon Carlos and Wendy E. Mackay (2014) PaperComposer: Creating Interactive Paper Interfaces for Music Composition. In IHM '14: Proceedings of the 26ème Conférence Francophone sur l'Interaction Homme-Machine. ACM, pages 1-8. (PDF) (google)
  2. Justin Mathew, Stéphane Huot and Alan Blum (2014) A Morphological Analysis of Audio Objects and their Control Methods for 3D Audio. In NIME '14: Proceedings of the 14th International Conference on New Interfaces for Musical Expression. Goldsmiths University of London, pages 415-420. (google)
  3. Joseph Malloch, Stephen Sinclair and Marcelo M. Wanderley (2014) Distributed Tools for Interactive Design of Heterogeneous Signal Networks. In Multimedia Tools and Applications, Springer, February 2014.

Events

Events related to the project or involving participants from both teams.

Quid Sit Musicus ?

MIDWAY's participants: Philippe Leroux, Jérémie Garcia.
After Voi(rex) and Apocalypsis, Quid sit musicus ? Leroux guides computer music towards the wake of Guillaume de Machaut, extracting the gestural essence from a 14th century manuscript, the perception of the ancient lines of calligraphy becomes the base of a sound production and the composition of a new work. Leroux uses the new augmented paper technology that, via an optical system and a Bluetooth pen, makes it possible to "recuperate"” data from Machaut’s score sung during the concert.

web page: http://manifeste.ircam.fr/events/event/philippe-leroux-i-2/?lang=en
video: http://medias.ircam.fr/x93c854