Introduction to Multimodal Interaction




Pradipta Biswas, PhD (Cantab)

Intelligent Inclusive Interaction Design Lab

Assistant Professor

Indian Institute of Science

Email: pradipta AT cpdm DOT iisc DOT ernet DOT in


·        Basic knowledge of Mathematics

·        C++ and preferably C# or VB.Net programming skill


·        Computer with Windows 7/8/10

·        MS Visual Studio 2010/2012/2015

·        MS Kinect SDK

·        Tobii EyeX SDK

·        LeapMotion SDK

·        Microsoft Speech SDK

What you get

·        Basic knowledge of multimodal interaction

·        Exposure to state-of-the-art eye gaze, hand, head and finger movement and EEG trackers

·        Developing new input modalities tracking eye gaze, hand, finger, head movement of users

·        Hands on experience on developing software using Tobii, Leap, Kinect and Speech SDKs

·        Multimodal fusion algorithms

·        Validating new input modality

·        Data analytics relevant to multimodal interaction


Course Material

1.     Introduction to Multimodal Interaction [Slides]

2.     A case study of developing multimodal middleware [Slides]

·        [Cambridge Simulator]

3.     Gesture recognition systems, hand and finger movement tracking, Affective computing [Slides]

·        [Sample software using LeapMotion]

·        [Sample software using Kinect]

4.     Multimodal interaction for limited mobility environment [Slides]

·        [Video demonstration]

5.     Multimodal fusion algorithms [Slides]

6.     Eye gaze tracking, iris recognition and head movement tracking [Slides]

·        [Sample software using Tobii EyeX]

·        [Sample software combining Tobii EyeX and LeapMotion]

7.     Multimodal interaction for automotive environment [Slides]

·        [Video demonstration]

8.     Image processing algorithm for face and eye detection [Slides]

9.     Experiment design and data analytics [Slides]

·        [Sample dataset and analysis]

Books for further reading


              B3_India                 9781447167075.jpg   


Instructor’s Bio: Pradipta Biswas is an Assistant Professor at the Centre for Product Design and Manufacturing of Indian Institute of Science. His research focuses on user modelling and multimodal human-machine interaction for aviation and automotive environments and for assistive technology. He set up and leads the Interaction Design Lab at CPDM, IISc. Earlier, he was a Senior Research Associate at Engineering Department, Research Fellow at Wolfson College and Research Associate at Trinity Hall of  University of Cambridge. He completed PhD in Computer Science at the Rainbow Group of University of Cambridge Computer Laboratory and Trinity College in 2010 and was awarded a Gates-Cambridge Scholarship in 2006. He undertook a master degree at the Indian Institute of Technology, Kharagpur. He conducted a course on Human Computer Interaction at Indian Institute of Technology, Mandi, guest lecture at Indian Institute of Technology, Madras and was a vice chairman of ITU-T Focus Group on Smart TV.


I3D Lab Demo Video