Workshop on Intelligent User Interface for Metaverse

At Co located with ACM IUI 24ACM International Conference on Intelligent User Interface 2024

Venue: Greenville, South Carolina, USA

Date: 18th March, 2024

Check out Aria Research Kit Demo and XR Demonstrations

Link for Online Participants


    Investigating UI/UX challenges in Metaverse

    Role of AI for improving user experience and scopes of generative AI in Metaverse

    Scope and challenges for new modalities like gesture and eye tracking in Metaverse

    Challenges in improving accessibility aspects of Metaverse for people with different range of abilities

    Identifying gaps in standardization for Metaverse

Call for Papers

  • Please click here to submit paper
  • Submission Deadline: Jan 16, 2024
  • Notification of Acceptance: Feb 9, 2024
  • Camera-Ready Papers Due: Mar 1, 2024
  • While Metaverse may be a new popular concept but researchers already explored interaction design for eXtended Reality systems. Use of AI in the form of object recognition in Augmented Reality system or speech recognition in Virtual Reality systems in widespread use but not explicitly considered as examples of Intelligent User Interface. However, AI technology has potential to introduce new interaction modality like eye and hand tracking for Metaverse and similarly improve interaction experience through emotion recognition, cognitive load estimation, target prediction or introducing new accessibility features in 3D media. Recent advancement in generative AI has been explored for generating synthetic data to improve object recognition accuracy in mixed reality system and Google RT2 explored using large language model to automatically generate instructions for robot manipulation, which can be extended to industrial metaverse. The organizers of this workshop have diverse range of expertise from developing VR crew cabin for Indias maiden human space flight mission, developing subtitles for VR media, designing industrial metaverse and marketing new headsets for data collection in Metaverse. In addition, the team is also actively involved in standardization activities through ISO and ITU and the workshop will directly contribute to assemble future use cases and associated challenges in Metaverse.

    Workshop Schedule

      Afternoon Session

    • EDT 1400 / CET 1900 / GMT 1800, Paper Presentations
      1. Towards and accessible metaverse: user requirements Anna Matamala (Universitat Autonoma de Barcelona)*; Estella Oncins (Universitat Autonoma de Barcelona)

        The Metaverse and the Future of Education, Anasol Pena-Rios (British Telecom, UK)

        Learning Human-to-Robot Handovers from Demonstrations Mukund Mitra (Indian Institute of Science, Bengaluru)*; Partha P Chakrabarti (Indian Institute of Technology Kharagpur); Pradipta Biswas (Indian Institute of Science)

      EDT 1500, Aria Research Kit Demo

      Project Aria glasses utilize groundbreaking technology to help researchers gather information from the users perspective, contributing to the advancement of egocentric research in machine perception and augmented reality. The sensors on the Project Aria glasses capture the wearers video and audio, as well as their eye tracking and location information. Example use cases include:
        Wayfinding for indoor and outdoor spaces ? See CMU Case Study
        Understanding daily life from a first-person POV ? Learn more about Ego4D
        Researching AR in-motion and in-vehicle use cases ? See BMW Case Study
        AI assistant with egocentric context from physical life
    • EDT 1700, Closing Remarks - Pradipta Biswas, IISc

    Workshop Pictures

    Video Demonstrations


      Prof Pradipta Biswas*, Indian Institute of Science,

      Dr Vinay Krishna Sharma, Siemens,

      Prof Pilar Orero, Universitat Autonoma De Barcelona,

      Dr Eryn Whitworth, Meta,

      Dr Anasol Pena-Rios, British Telecom,


    • S. Saren, A. Mukhopadhyay, D. Ghose, P. Biswas, Comparing alternative modalities in the context of multimodal human–robot interaction, Journal on Multimodal User Interfaces 2023, Springer
    • S. Raj, LRD Murthy, T Adhithiyan, A. Chakrabarti, P. Biswas, Mixed Reality and Deep Learning based System for Assisting Assembly Process, Journal on Multimodal User Interfaces 2023, Springer
    • AMC Rao, S. Raj, AK Shah, Harshitha BR, NR Talwar, VK Sharma, Sanjana M, H Vishvakarma, Development and comparison studies of XR interfaces for path definition in remote welding scenarios, Multimedia Tools and Applications 2023, Springer
    • A Hebbar, AK Shah, S Vinod, A Pashilkar, P Biswas (2023). Cognitive load estimation in VR flight simulator. Journal of Eye Movement Research, 15(3)
    • M Mitra, P Pati, VK Sharma, S Raj, PP Chakrabarti, P Biswas, Comparison of Target Prediction in VR and MR using Inverse Reinforcement Learning, ACM International Conference on Intelligent User Interfaces (IUI 23)
    • A Mukhopadhyay, VK Sharma, PG Tatyarao, AK Shah, AMC Rao, PR Subin, P Biswas, A comparison study between XR interfaces for driver assistance in take over request, Transportation Engineering (11),2023,100159, Elsevier, ISSN 2666-691X,
    • S Arjun, A Hebbar, S Vinod and P Biswas, VR Cognitive Load Dashboard for Flight Simulator, ACM Symposium on Eye Tracking Research and Applications (ETRA) 2022
    • A. Mukhopadhyay, GS Rajshekar Reddy, S. Ghosh, P. Biswas, Validating Social Distancing through Deep Learning and VR-Based Digital Twins, ACM VRST 2021
    • A. Mukhopadhyay, GS Rajshekar Reddy, KPS Saluja, S. Ghosh, A. PeƱa-Rios, G. K. Gopal, P. Biswas, A Virtual Reality-Based Digital Twin of Office Spaces with Social Distance Measurement Feature, Virtual Reality & Intelligent Hardware, Elsevier
    • S. Arjun, GS Rajshekar, A. Mukhopadhyay, S. Vinod and P. Biswas, Evaluating Visual Variables in a Virtual Reality Environment, British HCI 2021
    • A. Mukhopadhyay, GS Rajshekar Reddy, I. Mukherjee, G. K. Gopal, A. PeƱa-Rios, P. Biswas, Generating Synthetic Data for Deep Learning using VR Digital Twin, 3rd International Conference on Virtual Reality and Image Processing (VRIP 2021)