eXtended Reality (XR) based Aircraft Maintenance System with Cognitive Load Estimation Feature, Aeronautical Development Agency (INR 90.91L, 2024-2026)
eXtended Reality CoBot controller for Manufacturing Processes, DST, Govt of India (INR 50L, 2023-25)
AI - AR Integration for Manufacturing Assembly Process, Collins Aerospace Systems, USA (INR 70.8L, 2022-2023)
Holoportation and UI/UX Design for Metaverse, Siemens, Germany (INR 76.8L, 2022-2027)
In the context of Industry 4.0, we are working on mixed reality based component assembly, LLM controlled rovers for asset tracking and VR based digital twins. We developed novel synthetic data generation technique for machine component assembly and deployed solutions at Collins Aerospace and Aeronautical Development Agency. We published on novel 3D reconstruction and tactile reconstruction techniques for VR based digital twin development. Our project on VR Digital Twin in collaboration with British Telecom was finalist of UK AI for Innovation Award in 2024 .
Earlier, we developed an interactive sensor dashboard and multimodal CoBot for supporting Industry 4.0 initiative. We investigated different visualization techniques of sensor reading in 2D and 3D environment. We also developed image processing algorithms for automated inspection of rubber sheet for a shoe making MSME and Printed Circuit Boards (PCB) for detecting correct orientation and type of IC chips using webcam and shape detection algorithms. We have developed a prototype system using a Kinect that can detect bad leaning posture by measuring the difference between the neck-spine and hip-spine joints in the skeleton figure of Kinect. Following RULA guidelines, it raises an alert when the neck or torso is inclined more than 60 degrees.
Main Results
We developed and validated a new way of generating synthetic image using a bespoke Diffusion Model and video based data for training deep learning models like Convolutional Neural Networks using VR digital twins which offers more customization than Generative Adversial Network (GAN) and Variable Auto Encoders (VAE).
Using a 3D reconstructed Gaussian prior enables more accurate and truly model-free pose estimation. While Gen6D is also model-free, it relies on point cloud priors, whereas Gaussian splats offer more realistic rendering and better alignment with real-world observations. Unlike approaches like Pos3r, which require CAD models, this method remains model-free and leverages the expressive power of 3D Gaussians for robust pose estimation.
Monocular camera-based 3D reconstruction performs well in capturing relative depth, but it often lacks alignment with real-world scale, limiting its applicability in robotics and AR/VR contexts. To address this, we propose a LiDAR-camera fusion approach for 3D Gaussian mapping. Our method combines the strengths of monocular depth estimation—efficient at capturing relative depth—with LiDAR data, which provides accurate real-world scale. By using LiDAR to calibrate and scale the estimated depth, we generate an accurately scaled 3D Gaussian map (attached image). Additionally, utilizing LiDAR for robot localization reduces the computational burden of reconstruction, enabling higher frame rates.
We evaluated shape detection algorithms in different lighting conditions (indoor, outdoor, and controlled light source) and best shape detection result is obtained in YCbCr color space using bounding box shape descriptors for 2500 Lux using LED.
Publications
AK Sandula, S Damodaran, S Nagaraj, D Ghose, P Biswas, 3D Reconstruction via Camera-Lidar (2D) Fusion for Mobile Robots: A Gaussian Splatting Approach, IEEE International Conference on Robotics and Automation (ICRA 25)
Y Sinha, S Bhattacharya, YK Sahu, P Biswas, Blind Tactile Exploration for Surface Reconstruction, IEEE International Conference on Robotics and Automation (ICRA 25)
S. Raj, B. Karmakar, G. Kumar, A. Mukhopadhyay, R. Chandrahas, P. Biswas (2025) Comparing computer vision models for low resource dataset to develop a mixed reality based manual assembly assistant. Discovery Robotics 1(5), Springer
S Raj, N Beri, D Patel, Y Sinha, A Chakrabarti, P Biswas, HaM3D: Generalized XR-Based Multimodal HRI Framework with Haptic Feedback for Industry 4.0, Journal of Multimodal User Interface 2024
A. Mukhopadhyay, GS Rajshekar Reddy, S. Ghosh, P. Biswas, Validating Social Distancing through Deep Learning and VR-Based Digital Twins, ACM VRST 2021
A. Mukhopadhyay, GS Rajshekar Reddy, KPS Saluja, S. Ghosh, A. Peña-Rios, G. K. Gopal, P. Biswas, A Virtual Reality-Based Digital Twin of Office Spaces with Social Distance Measurement Feature, Virtual Reality & Intelligent Hardware, Elsevier
A. Mukhopadhay, I. Mukherjee and P. Biswas, Comparing Shape Descriptor Methods for Different Colour Space and Lighting Conditions, Artificial Intelligence in Engineering Design and Manufacturing, Cambridge University Press 33 (4) 2019
S. Arjun, GS Rajshekar, A. Mukhopadhyay, S. Vinod and P. Biswas, Evaluating Visual Variables in a Virtual Reality Environment, British HCI 2021
S. Arjun, LRD Murthy, P. Biswas, Interactive Sensor Dashboard for Smart Manufacturing, International Conference on Industry 4.0 and Smart Manufacturing, Procedia Computer Science, Elsevier 2022
V. K. Sharma and P. Biswas, System for Operating Joystick Indian Patent Application no. 201941044740, PCT International Application No. PCT/IB2020/059959
A. Mukhopadhyay, GS Rajshekar Reddy, I. Mukherjee, G. K. Gopal, A. Peña-Rios, P. Biswas, Generating Synthetic Data for Deep Learning using VR Digital Twin, 3rd International Conference on Virtual Reality and Image Processing (VRIP 2021)
P. Biswas, S. Roy, G. Prabhakar, J. Rajesh, S. Arjun, M. Arora, B. Gurumoorthy and A. Chakrabarti, Interactive Sensor Visualization for Smart Manufacturing System, Proceedings of the 31st British Human Computer Interaction Conference 2017 (British HCI 17)
A. Mukhopadhyay, LRD Murthy, M. Arora, A. Chakrabarti, I. Mukherjee and P. Biswas, PCB Inspection in the context of Smart Manufacturing, International Conference on Research into Design (ICoRD 2019)
LRD Murthy, Somnath Arjun, Kamalpreet Singh Saluja, Pradipta Biswas, Smart Sensor Dashboard, 7th International Conference on PLMSS (Product Life Cycle Modelling, Simulation and Synthesis) 2019, https://arxiv.org/abs/2005.05025
K. Puneeth, A. Sahay, G, Shreya, P. Biswas, M. Arora, A. Chakrabarti, Designing an Affordable System for Early Defect Detection Using Image Processing, Advances in Manufacturing Technology XXXIII: Proceedings of the 17th International Conference on Manufacturing Research, incorporating the 34th National Conference on Manufacturing Research, 2019
