In 1886, two German engineers, Benz and Daimler started the first automobile company, Mercedes-Benz. Mr. Daimler worked on IC (Internal Combustion) engine and initially he tested it on a bicycle while Mr. Benz designed a vehicle that was patented in 1886. Daimler’s vehicle appeared few months later. During similar time in North America, George B. Selden patented his automobile engine and its application in car in 1895 while Henry Ford introduced the iconic Model T (Figure 1) in 1911. In 1926, two-third of running cars were model T. It may be noted that although the first automotive was patented in late 18th century, it is not until 1924, Kelly’s Motors installed the car radio, which can be considered as the first secondary task in addition to driving. In 1939, Packard introduced the first air-conditioned car. In 1960s, the tape music players were installed in cars while in 1980s, the CD (Compact Disk) player was introduced in cars. In 2000s, MP3 players became common car. In 2011, Pioneer introduced the smartphone-based head unit with iOS support. In 2012, Tesla introduced the full size single digital display for secondary tasks.

Early cars like Model T had minimal controls to operate the vehicle. The user interface in cars became complex with the introduction of music player and air conditioner like Packard. The digital interfaces in Tesla cars offered the users to personalize interaction in terms of hierarchy of icons displayed. However, the single large display may increase visual search time of appropriate screen element and in turn increase the duration of eyes-off-road. This increase in complexity in car dashboards required researchers in improving the UI (User Interface) and decrease the distraction of drivers while operating the UI. This dissertation hypothesized that undertaking secondary tasks using eye gaze and virtual touch system can improve driving performance and in parallel can be used to accurately estimate drivers’ distraction and cognitive load. The literature review discussed the history of automotive HMIs (Human Machine Interfaces) and elaborated the challenges in current HMIs (Figure 2). It discussed literature survey on existing input modalities for interactive devices like touchscreen, speech, gesture recognition and eye gaze tracking systems that are already deployed or tested in commercial vehicles.

Consumer electronics like television and computer monitors already explored modalities like remote controller, mouse, keyboard, and touch. The touch interface has been used widely because of its ease of accessibility but it is challenging to use touch modality to operate at a distance. Virtual touch technology helps in achieving touch operation at a distance away from the display. I explored different modalities for operating a Graphical User Interface (GUI) through virtual touch. I proposed two new interaction devices using a laser pointer with hardware switch as well as laser pointer with eye gaze switch (Figure 3) which do not require drivers to physically touch a display. I compared their performances with a touchscreen in automotive environment by measuring the pointing and selection time for a secondary task, deviation from the lane, average speed, variation in steering angle, cognitive load, and system usability. Results found that the laser tracker system with eye gaze switch did not significantly degrade driving or pointing performance compared to the touchscreen in standard ISO 26022 lane changing task.

Though virtual touch is useful for existing HDD, they still involve drivers to take their eyes off the road for interaction. I investigated a see-through Heads-Up display (HUD) and explored hands free interaction techniques in the form of eye gaze-controlled interface. The gaze-controlled interface allows the user to look at the intended location on the screen to select the intended target. The proposed interactive HUD system (Figure 4) can be deployed on the windshield of a vehicle and does not occlude road vision. The gaze-controlled HUD system is tested in both driving simulator and actual vehicles. Drivers could operate the display as fast as existing touchscreen with improvement in driving performance in comparison to existing touchscreen-based HDD. In addition to developing new technologies to facilitatate automotive user interaction, I also proposed new technology to estimate cognitive load of drivers in real-time.

I investigated distraction detection of drivers by estimating their cognitive load while operating secondary tasks. My studies analyzed different physiological parameters involving pupil dilation, rate of fixation, saccadic intrusion, head movement, and EEG (Electroencephalogram). Initially, a series of studies were conducted in laboratory involving standard psychometric tests like N-back and Arithmatic tests followed by trials involving driving simulator. I also undertook studies involving professional drivers performing different secondary tasks while driving actual car to validate performance of the proposed system. I used machine learning methods to train on data of multiple drivers to learn a global threshold as it was difficult to find a single threshold for all the drivers to classify cognitive states. An intelligent system generating visual, auditory, and haptic alerts was developed and integrated to the proposed distraction detection system. I also developed a cognitive load monitoring dashboard (Figure 5) for comparing HMIs in automotive with real time graphical feedback on ocular parameters and cognitive load.

Despite quantitatively analyzing the efficacy of new technologies in real-time, they will be preferred and adopted in industries only if the technologies are accepted by the consumers (drivers). A qualitative study on wearable laser-controller based HDD and eye gaze controlled HUD contributed to explore and understand concerns of professional drivers in India and how these concerns could be addressed. Drivers reported about higher response and malfunctioning of touchscreen at crucial times like during picking up passengers and not meeting their psychological needs like desire for control, positive moods and so on. It was found that wearable laser-controller based HDD contributed to lowering technological anxiety by enhancing the notion of safety. Familiarity and learnability facilitate ease of use for eye gaze controlled HUD. The ability of participants to move freely in the seat to use an interaction system influences their notion of perceived risk of accidents, which could be further improved by accurate and quick selection of icons. Our study also gave insights about the purchasing power of professional drivers. The factors influencing buying motives were also discovered. All these findings helped to form a theory of acceptance and further improve wearable laser-controlled HDD and gaze-controlled HUD.

Finally, I summarized the dissertation and discussed the potential in future direction. I have discussed the application of my research outcomes in domains other than automotive like large screen interaction, aviation (Figure 7), and smart manufacturing. The cognitive load estimation system was used to estimate workload of pilots in both simulation and in-flight studies while the laser tracker based virtual touch system is integrated to an interactive sensor dashboard of a smart manufacturing system. The laser tracker system is also used for operating large screen displays with click and drag feature used for on-screen drawing. As part of this research I set up a flexible automotive UI testbed involving a driving simulator, eye gaze, EEG trackers and associated software that can be used to compare and test performances of automotive HMIs in a plug and play fashion. Finally, I have discussed about my future proposals for post-doctoral work.