A Study on Driver State Recognition Using CNN-based Multimodal Multi-input Learning

Accurate identification of a driver’s state is extremely important for the safety of both the driver and passengers. To date, many studies have demonstrated that identifying a driver’s state by creating an image of their face and upper body using a camera and then learning these CNN-based neural networks can be effective to a certain extent. In addition, efforts have been made to identify a driver’s level of fatigue using electroencephalogram (EEG), heart rate (ECG), and electrooculogram (EOG) data measurements, multiple sensors, and the RNN-LSTM model method. At present, increasing accuracy can be achieved using simple sensor devices and AI structures that can be installed in small devices such as on-device AI. In this study, we used a driver’s facial expression and upper body along with the sound data generated during driving to recognize the driver’s state more accurately. Subsequent experiments revealed that CNN-based neural network learning alone, using triple input elements, improved accuracy. To apply on-device AI, we proposed a CNN with a simple structure that can collect data employing only a camera and a recorder. We compared the proposed method with learning in ResNet50 and Xception, which revealed that it works effectively. These experimental results indicate that CNN can be used in multimodal applications and can be an efficient choice over other complex neural network learning methods that utilize multimodal learning data.