Recognition of emotional and cognitive states using physiological data

Abstract

This thesis presents the application of several pattern recognition techniques on phys­iological data as a means to provide useful information about human emotional or cognitive states. As these states may be correlated with the well-being and per­formance of subjects, knowledge of these states could improve the human-computer interaction, increase productivity, and reduce accidents. We first focus on a method for recognizing the emotional state of a person who is deliberately expressing one of eight emotions. Four physiological signals were mea­sured and six features of each of these signals were extracted. We investigated three methods for the recognition: (1) Sequential floating forward search (SFFS) feature selection with K-nearest neighbors classification, (2) Fisher Projection (FP) on struc­tured subsets of features with MAP classification, and (3) A hybrid SFFS-FP method. Each method was evaluated on the full set of eight emotions as well as on several sub­sets. The day-to-day variations within the same class often exceeded between-class variations on the same day. We present a way to take account of the day information, resulting in an improvement to the Fisher-based methods. The SFFS attained a rate of 88% for a trio of emotions, while the Fisher Projection attained the best perfor­mance on the full set of emotions, 81.25%. We extend the previous study by building an online classifier so that it can be used for real-time applications. The performance is comparable to that of the offiine version. These findings demonstrate that there is significant information in physiological signals for classifying the affective state of a person who is deliberately expressing a small set of emotions. We then look into cognitive load under different driving conditions. Subjects are asked to drive in a driving simulator around several curves. Messages appear on the screen prompting the driver to either brake immediately to a standstill or to continue driving. In parts of the experiment the driver is asked to perform a simple mathe­matical task on the phone. Several measures of the subjects' behavior are recorded, including driving parameters such as lane deviation, distance and time to lane cross­ing, steering entropy, and braking delay, mistakes in addition, and physiological data (EMG, BVP, GSR, HR, Respiration). Results show that although the majority of braking delays (irrespective of the phone task) lay between -0.5 and +0.5 seconds of the average no-phone delay, there were a few cases in which subjects pressed the brakes significantly later (0.5-2.5 seconds after the average no-phone delay). Out of 315 messages prompting subjects to brake while they were not engaged on a phone task, only twice did their breaking delay exceed the average; out of 642 messages prompting subjects to brake while they were engaged on a phone task, the delay ex­ceeded the average 41 times. The effect of the mathematical task can also be seen in a 10% higher mean reaction time and a four times larger variance when subjects were on the phone compared to when they were not on the phone. Furthermore, people were on the phone in 9 out of the 10 cases that subjects mistakenly pressed the brake pedal while the message prompted them to continue driving, as well as in 6 out of the 7 cases that subjects did not show any reaction while the message prompted them to brake. We separated the responses into 2 classes, a normal and a slow one. Using the physiological data and similar pattern recognition techniques as mentioned above we predicted the class of the next delay with 65% success for an individual subject. These results indicate that the existence of specific secondary tasks while driving may adversely affect the reaction time of the driver, while use of physiological data may help in predicting such potentially dangerous situations.