Automatic Detection of Unexpected AI Behavior from Human Cues

AI-powered features and autonomous systems can potentially make mistakes or behave unexpectedly. The research addresses the problem of detecting different types of expectation mismatches to system behaviors, recognizing that humans react differently based on the nature, severity, and context of the mismatch.

The approach involves collecting and analyzing user behavioral responses to unexpected events in a driving simulator. The study exposes users to stimuli designed to induce surprise, confusion, and frustration, emotional responses considered primary triggers for reacting to unexpected system behaviors. A validated multi-modal multi-camera dataset is collected, including video, audio, and heart rate data, specifically for subtle human emotional responses and reactions in a vehicle environment. Detecting negative emotional responses, even when subtle or from non-critical behaviors, is seen as a strong signal for refining machine learning models and preventing erosion of user confidence and trust.