Participation detection and Robot Mediation for Inclusive Child Group Interactions

Children with mixed abilities, such as those with visual or auditory impairments, often face challenges in fully participating in group interactions. This project, a collaboration between Carnegie Mellon University (USA) and Universidade de Lisboa (Portugal), seeks to address this issue by developing real-time multimodal group participation metrics and using these metrics to guide socially assistive robot behaviors that foster inclusion and equity in group dynamics.

Using synchronized cameras and microphones, we extract visual and auditory cues like gaze direction, head nods, speech turns, and interruptions. These are processed in real time using tools such as OpenFace, MediaPipe, etc., to generate composite metrics of group engagement.

These real-time metrics are used to inform different robot mediation strategies to promote equitable participation. The outcomes of this research are expected to support more inclusive educational, and social environments through intelligent, context-aware robotic systems. A related session-aware, dataset-agnostic engagement estimation architecture developed as part of this line of work was benchmarked in the ACM MultiMediate Grand Challenge (Enock et al., 2026).

Code: jskumaar/group_participation_detection (real-time participation metrics)  ·  Naveen-Enock/multimediate (MultiMediate engagement estimation architecture)