Pedestrian Behavior Modeling for Autonomous Vehicles

Pedestrian behavior prediction is one of the hardest open problems in AV perception — pedestrians are multimodal (they can cross or not cross), their intent is latent, and predictions must be accurate over 5–10 second horizons to be useful for planning. This work develops the Multimodal Hybrid Pedestrian (MHP) model: an explainable, probabilistic behavior model based on hybrid automaton theory that predicts long-horizon pedestrian actions and trajectories at unsignalized crosswalks, with direct integration into AV motion planning.

Model Structure

The MHP model is a hybrid automaton with four discrete behavioral modes:

• Approaching — pedestrian walking toward the crosswalk
• Waiting — pedestrian paused at the curb, evaluating a crossing gap
• Crossing — pedestrian actively traversing the road
• Walking away — pedestrian departing without crossing

Transition guards encode the pedestrian’s decision-making logic: the model separately represents intent to cross and execution of crossing, unlike prior work that assumed all pedestrians at a crosswalk would eventually cross. This separation produces significantly less conservative predictions for pedestrians who ultimately walk away.

Within each discrete mode, continuous motion evolves according to mode-specific dynamics (linear motion models with learned parameters). The model outputs a probability distribution over future mode sequences and associated continuous trajectories, represented as a set of forward reachable prediction envelopes.

Validation

The MHP model was validated against two baselines — a constant velocity model and a baseline hybrid automaton that does not separately model crossing intent — on two datasets:

• Real-world data: pedestrian trajectories collected on Michigan streets involving interactions with both human-driven and automated vehicles
• Virtual reality data: pedestrian behavior captured using an omnidirectional treadmill and VR headset in a simulated urban environment with AV interactions

Across both datasets, the MHP model’s probabilistic predictions more frequently contained the ground truth trajectory compared to both baselines, particularly in scenarios where pedestrians wait and then cross — the mode transition the constant velocity model cannot represent (Jayaraman et al., 2021) (Jayaraman et al., 2020).

Integration with AV Motion Planning

The probabilistic predictions from the MHP model feed directly into a Behavior-aware Model Predictive Controller (B-MPC) for AV motion planning. The B-MPC uses the predicted pedestrian mode probabilities and trajectory envelopes as constraints and costs, enabling the AV to plan paths that are safe across the distribution of possible pedestrian futures — not just the most likely one. This produces less conservative AV behavior compared to worst-case planning while maintaining safety guarantees (Jayaraman et al., 2020) (Jayaraman et al., 2021).

The model is applicable across urban scenarios: midblock crosswalks, signalized and unsignalized intersections, one-way and two-way streets.

Transferable Engineering

This work is directly applicable to any AV stack requiring long-horizon pedestrian prediction: the MHP model is a plug-in probabilistic predictor that outputs trajectory distributions compatible with standard MPC and sampling-based planners.