TL;DR
Current accessibility maps use a "one-size-fits-all" approach that fails to account for the vast differences in how individuals—even those using the same mobility aid—perceive sidewalk barriers. PASSCAL is a new framework that uses just eight personal answers to calibrate a vision-language model, significantly improving passability predictions and providing "depends on you" warnings when uncertainty is high.
What Problem Does This Paper Address?
If you use a manual wheelchair, a missing curb ramp might be a total blocker, while for a cane user, it’s a non-issue. However, the problem goes deeper: even two people using the same type of wheelchair often disagree on whether a specific cracked sidewalk is "passable."
Standard computer vision systems for urban accessibility typically assign a single label to a scene (e.g., "obstacle, severity 3"). This "crowd-average" label is often wrong for a significant portion of the population. The authors address the challenge of few-shot personal accessibility perception: how can a system learn a specific person's needs and "response style" from just a handful of examples?
Methods
The authors introduce PASSCAL, a modular system that separates the description of a scene from a person’s interpretation of it. It consists of three main components:
- Perception (The "What"):
- A Property Detector (fine-tuned DINOv2) trained on 14.9k public sidewalk audits to predict objective features: barrier type (e.g., curb ramp, surface problem), severity (1-3), and context tags (e.g., narrow, vegetation).
- A Scene Embedding from a frozen Vision-Language Model (Qwen3-VL-8B) to capture nuanced visual information that the explicit detector might miss.
- Personal Calibration (The "Who"):
- The system uses a proportional-odds model to combine general group weights (how manual wheelchair users usually feel) with personal weights learned from $K$ calibration answers.
- It includes a response calibration step (similar to temperature scaling) to account for individual personality traits, such as being generally "strict" or frequently "unsure."
- Statements with Controlled Risk:
- Instead of just giving a probability, the system uses Conformal Risk Control to output definitive statements: "passable," "blocked," or "depends on you." This ensures that the rate of "harmful errors" (saying it's passable when the user thinks it's blocked) stays below a user-defined bound (e.g., 5%).
Datasets
- Primary Evaluation Dataset: A public dataset by Li et al. containing 8,883 judgments from 135 verified mobility-aid users (cane, walker, manual/motorized wheelchair, scooter). Crucially, every participant rated the same 50 sidewalk scenes, allowing the model to isolate personal variation from scene variation.
- Training Pool: 14,893 (extended to 18k) sidewalk scenes from Project Sidewalk audits across nine cities, used to train the objective property detector.
Key Insights and Ideas
- VLMs are not enough: Interestingly, the authors found that state-of-the-art VLMs (like Qwen3 or LLaVA) often miss subtle but critical accessibility details, like the exact severity of a curb ramp. Adding explicit "audit-supervised" properties improved the performance of all 16 vision encoders tested.
- Style vs. Substance: A person's "response style" (how strictly they define "passable") explains more of the variance in the data than the images themselves for some groups. PASSCAL's ability to calibrate for this "strictness" was a major driver of its success.
- In-Context Learning Fails: Simply prompting a VLM with a user's previous answers (in-context learning) performed poorly. The VLM struggled to extract the "personal signal" from the examples, whereas PASSCAL’s explicit statistical calibration was much more effective.
- The "Depends on You" Safety Valve: By using conformal prediction, the system can admit when it doesn't know enough about a user to make a safe call, shifting from a binary "yes/no" to a more helpful "depends on you" for ambiguous scenes.
Key Results
- Accuracy Boost: With just 8 calibration answers, PASSCAL reduced the Brier score (a measure of prediction error) from 0.550 to 0.479.
- Decisive Utility: At a 5% error bound, providing 8 answers increased the share of "decisive" (passable or blocked) statements from 27% to 42%. Without calibration, the system has to say "depends on you" much more often to remain safe.
- Outperforming Baselines: PASSCAL outperformed in-context VLM personalization by a margin of 0.090 and per-person regression by 0.068 in Brier score.
- Generalization: The model successfully lowered error rates even when tested on a completely "unseen" barrier type that wasn't in its training set.
Limitations and Future Work
- Small Scene Sample: The evaluation was limited to 50 unique sidewalk scenes (the only dense dataset available). While the authors used bootstrapping to mitigate this, more diverse visual data is needed.
- Static vs. Dynamic: The study uses photographs. Real-world navigation involves traversals, where factors like weather or battery life might change a user's perception of passability in real-time.
- Marginal Guarantees: The risk control guarantees are "marginal" (averaged over the population). Future work could look into "conditional" guarantees that ensure safety for every single individual user.
- Privacy: The authors note that personal parameters reveal sensitive information about a user's disability and should ideally stay on the user's local device.
Summary updated