Passable for Whom? Few-Shot Personal Calibration of Sidewalk Accessibility Perception

Wesley Maia, Angel Martinez-Sanchez, Maitrayee Keskar, Ross Greer

TL;DR

Current accessibility maps use a "one-size-fits-all" approach that fails to account for the vast differences in how individuals—even those using the same mobility aid—perceive sidewalk barriers. PASSCAL is a new framework that uses just eight personal answers to calibrate a vision-language model, significantly improving passability predictions and providing "depends on you" warnings when uncertainty is high.

What Problem Does This Paper Address?

If you use a manual wheelchair, a missing curb ramp might be a total blocker, while for a cane user, it’s a non-issue. However, the problem goes deeper: even two people using the same type of wheelchair often disagree on whether a specific cracked sidewalk is "passable."

Standard computer vision systems for urban accessibility typically assign a single label to a scene (e.g., "obstacle, severity 3"). This "crowd-average" label is often wrong for a significant portion of the population. The authors address the challenge of few-shot personal accessibility perception: how can a system learn a specific person's needs and "response style" from just a handful of examples?

Methods

The authors introduce PASSCAL, a modular system that separates the description of a scene from a person’s interpretation of it. It consists of three main components:

  1. Perception (The "What"):
    • A Property Detector (fine-tuned DINOv2) trained on 14.9k public sidewalk audits to predict objective features: barrier type (e.g., curb ramp, surface problem), severity (1-3), and context tags (e.g., narrow, vegetation).
  2. A Scene Embedding from a frozen Vision-Language Model (Qwen3-VL-8B) to capture nuanced visual information that the explicit detector might miss.
  3. Personal Calibration (The "Who"):
    • The system uses a proportional-odds model to combine general group weights (how manual wheelchair users usually feel) with personal weights learned from $K$ calibration answers.
  4. It includes a response calibration step (similar to temperature scaling) to account for individual personality traits, such as being generally "strict" or frequently "unsure."
  5. Statements with Controlled Risk:
    • Instead of just giving a probability, the system uses Conformal Risk Control to output definitive statements: "passable," "blocked," or "depends on you." This ensures that the rate of "harmful errors" (saying it's passable when the user thinks it's blocked) stays below a user-defined bound (e.g., 5%).

Datasets

  • Primary Evaluation Dataset: A public dataset by Li et al. containing 8,883 judgments from 135 verified mobility-aid users (cane, walker, manual/motorized wheelchair, scooter). Crucially, every participant rated the same 50 sidewalk scenes, allowing the model to isolate personal variation from scene variation.
  • Training Pool: 14,893 (extended to 18k) sidewalk scenes from Project Sidewalk audits across nine cities, used to train the objective property detector.

Key Insights and Ideas

  • VLMs are not enough: Interestingly, the authors found that state-of-the-art VLMs (like Qwen3 or LLaVA) often miss subtle but critical accessibility details, like the exact severity of a curb ramp. Adding explicit "audit-supervised" properties improved the performance of all 16 vision encoders tested.
  • Style vs. Substance: A person's "response style" (how strictly they define "passable") explains more of the variance in the data than the images themselves for some groups. PASSCAL's ability to calibrate for this "strictness" was a major driver of its success.
  • In-Context Learning Fails: Simply prompting a VLM with a user's previous answers (in-context learning) performed poorly. The VLM struggled to extract the "personal signal" from the examples, whereas PASSCAL’s explicit statistical calibration was much more effective.
  • The "Depends on You" Safety Valve: By using conformal prediction, the system can admit when it doesn't know enough about a user to make a safe call, shifting from a binary "yes/no" to a more helpful "depends on you" for ambiguous scenes.

Key Results

  • Accuracy Boost: With just 8 calibration answers, PASSCAL reduced the Brier score (a measure of prediction error) from 0.550 to 0.479.
  • Decisive Utility: At a 5% error bound, providing 8 answers increased the share of "decisive" (passable or blocked) statements from 27% to 42%. Without calibration, the system has to say "depends on you" much more often to remain safe.
  • Outperforming Baselines: PASSCAL outperformed in-context VLM personalization by a margin of 0.090 and per-person regression by 0.068 in Brier score.
  • Generalization: The model successfully lowered error rates even when tested on a completely "unseen" barrier type that wasn't in its training set.

Limitations and Future Work

  • Small Scene Sample: The evaluation was limited to 50 unique sidewalk scenes (the only dense dataset available). While the authors used bootstrapping to mitigate this, more diverse visual data is needed.
  • Static vs. Dynamic: The study uses photographs. Real-world navigation involves traversals, where factors like weather or battery life might change a user's perception of passability in real-time.
  • Marginal Guarantees: The risk control guarantees are "marginal" (averaged over the population). Future work could look into "conditional" guarantees that ensure safety for every single individual user.
  • Privacy: The authors note that personal parameters reveal sensitive information about a user's disability and should ideally stay on the user's local device.

Summary updated

Abstract

Whether a sidewalk is passable depends on who is trying to pass: people who use the same mobility aid disagree about the same image, so one label per scene, or per aid, is wrong for many of them. We introduce few-shot personal accessibility perception: from K answers of a person never seen before, predict their judgment of sidewalk scenes never seen before. PassCal separates a shared description of the scene from the person's interpretation of it. The description comes from the internal state of a frozen vision-language model (VLM) and from barrier type, severity and context properties predicted by a detector trained on 14.9k public audits; the interpretation, a calibration of the person's response style and a weighting of the properties, is fit from their answers. On 8,883 judgments in which each of 135 verified mobility-aid users rated the same 50 sidewalk scenes, with people and scenes both held out, eight answers lower the Brier score from 0.550 to 0.479. The same VLM prompted with those answers reaches 0.562, and the same calibration without the scene 0.521. Conformal risk control turns predictions into passable, blocked or "depends on you" statements with a bounded expected rate of wrong statements over people; at 5%, eight answers raise the share of decisive scenes from 27% to 42%. Audit-supervised properties improve all sixteen vision and VLM encoders we test and outperform attributes from a sidewalk navigation VLM.
BibTeX
@misc{maia2026passable,
  author = {Wesley Maia and Angel Martinez-Sanchez and Maitrayee Keskar and Ross Greer},
  title = {{Passable for Whom? Few-Shot Personal Calibration of Sidewalk Accessibility Perception}},
  year = {2026},
  eprint = {2611.19839},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.19839},
}
Figure