TL;DR
This paper introduces a reproducible processing pipeline for Street View Imagery (SVI) that ensures automated urban audits are actually looking at the right building. By combining task-specific filtering and geometric target alignment, the researchers improved vacant house classification accuracy and, more importantly, provided a way for humans to verify the visual evidence behind a model's prediction.
What Problem Does This Paper Address?
Remote urban auditing using Street View Imagery (SVI) has become a staple for researchers tracking everything from street greenery to neighborhood decay. However, most automated assessments suffer from a "black box" input problem: they assume every image retrieved for a specific address clearly shows the target building.
In reality, SVI is messy. An image might be occluded by a delivery truck, taken from a mile away, or—most deceptively—show the neighbor’s house instead of the target. If a model predicts a house is "vacant" because it sees a dilapidated shed in the background or a neighbor's overgrown yard, the prediction might be "correct" for the wrong reasons. The authors argue that for SVI-based audits to be policy-relevant, we need to explicitly verify image suitability (is the image good?) and target identity (is this the right house?).
Methods
The authors propose a three-stage pipeline:
- Task-Suitability Filtering: Instead of a "one-size-fits-all" filter, the system uses SegFormer-B2 to segment images and K-means clustering to group them into three scene types: Built-up, Green, and Open-sky. It then applies group-specific thresholds (e.g., allowing more vegetation in rural areas) to ensure that rural or low-density parcels aren't accidentally filtered out of the dataset.
- Target-House Alignment: To find the specific house within a crowded street scene, the pipeline uses SAM 3 (Segment Anything Model) to generate building masks. It then projects the official parcel boundaries from a cadastral map into the 2D image space. Masks are scored based on their overlap with this "parcel band."
- Multi-View Consistency: The system checks if the selected buildings from different camera angles converge on the same physical point in space. If the "sight lines" from three different panoramas don't meet, the parcel is flagged for human review.
- Classification & Explainability: A ResNet-50 classifier predicts vacancy status. Crucially, the authors use Grad-CAM to visualize which pixels triggered the prediction, allowing them to check if the model was actually "looking" at the target house.
Datasets
- Primary Dataset: 10,311 images covering 1,609 detached-house parcels in South Korea, sourced from Naver Map and Kakao Map.
- Transferability Test: To prove the method isn't platform-dependent, they applied the pipeline to 19,612 Google Street View images of 5,000 parcels in Detroit, USA, using the Detroit Residential Parcel Survey (DRPS).
Key Insights and Ideas
- The "Most Central" Trap: A common heuristic in SVI research is to assume the building in the center of the frame is the target. This paper found that the most central building was the wrong house in roughly 32% of cases.
- Context-Sensitive Filtering: By adjusting filters based on whether a scene was urban or rural, the researchers retained 101 more usable parcels than a uniform filter would have. This prevents "coverage bias," where automated audits inadvertently ignore rural areas because they don't look like dense city centers.
- Right for the Wrong Reasons: The most striking finding was that even when the model was intentionally shown the wrong house in an image, it still achieved an AUROC of 0.709. This suggests that neighborhood-level cues (like a whole block being in decline) can mask the fact that a model isn't actually assessing the specific target building.
Key Results
- Alignment Accuracy: The proposed overlap-based alignment matched human labels for 85.6% of images, significantly outperforming the "Largest Building" (79.8%) and "Most Central" (68.1%) heuristics.
- Performance Boost: Moving from raw SVI to the full filtered and aligned pipeline raised the parcel-level AUROC from 0.684 to 0.719.
- Visual Evidence Gap: Grad-CAM analysis showed that in over one-third of correctly aligned images, more than half of the model's "attention" was actually on areas outside the target house (e.g., the street or adjacent buildings).
Limitations and Future Work
- Data Dependencies: The pipeline requires high-quality parcel boundaries and camera metadata (heading, FOV, coordinates), which are not available in all regions.
- Temporal Mismatch: SVI capture dates often don't align perfectly with official survey dates. A house might be occupied in a 2022 photo but vacant by a 2024 survey.
- Future Directions: The authors suggest integrating building footprints (where available) to further refine target selection and using the multi-view convergence points to re-rank candidate masks in complex urban environments.
Summary updated