Evaluating Pervasive and Multi-modal Information Access

You are reading version 1 of this direction, frozen 2026-10-04. The latest version is version 2.

Version 1 of 29 papers30 conceptsfrozen 2026-10-04

For decades, the progress of information retrieval was measured by a simple yardstick: how well a system could match a user’s keyword query to a static document. Metrics like Precision, Recall, and Average Precision were the gold standard. However, as our interaction with information shifts from a search box on a desktop to pervasive mobile assistants, multi-modal sensors, and generative AI, these legacy metrics are beginning to fracture.

The core insight across recent research is that evaluation is not merely a post-hoc measurement—it is the primary engine of technological advancement. To build more sophisticated systems, we must first build more sophisticated ways to measure them.

Evaluation as the Engine of Progress

In the field of Information Access (IA), a lack of robust evaluation frameworks acts as a literal bottleneck. As Ruihua Song and William Webber argue in Preface (EVIA@NTCIR 2013), evaluation frameworks provide the feedback loop necessary for algorithmic evolution. Without precise metrics to quantify growth, researchers cannot validate whether a new architecture is actually better or just different.

This becomes a critical problem when the "task" itself changes. When Hsin-Hsi Chen introduced sessions on sentiment analysis and opinion mining in Preface for Open Submission Session, it became clear that subjective information needs cannot be captured by binary relevance. A system might retrieve a "relevant" document that completely misses the nuance of the user's intent or the sentiment they are searching for.

The Failure of Desktop-Centric Metrics

Most traditional metrics assume a "stationary" user—someone sitting at a desk, looking at a list of links, and clicking on the most promising one. But modern information access is pervasive.

In Evaluating Personal Assistants on Mobile devices, Julia Kiseleva and Maarten de Rijke point out that on mobile devices, traditional signals like click-through rates and dwell time are often misleading. A user might find exactly what they need from a notification or a quick glance at a snippet without ever "clicking." To measure success here, we need to look at "hidden" signals: touch gestures, device orientation, and even the environmental context.

Similarly, Ruihua Song and Stefano Mizzaro emphasize in Preface (EVIA@NTCIR 2014) that mobile search is highly situational. A metric that ignores device constraints or the user's physical location fails to capture the true utility of the system.

High-Dimensional and Multi-modal Challenges

As we move toward "Digital Ecosystems," the complexity of what we are retrieving increases. Hai Dong, Farookh Khadeer Hussain, and Elizabeth Chang 0001 suggest in A Service Search Engine for the Industrial Digital Ecosystems that we must move toward evaluating "Quality-of-Service" (QoS) and reputation, rather than just topical relevance.

This trend reaches its peak in multi-modal environments. In the Overview of NTCIR-15, Noriko Kando and Yiqun Liu describe tasks involving biosignals and screen captures for activity tracking. In these high-dimensional domains, a single score like Average Precision is no longer sufficient to describe how well a system understands human behavior or complex data structures.

Depth, Breadth, and Precarity

How should the field move forward? Fernando Diaz provides a roadmap in The Principles of Depth, Breadth, and Precarity in Developing Metrics for Information Retrieval Evaluation:

  1. Depth: We must refine existing metrics to handle the nuances of modern outputs, such as those from generative AI.
  2. Breadth: We must expand evaluation to include social considerations, such as fairness and bias, which traditional IR metrics completely ignore.
  3. Precarity: We must acknowledge that our metrics are "brittle." As Emine Yilmaz and Charles L. A. Clarke warn in Preface (EVIA@NTCIR 2016), if our measurement tools do not co-evolve with our technological capabilities, we risk a stagnation where we can no longer tell if we are making progress at all.

The shift in research is clear: the community is moving away from "one-size-fits-all" metrics toward context-aware, multi-faceted evaluation frameworks that treat the user's situational experience as the ultimate ground truth.

Papers in this version

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 15 papers
  2. Version 1 · 2026-10-04 · 9 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_metric_obsolescence_4a814c9f,
  title = {metric obsolescence},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/metric-obsolescence/rdr_4a814c9f4c4c48e687bf83a72e236346},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_evaluating_pervasive_and_multi_modal_inf_4a814c9f_v1,
  title = {Evaluating Pervasive and Multi-modal Information Access},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/evaluating-pervasive-and-multi-modal-information-access/rdr_4a814c9f4c4c48e687bf83a72e236346/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}