In the pursuit of general artificial intelligence, researchers often seek a "universal representation"—a single, compressed encoding of data that contains everything an agent needs to know about the world. However, a growing body of evidence across robotics, computer vision, and molecular biology suggests that such a representation may be a mathematical impossibility. Instead, the "optimal" way to represent data is inextricably tied to the specific task the system is trying to solve.
The Relativity of Noise
The fundamental challenge of representation learning is deciding what to keep and what to throw away. In their theoretical work, Stefano Soatto and Alessandro Chiuso define a visual representation as a "minimal sufficient statistic" that is invariant to "nuisance" variables.
The catch is that the definition of a nuisance is entirely task-dependent. As Soatto and Chiuso argue, spatial orientation might be irrelevant noise for a model trying to recognize a breed of dog, but it is the primary signal for a model trying to navigate a room. Because one task's signal is another task's noise, a single universal representation must either be inefficiently large (keeping everything) or inevitably lose information critical for certain objectives.
This "explanation-based paradox" was identified as early as 1987 by Richard M. Keller, who noted that learning isn't just about acquiring facts, but about operationalizing knowledge—transforming it into a form that is effective for a specific context, such as recognition versus design.
Geometric Precision versus Semantic Context
This tension is most visible in robotics. A robot tasked with "cleaning a room" requires a different understanding of its environment than a robot tasked with "inserting a key into a lock."
Research by Jonathan Yang and Dorsa Sadigh demonstrates that robotic policies benefit from "Mid-Level Representations." They found that object-centric data (like bounding boxes) is optimal for organizing clutter, while motion-centric trajectories are required for high-precision tasks. Similarly, Amir Zamir showed that geometric priors like depth estimation excel at spatial exploration, whereas semantic features like object classification are better for target-driven navigation.
Even within a single domain like autonomous driving, the requirements shift second-by-second. Luca Cultrera and colleagues found that the visual cues needed for lane following are distinct from those needed for turning at an intersection. A monolithic encoder often fails to capture these shifting priorities, leading to a "representation collapse" where the model ignores the fine-grained details necessary for complex maneuvers.
Domain-Specific Evidence: Molecules and Networks
The task-dependency of representations extends beyond visual pixels into abstract data structures:
- Molecular Chemistry: Henry Moss and Ryan-Rhys Griffiths found that string-based representations of molecules excel at predicting physical properties like solubility, while fingerprint-based representations are better for electronic properties. Jiaxin Ju further showed that expert-derived rules are critical for regression tasks (predicting values) but less so for classification.
- Action Recognition: For human activity, Zhenyue Qin observed that fine-grained hand movements require precise bone geometry, while cyclic actions like running are better captured by temporal motion vectors.
- Network Science: Even the "network" itself is a choice. Research on inferring network structure suggests that a graph optimized for one analytical objective often performs poorly on another, proving that there is no single "correct" way to represent connectivity in a dataset.
From Monoliths to Dynamic Routing
If no single representation is optimal, how do we build generalist agents? The field is moving toward Dynamic Selection and Mixture-of-Experts (MoE).
Instead of a single encoder, models like VER, developed by Yixiao Wang and Haotian Lin, use a library of vision experts. A tiny, dynamic router selects the most relevant features for the current task on the fly. This allows the system to use geometric precision when grasping and semantic context when searching.
Similarly, the Chain of Action (CoA) framework in OpenHA treats different action representations—like coordinate grounding for visible targets and relative motion for off-screen objectives—as tools to be selected based on the task context.
Conclusion
The evidence suggests that the "richness" of a representation is not a fixed property of the data, but a relationship between the data and the goal. As Arunkumar Byravan notes, a representation that is perfect for perception might be useless for exploration. By moving away from the search for a universal embedding and toward flexible, task-aware architectures, we can create AI systems that better mimic the adaptive focus of human intelligence.