Abstract
Having a rich multimodal inner language is an important component of human intelligence that enables several necessary core cognitive functions such as multimodal prediction, translation, and generation. Building upon the Conscious Turing Machine (CTM), a machine model for consciousness proposed by Blum and Blum (2021), we describe the desiderata of a multimodal language called Brainish, comprising words, images, audio, and sensations combined in representations that the CTM's processors use to communicate with each other. We define the syntax and semantics of Brainish before operationalizing this language through the lens of multimodal artificial intelligence, a vibrant research area studying the computational tools necessary for processing and relating information from heterogeneous signals. Our general framework for learning Brainish involves designing (1) unimodal encoders to segment and represent unimodal data, (2) a coordinated representation space that relates and composes unimodal features to derive holistic meaning across multimodal inputs, and (3) decoders to map multimodal representations into predictions (for fusion) or raw data (for translation or generation). Through discussing how Brainish is crucial for communication and coordination in order to achieve consciousness in the CTM, and by implementing a simple version of Brainish and evaluating its capability of demonstrating intelligence on multimodal prediction and retrieval tasks on several real-world image, text, and audio datasets, we argue that such an inner language will be important for advances in machine models of intelligence and consciousness.
1. Introduction: The Need for an Inner Language
Human intelligence is not siloed; we do not simply "see" or "hear" in isolation. Our brains seamlessly integrate different sensory streams into a cohesive internal experience. In contrast, traditional AI often builds specialized models for specific tasks without a unified protocol for these modalities to communicate.
This paper addresses the need for a formal communication protocol—an "inner language"—within machine models of intelligence. Specifically, it builds upon the Conscious Turing Machine (CTM), a theoretical model for consciousness. The CTM requires a way for its many sub-processors to broadcast "gists" of information to a central "stage." Brainish is the proposed language for those gists, providing the syntax (structure) and semantics (meaning) necessary for a machine to achieve a holistic understanding of the world.
2. Operationalizing Brainish through Multimodal AI
We operationalize Brainish through a three-step machine learning framework that addresses the core challenges of multimodal intelligence: representation, fusion, alignment, translation, and co-learning.
The framework consists of:
- Unimodal Encoders: Raw data (images, text, audio) is processed by specialized encoders (e.g., ResNet, BERT). These segment data into "atomic units"—the "words" of the inner language.
- Coordinated Representation Space: This is the core of Brainish. The framework learns a shared space where atomic units from different modalities are aligned if they share the same meaning.
- Decoders: These map coordinated multimodal representations back into useful outputs, such as category predictions (fusion) or raw data in a different modality (translation/generation).
3. Key Concepts: Syntax, Semantics, and Co-learning
A crucial distinction in Brainish is between semantics and syntax. While semantics focuses on shared meaning (e.g., aligning the vector for the word "dog" with an image of a dog), syntax involves composition. For example, a text description of a "teacup on the right of a laptop" helps resolve probabilistic ambiguity in a visual scene through structured composition.
Furthermore, the shared space enables Co-learning, where a machine can learn about a "low-resource" modality (like a rare sound) by leveraging its knowledge of a "high-resource" modality (like text). By formalizing these "gists," the paper moves consciousness research from philosophy toward engineering.
4. Experimental Results
The framework was evaluated on real-world datasets including Yummly-28K (recipes) and a custom CIFAR-10/ESC-50 benchmark (images and sounds).
- Superior Fusion: In recipe classification, the Brainish model achieved 68.0% accuracy, significantly outperforming unimodal baselines (60.0% for text, 55.0% for images).
- Effective Alignment: The model successfully performed cross-modal retrieval, accurately matching complex text descriptions with corresponding images.
- Few-Shot Co-learning: In transfer tests, Brainish classified images or audio using only 1–10 labeled examples, performing on par with "Oracle" models trained on thousands of samples.
5. Conclusion and Future Work
Brainish provides a path toward more integrated machine intelligence. While current implementations focus on prediction and retrieval, future work will explore high-fidelity multimodal generation—the machine equivalent of dreaming. Additionally, moving from static to "plastic" encoders will better mimic the adaptability of the human brain, facilitating decision-making in complex, real-time environments.
Summary updated