Many social media posts in India, Pakistan, Bangladesh and elsewhere mix two languages inside a single sentence. A common example is Hindi or Tamil typed in Latin letters and interleaved with English. Language models handle this text poorly. Their word-piece vocabularies break a transliterated word like achha into meaningless fragments, and spelling varies from writer to writer. Yet these posts are usually memes, videos or ads, and their images carry emotional cues that do not depend on any script. One such cue is deliberate, non-realistic color: saturated warm palettes, washed-out tones, stark color blocks. Film and design scholarship treats these choices as a channel for mood and subtext. This note proposes to test one specific claim. An explicit description of an image's palette, read through the viewer's cultural frame, adds affective signal that general image encoders do not already supply. That added signal should grow as the text becomes less reliable, which is exactly the code-mixed case.
The stakes are practical and diagnostic. Meme classifiers for Tamil code-mixed content score near chance on held-out data. If color carries information that is cheap to compute and independent of the script, it is a low-cost fix. If it does not, then we learn that current image encoders already absorb color, and future work should look elsewhere. The study is designed so that either outcome is informative.
Core Claim
The claim has three parts, and each can fail on its own. First, a low-dimensional palette descriptor carries affect or intent information that is not linearly recoverable from a pretrained image embedding. The descriptor covers hue distribution in a perceptual color space, saturation and lightness statistics, and dominant-palette clusters. Second, the mapping from palette to affect differs across cultures. A color-to-sentiment model trained on English ads should transfer imperfectly to Chinese ads, and conditioning on the cultural source should recover the gap. Third, the benefit of adding the palette channel grows as the text channel degrades. That benefit is therefore largest on transliterated, code-mixed posts and smallest on clean monolingual text.
The four research directions play different roles here. Subjective color expression supplies the mechanism. Cross-cultural aesthetics predicts why a single universal color-to-emotion mapping should fail. Code-mixing is the regime where the mechanism should matter most. SWOT analysis is used only as an analysis lens, described below; it is not part of the model.
Grounding in Prior Work
Humanities work motivates the idea that color is an intentional affective channel. That literature includes the quantitative study of chromatic regimes in wuxia cinema, Cultural anchoring in cinematic digitalization, which applies stratified principal component analysis to 30 films across three technological eras. It also includes analysis of genre-signaling palettes in Xi'an Film Studio posters. Both treat color descriptively and never test whether it helps prediction. That test is our contribution.
The wuxia study also constrains the design. Its finding that editing speed dominates the first component warns that color effects can be swamped by other visual structure. We therefore need controls that separate color from form, which the research file names as an open problem.
On the multimodal side, Cultural Bias Matters builds MultiMM, a set of 8,461 Chinese and English ads. It shows that visual metaphors diverge across cultures and that translating Chinese text into English hurts accuracy. That result motivates culture-conditioning. Its model, called SEMD, fuses text features from BERT, image features from a Vision Transformer (ViT), and sentiment embeddings. It never isolates color from objects, and it covers only monolingual pairs. MultiMET provides English image-text metaphor and sentiment labels. It finds that fusing text and image beats either alone by about 6%, but its image encoders, such as ResNet-50, are generic.
Code-mixed multimodal work treats pixels generically too. ToxVidLM fuses a Hinglish text model with audio and video encoders and shows that text alone misses toxicity in Hinglish videos. The two Tamil meme systems each discard one modality: TrollMeta uses images only and Simon uses text only. Both land near 0.48–0.49 test F1.
The weakness of the text channel on code-mixed data is documented by Reed at SemEval-2020 and TueMix. In both, bag-of-words or hand-built features rival fine-tuned transformers on Hinglish sentiment. These papers provide our measure of how unreliable text is in each setting.
Research Program
We build a palette descriptor per image, or per sampled frame for video. It combines three components:
- a hue histogram in the lightness–chroma–hue color space, weighted by chroma;
- the mean and spread of saturation and lightness;
- a small set of dominant colors from clustering.
The descriptor is deliberately tiny, well under a hundred numbers. That keeps it from simply duplicating an image encoder. We add it as a gated side input to otherwise standard text-plus-image fusion models.
The central tool is a hue-rotation control. Rotating every pixel's hue by a fixed angle leaves lightness, chroma, edges and object layout unchanged and alters only the chromatic identity. If a trained model's predictions shift under hue rotation, it is using color. Comparing that shift with and without the palette channel tells us whether the channel adds sensitivity the backbone lacked. A lightness-only variant of the descriptor gives a second check. It separates true color information from brightness, which is plausibly redundant with any encoder.
Next comes a redundancy probe. We fit a linear regressor from a frozen pretrained image embedding to the palette descriptor. If the descriptor is recovered almost perfectly, the first claim is probably false, and we will say so.
Cultural conditioning uses MultiMM's Chinese and English splits. We fit a palette-to-sentiment model on one culture and evaluate it on the other, then compare against culture-specific and culture-conditioned versions. The quantity of interest is how much in-culture accuracy is lost when transferring across cultures.
SWOT enters only as an analysis lens. Its useful idea is the cross-referenced matrix, which crosses internal factors a designer controls against external conditions they do not. Here the internal factor is whether the palette channel is included. The external factors are text reliability and cultural match. We report results in this two-by-two form: palette channel present or absent, crossed with reliable versus unreliable text and matched versus mismatched culture. The cell where text is weak and culture is matched is where the channel should pay off. The cell where culture is mismatched is where it might hurt. We also report compute and latency beside accuracy, following the warning in the Project Loon post-mortem that technically strong systems fail when deployment constraints are ignored. The palette channel's main argument is that it is cheap.
Baselines
The naive reference points are the majority class and the published unimodal systems. On the Tamil Troll Memes test set, TrollMeta's image-only ResNet-50 reports F1 0.48 and Simon's text-only XLM-RoBERTa system reports F1 0.49, per their papers. We rerun both on the official split, so that any gain is measured against numbers we reproduce ourselves.
A text-only bag-of-words model follows Reed, whose best bag-of-words network reached 60.0% validation accuracy on SentiMix Hinglish. It tells us how much any image signal is worth on top of cheap text features. We train it on the same splits as every other model.
The strongest joint baseline on each dataset is a text-plus-image fusion model with a pretrained vision encoder and a code-mixing-aware text encoder. It is trained with the same data, hyperparameter search budget, seeds and GPU hours as our model; the only difference is that it lacks the palette channel. For MultiMM, the reference is SEMD, which Cultural Bias Matters reports at F1 80.16% on English and 77.79% on Chinese metaphor detection. We reimplement it and add the palette channel to it. For ToxCMM, the reference is ToxVidLM at 94.29% accuracy and 94.35% weighted F1. There, palette features from sampled frames are added to its gated fusion. For joint fusion on Tamil memes, no reported result was found on Lacuna, so we establish one.
We count our model as beating a baseline if it improves macro-F1 by more than the spread across five seeds, on at least the code-mixed datasets. We will report it plainly if SEMD or ToxVidLM show no gain.
Ablations isolate the key ingredient. We compare the full palette against lightness-only, a culture-agnostic against a culture-conditioned palette head, and a palette computed on hue-rotated images. The last is a placebo. It keeps the input format and parameter count while destroying chromatic identity, so if it matches the real palette, the gain came from extra capacity rather than color.
Why not just…?
Why not just use a stronger image encoder? This is the real threat. CLIP- or ViT-scale encoders see color, and a larger backbone may make an explicit palette redundant. The redundancy probe and the hue-rotation test settle it directly. If the palette is linearly recoverable and adding it leaves hue sensitivity and F1 unchanged over the strong-backbone baseline, the first claim fails. The study then reports that negative result. The cultural-transfer analysis still stands as a separate finding.
Why not just translate or normalize the code-mixed text into English and use a standard model? Cultural Bias Matters reports that translating Chinese metaphor text into English before processing caused a significant accuracy drop. MixSarc reports that monolingual pretraining fails on transliterated implicit meaning. Normalization is still cheap, so we include a normalize-then-classify baseline. If it closes the text gap, then the third claim predicts the palette benefit should shrink on normalized text, which is itself a test of that claim.
Why not just prompt a large vision-language model? Hanfu-Bench finds closed models about 10% behind human experts on cultural visual understanding. Cultural Bias Matters reports that SEMD outperformed GPT-4o and Gemini on specific metaphor detection tasks. We run one zero-shot prompted model as a reference on each test set. It is not tuned, so it cannot be a fair head-to-head comparison, but it bounds what scale alone buys.
Why not just add more text features, since bag-of-words already rivals transformers? Reed and TueMix show strong text-feature baselines, but those are text-only tasks. On the meme data, the text-only ceiling is exactly what sits near 0.49. The fusion comparison measures whether any visual channel helps beyond text features.
Proposed Experiments and Decisive Checks
The datasets are Tamil Troll Memes, MultiMM, MultiMET for English sentiment and intent, and ToxCMM, where palettes are computed from frames. The metrics are macro-F1, plus weighted F1 where prior work reports it, with five seeds each.
The first decisive check is the interaction test. We regress the palette-channel gain on a per-dataset text-reliability score, defined as the accuracy of the text-only model. The third claim predicts a negative slope: the palette should help most where text is least reliable, meaning the largest gain on Tamil memes and the smallest on English MultiMET. A flat or positive slope refutes it.
The second decisive check is the cross-cultural transfer gap on MultiMM, together with whether culture conditioning closes it.
The third is the hue-rotation placebo. If rotated-hue palettes perform as well as real palettes, any observed gain came from extra capacity, not color, and the core claim is rejected.
References
- Cultural anchoring in cinematic digitalization: Stratified PCA reveals chromatic adaptation regimes in wuxia cinema evolution
- STYLISTIC FEATURES OF THE DESIGN OF POSTERS OF THE XI'AN FILM STUDIO
- Cultural Bias Matters: A Cross-Cultural Benchmark Dataset and Sentiment-Enriched Model for Understanding Multimodal Metaphors
- MultiMET: A Multimodal Dataset for Metaphor Understanding
- ToxVidLM: A Multimodal Framework for Toxicity Detection in Code-Mixed Videos
- TrollMeta@DravidianLangTech-EACL2021: Meme classification using deep learning
- Simon @ DravidianLangTech-EACL2021: Meme Classification for Tamil with BERT
- Reed at SemEval-2020 Task 9: Fine-Tuning and Bag-of-Words Approaches to Code-Mixed Sentiment Analysis
- TueMix at SemEval-2020 Task 9: Logistic Regression with Linguistic Feature Set
- Lessons From the Commercial Failure of Project Loon for 6G Research Roadmap Design
- MixSarc: A Bangla-English Code-Mixed Corpus for Implicit Meaning Identification
- Hanfu-Bench: A Multimodal Benchmark on Cross-Temporal Cultural Understanding and Transcreation