Truth-Conditional Captions for Time Series Data

Harsh Jhamtani, Taylor Berg-Kirkpatrick

Introduction

Generating natural language descriptions from structured data—a task known as Data-to-Text—remains a significant challenge in artificial intelligence. While modern large language models produce fluent text, they frequently "hallucinate" facts, especially when interpreting numerical data like stock market trends. A standard model might observe a stable chart and incorrectly report a "sharp decline."

To address this, researchers have developed TRUCE (TRUth Conditional gEneration). TRUCE mitigates hallucinations by introducing a "logical bottleneck": the model must first identify a mathematically true "program" (e.g., "a peak at the end") before the text generator is allowed to write a caption. This ensures that every generated sentence is factually grounded in the underlying data.

The TRUCE Architecture

The core innovation of TRUCE is its truth-conditional architecture. Unlike traditional encoder-decoder models that map raw numerical signals directly to words—often learning spurious correlations—TRUCE forces the decoder to condition only on a verified latent program.

The TRUCE graphical model and decoder architecture. The decoder conditions only on the sampled latent program z, not the raw input x.

The framework operates in four distinct stages:

  1. Module Composition: The system utilizes a library of learnable neural modules. Pattern modules use 1D convolutions to detect shapes (e.g., peaks, dips, increases), while Locate modules use a mixture of Gaussians to identify temporal windows (e.g., "the beginning," "the middle").
  2. Program Execution: These modules are combined into programs, such as Decrease AND Middle.
  3. Truth Scoring: Each program is executed against the input time series to produce a "soft truth value" between 0 and 1, indicating how well the pattern matches the data.
  4. Conditioned Generation: The model samples a program that is "true" for the data. The text decoder then generates a caption based only on this program. By hiding the raw, noisy data from the decoder, the model is prevented from hallucinating details not present in the verified program.

Experimental Evaluation

The researchers evaluated TRUCE using two primary datasets: the SYNTH dataset, containing 720 time series with six ground-truth patterns, and the STOCK dataset, comprising 1,900 real-world stock price sequences paired with 5,700 human-written descriptions.

Key Results

The primary metric for success was Correctness (COR), a human-evaluated measure of whether the caption accurately describes the chart.

Qualitative comparison showing TRUCE ("Ours") correctly identifying a peak at the end using a logical formula, while the baseline incorrectly describes a sharp decrease at the start.
  • Accuracy: On synthetic data, TRUCE achieved 92% correctness, significantly outperforming the best neural baseline (53%). On the noisy STOCK dataset, TRUCE maintained 88.4% correctness, while baselines reached only 65.9%.
  • Zero-Shot Generalization: TRUCE demonstrated robust performance on time series of varying lengths. When tested on sequences twice as long as those in the training set, TRUCE's accuracy remained high (97% on SYNTH), whereas baselines struggled to adapt to the new temporal scale.

Discussion and Future Directions

TRUCE's success stems from its interpretability by design. Because the model selects a latent program, users can inspect the specific pattern the AI detected, making the "thought process" transparent. Furthermore, the compositionality of the modules allows the model to learn efficiently; a "peak" module learned in one context can be reused across different timeframes.

However, TRUCE is currently limited by its predefined module library. While it excels at describing trends and peaks, it may struggle with complex financial behaviors that do not fit the pattern + locate template. Future work involves expanding the module library to include complex mathematical operations and more sophisticated temporal logic to capture a wider range of data nuances.

Summary updated

Abstract

In this paper, we explore the task of automatically generating natural language descriptions of salient patterns in a time series, such as stock prices of a company over a week. A model for this task should be able to extract high-level patterns such as presence of a peak or a dip. While typical contemporary neural models with attention mechanisms can generate fluent output descriptions for this task, they often generate factually incorrect descriptions. We propose a computational model with a truth-conditional architecture which first runs small learned programs on the input time series, then identifies the programs/patterns which hold true for the given input, and finally conditions on only the chosen valid program (rather than the input time series) to generate the output text description. A program in our model is constructed from modules, which are small neural networks that are designed to capture numerical patterns and temporal information. The modules are shared across multiple programs, enabling compositionality as well as efficient learning of module parameters. The modules, as well as the composition of the modules, are unobserved in data, and we learn them in an end-to-end fashion with the only training signal coming from the accompanying natural language text descriptions. We find that the proposed model is able to generate high-precision captions even though we consider a small and simple space of module types.
BibTeX
@inproceedings{jhamtani2021truth,
  author = {Harsh Jhamtani and Taylor Berg-Kirkpatrick},
  title = {{Truth-Conditional Captions for Time Series Data}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2021},
  doi = {10.18653/v1/2021.emnlp-main.55},
}
Figure