Truth-Conditional Captioning of Time Series Data

Harsh Jhamtani, Taylor Berg-Kirkpatrick

Abstract

In this paper, we explore the task of automatically generating natural language descriptions of salient patterns in a time series, such as stock prices of a company over a week. A model for this task should be able to extract high-level patterns such as the presence of a peak or a dip. While typical contemporary neural models with attention mechanisms can generate fluent output descriptions for this task, they often generate factually incorrect descriptions. We propose a computational model with a truth-conditional architecture which first runs small learned programs on the input time series, then identifies the programs/patterns which hold true for the given input, and finally conditions on only the chosen valid program (rather than the input time series) to generate the output text description. A program in our model is constructed from modules, which are small neural networks that are designed to capture numerical patterns and temporal information. The modules are shared across multiple programs, enabling compositionality as well as efficient learning of module parameters. The modules, as well as the composition of the modules, are unobserved in data, and we learn them in an end-to-end fashion with the only training signal coming from the accompanying natural language text descriptions. We find that the proposed model is able to generate high-precision captions even though we consider a small and simple space of module types.

Introduction: The Faithfulness Problem

Standard neural models, such as Transformers or LSTMs, are remarkably adept at generating fluent natural language. However, when tasked with describing structured data like time series, they frequently suffer from the "faithfulness" problem—generating descriptions that sound plausible but are factually incorrect. For example, a model might describe a stock price as "rising steadily" simply because that is a common linguistic pattern, even if the underlying data shows a sharp decline.

Comparison of TRUCE (Ours) vs Baseline on a time series. The baseline hallucinates a decrease at the start, while TRUCE correctly identifies the peak at the end using a logical program.

This issue arises because standard attention mechanisms often bypass complex numerical reasoning in favor of simple linguistic associations. To address this, we introduce TRUCE (TRUth-Conditional gEneration), a framework that forces the model to verify a pattern's "truth" before it is permitted to describe it.

The TRUCE Framework

Instead of a direct mapping from time series to text, TRUCE utilizes a three-step latent variable approach designed to ensure factual grounding:

  1. Program Composition: The model employs a library of "neural modules"—small networks specialized for specific tasks. Pattern modules (using 1D convolutions) detect shapes like slopes or dips, while Locate modules (using Gaussian mixtures) identify where in the timeline these shapes occur (e.g., beginning, middle, end).
  2. Truth Verification: Each composed program (e.g., INCREASE AND BEGINNING) operates on the input data to produce a "soft truth value" between 0 and 1, representing the model's confidence in the pattern's existence.
  3. Bottlenecked Generation: Crucially, the text decoder is only allowed to see the selected program, not the raw time series data. This functional bottleneck prevents the decoder from "hallucinating" based on raw data noise, forcing it to rely solely on verified logical patterns.
The TRUCE architecture: A graphical model where the decoder is bottlenecked by a latent program z, ensuring text is generated only from verified patterns.

To train the system without explicit labels for which programs are "true," we utilize an Inference Network during training. This network analyzes human-written captions to help the model learn which latent programs correspond to specific natural language phrases.

Experimental Setup and Results

We evaluated TRUCE on two benchmarks: SYNTH, a synthetic dataset with six perfectly defined patterns, and STOCK, a real-world dataset containing 1,900 sequences of actual stock prices paired with human-written descriptions.

Key Results

  • Factual Correctness (COR): On synthetic data, TRUCE achieved 92% correctness, significantly outperforming the best neural baseline (CONVENC) at 53%. On real stock data, TRUCE maintained a high lead with 88.4% correctness, compared to baseline models which hovered between 61% and 65%.
  • Zero-Shot Generalization: TRUCE proved remarkably robust to changes in data length. When trained on sequences of length 12 and tested on length 24, TRUCE’s correctness remained at 97%, while baseline performance dropped significantly.
  • Interpretability: Unlike "black-box" models, TRUCE is inherently interpretable. Users can inspect exactly which program the model selected and the "truth score" assigned to the data, providing a clear audit trail for the generated caption.

Conclusion

TRUCE demonstrates that incorporating truth-conditional logic into neural generation can significantly improve the faithfulness of time series descriptions. While currently limited by a relatively small program space, the model's modular architecture allows for future expansion into more complex mathematical operations and automated module discovery. By enforcing a logical bottleneck, we can move closer to AI systems that are not just fluent, but reliably truthful.

Summary updated

Abstract

In this paper, we explore the task of automatically generating natural language descriptions of salient patterns in a time series, such as stock prices of a company over a week. A model for this task should be able to extract high-level patterns such as presence of a peak or a dip. While typical contemporary neural models with attention mechanisms can generate fluent output descriptions for this task, they often generate factually incorrect descriptions. We propose a computational model with a truth-conditional architecture which first runs small learned programs on the input time series, then identifies the programs/patterns which hold true for the given input, and finally conditions on only the chosen valid program (rather than the input time series) to generate the output text description. A program in our model is constructed from modules, which are small neural networks that are designed to capture numerical patterns and temporal information. The modules are shared across multiple programs, enabling compositionality as well as efficient learning of module parameters. The modules, as well as the composition of the modules, are unobserved in data, and we learn them in an end-to-end fashion with the only training signal coming from the accompanying natural language text descriptions. We find that the proposed model is able to generate high-precision captions even though we consider a small and simple space of module types.
BibTeX
@inproceedings{jhamtani2021truth,
  author = {Harsh Jhamtani and Taylor Berg-Kirkpatrick},
  title = {{Truth-Conditional Captioning of Time Series Data}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2021},
  doi = {10.18653/v1/2021.emnlp-main.55},
}
Figure