Reverse TDNN: An Architecture For Trajectory Generation

Patrice Y. Simard, Yann LeCunNeurIPS 1991 · OpenReview

Overview

Researchers Patrice Simard and Yann Le Cun propose the Oversampled Reverse Time Delay Neural Network (ORTDNN), an architecture specifically designed to generate complex time-series trajectories like handwriting. By "flipping" the structure of successful recognition networks and using "differential units" that control the rate of change rather than absolute position, they demonstrate a highly efficient way to map abstract inputs to fluid, hierarchical motor patterns.

The Challenge of Trajectory Generation

Generating smooth, complex trajectories—such as the path of a pen drawing a letter or a robotic arm moving through space—is a notoriously difficult task for standard neural networks. While recurrent neural networks (RNNs) are often used, they are not inherently tailored for the hierarchical nature of movement.

In human motor control, we do not consciously process every millisecond of muscle tension. Instead, we rely on an abstract "plan" (e.g., "draw the letter B") that decomposes into sub-programs (strokes) and eventually into high-frequency muscle activations. Standard fully connected networks struggle to learn multiple, distinct trajectories simultaneously because they lack this hierarchical structure. They often end up "averaging" their outputs, failing to distinguish between different input commands.

Proposed Methods

To address these limitations, the authors introduce two primary innovations:

1. Differential Units

Instead of the network's output directly determining an absolute position (e.g., "the pen is at coordinate X, Y"), the input to a neuron controls the rate of change (the time derivative) of that unit's state.

Architecture of the differential units

This is implemented by discretizing continuous-time equations into a feed-forward structure that can be trained via standard backpropagation. This approach inherently favors smoother, more stable trajectories and avoids the instability often found in standard recurrent networks.

2. Oversampled Reverse TDNN (ORTDNN)

This architecture is essentially the "inverse" of a Time Delay Neural Network (TDNN) typically used for speech or character recognition.

  • In a recognition TDNN, the network reduces high-frequency data (like audio waveforms or pixels) into abstract categories through subsampling.
  • In the Reverse TDNN, the process is flipped: an abstract, low-frequency input (a code for a specific letter) is "oversampled" through layers. Higher layers (closer to the input) update slowly, representing the overarching "plan," while lower layers (closer to the output) update faster to produce the high-frequency details of the trajectory.

Datasets and Key Insights

The researchers validated their model using a handwritten letter database containing pen trajectories. These trajectories were recorded using a tablet that captured triplets of data: time, X-position, and Y-position. The dataset included the 26 letters of the alphabet, where each letter trajectory covered up to 90 time stamps.

  • Architecture as a Prior: Just as convolutional layers act as a structural "prior" for images, the ORTDNN structure acts as a prior for motor control. It forces the network to learn a hierarchy where abstract plans gradually unfold into specific movements.
  • Efficiency through Weight Sharing: Because the network uses time-delay connections and weight sharing (similar to a 1D convolution), it is incredibly compact. Remarkably, one version of the network used to generate the entire alphabet contained only 94 free weights.
  • Breaking Correlation: One of the most impressive findings was that the ORTDNN could produce totally different trajectories for inputs that were nearly identical (e.g., the input codes for 'A' and 'Z' might only differ by two bits). Fully connected models completely failed to achieve this separation.

Key Results

While a standard fully connected network was unable to learn even two letters ('A' and 'B') simultaneously—essentially ignoring the input and producing a garbled average—the ORTDNN excelled.

2D trajectory generated by the network
  • Alphabet Mastery: The ORTDNN successfully learned to generate all 26 letters of the alphabet after 10,000 training iterations.
  • Smoothness: By setting specific time constants in the differential units, the network produced fluid, human-like handwriting trajectories rather than jagged or robotic movements.

Limitations and Future Work

  • Recurrent Connections: The authors note that for tasks requiring information to be stored for an arbitrary length of time, explicit recurrent connections (moving "backward" or "sideways" through the architecture) would be necessary.
  • Sensor Integration: The current model is "open-loop," meaning it does not observe its own output. The authors suggest adding sensor inputs at various stages to allow the network to adjust trajectories in real-time based on external feedback, which is critical for robotics applications.
  • Auto-Encoders: They propose a "back-to-back" architecture—combining a TDNN for recognition with an ORTDNN for generation—to create a powerful system for the compact encoding and reconstruction of time-series data.

Summary updated

Abstract

The backpropagation algorithm can be used for both recognition and gen(cid:173) eration of time trajectories. When used as a recognizer, it has been shown that the performance of a network can be greatly improved by adding structure to the architecture. The same is true in trajectory generation. In particular a new architecture corresponding to a "reversed" TDNN is proposed. Results show dramatic improvement of performance in the gen(cid:173) eration of hand-written characters. A combination of TDNN and reversed TDNN for compact encoding is also suggested.
BibTeX
@inproceedings{simard1991reverse,
  author = {Patrice Y. Simard and Yann LeCun},
  title = {{Reverse TDNN: An Architecture For Trajectory Generation}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {1991},
  url = {https://openreview.net/forum?id=S1WbsdbdbH},
}
Figure