A Winner-Take-All Circuit with Controllable Soft Max Property

Shih-Chii LiuNeurIPS 1999 · OpenReview

This paper, authored by Shih-Chii Liu from the Institute for Neuroinformatics (ETH/UNIZ), presents a clever and highly efficient modification to a classic neuromorphic circuit. By adding just one transistor and a reference voltage to a standard "Winner-Take-All" (WTA) design, Liu creates a system that can smoothly transition between picking a single winner and performing "soft-max" normalization. This tunable behavior is critical for mimicking biological vision systems in hardware.

The Challenge: Beyond Binary Competition

In the world of neuromorphic engineering—building hardware that mimics the brain—the Winner-Take-All (WTA) circuit is a fundamental building block. Standard WTA circuits, such as the famous Lazzaro circuit, are "hard" competitors: they identify the single strongest input and completely suppress all others.

However, biological systems are rarely that binary. For example, the wide-field direction-selective cells in a fly's visual system perform normalization. Rather than just picking one "winning" pixel, they scale their output based on the relative strength of all inputs. This ensures the signal remains consistent and codes for the strength of the inputs, regardless of how many inputs are active. Before this paper, implementing this kind of controllable "soft" competition in a compact, tunable hardware format was a significant challenge.

The Circuit Innovation

The author proposes a recurrent inhibitory network consisting of excitatory neurons and a single global inhibitory neuron. The core innovation is a subtle but powerful modification of the traditional Lazzaro WTA cell.

Modified WTA Circuit

Each excitatory neuron acts as a linear threshold unit. To achieve the tunable soft-max property, the author adds a diode-connected transistor ($M_2$) and an external bias voltage ($V_\alpha$) to the standard circuit.

This $V_\alpha$ acts as a master gain controller:

  • Soft-Max Regime: When $V_\alpha$ is low, the output nodes have low impedance (low gain). This allows multiple neurons to remain active simultaneously, representing their relative input strengths without entirely suppressing their neighbors.
  • Hard WTA Regime: When $V_\alpha$ is high, the added transistor $M_2$ becomes negligible. The circuit effectively reverts to a standard Lazzaro WTA. The output nodes shift to a high-impedance (high gain) state, causing the circuit to aggressively suppress all but the single strongest input.

The paper explores two variants for generating the inhibitory signal: one using a fixed current source (creating a linear combination of states) and another using a diode-connected transistor (resulting in a square-root dependence on neuron states).

Experimental Validation on Silicon

Because this is a hardware design paper, the "dataset" consists of controlled electrical inputs applied to a physical, fabricated silicon chip. The author tested a chip containing 20 neurons fabricated using a 1.2$\mu$m CMOS technology, measuring output currents and common-node voltages while varying the number of active inputs and the control voltage $V_\alpha$.

Key Insights and Results

  • Tunable Competition: The most significant insight is that the transition from soft normalization to hard competition does not require a different architecture—it is simply a shift in the circuit's gain state. Soft-max is a low-gain/low-impedance state, while hard WTA is a high-gain/high-impedance state where small input differences are immediately amplified.
  • Biological Realism (Normalization): Experimental data from the 20-neuron chip showed that as the number of active inputs increased, the inhibitory signal ($V_T$) saturated. This confirmed the normalization property, successfully modeling the "aggregation property" of fly visual cells.
  • Controllable Splitting: In tests comparing two inputs, the author demonstrated the smooth transition between operating modes.
Soft-max to Hard WTA Transition

As shown in the experimental data, at low $V_\alpha$ (e.g., 0.4V or 0.5V), both outputs remain active over a wide range of input differences, exhibiting a linear differential input range of approximately 100mV. As $V_\alpha$ increases past a critical threshold (around 0.6V to 0.7V), the transition sharpens dramatically, and one output quickly drops to zero, leaving a single winner.

Limitations and Future Work

While the circuit is highly compact—requiring very little extra "silicon real estate" per neuron—there are practical limitations to consider for large-scale deployment:

  • Transistor Mismatch: The author notes that "backgate effects" and natural transistor variations during fabrication can influence the critical value of $V_\alpha$ where the WTA behavior kicks in. In larger arrays, device mismatch might make uniform tuning across the entire chip difficult.
  • Scaling Dynamics: While a 20-neuron array successfully demonstrates the principle, real-world visual or cortical models require thousands of neurons. The paper does not extensively detail how the global inhibitory signal scales or if parasitic capacitance slows down the network as it grows significantly larger.

Ultimately, this paper provides a highly versatile, compact building block for neuromorphic engineering. Future work would naturally involve integrating this tunable circuit into a functional, large-scale sensory system, such as a silicon retina or a hardware-based neural network.

Summary updated

Abstract

I describe a silicon network consisting of a group of excitatory neu(cid:173) rons and a global inhibitory neuron. The output of the inhibitory neuron is normalized with respect to the input strengths. This out(cid:173) put models the normalization property of the wide-field direction(cid:173) selective cells in the fly visual system. This normalizing property is also useful in any system where we wish the output signal to code only the strength of the inputs, and not be dependent on the num(cid:173) ber of inputs. The circuitry in each neuron is equivalent to that in Lazzaro's winner-take-all (WTA) circuit with one additional tran(cid:173) sistor and a voltage reference. Just as in Lazzaro's circuit, the outputs of the excitatory neurons code the neuron with the largest input. The difference here is that multiple winners can be chosen. By varying the voltage reference of the neuron, the network can transition between a soft-max behavior and a hard WTA behav(cid:173) ior. I show results from a fabricated chip of 20 neurons in a 1.2J.Lm CMOS technology.
BibTeX
@inproceedings{liu1999winner,
  author = {Shih-Chii Liu},
  title = {{A Winner-Take-All Circuit with Controllable Soft Max Property}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {1999},
  url = {https://openreview.net/forum?id=BkWoiL-OZH},
}
Figure