Active speaker detection is the task of deciding, frame by frame, which visible face in a video is producing the speech on the audio track. The best systems learned that faces cannot be judged one at a time. Instead, all visible faces must compete for the single shared audio stream, so that the true speaker's lip motion suppresses a bystander who happens to be chewing or laughing. That competition is expensive. It is usually implemented as dense attention or graph message passing that compares every face with every other face and with the audio. The cost grows with the number of faces, which is exactly the crowded regime where competition helps most.
Spiking neural networks are networks whose units exchange sparse binary pulses rather than dense floating-point values. Their energy use scales with how many pulses are fired. This proposal builds an active speaker detector in which the competition among faces is carried out by the spikes themselves. Each face track drives a small population of spiking units. A single pooled inhibitory unit sums the activity of all faces and feeds inhibition back to each one. Faces that lose the competition stop firing, so the network's activity, and therefore its energy, should track the number of people actually speaking rather than the number of faces on screen. The bet is that one mechanism can supply both the relational accuracy of competitive models and the low cost that edge devices such as hearing aids and conference webcams need.
Core Claim
The claim is that competitive exclusion among candidate faces can be implemented as learned, pooled lateral inhibition between spiking candidate units, with three consequences. First, the cost of competition is linear in the number of faces rather than quadratic. Second, silenced losers contribute almost no further computation, so spike count stays nearly flat as bystanders are added at a fixed number of speakers. Third, this spiking competition recovers most of the crowded-scene accuracy gain that dense relational models obtain over single-candidate models.
The second consequence is the one that distinguishes the idea from simply shrinking a network. It is also directly measurable, because the prediction is that spike count follows the number of active speakers, not the number of faces.
The core update for candidate unit $k$ at timestep $t$ is a standard leaky integrate-and-fire neuron with one added term:
$$U_k[t] = \lambda U_k[t-1] + I_k[t] - V_{th}\, s_k[t-1] - w_{inh}\, g\Big(\sum_{j} s_j[t-1]\Big)$$
The symbols are defined as follows:
- $U_k$ is the membrane potential, the unit's running internal state.
- $\lambda$ is the leak factor.
- $I_k[t]$ is the input current from that face's audio-visual features.
- $V_{th}$ is the firing threshold.
- $s_k[t] \in \{0,1\}$ is the emitted spike.
- $w_{inh}$ is a learned inhibition strength.
- $g$ is the output of a single inhibitory neuron that integrates the spikes of all faces.
Because the inhibition passes through one shared sum rather than a pairwise matrix, adding a face adds one term to that sum. With a learned, adaptive threshold, the circuit acts as a soft k-winners-take-all rather than a hard argmax. This matters because overlapping speech means that sometimes two faces should both win.
Relation to Prior Work
Three competitive detectors motivate the design. MAAS builds a per-frame graph with one audio node and one node per face, and uses graph convolutions to let the true speaker suppress distractors. It reports large gains for scenes with three or more people. UniCon adds permutation-equivariant relational scoring and an explicit non-active-speaker suppression module, gaining 14.7 mAP in scenes with at least three candidates. LoCoNet separates long-range within-speaker attention from short-range between-speaker convolution. All three establish that competition is what fixes crowded scenes. All three implement it with dense floating-point operations over every candidate at every frame. We keep the functional role of UniCon's suppression module and replace its dense implementation with spiking inhibition whose cost falls as losers go silent.
The single-candidate lightweight detector of Liao et al. constrains the proposal. It factorizes 3D convolutions and uses a small bidirectional GRU, reaching 94.1% mAP on AVA-ActiveSpeaker with 1.0M parameters and 0.6G FLOPs. This shows that per-face models are already strong and cheap on movie footage, so any spiking system must justify itself on crowded and in-the-wild data, not on AVA overall. UniTalk supplies that harder regime: models near 95% on AVA fall to roughly 83% mAP on its in-the-wild data and about 77% on its mixed-difficulty subset.
The closest bio-inspired detector is Assunção et al.'s superior colliculus fusion. It uses Mexican-hat lateral excitation and inhibition between audio and visual populations, but it is continuous rather than spiking, uses hand-tuned lateral weights, and scores faces in isolation. We move the lateral inhibition from within one face's fusion to between faces, make it spiking, and learn it end to end.
On the spiking side, audio-visual spiking networks exist, but they target whole-video classification: STFT and MDST++ for zero-shot action recognition, and S-CMRL for single-speaker emotion recognition. None of them assigns speech to one of several faces. We reuse several of their techniques:
- MDST++'s idea of converting frames to change events, so static faces generate no input.
- S-CMRL's observation that a noisy modality can poison a clean one in spiking fusion. This makes it a named risk for noisy UniTalk audio.
- SpiLiFormer, which uses lateral inhibition inside spiking attention to suppress background tokens in image classification. Its success is evidence that inhibition and spiking training coexist, though it never applies inhibition across competing objects.
- DeepEISNN, which replaces batch normalization with excitatory-inhibitory circuits. We adopt its approach because batch statistics are ill-defined when the number of faces per frame varies.
S²M-Former decodes the attended talker from EEG at 0.0779 mJ per inference. It shows that fully spiking multi-talker attention can beat its non-spiking twin, but it works on brain signals, not video.
Research Program
The system has four stages.
Audio encoding. Log-mel frames are delta-encoded into spikes, so silence produces almost no activity.
Visual encoding. Face crops from an off-the-shelf tracker are converted to change events by thresholded frame differencing, in the spirit of MDST++. This keeps the lip and jaw motion and drops static texture. A small spiking encoder, trained directly with surrogate gradients, processes each face's events. Its architecture mirrors Liao et al.'s factorized spatial and temporal kernels, so that the only intended difference from that baseline is spiking versus dense.
Per-face fusion. Each face's spikes are fused with the shared audio spikes through a spiking cross-modal residual in the style of S-CMRL, producing the input current $I_k$.
Competition and identity tracking. Candidate units compete through the pooled inhibitory neuron above. A per-identity recurrent leak across frames provides the temporal smoothing that MAAS obtains from its temporal graph. Speaking scores are read from candidate firing rates over a short window.
Training uses the standard frame-level speaking labels plus a small spike-count penalty. We sweep the penalty to trace an accuracy-energy curve rather than report a single point.
The analysis program goes beyond accuracy. We will measure how total spikes and synaptic operations scale with the number of faces at a fixed number of speakers, and how quickly losers are silenced after speech onset. The latency question is whether inhibition needs many timesteps to settle, which would erase the savings. We will also measure how often the soft winner-take-all correctly lets two faces win during overlapped speech.
Energy is estimated from counted accumulate and multiply-accumulate operations using per-operation costs, following the counting methodology of Barchid et al.. That study reported 87.94 mJ against 11,129.44 mJ for an equivalent ANN on localization. Such estimates are known to be generous to spiking models when the comparison is against FP32. For that reason every energy comparison below also includes an 8-bit integer ANN, and memory access for membrane state is counted rather than ignored. If a Loihi-class chip is accessible, a small model will be deployed to check the estimates, as Tang et al. did for control. The claims do not depend on hardware access.
Baselines
The naive reference is an independent spiking detector: our architecture with $w_{inh}$ fixed to zero, so each face is scored alone. This isolates what competition adds on top of spiking. A secondary naive reference is Liao et al.'s single-candidate ANN run as-is.
The strongest efficiency competitor is the lightweight detector of Liao et al.. It reports 94.1% mAP on AVA-ActiveSpeaker at 1.0M parameters and 0.6G FLOPs, and 81.1% average F1 zero-shot on Columbia. It is the direct answer to "why spike at all." We retrain it from its published recipe on the same AVA training split with the same face tracks and the same tuning budget, and report it at FP32 and at 8-bit integer precision.
The strongest accuracy competitor is LoCoNet. It reports 95.2% mAP on AVA-ActiveSpeaker, 97.2% on Talkies and 68.4% on Ego4D, with a 3.0 mAP margin on frames with three or more faces. UniCon, at 92.0% on AVA, is included because its suppression module is the functional analog of our inhibition. We do not expect to beat LoCoNet's accuracy. Success against it is defined as matching its crowded-scene gain over Liao et al. to within half, at a small fraction of its operation-count energy. No reported result was found on Lacuna for per-inference energy of LoCoNet, UniCon or Liao et al., so all energies will be measured by us under one counting protocol.
Two ablations isolate the key ingredient. The first replaces pooled spiking inhibition with dense spiking all-to-all attention between faces. The prediction is similar accuracy but spike count growing with the number of faces. The second is described in the next section.
Beating the field means one of two outcomes. The spiking model could lie strictly above both Liao et al. variants on the curve of crowded-scene mAP against estimated energy. Or it could show near-flat energy growth with bystander count where every baseline grows linearly or worse.
Why not just…?
Why not just add a softmax over faces to the lightweight ANN? This is the most serious threat. A per-frame softmax across the scores of Liao et al.'s model gives cheap, linear-cost competition, and it may recover much of UniCon's crowded-scene gain. We treat it as a first-class ablation: Liao et al. plus a learned softmax and a sigmoid top-k readout, trained identically.
If it matches our crowded-scene accuracy, then competition alone does not need spikes. The contribution then narrows to energy, and in particular to the claim that losers stop computing. A softmax head still runs the full dense encoder on every bystander at every frame. Inhibition that feeds back into the encoder's candidate layers can silence them upstream. The per-face energy measurement settles which account holds. If feedback inhibition does not reduce encoder spikes for losing faces, the idea collapses to "a spiking Liao et al. with a softmax." We would report it as such.
Why not just quantize and prune an ANN? An 8-bit, pruned version of Liao et al. may already sit at energy levels comparable to a spiking model. Published spiking-versus-ANN energy ratios, such as Barchid et al.'s 44x to 126x, are measured against FP32. For this reason the integer ANN is a mandatory baseline, not an afterthought.
The structural argument for spikes is input-dependent silence. A quantized ANN does the same work on a static bystander as on a speaker, while event encoding plus inhibition should not. The bystander-scaling experiment is where this either shows up or does not.
Why not just convert a trained LoCoNet to spikes? Conversion typically needs many timesteps to approximate continuous activations. It would inherit dense all-to-all attention, whose spike count grows with the number of faces. This is exactly the tension between global relational context and sparsity that motivates the proposal. We will run a converted-attention variant through the dense-spiking-attention ablation to show the scaling directly rather than assert it.
Proposed Experiments
All models train on AVA-ActiveSpeaker. We evaluate on the AVA validation set, with its small-face and three-or-more-face breakdowns, and on Talkies. Zero-shot evaluation covers Columbia and the UniTalk test set, including its crowded-scenes, noisy-backgrounds and mixed-difficulty subsets. For UniTalk, the relevant reference is that an AVA-trained model drops to 77.5 mAP zero-shot. A second round fine-tunes on UniTalk training data. Metrics are mAP, per-inference synaptic-operation energy, spikes per face per second, and the number of timesteps needed for scores to stabilize.
Four checks decide the project.
Bystander scaling. We take AVA and UniTalk clips with exactly one speaker and bin them by face count, from one to five or more. We plot energy per frame against face count for our model, the dense-spiking-attention ablation, Liao et al. and LoCoNet. The hypothesis predicts a near-flat curve for ours and linear or steeper curves for the others. A clearly linear curve for our model falsifies the central claim.
Crowded-scene accuracy. On three-or-more-face frames and on UniTalk crowded scenes, the pooled-inhibition model must beat its own zero-inhibition version and approach the softmax-augmented Liao et al. Failing to beat the zero-inhibition version means the circuit is not competing.
Overlap handling. On overlapped-speech segments annotated in UniTalk, the soft winner-take-all should retain recall for concurrent speakers where a hard softmax readout loses it.
Noise asymmetry. On UniTalk's noisy-backgrounds subset, we check whether noisy audio spikes poison competition, as S-CMRL's findings would predict. The failure to look for is inhibition driven by noise silencing the true speaker. We compare with and without the spiking residual fusion.
Together these tests show whether competition can be made cheap by letting it silence the computation it rules out. They also show, separately, whether spiking is necessary for that.
References
- MAAS: Multi-modal Assignation for Active Speaker Detection
- UniCon: Unified Context Network for Robust Active Speaker Detection
- LoCoNet: Long-Short Context Network for Active Speaker Detection
- A Light Weight Model for Active Speaker Detection
- UniTalk: Towards Universal Active Speaker Detection in Real World Scenarios
- Bio-Inspired Modality Fusion for Active Speaker Detection
- Spiking Tucker Fusion Transformer for Audio-Visual Zero-Shot Learning
- Multi-Timescale Motion-Decoupled Spiking Transformer for Audio-Visual Zero-Shot Learning
- Enhancing Audio-Visual Spiking Neural Networks through Semantic-Alignment and Cross-Modal Residual Learning
- SpiLiFormer: Enhancing Spiking Transformers with Lateral Inhibition
- Training Deep Normalization-Free Spiking Neural Networks with Lateral Inhibition
- S$^2$M-Former: Spiking Symmetric Mixing Branchformer for Brain Auditory Attention Detection
- Spiking neural networks for frame-based and event-based single object localization
- Reinforcement co-Learning of Deep and Spiking Neural Networks for Energy-Efficient Mapless Navigation with Neuromorphic Hardware