Online conferences and remote work keep the scheduled talks and lose the hallway. People cannot see a conversation from across the room, hover at its edge, and drift into it. Reports from virtual events describe the result as "coldness" and the failure of forced breakout rooms (ECIR 2020 workshops: assessing the impact of going online). A natural repair for hybrid events is to let remote attendees perceive and join the physical conversational groups forming in a real hall. That requires knowing, live and privately, who is talking with whom. Today this is done offline. Sociometric badges log raw signals that are analyzed weeks later (Mining Face-to-Face Interaction Networks Using Sociometric Badges), and group detectors run on stored camera frames (F-Formation Detection).
We propose to infer conversational groups directly on cheap wearable badges. Each badge turns its microphone and accelerometer stream into a short string of bits: who seems to be speaking, and when. Badges then compare these strings with one another using two machine instructions, XOR and popcount. This is where we meet the hardware-software gap in binary neural networks, which keep failing to deliver their promised speedups in practice. We argue that for this task the bits are not a lossy compression of floating-point features. Speaking or not speaking is a binary social fact, and a binary code is what the badges would naturally exchange over a low-power radio. The engineering question then becomes where the real cost sits on a microcontroller, and whether the bitwise design actually wins once that cost is measured honestly.
Core Claim
Membership in a free-standing conversation (an "F-formation," the circle or arc people form when talking while standing) leaves a temporal signature in sound. Members take turns, rarely overlap for long, and hear correlated ambient audio, while people in different groups speak independently of one another. Our claim has two parts. First, binary voice-activity and motion codes of roughly one to two seconds, produced on the badge and compared pairwise in Hamming space, recover F-formation membership on the SALSA poster and cocktail-party sessions at least as well as the camera-based pipeline. That pipeline suffers 42–49° head-pose and 45–54° body-pose errors (SALSA). Second, on a GAP8-class microcontroller the end-to-end cost is set by the audio front-end and the full-precision first layer, not by the binary layers.
The second part has direct precedent. On GAP8, Sound Event Detection with Binary Neural Networks on Tightly Power-Constrained IoT Devices reports 4.6 GMAC/s on binary layers but only 1.5 GMAC/s once spectrogram preprocessing is included. A co-design that ignores the front-end therefore optimizes the wrong thing.
Relation to Prior Work
The motivating evidence comes from the social side. Wu et al. show that face-to-face cohesion predicts faster completion of complex tasks, with a one-standard-deviation increase in cohesion giving 9.5% faster completion. Large-Scale Analysis of New Employee Network Dynamics shows that remote hires stay structurally isolated. Towards a Social Virtual Reality Learning Environment in High Fidelity shows that distance-attenuated spatial audio lets virtual groups form without breakout rooms, but only on tethered workstations and with no sensing of physical people.
The task and its ground truth come from SALSA, which has synchronized badges, video, and F-formation annotations every 3 seconds. The evaluation protocol comes from GCFF, which clusters people from positions and orientations using graph cuts over a shared inner space and reports F1 of 0.89. GCFF needs orientation estimates that, from cameras, are badly noisy. Our design needs no cameras at all.
The method draws on the binary-computing side. From Binary Graph Neural Networks we reuse the idea of building neighbor graphs in Hamming space with XOR and popcount. Their measured gain on a Raspberry Pi was only 2x, against a theoretical 64x for the neighbor-search step. That gap warns us to report measured latency, not counted operations. From the GAP8 work we reuse memory tiling and the native popcount instruction. Espresso supplies the bit-packing layout. The cautionary literature, namely Reducing the Computational Cost of Deep Generative Models with Binary Neural Networks and Resource-Efficient Speech Quality Prediction, constrains our claims. Neither paper's runtime packs sub-byte values, so without custom kernels the theoretical savings vanish. We treat this as a hypothesis to be measured stage by stage.
Prior binary-network work evaluates on CAD models, digits, and environmental sounds. The F-formation work never runs on-device and never fuses audio timing. Neither line has asked whether a socially meaningful binary code makes the "binarization gap" moot.
Research Program
The system has three stages, all running on the badge.
The first stage is a small acoustic and motion front-end. A binary network takes a full-precision first layer over log-energy or Mel features and emits, every 100 ms, a self-speech bit, a nearby-speech bit, and a motion bit. Self-speech labels can be bootstrapped from the strong energy asymmetry of the badge's own wearer on SALSA audio, and checked against SALSA's speaker annotations wherever those are available.
The second stage packs the last $T$ frames into a code $b_i$ of length $3T$ bits. Badges in Bluetooth range broadcast their codes, and each badge computes a pairwise affinity:
$$s_{ij} = \mathrm{popcount}(b_i \oplus \pi(b_j)),$$
Here $\pi$ is a learned fixed bit permutation that lets turn-taking show up as agreement. Concretely, $\pi$ aligns my-speech bits with your nearby-speech bits.
The third stage turns these affinities, together with Bluetooth and infrared encounter evidence, into a graph. A tiny clustering step, either correlation clustering or GCFF's graph cut with affinities in place of geometric costs, splits the graph into groups.
We will profile every stage in cycles and joules on GAP8 and on an ARM Cortex-M4. The goal is to locate where the budget actually goes and to redesign that stage. For example, a fixed-point filterbank could replace the Mel spectrogram, or a learned 1-D binary front-end could read raw audio directly.
The application layer is a small hybrid pilot. Inferred group state drives distance-attenuated audio "windows" that let remote participants overhear and join physical groups, in the style of the High Fidelity system but fed by real sensing.
Baselines
The naive reference uses proximity thresholding: two people count as a group if their badges register Bluetooth proximity or an infrared encounter in the window, followed by connected components. It costs almost nothing and uses the same SALSA badges. Our method beats it if it raises F1 meaningfully, specifically in the dense Party session where proximity over-merges neighboring groups.
The strongest existing method is GCFF from F-Formation Detection, which reports combined F1 of 0.89 against 0.83 for CTCG at tolerance 2/3. Those figures come from its own benchmark suite. We found no reported GCFF figure on SALSA on Lacuna. We will run GCFF on SALSA twice, once with ground-truth positions and orientations as an upper reference and once with automatically estimated poses as the realistic camera pipeline. We will use the same frames, the same GTM tolerance of 2/3, and GCFF's default parameters plus an equal-sized grid search. Beating the automatic-pose GCFF with no cameras would be a clear win. Approaching the ground-truth GCFF would be strong.
Two ablations isolate the key ingredient. The first is a float version that computes the same temporal features in fixed-point and full precision and uses cross-correlation, which tests whether binarization costs accuracy. The second is an INT8 front-end built with standard microcontroller kernels on the Cortex-M4. Both get the same training data and tuning budget as our method, and we report F1, latency, energy, and bytes sent over the radio.
For hardware, the reference point is the GAP8 result of 77.9% accuracy at 511 ms per 3.2 s window in 262 kB. To be useful in real time, our full pipeline must refresh group state within one second, inside the same memory limit.
Why not just…?
Why not just run GCFF on badge proximity and skip learning entirely? This is the most serious threat. Infrared encounters are directional, and Bluetooth roughly encodes distance. In sparse poster sessions this may already work well, and if so we will say so. Our bet is the crowded Party session, where SALSA reports rising tracking failures and where adjacent groups are physically close but acoustically independent. If adding audio-timing codes does not raise Party F1 over the proximity-only GCFF, the acoustic stage is not worth building.
Why not just use INT8 instead of 1-bit codes? Compute for a three-bit-per-frame code is trivial at any precision, so INT8 may match us on the front-end. The GAP8 results suggest binary layers are where the memory and energy savings concentrate: weights dropped from 815 kB to 58 kB, and the BNN ran with 51.1x the energy efficiency of an ARM baseline. Still, the more honest argument for bits is radio cost. Badges broadcast continuously, and packed bits shrink every exchange. The INT8 ablation will show whether the binary advantage is in computation, in communication, or nowhere.
Why not just stream audio to a server? The badge studies were acceptable to participants because raw speech never left the device (Wu et al.). A server pipeline is the natural accuracy upper bound, so we will run the full-precision model offline as that bound. It is not a deployable design for continuous use in a workplace or conference hall.
Proposed Experiments and Decisive Checks
All accuracy experiments use SALSA, with leave-one-session-out and leave-participants-out splits. Metrics are F1 under GTM tolerance 2/3, group-change detection delay, and per-pair affinity AUC. Hardware experiments deploy on GAP8 and on an STM32F469I Cortex-M4 as in the GAP8 study. We report measured cycles per stage, energy per refresh, peak SRAM, and radio bytes per second, and never counted operations alone.
Three results decide the project. The first is whether binary audio-timing codes plus proximity beat proximity-only GCFF in the Party session. The second is whether the binary codes stay within a few F1 points of the float ablation. The third is whether profiling confirms that the front-end dominates cost, and whether our redesign of it brings the full pipeline under one second and 512 kB.
A small hybrid pilot will measure how often remote participants join physical groups through audio windows compared with a link-based breakout baseline. It uses presence questions in the style of the High Fidelity study. Presence is a secondary outcome and does not decide the project.
References
- ECIR 2020 workshops: assessing the impact of going online
- Mining Face-to-Face Interaction Networks using Sociometric Badges: Predicting Productivity in an IT Configuration Task
- F-Formation Detection: Individuating Free-Standing Conversational Groups in Images
- SALSA: A Novel Dataset for Multimodal Group Behavior Analysis
- Sound event detection with binary neural networks on tightly power-constrained IoT devices
- Large-Scale Analysis of New Employee Network Dynamics
- Towards a social virtual reality learning environment in high fidelity
- Binary Graph Neural Networks
- Espresso: Efficient Forward Propagation for Binary Deep Neural Networks
- Reducing the Computational Cost of Deep Generative Models with Binary Neural Networks
- Resource-Efficient Speech Quality Prediction through Quantization Aware Training and Binary Activation Maps