Studies of brains in natural behavior pair a neural recorder with cameras. Examples are a wireless EEG cap worn while someone cooks, or an intracranial array in a patient who moves freely around a hospital room. Every such study must line up the neural signal and the video in time. Today this alignment usually relies on software clocks and sliding windows, which drift and jitter by unknown amounts. The proposal is to make the room itself carry the time. A ceiling LED or a display backlight would emit short light pulses that encode a running timestamp, at rates too fast for people to see. Every camera in the room, and a small light sensor on each EEG headset, would record the same pulses. Decoding them afterward puts all streams on one shared clock.
The second half of the proposal matters as much as the first. "Too fast to see" has so far been checked by asking people whether they notice anything, or by computing image-quality scores. A neuroscience experiment needs a stricter test: the code must leave no trace in the brain activity being recorded. We therefore treat neural invisibility as a measurable design constraint. We would measure it with the EEG the study already collects, and only adopt code designs that pass it.
Core Claim
Sub-perceptual optical codes can deliver sub-millisecond, drift-free alignment across many wireless neural recorders and ordinary rolling-shutter cameras at once, with no wires to the subject. This holds only if the code keeps its mean luminance constant at every timescale the visual system can follow. Codes that are invisible to observers but let average brightness vary at the packet rate will produce detectable evoked responses in the EEG. We predict this, and it is the specific failure mode that existing imperceptibility metrics miss.
Two things are new. First, nobody has used ambient sub-perceptual light as a shared timebase for neural and behavioral streams. The research file's survey of the intersection finds no study connecting these directions. Second, the field validates imperceptibility by perception and image quality. We validate it by the absence of a neural response.
Why this intersection, and why this gap rather than head tracking
The obvious idea here is head tracking for mobile PET using blinking LED markers and an event camera, which is a sensor that reports per-pixel brightness changes with microsecond timing. We decided against it. In Method for robotic motion compensation during PET imaging of mobile subjects, a robot arm carries a mock scanner ring, and six string encoders measure head-to-ring motion at 1 kHz with 0.5 mm static accuracy. The 80 ms latency and 5.8 mm walking error come from the robot control loop, not from the sensing. Faster optical sensing would not fix the bottleneck. Infrared active markers would also make the "sub-perceptual visible light" ingredient unnecessary.
Synchronization is different. Here the perceptual property is the point: the signal must be inside the subject's visual scene, seen by consumer cameras that block infrared, and harmless to the brain being recorded.
The motivating gap comes from two datasets. EgoBrain recorded 61 hours of 30 Hz helmet video and 256 Hz wireless EEG from 40 people. It aligned the streams with sliding windows to absorb drift and sub-second jitter. Its reported fusion gain is modest: 66.70% versus 63.40% vision-only. That leaves open whether misalignment is eating part of the neural contribution. AJILE12 aligned 1,280 hours of 500 Hz ECoG with clinical room video and 2D pose for 6,931 wrist movements. Movement-onset analyses there are only as sharp as that alignment. Group recordings such as the wearable group-synchronized EEG system raise the stakes, because many headsets must share one clock.
The mechanism comes from the optical signaling literature. Bhardwaj et al. send data with ultrashort pulse-position modulation (bits encoded in when a brief pulse occurs within a slot) through room lighting, without degrading perceived illumination. Guri et al. switch indicator LEDs at hundreds of microseconds without observers noticing. Zhou et al. show that light drops shorter than 40 ms go undetected. For displays, KOMURO et al. hide patterns in blue-channel temporal modulation. Revelio embeds 48-bit codes in video with VMAF above 95, decodes them through rolling-shutter distortion, and reaches zero bit errors up to 2.0 m.
All of these judge success by bit error and by human perception or image quality. None asks whether a recorded brain responds to the code. That omission is acceptable for advertising, but it is a serious risk when the receiver sits beside an EEG amplifier pointed at visual cortex.
The constraining evidence points the same way. Fast neural population dynamics in primate V1 captured by a genetically-encoded voltage indicator shows that early visual cortex carries millisecond-scale structure. Fast luminance structure that observers cannot report may still drive it. The hemispheric dome for head-fixed mice reminds us that "invisible" depends on the species: mouse vision runs on UV and green opsins. We keep the main program in humans and treat animal transfer as a later extension.
Research Program
We would build a broadcast unit that drives one or more room LED fixtures, or a display backlight, from a GPS- or crystal-disciplined clock. The unit emits a pulse-position timecode: one packet per second carrying an absolute counter, with error-correcting coding in the spirit of Revelio's Reed-Solomon layer. Two families of code are built.
In the naive family, average luminance dips slightly whenever a packet is sent. This is typical of off-the-shelf visible-light communication.
In the balanced family, the number of pulses is held constant in every window longer than about 1 ms. Average brightness is then flat at every frequency below a few hundred hertz, where cortex and pupil can respond. We also add a chromatic variant that alternates along the blue-yellow axis, following KOMURO et al. and the complementary-color markers of Yamamoto et al.. This variant is for display-based setups.
There are two kinds of receiver. The first is the cameras themselves. A rolling-shutter sensor reads its rows one after another, so a 30 fps 1080-line camera samples the scene roughly every 30 µs down the frame (33 ms divided by about 1,000 rows). Locating a pulse edge to within a few rows should therefore give sub-millisecond timing from an ordinary camera. We reuse Revelio's idea of differencing frames to isolate the modulation from scene content. The second receiver is a small photodiode with a microcontroller clipped to each EEG headset. It decodes packets at full speed and writes event markers into the amplifier's trigger input. Its fixed latency is calibrated once on the bench.
A third component is analytic. We would measure the neural and physiological footprint of each code using the experiment's own EEG and an eye-tracker for pupil size. We then state formal equivalence bounds on how much the code can change any evoked response.
Baselines
The do-nothing reference is the approach EgoBrain uses: device timestamps and software clocks, with sliding-window segmentation to absorb drift. We would run it on exactly the recordings where our timecode is present, by simply ignoring the code. This makes the comparison paired. No reported synchronization-error figure for EgoBrain or AJILE12 was found on Lacuna, so we will measure this baseline's error ourselves against ground truth.
The strongest practical competitor is network-based synchronization of the Lab Streaming Layer kind, plus per-device clock-offset estimation. It is free, widely used, and the default a reviewer will propose. It gets the same hardware, the same session lengths of up to several hours, and its recommended configuration.
The strongest hardware competitor is a wired visible or infrared LED mounted at the edge of each camera's view and driven by the same TTL line that marks the EEG. This is the lab-standard approach and the most accurate one available. We use it as ground truth in validation sessions. Because it is the reference, it cannot be "beaten" on accuracy. Beating it means matching it to within 0.5 ms median error while needing no wire per device and scaling to many subjects and cameras.
Our method beats the software baselines if it reaches sub-millisecond median error with no drift growth over multi-hour sessions, where software clocks drift.
Two ablations isolate the key ingredients. Comparing balanced and naive codes isolates the constant-mean-luminance design, and its main outcome is neural leakage. Comparing row-level and frame-level decoding of the same camera video isolates the rolling-shutter contribution to timing precision.
Why not just…?
Why not just use Lab Streaming Layer or network time? For tethered lab hardware this may be good enough, and we will report so honestly if the data show it. The open question concerns wireless amplifiers and consumer cameras with their own internal buffering. That is the regime where EgoBrain needed sliding windows to cope with drift and sub-second jitter. The paired design decides the question: one recording is aligned three ways and scored against the wired reference.
Why not just use infrared LEDs, which nobody can see? This is a real threat. For room cameras without infrared-cut filters, with human subjects, and without optical neural recording, infrared may work equally well. We will include an infrared arm on those cameras. The case for visible light rests on two things. The egocentric cameras used in practice, such as EgoBrain's GoPro, filter infrared. And infrared illumination is a known nuisance in optical neural recording setups, as the research file notes for fluorescence pathways. If the infrared arm matches our method on every camera that can see it, we will narrow our claim to filtered-camera and optical-recording settings.
Why not just align the streams after the fact by cross-correlating head motion? One could correlate head motion measured from video with motion artifacts or headset accelerometer signals. This needs no new hardware, and it is a serious competitor whenever the subject moves vigorously. It fails during stillness, which is exactly when cognitive EEG analyses happen, and its precision depends on how much motion there is. We will run it on the same sessions and report error as a function of motion level. We expect it to beat software clocks during active periods and to fall apart in still periods.
Why not just blink a visible LED once a minute, the clapperboard approach? This works for offset but not for drift, unless the blinks are frequent. Frequent visible blinks are salient visual events that evoke responses and pull attention. Our naive-code and visible-contrast ablations measure exactly this cost.
Proposed Experiments
The first experiment is bench timing. A wired TTL reference marks the EEG amplifier and a photodiode-verified LED in view of each camera. We would record 3-hour sessions with four consumer cameras (including a GoPro-class helmet camera and a clinical room camera) and two wireless EEG headsets. The metrics are median and 95th-percentile absolute timing error against the reference, error growth per hour, and the fraction of seconds with a successfully decoded packet. Decoding is also measured under the conditions Revelio found hard, namely dark scenes, oblique angles, and distance.
The second experiment, the decisive neural-invisibility test, is a within-subject study of about 24 adults. Each person sits under the room lighting while performing a passive viewing task and an oddball task, during blocks of sham light (identical average brightness, no code), naive code, balanced code, and a deliberately visible low-frequency control. Outcomes are taken from scalp EEG: power at the packet rate, at its harmonics, and at any beat with display refresh; changes in oddball evoked responses; pupil size; and two-alternative forced-choice detection of whether the code is present.
The prediction is specific. The balanced code should be statistically equivalent to sham on every measure. The naive code should be undetectable in the forced-choice task yet show a measurable packet-rate EEG response. If the balanced code also leaves an EEG footprint, the central premise fails and the project reports that result. If the naive code leaves none, the constant-mean design is unnecessary, which simplifies deployment.
The third experiment asks whether alignment matters downstream. Before collecting new data, we inject controlled timing offsets of 0 to 500 ms into the public EgoBrain recordings. We then measure how Brain-TIM's reported 66.70% cross-subject accuracy degrades. We do the same with movement-onset-locked ECoG averages in AJILE12 against its verified wrist events. This tells us how much synchronization precision these analyses actually need. If both are flat out to hundreds of milliseconds, we would say plainly that coarse action decoding does not need our method. We would then redirect the claim to analyses that are sensitive at the millisecond scale, such as movement-onset-locked responses and cross-subject coupling in group EEG.
Finally, a small naturalistic recording repeats EgoBrain's protocol in an instrumented room. Fusion accuracy and onset-locked averages are compared under each synchronization method, all on the same data.
References
- Method for robotic motion compensation during PET imaging of mobile subjects
- EgoBrain: Synergizing Minds and Eyes For Human Action Understanding
- AJILE12: Long-term naturalistic human intracranial neural recordings and pose
- A wearable group-synchronized EEG system for multi-subject brain–computer interfaces
- HotMobile 2016
- CTRL-ALT-LED: Leaking Data from Air-Gapped Computers via Keyboard LEDs
- Optical Exfiltration of Data via Keyboard LED Status Indicators to IP Cameras
- Mimetic Code Using Successive Additive Color Mixture
- Revelio: A Real-World Screen-Camera Communication System with Visually Imperceptible Data Embedding
- Fast neural population dynamics in primate V1 captured by a genetically-encoded voltage indicator
- A hemispheric dome setup for naturalistic visual stimulation in head-fixed mice
- Imperceptible On-Screen Markers for Mobile Interaction on Public Large Displays