Electron microscopes can now film individual atoms in a single sheet of a layered semiconductor such as MoSe₂ while the electron beam knocks atoms out. The resulting movies show vacancies forming, merging and migrating. Analyzing them still relies on hand-built pipelines: blur the image, find bright spots, and link each spot to its nearest neighbor in the next frame. This note proposes a single neural network that, in one pass over each frame, finds every atomic column and outputs a short vector describing the column's local atomic environment. That environment can be intact lattice, a column missing one chalcogen atom, a metal vacancy, a vacancy cluster, and so on. The network is trained only on synthetic movies. These are made by relaxing defect structures with dispersion-corrected density functional theory and then computing the microscope image each structure would produce.
The method borrows from two fields. From multi-object tracking it takes joint detection and embedding: one network both locates objects and emits a descriptor per object for linking across frames. We change what that descriptor means. Pedestrian trackers learn a descriptor that tells individuals apart. Two molybdenum atoms are physically identical, so that goal makes no sense for atoms. The useful descriptor encodes local configuration. From quantum chemistry it takes dispersion-corrected density functional theory. Density functional theory (DFT) is the standard method for computing atomic structures from electronic structure. The "dispersion-corrected" variants add the weak long-range van der Waals attraction that the common approximations omit. Here DFT serves as the label generator, and we test directly whether the dispersion treatment matters for what the tracker learns.
Core Claim
The claim has three parts. First, a single-stage detector with an environment-supervised embedding head, trained only on DFT-relaxed and multislice-simulated defect movies, can localize atomic columns, classify vacancy configurations, and link columns through beam-induced hops on real STEM (scanning transmission electron microscopy) data. Second, it does better than both classical blob tracking and a pedestrian-style tracker fine-tuned on a small set of real labels. Third, the gain comes specifically from supervising the embedding with physical environment labels instead of instance identities.
The dispersion question is a secondary, testable claim, not an assumption. Our expectation is that relaxing structures with or without a dispersion correction makes little difference for an isolated monolayer but a measurable one for bilayers. We will report the result whichever way it falls.
Where this sits in prior work
Three papers motivate and define the method.
Towards Real-Time Multi-Object Tracking introduced the shared-backbone design that predicts boxes and identity embeddings together. It also showed that identity cross-entropy beat triplet loss and that the different losses need uncertainty-based weighting. We reuse both findings.
FairMOT moved to anchor-free center-point detection. A center point is a natural fit for an atomic column. FairMOT also showed that pre-training on single images gives large gains (+3.6 MOTA on MOT17), which motivates our synthetic pre-training.
DefectTrack carried this design to in-situ TEM of irradiated nickel, reaching 66.43% MOTA. It needed 20 weeks of manual annotation, and its embeddings carry no physical meaning. We remove the annotation cost and give the embedding a physical meaning.
The atom-tracking precedent is Computer Vision-aided Atom Tracking in STEM Imaging. It tracked 124 atoms over 49 frames of beam-irradiated MoSe₂ using Gaussian blur, Hessian blob detection, nearest-neighbor linking within a lattice cutoff, and intensity voting (Se columns appear about 8% brighter than Mo). That is the classical pipeline we must beat. Its fixed cutoff is the specific weakness we target: a hop longer than the cutoff breaks the track.
Deep Learning-Enabled STEM Imaging for Single-Molecule Identification in Zeolites supplies the simulation route. It used DFT configurations and abTEM multislice simulation to make synthetic training images for a two-stage Faster R-CNN detector. We extend that route from static molecules to dynamic defect sequences, and we use a single-stage tracker rather than a two-stage detector.
The physics side supplies the labels and constrains the design. Vacancy impacts on MX2 monolayers relaxed six vacancy types (X, 2X, 2X_s, M, MX₃, MX₆) in four TMDs (transition metal dichalcogenides) with PBE plus Grimme's D3 correction. It found large structural and electronic changes; for example, C₁₁ in MoS₂ drops from about 130 to about 78 N/m with an MX₃ vacancy. Those six types become our environment classes.
Stability of charged sulfur vacancies in MoS₂ showed two things relevant here. SCAN+rVV10, a functional that computes dispersion directly from the electron density, fixes PBE's interlayer errors. The −1 charged sulfur vacancy also undergoes a Jahn-Teller distortion, a symmetry-breaking relaxation. This means relaxed geometry can depend on charge state, so we include both charge states as data-generation variants.
Machine Learning Interatomic Potentials and Long-Range Physics warns that cheap short-range potentials miss dispersion. This is why we use DFT relaxations directly instead of a learned potential to generate intermediate frames.
Research Program
Data generation
We build a library of defect supercells for monolayer MoS₂ and MoSe₂ covering the six vacancy classes, pairs of interacting vacancies, and the neutral and −1 sulfur vacancy. A smaller set of 2H-stacked bilayers is added. Each structure is relaxed three ways: plain PBE, PBE-D3 with Becke-Johnson damping, and SCAN+rVV10.
Movies are assembled from these relaxed snapshots. Each movie follows a scripted sequence: an atom is removed, the neighboring atoms relax, a vacancy hops to an adjacent site, and two vacancies merge. Simple linear interpolation between relaxed endpoints stands in for the true path, and we state plainly that this is a simplification.
Each frame is passed through abTEM, the multislice image simulator, to produce an annular dark-field STEM image. The simulated images then receive realistic imperfections: Poisson dose noise, scan distortion, probe blur, and sample drift. Every atomic column in every frame carries three labels: its true position, a persistent track ID, and an environment class.
Model
The backbone is high-resolution, as in DefectTrack, because atomic columns are only a few pixels across. On top of it sit two heads. A center-heatmap head marks where each column is. A second head produces, at each detected center, a low-dimensional embedding vector.
The embedding is trained with two losses:
$$\mathcal{L}_{emb} = \mathcal{L}_{env} + \lambda\, \mathcal{L}_{link},$$
Here $\mathcal{L}_{env}$ is cross-entropy over environment classes, which takes the place of JDE's per-person identity classes. $\mathcal{L}_{link}$ is a contrastive loss that pulls together the embeddings of the same column in adjacent frames and pushes apart those of neighboring columns. Because a column's neighborhood differs from its neighbors' neighborhoods, this loss can learn to tell them apart even though the atoms themselves are identical. The weight $\lambda$ is tuned against the detection loss with JDE's uncertainty weighting.
Linking frames
Frame-to-frame matching first fits a lattice-registered motion model, absorbing drift with an affine transform. It then solves a Hungarian assignment whose cost combines distance after registration with embedding distance. The embedding term is what lets the tracker follow a column that has hopped beyond the lattice cutoff.
Predicted vacancy classes are then mapped to their DFT formation energies. This yields energy-annotated defect trajectories, so we can check whether the observed sequence of events follows the computed energy ordering.
Baselines
The naive reference is static lattice assignment. We fit an ideal hexagonal lattice to the first frame and assign every later bright spot to its nearest lattice site. Any method that does not beat this on hop events adds nothing.
The classical baseline is the pipeline of Computer Vision-aided Atom Tracking, reimplemented and run on identical inputs, with its cutoff and blur scales tuned on our validation split. On the SMC'17 dataset its paper reports only that it tracked "124 unique atoms" over 49 frames. No reported MOTA, IDF1 or HOTA was found on Lacuna for that pipeline, so we will measure them ourselves. Beating it means higher HOTA and higher recall of hop events on the annotated test frames.
The two-stage baseline keeps detection and classification separate. A CenterNet-style detector feeds a separate small classifier that sees a patch around each column, followed by the same lattice-registered Hungarian linker. It gets the same synthetic data and the same tuning budget in GPU-hours. This baseline avoids the conflict between localization and discrimination identified in RelationTrack.
The strongest existing learned method is FairMOT fine-tuned on a small hand-annotated set of real frames, which is how DefectTrack was built. Fine-tuning matters a great deal on new domains: SoccerNet-Tracking reports FairMOT rising from 43.9 to 57.8 HOTA after fine-tuning. We give this baseline the same real annotations we hold out for our own optional fine-tuning. DefectTrack's 66.43% MOTA was measured on nickel TEM data and is not directly comparable, so we cite it only as a sense of scale.
The key ablation keeps our architecture and data unchanged but trains the embedding with JDE's per-instance identity cross-entropy instead of environment labels. A second ablation varies only the functional used to relax the training structures: PBE, PBE-D3, or SCAN+rVV10.
Our method counts as winning if it beats the fine-tuned FairMOT on hop-event recall and on defect-class macro-F1 while using no real labels. It should also beat the identity-embedding ablation by a margin larger than the spread across random seeds.
Why not just…?
Why not just use the classical blob-and-nearest-neighbor pipeline? It works on stable lattice and is cheap. Its fixed cutoff, however, breaks tracks whenever a hop exceeds it, and closely spaced columns merge into one blob, a problem the original paper patched with nested Hessian routines and voting heuristics. Our hop-event split measures exactly these failures. If the classical pipeline matches us there, the learned model is unjustified, and we will say so.
Why not detect with a network, classify each column separately, and link by lattice geometry? This is the real threat. In a crystal, association is mostly registration, so a learned linking embedding may contribute little. The two-stage baseline above tests this directly. We expect the shared model to win mainly on hops and merging vacancies, where position alone is ambiguous. If it does not, the honest result is that environment classification is useful but joint embedding is not, and we will report it that way.
Why not skip DFT and simulate unrelaxed or PBE-only structures? For monolayers this may truly be sufficient, since dispersion mostly acts between layers. The functional ablation settles the question. We expect PBE and PBE-D3 to perform similarly on monolayers and to differ on bilayers, where PBE overestimates layer spacing (charged S vacancies). Unrelaxed, ideal-lattice vacancies should hurt classification of reconstructed clusters such as MX₃ and MX₆. If no relaxed variant beats unrelaxed structures, that negative result is itself useful to people building simulation-trained microscopy models.
Why not use a fast machine-learned potential to run real dynamics? Short-range potentials miss dispersion (MLIPs and long-range physics). Interpolated DFT snapshots avoid inventing physics in the training data. We acknowledge that interpolation is crude, and the gap between simulated and real data is measured below.
Proposed Experiments
Real-data evaluation. On the SMC'17 MoSe₂ series, two annotators will label column positions, track IDs, hop events and vacancy classes in all 49 frames; agreement between them sets the ceiling on achievable accuracy. We report detection F1 at a half-lattice-spacing match radius, HOTA and IDF1, recall of hop events, and macro-F1 over defect classes.
Simulation checks. On held-out synthetic movies, we report the same metrics separately by vacancy class and by electron dose.
Decisive checks.
- Environment versus identity embeddings. The environment-supervised embedding must beat the identity-supervised ablation on real hop recall.
- Learned embedding versus geometry alone. The joint model must beat the two-stage, geometry-linked baseline on frames where vacancies merge.
- Effect of the dispersion treatment. Real-data classification accuracy is measured as a function of which functional relaxed the training structures. The prediction is little difference for monolayers and a gap for bilayers. Bilayer real data are not in the file, so the bilayer half of this check is limited to simulation unless suitable data are acquired.
- Transfer gap. We compare models trained on synthetic data alone with the same models after fine-tuning on 10% of the real labels.
Physical consistency check. We test whether vacancy sequences recovered from real movies follow the DFT formation-energy ordering, with the neutral sulfur vacancy at about 1.5 eV under Mo-rich conditions as one anchor. This is where the energy-annotated trajectories become useful in their own right.
References
- Towards Real-Time Multi-Object Tracking
- FairMOT: On the Fairness of Detection and Re-identification in Multiple Object Tracking
- DefectTrack: a deep learning-based multi-object tracking algorithm for quantitative defect analysis of in-situ TEM videos in real-time
- Computer Vision-aided Atom Tracking in STEM Imaging
- Deep Learning‐Enabled STEM Imaging for Precise Single‐Molecule Identification in Zeolite Structures
- Vacancy impacts on electronic and mechanical properties of MX2 (M = Mo, W and X = S, Se) monolayers
- Stability of charged sulfur vacancies in 2D and bulk <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"><mml:msub><mml:mi>MoS</mml:mi><mml:mn>2</mml:mn></mml:msub></mml:math> from plane-wave density functional theory with electrostatic corrections
- Machine Learning Interatomic Potentials and Long-Range Physics
- RelationTrack: Relation-Aware Multiple Object Tracking With Decoupled Representation
- SoccerNet-Tracking: Multiple Object Tracking Dataset and Benchmark in Soccer Videos