Abstract Visual surveillance is the latest paradigm for social security through machine intelligence. It includes the use of visual data captured by infrared sensors or visible-light cameras mounted in cars, corridors, traffic signals etc. Visual surveillance facilitates the classification of human behavior, crowd activity, and gesture analysis to achieve application-specific objectives.
Introduction
This special section of IEEE Access presents a curated collection of 20 research articles exploring the rapidly evolving intersection of visual surveillance and biometrics. The central theme is the transition from traditional, controlled biometric identification to "intelligent" security systems capable of monitoring behavior and identifying individuals in unpredictable, real-world environments.
The Convergence of Identification and Monitoring
Historically, biometrics and visual surveillance operated as distinct disciplines. Biometrics focused on high-precision identification in controlled settings, such as fingerprinting at border crossings. Conversely, surveillance focused on broader activities, like monitoring traffic flow or crowd behavior.
The current "holy grail" for security technology is the seamless merger of these fields. This involves identifying specific individuals covertly or under adverse conditions—such as poor lighting, long distances, or significant occlusions—while simultaneously interpreting their actions and emotional states. This editorial highlights research addressing the technical gaps in this merger, including image degradation, the requirement for real-time processing, and the complexities of non-cooperative recognition (e.g., when a subject is not looking at the camera).
Methodological Overview
The summarized research utilizes a diverse array of advanced computational techniques:
- Deep Learning Architectures: Convolutional Neural Networks (CNNs) are extensively employed for feature extraction in iris and face recognition, while Long Short-Term Memory (LSTM) networks are used to generate natural language descriptions of video streams.
- Multimodal & Multispectral Fusion: Techniques such as "compact multi-linear pooling" are used to combine data from disparate sensors (e.g., infrared and visible light) or different biometric traits (e.g., the iris and the periocular region).
- Bio-inspired Algorithms: Innovative studies utilize "artificial dorsal stream" models and "brain programming" to simulate human visual tracking, specifically for head tracking applications.
- Signal Processing & Optimization: The use of Fast Fourier Transforms (FFTNet) enables efficient tracking, while Continuous Wavelet Transforms facilitate "no-reference" image quality assessment—evaluating image degradation without a reference original.
Data and Benchmarking
The special section emphasizes the importance of robust datasets for training and validation:
- iCV-MEFED: A new dataset introduced by Guo et al. for emotion recognition, specifically designed to address unbalanced data distributions where certain emotions are over-represented.
- Dataset Optimization: Research by Trichet and Bremond investigates how suboptimal training data can degrade pedestrian detectors, proposing an expectation-maximization algorithm for improved data weighting.
Key Insights
- The Utility of "Off-the-Shelf" Features: Research by Nguyen et al. demonstrates that CNN features learned for general image tasks can be highly effective for specialized biometric tasks like iris recognition, significantly improving recognition rates.
- Proactive Surveillance: There is a shift from simple identification to proactive monitoring. This involves recognizing a subject's perceived emotional state and actions to predict potential security incidents.
- Quality as a Pre-requisite: Several studies argue that image quality assessment must be a pre-processing step, making biometric systems more robust against real-world blur and geometric distortion.
- Contextual Tracking: New methods utilize spatiotemporal descriptors to understand the relationship between a target and its environment, enhancing resilience to occlusions.
Results and Forensic Impact
The articles report significant quantitative and qualitative advancements:
- Iris and Face Recognition: CNN-based approaches have proven critical for large-scale applications, while the introduction of FFTNet and multi-order statistical descriptors allows for real-time, computationally efficient processing.
- Forensic Validation: The application of "graphonomics" to handwritten signatures provides a scientifically grounded method for signature validation in forensic contexts.
Challenges and Future Directions
Despite significant progress, several hurdles remain:
- Adverse Environments: Variations in illumination, pose, scale, and heavy occlusions continue to challenge robust object tracking.
- Data Scarcity: Complex tasks, such as compound emotion detection, suffer from a lack of diverse and balanced datasets.
- System Integration: The editors conclude that further work is required to create truly seamless, cross-platform systems that operate reliably without human intervention.
Summary updated