Supervised Mixed Norm Autoencoder for Kinship Verification in Unconstrained Videos

Naman Kohli, Daksha Yadav, Mayank Vatsa, Richa Singh, Afzel Noore

Abstract

Identifying kinship relations has garnered interest due to several applications such as organizing and tagging the enormous amount of videos being uploaded on the Internet. Existing research in kinship verification primarily focuses on kinship prediction with image pairs. In this research, we propose a new deep learning framework for kinship verification in unconstrained videos using a novel Supervised Mixed Norm regularization Autoencoder (SMNAE). This new autoencoder formulation introduces class-specific sparsity in the weight matrix. The proposed three-stage SMNAE based kinship verification framework utilizes the learned spatio-temporal representation in the video frames for verifying kinship in a pair of videos. A new kinship video (KIVI) database of more than 500 individuals with variations due to illumination, pose, occlusion, ethnicity, and expression is collected for this research. It comprises a total of 355 true kin video pairs with over 250,000 still frames. The effectiveness of the proposed framework is demonstrated on the KIVI database and six existing kinship databases. On the KIVI database, SMNAE yields video-based kinship verification accuracy of 83.18% which is at least 3.2% better than existing algorithms. The algorithm is also evaluated on six publicly available kinship databases and compared with best-reported results. It is observed that the proposed SMNAE consistently yields best results on all the databases.

Introduction

Kinship verification—the task of determining if two individuals are biologically related—is a burgeoning area of computer vision. Its applications range from automated family photo organization and social media tagging to critical societal issues like border control and the prevention of human trafficking. While significant progress has been made using static image pairs, real-world "in the wild" scenarios are often captured in video format, where lighting, pose, and expressions vary dynamically.

Examples of kinship pairs: Father-Son relations (Bush family and Smith family).

Traditional methods often overlook the rich spatio-temporal information inherent in video sequences. A single frame might fail to capture the subtle facial dynamics that reveal kinship, whereas a sequence of frames can provide a more robust signature of relatedness. This research introduces a novel deep learning framework designed specifically for unconstrained videos.

Proposed Method: Supervised Mixed Norm Autoencoder (SMNAE)

The core of this framework is the Supervised Mixed Norm Autoencoder (SMNAE). Unlike standard autoencoders that focus solely on input reconstruction, SMNAE incorporates two specialized penalties to enhance discriminative power:

  1. Class-Specific Sparsity ($l_{2,p}$ norm): This penalty forces the network to learn representative features specific to either "kin" or "non-kin" classes. The use of the $l_{2,p}$ norm (where $0 < p < 1$) was found more effective than the standard $l_{2,1}$ norm for finding sparse, distinct features.
  2. Pairwise Reconstruction Penalty: Utilizing a Laplacian matrix, this penalty enforces similarity between representations of the same class while pushing different classes apart in the latent space.

The framework processes video data through a three-stage hierarchical pipeline:

  • Spatial Stage: Learns kinship-specific features from individual frame pairs.
  • Local Spatio-temporal Stage: Analyzes "vidlets"—small groups of contiguous frames—to capture short-term facial dynamics.
  • Global Vidlet Stage: Aggregates these local features into a compact representation for the entire video segment.

Datasets and Evaluation

To evaluate the framework, the authors introduced the KIVI (Kinship Video) Database, the largest unconstrained kinship video database to date. It features 503 individuals from 211 families, containing 355 true kin pairs across seven relations and over 250,000 frames.

Sequence of frames from unconstrained videos showing variations in pose and expression.

The model was also benchmarked against the UvA-NEMO Smile video database and five image-based databases (Cornell, UB KinFace, KFW-I, KFW-II, and WVU Kin) to demonstrate its versatility across different media types.

Key Results

The SMNAE framework achieved state-of-the-art performance across all tested benchmarks:

  • KIVI Dataset: Achieved 83.18% accuracy, outperforming existing algorithms like NRML and VGG-Face by at least 3.2%.
  • UvA-NEMO Smile: Reached 96.07% accuracy, a 2.42% improvement over previous visual transformation techniques.
  • Relation Sensitivity: Verification was most accurate for Mother-Daughter pairs (92.31%) and most challenging for Mother-Son pairs (77.78%), aligning with psychological findings that female kinship cues are often more recognizable.

Conclusion

This research demonstrates that temporal dynamics are a vital cue for kinship verification. The SMNAE framework effectively leverages these cues through class-specific sparsity and spatio-temporal feature aggregation. Future work will focus on expanding the KIVI database to include more distant relations and implementing intelligent keyframe selection to improve computational efficiency in "wild" video sequences.

Summary updated

Abstract

Identifying kinship relations has garnered interest due to several applications such as organizing and tagging the enormous amount of videos being uploaded on the Internet. Existing research in kinship verification primarily focuses on kinship prediction with image pairs. In this research, we propose a new deep learning framework for kinship verification in unconstrained videos using a novel Supervised Mixed Norm regularization Autoencoder (SMNAE). This new autoencoder formulation introduces class-specific sparsity in the weight matrix. The proposed three-stage SMNAE based kinship verification framework utilizes the learned spatio-temporal representation in the video frames for verifying kinship in a pair of videos. A new kinship video (KIVI) database of more than 500 individuals with variations due to illumination, pose, occlusion, ethnicity, and expression is collected for this research. It comprises a total of 355 true kin video pairs with over 250,000 still frames. The effectiveness of the proposed framework is demonstrated on the KIVI database and six existing kinship databases. On the KIVI database, SMNAE yields video-based kinship verification accuracy of 83.18% which is at least 3.2% better than existing algorithms. The algorithm is also evaluated on six publicly available kinship databases and compared with best-reported results. It is observed that the proposed SMNAE consistently yields best results on all the databases
BibTeX
@article{kohli2019supervised,
  author = {Naman Kohli and Daksha Yadav and Mayank Vatsa and Richa Singh and Afzel Noore},
  title = {{Supervised Mixed Norm Autoencoder for Kinship Verification in Unconstrained Videos}},
  journal = {IEEE Trans. Image Process.},
  year = {2019},
  url = {https://openreview.net/forum?id=bN_bcxCJ4Bx},
}
Figure