Supervised Mixed Norm Autoencoder for Kinship Verification in Unconstrained Videos

Naman Kohli, Daksha Yadav, Mayank Vatsa, Richa Singh, Afzel Noore

TL;DR

This paper introduces a novel deep learning framework for kinship verification—determining if two people are biologically related—using unconstrained videos rather than static images. The authors propose the Supervised Mixed Norm Autoencoder (SMNAE), which utilizes class-specific sparsity and spatio-temporal feature learning to achieve a state-of-the-art accuracy of 83.18% on their newly released KIVI (Kinship Video) dataset.

What Problem Does This Paper Address?

Kinship verification is a challenging computer vision task with applications in family photo organization, social media tagging, and even sensitive areas like border control and human trafficking prevention. While significant progress has been made using pairs of static images, real-world "in the wild" scenarios often involve videos where lighting, pose, and expressions vary wildly.

Existing research has largely ignored the rich spatio-temporal information available in video sequences. The authors argue that a single frame might not capture the subtle facial dynamics that reveal kinship, whereas a sequence of frames (a "vidlet") can provide a much more robust signature of relatedness.

Methods

The core of the research is the Supervised Mixed Norm Autoencoder (SMNAE). Unlike standard autoencoders that simply aim to reconstruct their input, SMNAE incorporates two specialized penalties into its loss function:

  1. Class-Specific Sparsity ($l_{2,p}$ norm): This forces the network to learn representative features that are specific to either "kin" or "non-kin" classes, enhancing the discriminative power of the latent representation.
  2. Pairwise Reconstruction Penalty: This uses a Laplacian matrix to enforce similarity between representations of the same class while pushing different classes apart.

The framework operates in three hierarchical stages:

  • Stage 1 (Spatial): Learns kinship-specific features from individual pairs of frames.
  • Stage 2 (Local Spatio-temporal): Analyzes "vidlets"—small groups of contiguous frames—to capture how facial features change over a short window (e.g., a pivot frame and its immediate neighbors).
  • Stage 3 (Global Vidlet): Aggregates these local features into a final compact representation for the entire video segment.

Finally, a Support Vector Machine (SVM) with a Radial Basis Function (RBF) kernel is used to classify the aggregated features as "Kin" or "Non-Kin."

Datasets

The authors evaluated their method on several benchmarks but notably introduced a new, large-scale dataset:

  • KIVI (Kinship Video) Database: The largest unconstrained kinship video database to date, featuring 503 individuals from 211 families. It contains 355 true kin pairs across seven relations (e.g., Father-Son, Sister-Sister) and over 250,000 frames.
  • Existing Benchmarks: The model was also tested on the UvA-NEMO Smile (video) database and five image-based databases (Cornell, UB KinFace, KFW-I, KFW-II, and WVU Kin) to demonstrate its versatility.

Key Insights and Ideas

  • Video > Images: The results confirm that temporal dynamics (how a face moves or changes expression) are a vital cue for kinship that static images simply miss.
  • The Power of "Vidlets": By breaking videos into small, overlapping temporal windows, the model can handle occlusions or poor frames more gracefully than a model looking at the entire video at once.
  • Sparsity Matters: The use of the $l_{2,p}$ norm (where $0 < p < 1$) was found to be more effective than the standard $l_{2,1}$ norm. It allows the model to find "sparser" and more distinct features for kin vs. non-kin pairs.
  • Optimal Window Size: Through ablation studies, the authors found that using a pivot frame with two neighbors on each side ($z=2$, total 5 frames per vidlet) provided the best balance between capturing motion and maintaining feature relevance.

Key Results

  • State-of-the-Art Performance: On the KIVI dataset, the SMNAE framework achieved 83.18% accuracy, outperforming existing algorithms like NRML and VGG-Face by at least 3.2%.
  • Consistency: The proposed method consistently yielded the best results across all six public databases tested. For example, it achieved 96.07% on the UvA-NEMO Smile dataset, a 2.42% improvement over previous visual transformation techniques.
  • Relation Sensitivity: The model performed best on Mother-Daughter pairs (92.31%) and found Mother-Son pairs (77.78%) the most difficult to verify, echoing findings in human psychology that female subjects are often easier to verify for kinship.

Limitations and Future Work

While the results are impressive, the authors note a few areas for growth:

  • Relation Diversity: The current work focuses on seven primary relations. Future iterations could expand to more distant relatives like cousins, aunts, and uncles.
  • Frame Selection: The current method uses all frames in a vidlet. Implementing an intelligent "keyframe" selection process might reduce computational load and filter out low-quality or blurry frames.
  • Dataset Expansion: The authors plan to further expand the KIVI database to include more diverse ethnicities and even more "wild" variations to further stress-test the algorithm.

Summary updated

Abstract

Identifying kinship relations has garnered interest due to several applications such as organizing and tagging the enormous amount of videos being uploaded on the Internet. Existing research in kinship verification primarily focuses on kinship prediction with image pairs. In this research, we propose a new deep learning framework for kinship verification in unconstrained videos using a novel Supervised Mixed Norm AutoEncoder (SMNAE). This new autoencoder formulation introduces class-specific sparsity in the weight matrix. The proposed three-stage SMNAE based kinship verification framework utilizes the learned spatio-temporal representation in the video frames for verifying kinship in a pair of videos. A new kinship video (KIVI) database of more than 500 individuals with variations due to illumination, pose, occlusion, ethnicity, and expression is collected for this research. It comprises a total of 355 true kin video pairs with over 250 000 still frames. The effectiveness of the proposed framework is demonstrated on the KIVI database and six existing kinship databases. On the KIVI database, SMNAE yields video-based kinship verification accuracy of 83.18% which is at least 3.2% better than existing algorithms. The algorithm is also evaluated on six publicly available kinship databases and compared with best reported results. It is observed that the proposed SMNAE consistently yields best results on all the databases.
BibTeX
@article{kohli2019supervised,
  author = {Naman Kohli and Daksha Yadav and Mayank Vatsa and Richa Singh and Afzel Noore},
  title = {{Supervised Mixed Norm Autoencoder for Kinship Verification in Unconstrained Videos}},
  journal = {IEEE Trans. Image Process.},
  year = {2019},
  url = {https://openreview.net/forum?id=bN_bcxCJ4Bx},
}

Loading related content...

Figure