Movie Recommendation System using Apache Spark

Ankit Maheshwari, Anuradha Kumari, Anjali Kumari, Neeraj Kumar, B M Nandini2018 · DOI

Abstract

This volume contains contributed articles presented in the conference NCICCNDA 2018, organized by the Department of Computer Science and Engineering, GSSS Institute of Engineering and Technology for Women, Mysore, Karnataka (India) on 28th April 2018.

Introduction

In the modern era of "information overload," consumers are flooded with choices. Manually searching through thousands of movies to find something of interest is a tedious, often frustrating task. While recommendation systems are now common in e-commerce and streaming, they face a significant technical hurdle: scalability. As datasets grow into the "mammoth" category, traditional single-machine algorithms or disk-heavy batch processing (like standard Hadoop MapReduce) become too slow for real-time or near-real-time suggestions. This paper details the design and implementation of a movie recommendation system built on the Apache Spark framework using Scala to address these scalability challenges.

Methodology

The authors implement an item-based collaborative filtering model. Unlike content-based filtering, which looks at movie metadata, collaborative filtering relies on analyzing patterns in how different users rate different items. The system leverages Apache Spark and Scala to maximize performance, utilizing Resilient Distributed Datasets (RDDs) to keep data in-memory during processing.

Workflow of the Movie Recommendation System

The general workflow involves reading and tabulating input data, mapping movie IDs to names, and creating RDDs of movie-rating pairs. The core mathematical engine is the Cosine Similarity metric, which treats movies as vectors in a high-dimensional space of user ratings.

Proposed Approach

The proposed approach refines the recommendation process by incorporating specific user inputs and filtering mechanisms.

Architecture of the Proposed Recommendation Approach

As shown in the architecture, the system takes inputs such as User ID, Movie ID, Genre, and the desired number of suggestions. It applies cosine similarity to identify similar and dissimilar movies, ultimately producing a final filtered result. To reduce computational overhead, the system performs a "self-join" on user-rating data and eliminates duplicate movie pairs before calculating similarities. A minimum threshold rating is also applied to ensure only high-quality recommendations reach the final "Top 10" list.

Datasets

The researchers utilized the GroupLens (MovieLens) dataset from the University of Minnesota. The data includes userID, movieID, user ratings, and an item file containing movie titles and release dates across 19 distinct genres. The authors employed a standard 80% training and 20% testing split to validate the model's accuracy.

Results and Discussion

The primary advantage of this system is the efficiency gained by Spark’s RDDs. By caching similarity values in memory, the system avoids re-calculating expensive operations. While Spark’s MLLib has built-in algorithms, the authors formulated their own logic for more granular control. The item-based choice provides stability, as movie similarities change less rapidly than individual user tastes.

The paper concludes that the model provides "really good performance" and is inherently designed to be highly scalable. The system successfully generates a refined "Top 10" list of recommended movies for specific users based on historical ratings.

Conclusion and Future Work

While successful on a local machine, moving to a distributed online cluster is a logical next step. Future work could involve testing other metrics like Pearson Correlation or Jaccard Coefficients. Additionally, addressing the "cold start" problem for new users and exploring hybrid systems that combine collaborative and content-based filtering could further enhance recommendation quality.

Summary updated

Abstract

This volume contains contributed articles presented in the conference NCICCNDA 2018, organized by the Department of Computer Science and Engineering, GSSS Institute of Engineering and Technology for Women, Mysore, Karnataka (India) on 28th April 2018.
BibTeX
@misc{maheshwari2018movie,
  author = {Ankit Maheshwari and Anuradha Kumari and Anjali Kumari and Neeraj Kumar and B M Nandini},
  title = {{Movie Recommendation System using Apache Spark}},
  year = {2018},
  doi = {10.21467/proceedings.1.45},
}
Figure