Abstract
This volume contains contributed articles presented in the conference NCICCNDA 2018, organized by the Department of Computer Science and Engineering, GSSS Institute of Engineering and Technology for Women, Mysore, Karnataka (India) on 28th April 2018.
Introduction
In the modern era of "information overload," consumers are flooded with choices. Manually searching through thousands of movies to find something of interest is a tedious, often frustrating task. While recommendation systems are now common in e-commerce and streaming, they face a significant technical hurdle: scalability. As datasets grow into the "mammoth" category, traditional single-machine algorithms or disk-heavy batch processing (like standard Hadoop MapReduce) become too slow for real-time or near-real-time suggestions. This paper details the design and implementation of a movie recommendation system built on the Apache Spark framework using Scala to address these scalability challenges.
Methodology
The authors implement an item-based collaborative filtering model. Unlike content-based filtering, which looks at movie metadata, collaborative filtering relies on analyzing patterns in how different users rate different items. The system leverages Apache Spark and Scala to maximize performance, utilizing Resilient Distributed Datasets (RDDs) to keep data in-memory during processing.
The general workflow involves reading and tabulating input data, mapping movie IDs to names, and creating RDDs of movie-rating pairs. The core mathematical engine is the Cosine Similarity metric, which treats movies as vectors in a high-dimensional space of user ratings.
Proposed Approach
The proposed approach refines the recommendation process by incorporating specific user inputs and filtering mechanisms.
As shown in the architecture, the system takes inputs such as User ID, Movie ID, Genre, and the desired number of suggestions. It applies cosine similarity to identify similar and dissimilar movies, ultimately producing a final filtered result. To reduce computational overhead, the system performs a "self-join" on user-rating data and eliminates duplicate movie pairs before calculating similarities. A minimum threshold rating is also applied to ensure only high-quality recommendations reach the final "Top 10" list.
Datasets
The researchers utilized the GroupLens (MovieLens) dataset from the University of Minnesota. The data includes userID, movieID, user ratings, and an item file containing movie titles and release dates across 19 distinct genres. The authors employed a standard 80% training and 20% testing split to validate the model's accuracy.
Results and Discussion
The primary advantage of this system is the efficiency gained by Spark’s RDDs. By caching similarity values in memory, the system avoids re-calculating expensive operations. While Spark’s MLLib has built-in algorithms, the authors formulated their own logic for more granular control. The item-based choice provides stability, as movie similarities change less rapidly than individual user tastes.
The paper concludes that the model provides "really good performance" and is inherently designed to be highly scalable. The system successfully generates a refined "Top 10" list of recommended movies for specific users based on historical ratings.
Conclusion and Future Work
While successful on a local machine, moving to a distributed online cluster is a logical next step. Future work could involve testing other metrics like Pearson Correlation or Jaccard Coefficients. Additionally, addressing the "cold start" problem for new users and exploring hybrid systems that combine collaborative and content-based filtering could further enhance recommendation quality.
Summary updated