
Elliot: A Comprehensive and Rigorous Framework for Reproducible Recommender Systems Evaluation
ELLIOT is a novel kind of recommendation framework aimed to overcome obstacles about reproducibility in the RecSys field by proposing a fully declarative approach(using a configuration file) to the set-up of an experimental setting. It analyzes the recommendation problem from the researcher’s perspective as it implements the whole experimental pipeline, from dataset loading to results gathering in a principled way. The main idea behind ELLIOT is to keep an entire experiment reproducible and put the user(in our case, a researcher or RS developer) in control of the framework. According to the recommendation model, ELLIOT allows, to date, the choice among 27 similarity metrics, the definition of multiple neural architectures, and 51 hyperparameter tuning combined approaches, unleashing the full potential of the HyperOpt library (Bergstra et al., 2013).To enable evaluation for the diverse tasks and domains, ELLIOT supplies 36 metrics (including Accuracy, Error-based, Coverage, Novelty, Diversity, Bias, and Fairness metrics), 13 splitting strategies, and 8 prefiltering policies.
- 15 Luglio 2021