MSc
Online News Popularity
A graded machine-learning challenge where every model scored 80% accuracy by predicting one class for every article, and the work was making that failure visible and then fixing it.
A course challenge: predict how widely a Mashable article will be shared, from a deliberately corrupted version of a public dataset, with the share count discretised into five ordered classes.
- Context
- Sapienza Machine Learning challenge, 2021/22
- Data
- 39,648 rows supplied → 37,638 after cleaning and outlier removal
- Result
- 0.65 accuracy, 0.25 F1-macro on a held-out third
- Repository
- VincenzoImp/online-news-popularity
What it was
The dataset was supplied broken on purpose – malformed rows, out-of-domain values – so cleaning carried its own marks. Feature importance from a random forest then cut fifty-nine predictors to the eighteen scoring above a threshold.
The real problem appeared at the first evaluation. Every model returned about 80% accuracy and an F1-macro of 0.18, and the confusion matrix showed why: the classifier predicted the majority class for all 12,546 test articles. It had learned the base rate and nothing else, and accuracy was rewarding it.
The fix was resampling to a graded distribution rather than a uniform one: undersampling and then SMOTE to class targets of one third, one quarter, one fifth, one sixth and one seventh. The reasoning is written in the notebook: non-viral articles genuinely outnumber viral ones, so flattening the classes entirely trains for a world that does not exist. Uniform balancing was tried and rejected on measured results.
What I took from it
Accuracy at 80% and accuracy at 65% were the same model getting better. Choosing the metric before looking at the numbers is what stops a result like the first one from being reported as a success, and the honest finish is that even after balancing, the top class scores an F1 of 0.02. The model still cannot find a viral article; it just no longer pretends none exist.