Reliable or not? An automated classification of webpages about early childhood vaccination using supervised machine learning

Objective
To investigate the applicability of supervised machine learning (SML) to classify health-related webpages as ‘reliable’ or ‘unreliable’ in an automated way.

Methods
We collected the textual content of 468 different Dutch webpages about early childhood vaccination. Webpages were manually coded as ‘reliable’ or ‘unreliable’ based on their alignment with evidence-based vaccination guidelines. Four SML models were trained on part of the data, whereas the remaining data was used for model testing.

Results
All models appeared to be successful in the automated identification of unreliable (F1 scores: 0.54–0.86) and reliable information (F1 scores: 0.82–0.91). Typical words for unreliable information are ‘dr’, ‘immune system’, and ‘vaccine damage’, whereas ‘measles’, ‘child’, and ‘immunization rate’, were frequent in reliable information. Our best performing model was also successful in terms of out-of-sample prediction, tested on a dataset about HPV vaccination.

Conclusion
Automated classification of online content in terms of reliability, using basic classifiers, performs well and is particularly useful to identify reliable information.

Practice implications
The classifiers can be used as a starting point to develop more complex classifiers, but also warning tools which can help people evaluate the content they encounter online.