Auditing Recommender Systems. An Overview Of Existing Audits, Risk Assessments and Studies on Potential "VLOPs"
The "Digital Services Act" (DSA) requires "Very Large Online Platforms" (VLOPs) operating in the European Union to conduct different audits and risk assessments. Our paper "Auditing Recommender Systems. Putting the DSA into practice with a risk-scenario-based approach"  provides general guidelines on conducting these audits, taking the operationalisation of systemic risks into account.
As mentioned in our paper, there is no clear "one size fits all" approach. Not every method can measure every risk, and each approach has different benefits and limitations. The decision on the technical method depends on the systematic risk that will be audited and the platform elements involved. Auditors and researchers must share their experiences and findings on different audit methods to get results that are as precise as possible.
This mapping intends to support this exchange by providing an overview of existing audits, risk assessments and studies of potential "Very Large Online Platforms". The document thereby only provides an overview of existing research.
- Code Audits
- Crowdsourced Audits
- Document Audits
- Architecture Audits
- Sock Puppet Audits
- API Audits
- Scraping Audits
- User Surveys
Different choice architecture elements that can be involved are, for example:
- User experience
- User interface
- e.g., Recommendation or Search algorithms
- Content Moderation
- Terms & Conditions
- Report Mechanisms
- Data-related practices
Are you conducting research or audits on “VLOPs” and would like to be mentioned in the mapping?
Are there audits and studies we should mention? We are pleased to receive references and other feedback. Therefore, please write an e-mail to: firstname.lastname@example.org
 Christian Sandvig et al., ‘Auditing Algorithms : Research Methods for Detecting Discrimination on Internet Platforms’, in Data and Discrimination: Converting Critical Concerns into Productive Inquiry, 2014
 Ada Lovelace Institute, ‘Technical Methods for Regulatory Inspection of Algorithmic Systems’ (London: Ada Lovelace Institute, 9 December 2021)