Roles of Standardised Criteria in Assessing Societal Impact of AI

This paper proposes a set of criteria to evaluate organizational processes for artificial intelligence (AI) impact assessment, in order to facilitate public and private organizations to select the most appropriate impact assessment methodology for their specific context. To build this reference, the paper employs a dual research methodology: reviewing research on existing impact assessment in diverse domains including privacy, health, environment, and examining recent discussions on potential elements and methods for AI-related impact assessment. We consolidate key findings found in the literature and organize them in five dimensions: normative framework, process rules, methodology, engagement, and oversight. Within each of these dimensions, we propose a set of critical questions for meaningful impact assessment by integrating reflections on the challenges raised in the examined research, discussions on the ongoing impact assessment methodology, as well as investigations on impact assessment pitfalls. The resulting criteria are potentially useful to developers, regulators, and as a backbone of comparison across various standardization initiatives.