Executive Summary
With the “Atlas of Automation“ we want to show how everyone‘s daily life is already immersed in automated decisions. We do not necessarily perceive them as such – but they have consequences. In today’s automated decision-making (ADM), neural networks (that are often referred to as “Artificial Intelligence”) are rarely employed. Instead we find more or less complex software applications that calculate, weigh and sort data according to sets of rules. We speak of decision-making systems because the respective software only selects from preset decision options. However, these decisions are determined by people who take part in the design and the programming, as well as the employment of ADM software.
The scope of this Atlas is limited in two ways: firstly, it focusses solely on Germany and on ADM systems specifically in regard to their relevance for participation. We are interested in the way ADM limits (or enhances) access to public goods and services, and the ability to exercise individual rights. In this context, people can experience discrimination in many ways: based on age, sex, or their social or geographical origin.
For this reason we address specific key areas to look at various issues. In regard to labor we examine automized recruitment procedures, ADM in personnel management and the administration of unemployment. In the chapter “Health and Medicine“ we focus on diagnostic systems and health apps. When our attention comes to the Internet we include aspects such as upload-filters and platform regulation. The chapter “Security & Surveillance” relates to face recognition and speech recognition used on asylum seekers and in “predictive policing“. Whereas in the “Education, Stock Trading, Cities & Traffic“ chapter we delve into topics such as education and traffic. Further chapters give an overview of the legal regulation of ADM and of relevant actors.
Another part of the “Atlas of Automation“ consists of a free online database: atlas-of-automation.net. In its initial form it contains about 150 entries on ADM products and technologies, on actors and on regulations, all of which are relevant to the subject of participation. At this stage the database entries are all in the German language, but the text of this report is in English.
The research and analysis we did for the “Atlas of Automation“ brought us to a series of conclusions. We intend our recommendations for action to spur discussion and to inspire politicians and decision-makers in authorities, companies and civil society organizations. Briefly, these are our recommendations:
ASSESS IMPACT
In the development and application of systems for automated decision-making (ADM), the guiding principle should be: to do no harm. (Primum non nocere). This means: The design and implementation of ADM systems should accompany an impact assessment and, if possible, a risk assessment of such technologies.
SPEAK OF ADM INSTEAD OF AI
Instead of using the term “Artificial Intelligence” (AI) – which is loaded with expectations – we consider the term algorithm-based decision making (ADM) to be more appropriate. We thus want to highlight the issue of accountability because the responsibility for decisions still lies with the humans who commission, develop and approve ADM systems.
EMPOWER CITIZENS AND CIVIL SOCIETY
Citizens should be empowered to more competently assess the results and the potential of automated decisions. It is particularly important to develop materials and programs for schools, job training and further education. Germany‘s federal government should now let actions follow the promises it made in the AI strategy, specifically: “The state has to enable academia and civil society to contribute independent and competence based input to the societal discourse.”
EXTEND REPORTING
Journalists, editors and publishers should make ADM a subject of their research and reporting. In addition, competences should be built and extended in order to enable responsible reporting on algorithms in decision-making (“Algorithmic Accountability Reporting”).
STRENGTHEN ADMINISTRATION
Municipalities, federal states and the national government in Germany should create a register of all software systems used in their administrations and which documents the degree of automation and its effect on participation and on society. Employees should be sensitized to the various possible impacts of decision-making software. At the same time, they should also be enabled so that they can identify potential uses of ADM for administrative purposes.
REGULATE INTELLIGIBILITY
We have our doubts whether a centralized “Algorithm TÜV” (software testing and certifying institute) would be able to match the need for regulation in all the different sectors. Instead of demands for more transparency, the focus should be on the intelligibility of ADM systems: Transparency on its own is of little help when dealing with complex software and large amounts of data; it also needs to be openly documented as to which data was processed and how the procedure worked.
ENSURE ENFORCEABLE SUPERVISION
There already exist numerous regulations that allow the use of ADM systems and control it. It must be ensured, that these regulations are also reviewed and enforced. To this end, the supervisory authorities must be adequately equipped and qualified to proactively pursue this task.
OBLIGATE PRIVATE BUSINESS
Private companies should be subject to regulation if their ADM products can have collective effects, e.g. when it is used in an administration. Besides staff training, self-regulation and certification programs, audit procedures–defined by the state and relating to the above mentioned accountability–should be considered.
CONSIDER ECOLOGICAL ASSESSMENT
Apart from the software, automated decision-making processes need hardware and Internet infrastructure. This goes along with energy consumption. It should therefore be considered as to whether the expected benefit of ADM justifies the negative effects on the environment.