Life was quiet in the small town of Wanzer close to the former GDR border until shortly after the turn of the millennium. However, a few years later, everything changed: On week days, the remote country road that winds through the residential area of the town became filled with trucks. The introduction of an HGV toll on German motorways meant lorry drivers were re-routing through Wanzer. The drivers were using GPS navigation systems and smartphones with routing algorithms that sent them to the small town. The algorithms suggested that this section of previously quiet road would help drivers avoid paying a few Euros at the toll, while at the same time keeping traveling time between destinations in Eastern and Western Europe to a minimum. The former serenity and joy in nature behind the dike along the Elbe river in Wanzer was replaced by traffic noise and the scent of exhaust fumes.

The inhabitants of the town experienced first hand how automated decisions – that they have no influence over – can touch daily life in unexpected ways. We placed this example at the beginning of our report because we do not want to merely cover the familiar use cases of automated decision-making, such as Predictive Policing, in our Atlas of Automation. We also want to show that our entire everyday life is interfused with small or big automated decisions that we might not even register as such – and yet have consequences.

WHAT IS THE SCOPE OF THIS ATLAS?

A comprehensive inventory of automation across all sectors of society would be a massive undertaking. To present the perennial transformation of industry alone would need a separate and extensive deliberation. With this Atlas, we therefore chose to concentrate on one specific aspect which we consider to be particularly relevant: The relevance for social participation. Other than the term discrimination, the term participation does not only examine the systematic disadvantage of specific groups in an abstract way, it also brings our attention to the field in which this disadvantage can take place. Participation means access to public goods and services as well as the ability to claim one’s rights. Examining systems of automated decision-making, in respect to their relevance for social participation, means looking at ways in which these systems could impede access to public goods and services as well as the ability to exercise one’s rights – especially for persons who are already destitute or can be described as being disadvantaged. We consider the extent to which such groups of people are enabled or refused from participating socially to be a decisive indicator for the state of democracy in a society.

Despite all the criticism, we do not want to give the impression that automation should be rejected. In its 200 year long history, automation has brought about enormous social progress and improved quality of life to such an extent that no-one would want to roll back the clock. Hence, the Atlas not only points out the potential for discrimination that can accompany automated decisions, it also illustrates the opportunities and benefits that the use of automated decisions make possible or imaginable.

WHY NOT SPEAK OF ARTIFICIAL INTELLIGENCE?

The term “Artificial Intelligence” (AI) has had something of a renaissance over the past two years. It was first coined more than sixty years ago, and ever since there have been repeated periods during which it was en vogue for some time to use the term “Artificial Intelligence”. We have our doubts as to whether the current excitement for this term, which is mainly based on progress in “Machine Learning” and “Neural Networks”, will last very long. Outside of the world of experts, the use of the term AI is rather unclearly defined and some people award it almost magic capabilities. In the German government’s AI strategy, almost anything in connection to digitalization is summed up under “AI”. To which extent machines will ever be able to show “intelligence” that equals human autonomy and intentionality has long been disputed. However, it is agreed that the AI of today is still a very long way from human capability.

Mixing up automated decision-making with the term artificial intelligence rather leads us down the wrong path. When it comes to automated decision-making, what is essential is the fact that humans delegate machines to prepare decision-making or even to implement decisions. These machines can consist of highly complex neuronal networks, but they can also be made up of fairly simple software which calculates, weighs and sorts data according to a plain set of rules. In this Atlas, we therefore consistently speak of automated decision-making (ADM – see box) instead of artificial intelligence – even though many of the phenomena that we group under the term ADM could also be called AI applications.

In the past ten years, we have seen an unprecedented increase in automation through software. The amount of available data that helped make ADM possible in the first place, together with the proliferation of devices, equipment and infrastructure that are essential for ADM, have grown exponentially. With this, the influence of ADM on various aspects of society has inevitably expanded. As a result, we want our Atlas of Automation to initiate debate in society on the consequences of ADM. This is something that we believe to be an urgent necessity.

THE ELEMENTS OF THIS ATLAS

The current Atlas is a collection, not of geographical maps and diagrams, but of topics that in our view are particularly relevant in regard to the issue of ADM in connection with participation. Following this introduction, we present a set of recommendations which we developed based on our research and analysis. In the chapter “Methodology” we describe how we determined and defined the elements of this Atlas. The chapter “Actors” presents which authorities, research institutes, interest groups and NGOs decisively shape the discourse on ADM. This is followed by an overview of existing approaches to regulation of ADM that are relevant to participation. Individual chapters on social infrastructure, health and medicine, labour, consumer protection, the Internet, and security and control deal with the use of ADM in specific sectors of society.

Part of the Atlas project consists of a free digital database in which we describe about 150 actors and regulations, software systems and technologies. We will continue to expand this database. At the end of the report you can read more in the chapter entitled “Database” which also includes a selected bibliography.

We do not claim that our Atlas is exhaustive. We mainly want to illustrate ways in which ADM shapes society in the age of digitalization. In this we follow the Mission Statement which AlgorithmWatch set out in 2016: “It is not a law of nature that ADM processes are concealed from those affected by them. This has to change.” The statement goes on to say: “We have to decide how much freedom we want to transfer to ADM.” In this sense, this current Atlas of Automation is meant to help with taking the right human decisions.

 

---

ADM – Systems of automated decision-making

Systems of automated decision-making (ADM) are always a combination of the following social and technological parts:

• A decision-making model
• Algorithms that make this model applicable in the form of software code
• Data sets that are entered into this software, be it for the purpose of training via Machine learning or for analysis by the software
• The whole of the political and economic ecosystems that ADM systems are embedded in (elements of these ecosystems include: the development of ADM systems by public authorities or commercial actors, the procurement of ADM systems, and their specific use).

---

 

---

Participation

Participation means actively and passively using or exercising social opportunities and rights. People who are hindered in or denied access to public goods and services, and exercising their rights solely based on:

• their sex
• their sexual orientation
• their age
• their religion and/or their world view
• their origin (geographic, social)
• their health status
• their social status (education, occupation)

are limited in their participation in society. (In the German report we use the term „Teilhabe“, which has a slightly different connotation than ”Partizipation” in German language.)

We consider ADM-systems, technologies, regulations and actors as relevant to participation if their actions or their application hinders or enhances participation.

To preserve freedom of expression and of information within the framework of a common public sphere is, in this sense, an integral part of participation. Similarly, a functioning ecosystem is relevant to participation because, by providing a common livelihood for all of us, it is an important public good.

---

Next chapter: Recommendations

Previous chapter: Executive Summary

Contents

With the “Atlas of Automation“ we want to show how everyone‘s daily life is already immersed in automated decisions. We do not necessarily perceive them as such – but they have consequences. In today’s automated decision-making (ADM), neural networks (that are often referred to as “Artificial Intelligence”) are rarely employed. Instead we find more or less complex software applications that calculate, weigh and sort data according to sets of rules. We speak of decision-making systems because the respective software only selects from preset decision options. However, these decisions are determined by people who take part in the design and the programming, as well as the employment of ADM software.

The scope of this Atlas is limited in two ways: firstly, it focusses solely on Germany and on ADM systems specifically in regard to their relevance for participation. We are interested in the way ADM limits (or enhances) access to public goods and services, and the ability to exercise individual rights. In this context, people can experience discrimination in many ways: based on age, sex, or their social or geographical origin.

For this reason we address specific key areas to look at various issues. In regard to labor we examine automized recruitment procedures, ADM in personnel management and the administration of unemployment. In the chapter “Health and Medicine“ we focus on diagnostic systems and health apps. When our attention comes to the Internet we include aspects such as upload-filters and platform regulation. The chapter “Security & Surveillance” relates to face recognition and speech recognition used on asylum seekers and in “predictive policing“. Whereas in the “Education, Stock Trading, Cities & Traffic“ chapter we delve into topics such as education and traffic. Further chapters give an overview of the legal regulation of ADM and of relevant actors.

Another part of the “Atlas of Automation“ consists of a free online database: atlas-of-automation.net. In its initial form it contains about 150 entries on ADM products and technologies, on actors and on regulations, all of which are relevant to the subject of participation. At this stage the database entries are all in the German language, but the text of this report is in English.

The research and analysis we did for the “Atlas of Automation“ brought us to a series of conclusions. We intend our recommendations for action to spur discussion and to inspire politicians and decision-makers in authorities, companies and civil society organizations. Briefly, these are our recommendations:

ASSESS IMPACT

In the development and application of systems for automated decision-making (ADM), the guiding principle should be: to do no harm. (Primum non nocere). This means: The design and implementation of ADM systems should accompany an impact assessment and, if possible, a risk assessment of such technologies.

SPEAK OF ADM INSTEAD OF AI

Instead of using the term “Artificial Intelligence” (AI) – which is loaded with expectations – we consider the term algorithm-based decision making (ADM) to be more appropriate. We thus want to highlight the issue of accountability because the responsibility for decisions still lies with the humans who commission, develop and approve ADM systems.

EMPOWER CITIZENS AND CIVIL SOCIETY

Citizens should be empowered to more competently assess the results and the potential of automated decisions. It is particularly important to develop materials and programs for schools, job training and further education. Germany‘s federal government should now let actions follow the promises it made in the AI strategy, specifically: “The state has to enable academia and civil society to contribute independent and competence based input to the societal discourse.”

EXTEND REPORTING

Journalists, editors and publishers should make ADM a subject of their research and reporting. In addition, competences should be built and extended in order to enable responsible reporting on algorithms in decision-making (“Algorithmic Accountability Reporting”).

STRENGTHEN ADMINISTRATION

Municipalities, federal states and the national government in Germany should create a register of all software systems used in their administrations and which documents the degree of automation and its effect on participation and on society. Employees should be sensitized to the various possible impacts of decision-making software. At the same time, they should also be enabled so that they can identify potential uses of ADM for administrative purposes.

REGULATE INTELLIGIBILITY

We have our doubts whether a centralized “Algorithm TÜV” (software testing and certifying institute) would be able to match the need for regulation in all the different sectors. Instead of demands for more transparency, the focus should be on the intelligibility of ADM systems: Transparency on its own is of little help when dealing with complex software and large amounts of data; it also needs to be openly documented as to which data was processed and how the procedure worked.

ENSURE ENFORCEABLE SUPERVISION

There already exist numerous regulations that allow the use of ADM systems and control it. It must be ensured, that these regulations are also reviewed and enforced. To this end, the supervisory authorities must be adequately equipped and qualified to proactively pursue this task.

OBLIGATE PRIVATE BUSINESS

Private companies should be subject to regulation if their ADM products can have collective effects, e.g. when it is used in an administration. Besides staff training, self-regulation and certification programs, audit procedures–defined by the state and relating to the above mentioned accountability–should be considered.

CONSIDER ECOLOGICAL ASSESSMENT

Apart from the software, automated decision-making processes need hardware and Internet infrastructure. This goes along with energy consumption. It should therefore be considered as to whether the expected benefit of ADM justifies the negative effects on the environment.

Next chapter: Introduction

Contents