An Initial Introduction to the European Union's AI Act

It is worth reviewing why the Act was deemed necessary by the European Parliament. The Act responds to identified risks with AI, and organizations that use AI cannot plead ignorance of these risks.

Artificial intelligence refers to the ability of an IT system to reason or make decisions in a manner similar to the human brain. The term was coined by computer scientist John McCarthy in 1955, a very long time before the emergence of ChatGPT in 2022!

AI should arguably be called Advanced Algorithms, as the last decades have seen the design of algorithms that try to mirror human decision-making processes. One field of the domain is machine learning, where probability-based algorithms are developed by training them on observed real-world data. This approach has worked really well, and with the explosion of the Web in the last 30 years, the data available to train AI algorithms is huge. This data availability, coupled with a continued improvement in computing power which these algorithms require (often thanks to the Graphics Processing Units designed for gaming consoles!), has brought us to the situation today where AI processing is widely available and used.

Our goal here is to highlight salient features of the Act that can have an impact on organizations.

The key roles defined by the AI Act are providers and deployers.

• Providers are people or organizations that develop an AI system, or who put an AI system into service. For instance, Github is a provider for the Copilot AI-programming tool.
• Deployers are people or organizations that employ an AI system in the provision of some service, perhaps to clients of theirs. For instance, an organization using Github Copilot or a bank using an AI-powered chatbot to communicate with customers is a deployer.

In the following, we refer to a user as any person or organization that avails of AI-based services. A programmer working with an organization that has deployed Github Copilot is a user.

A common use-case scenario today is for an organization to choose and install a basic AI model, and to adapt this model for their own needs, for instance, by re-training or fine-tuning the model with organizational data, or by optimizing the internal parameters of the model. In certain cases, the organization may be a provider and deployer, with its clients and employees being users.

The obligations for organizations under the Act depend on their role. Providers must demonstrate the safety and trustworthiness of the AI tools they develop. The deployers are subject to a lower set of obligations, mostly related to transparency of AI use.

A fundamental requirement of the Act is that the risk of each AI system must be evaluated, and the obligations of deployers and providers depend on the result of the risk classification. As shown in the following figure, four levels of risk are defined by the Act. The Act also defines the use cases which fall within each category.

1. Unacceptable risk. AI systems that facilitate exploitation, manipulation or social scoring fall into this category. Commonly cited scenarios include indiscriminate biometric identification of people, social scoring leading to later discrimination, emotion recognition software in the workplace, and any AI that detects and exploits a person’s vulnerabilities. AI systems in this category are simply banned.
2. High-risk. AI systems in this category are those that can cause potential harm to citizens if they are used in an appropriate manner. The domain of use is important, with education, job application processing, immigration and border control, healthcare, democratic and judiciary processes being particularly surveyed. The obligations on the provider include registering the AI system in an EU database as well as implementing a range of governance procedures. We return to these procedures later on.
3. Limited risk. These are systems where risks do not outweigh basic deception or trickery. Most chatbots, like ChatGPT, fall into this category. The main obligation for providers and deployers under the act is transparency - the requirement that users be made aware that the IT service they are interacting with employs AI.
4. Minimal risk. These systems pose no direct risks, with commonly cited examples being SPAM filters and gaming applications. No specific requirements are mentioned for these systems.

The GDPR came into effect in 2018, and many organizations spent considerable resources adapting their organizational process to be compliant. It would be convenient if the effort made to implement GDPR could serve to help implement compliance with the AI Act. We believe that this is the case. The GDPR Act put forward two measures that are obligatory for large organizations, and highly recommended for others: the register of processing activities (ROP) and the data protection officer (DPO) role. These measures could serve as a basis for AI Act compliance.

In its simplest form, a register of processing activities is a record-kept log of all processing activities within the organization (e.g., payroll, CRM, Website management, etc.). Each activity’s description explains what the purpose of the activity is, which data is processed, what the legal basis for that processing is when personal data is involved, why processing is absolutely necessary, what tools are used for the processing, and how the processing is made secure. The advantage for organizations which have implemented the ROP are that they:

• Have an up-to-date account of all processing activities in their organization. This allows organizations to fulfill their duty of information towards EU residents, since the latter have the right to know precisely how their personal information is processed.
• Have identified the data used by each activity. A risk analysis will have been done for those activities that process personal data. This allows them to implement the required security measures, which need to be tailored to the level of risk.

The ROP does not have a fixed structure even though several organizations have proposed templates. Therefore, for deployer organizations, the ROP can be extended to include AI-related information like the justification for using AI for the activity, an AI related risk analysis, and a description of the measures to monitor the safe usage of the AI tool. Likewise, such an expanded ROP could also be used to meet the transparency requirements set forth in the AI Act at least with respect to the AI tools which rely on personal data (which, incidentally, is often done by integrating a specific AI-related section in the GDPR privacy statement, yet another situation where the data protection requirements and the AI-related regulatory requirement present similarities).

Under the GDPR, the DPO is an organizational role whose brief is to oversee the implementation of measures for GDPR conformity in an organization, from the ROP’s maintenance to employee awareness programs. The role involves challenging organizational management over plans to implement activities when there is a risk of GDPR non-compliance. The DPO has a broad view over all activities within the organization.

An equivalent role, let us call it AI Officer, can be attributed for the AI Act as the requirements are the same: ensuring a complete view of all AI-related projects, ensuring governance measures for activities using AI tools, challenging management and implementing awareness programs.

Once the AI officer has been appointed, he or she will work with other people in the organization, like the DPO, IT and IT security (though these might of course be the same people). The collaboration with the DPO is useful because, through justifications for the necessity and legality of each activity’s processing, the collaboration helps avoid the urge to over-digitalize processes. The collaboration with IT can help improve organizational data management. Many AI projects are known to fail because of poor data quality in organizations. For both AI officers and DPOs, verifying data provenance for services is crucial to ensure compliance so the AI Act, GDPR and data management projects are intricately linked.

One recommended tool much talked about recently is the Organizational AI Charter. The idea of a charter is to permit an organization to demonstrate its awareness of AI risks, and to mention the restrictions it imposes on itself to address these risks (for example, by defining the kinds of AI processing that are unacceptable to the organization). The charter is a means of communicating values to employees, clients and to the public at large. Such a Charter can also serve as a useful tool to instruct employees about responsible use of AI in their day-to-day activities.

Another measure to take is a Documented AI Policy. This should define the test procedures put in place to validate AI safety and to ensure the terms and conditions of AI services are clear and up to date. The policy should outline the awareness program for employees, explain the steps taken by the organization to ensure that promises made in the AI Charter are respected, and define an incident-disclosure process.

A number of AI test practices are common today. In red-teaming, a group of people from within or outside of the organization test the AI system until it breaks, in an attempt to detect problems. This is useful for both deployers and providers. In field-testing, the system is tested in the environment of the deployer. In A/B testing, two model versions, say a model and a fine-tuned version of the model, are compared for the quality of their outputs for a benchmark of input questions.

Let us consider briefly three scenarios where an organization may (be tempted to) use AI.

It is quite common today for companies to use software for automated processing of CVs, given the large number of on-line applications for positions. Job applicants are using AI also - notably to help write CVs and cover letters.

Some processing might appear not to be “intelligent”, e.g., the software might be a rule-based program that eliminates CVs based on the age of the candidate. Our advice is that there is no point trying to exploit such loopholes. It is best to treat all processing activities as activities that require a risk analysis.

In any case, processing CVs might be an instance of high-risk processing because people are being profiled. The software provider must register his software with the EU database when this integrates AI. The deployer organization must test and monitor the tool. Automated CV processing might be an area where one should avoid over-digitalization (activities that are not really needed). Justification for automated processing of CVs is generally weak.

In such cases, a provider organization must ensure measures are taken to ensure training data and output do not violate IP license restrictions, or contain personal data whose owner did not consent to having this data used. This can typically be done by inserting filters to provide transparency on the level of risk that the output contains IP-protected content.

For marketing content, a risk assessment must be made though the risk level is most likely limited. If there is a risk of trickery, e.g., an image of a person’s face with smooth skin is generated to market anti-aging cream, then for transparency, the company must indicate that AI has been used.

A risk analysis is needed before deploying a chatbot, because by definition, a chatbot’s role is to influence the customer, even if this relates to the choice of product to buy.

You should also verify the age of the customer, and make the customer aware that he or she is conversing with an AI. Finally testing and monitoring the chatbot is obligatory.

This article has looked at implications of the EU’s AI Act for organizations. Key take-aways are:

• Organizations should use their register of processing activities as a basis to identify activities using AI and to conduct their risk analysis. The GDPR offers a helpful start on the road to compliance with the AI Act.
• Documented testing procedures are obligatory for model providers; monitoring and reporting is obligatory for organizations deploying AI systems. Transparency is always required. AI Providers must assign a risk level to their systems. Depending on the risk level, the system might be banned or it may have to be registered in a database maintained by the EU.
• Organizations cannot plead ignorance in the event of violations of the AI Act. Any organization deploying an AI-based service should conduct a risk analysis required prior to its launch.