Reflections from a judicial conference in Rio de Janeiro on the legality of algorithms in the courtroom.

Egyptian judge on leave and alignAI doctoral candidate, Mohaned Bahr, was among the panelists at the EMERJ, at the Rio de Janeiro State Court School, upon an invitation from Instituto de Tecnologia e Sociedade do Rio de Janeiro (ITS Rio), and the EMERJ, on a panel titled “Direito e Tecnologia: A Justiça 4.0” (Law and Technology: Justice 4.0), where he presented his research on developing AI governance frameworks, inspired by the regulatory mechanisms devised in the EU AI Act.

The panel brought together an outstanding set of Brazilian and Portuguese law professors, Armando Guio, a Harvard-trained lawyer and the Executive Director of the Network of Centers, and other Brazilian practicing advocates. This news blog post reflects Mr. Bahr’s selected outcomes and commentary on the discussions from the plenary sessions and the project presentations.

Attending this forum was overwhelming, with the responsibility of wearing two hats at the time. One was a legal practitioner who used to lead the bench, considering and ruling in cases, while the other hat was coming to this forum as an alignAI doctoral candidate, at the Chair of Public Policy, Governance and Innovative Technology, Chaired by Prof. Urs Gasser, at the School of Social Science and Technology, at the Technical University of Munich.

In this regard, I flew to Rio de Janeiro wearing both of those hats and I am not sure I have ever felt the weight of that combination more than I did in that room. Here, I share some of the questions highlighted at the forum:

Unanswered Questions That Do Not Parallel AI Developments

I started my speech by pondering some questions, some of which emerged during the course of my research, and some had followed me long before I started this PhD. The core question evolved around accountability, specifically, when AI systems make mistakes, either in social life or in a highly sensitive use case, such as deploying an AI system in a court room either as an aid tool to judges or an autonomous decision maker system in the court, who should be responsible, and who should be accountable to pay the price of making decisive decisions without the intervention of human reasoning? In an attempt to answer these questions, I presented my role in the alignAI (MSCA) project, that goes in collaboration with my colleagues, who come from interdisciplinary fields within the alignAI projects, where we are all motivated to land answers for how LLMs can be aligned with the human values, and ensure the implementation of the two core principles of fairness and explainability in the design and pre-development stages of AI development for the benefit of humanity. Indeed, this was not a rhetorical question. Yet, the sensitivity of this question is crucial, since it touches on people’s fundamental rights; the honest answer is that governance systems still struggle with the dilemma of uncertainty to this day.

Let’s face the truth: it is too late to contemplate whether AI is prevailing in our lives, because AI is already infiltrating the majority of aspects of human life. Yet, using it in our court systems, either in the form of case management systems, risk assessment tools or document analysis software, leads to the question of how to ensure the protection of principles fairness, justice and due process, and operating amid AI systems that are explainable to their users without giving up on the human reasoning in the court system, which is one of the fundamental guarantees to ensure the protection of these aforementioned rights in the due process systems. The question that is facing judiciaries everywhere is no longer whether to use these tools, but whether the people who use them actually understand what they are doing on their behalf. As I told the audience that day, a judge cannot justify a ruling based on a process they cannot explain. Due process is not only a right belonging to the defendant, but it is a structural requirement of the legal system itself. Explainability is not a technical luxury. It is a legal necessity, and delegating this responsibility to algorithms hinders the process entirely and makes humanity vulnerable to the technology.

The Governance Gap Nobody Designed on Purpose

Our governance frameworks from the Global East to the Global South have not yet kept pace with the speed of AI deployment. The asymmetry in knowledge, skills and information about AI gives technology developers a superiority that enables them to design and implement control over those systems, posing a significant risk that goes to the heart of the problem. It cannot be denied that there is a wide range of decision-makers and legislators across various jurisdictions who often have almost no visibility into the development and design choice processes implemented by developers in these AI systems, and this is where my research lives. In that sense, I have been developing a way of thinking about AI governance in three coordinated layers: a national level, where courts and regulators make deployment decisions; a regional level, where coordination between jurisdictions prevents a race to the bottom; and a mutual recognition mechanism, so that a system certified as trustworthy in one country is not simply exported, unscrutinised, into another. I shared this not as a finished answer, but as a framework worth testing. In addition, I suggested, in context, examining the external contextualisation of the EU AI Act beyond EU borders. For instance, I referred to Brazil, with its own evolving AI legislation and regional ties through Mercosur, which is, in many ways, well-positioned to think about governance regionally rather than alone.

Not a Problem the Global South Should Inherit Secondhand

Much of the global conversation on AI governance has so far been dominated by Europe and the United States. It holds true that the EU AI Act is a serious and sophisticated instrument, yet it is still a fledgling instrument, which I enjoy studying closely, but it was mainly built for the European market and its systems, with the aim of bringing it to the world as a regulatory model worth considering. However, herein, I raise the question: when that model, or the regulatory instinct underlying it, is exported to Egypt, Brazil and the wider Global South, it carries assumptions that do not always travel well. This requires asking questions before considering this model’s contextualisation, such as: who counts as a vulnerable user in a different legal culture? What does due process look like outside the European frame? What institutional capacity actually exists that is capable of enforcing any of this?

I call this the legal transplant problem in AI governance. You can copy a law. You cannot copy the ecosystem that makes a law work. What jurisdictions in the Global South need is not a mere copy-paste regulation, but adaptive governance—frameworks that learn from the EU’s experience while being built on their own legal traditions and institutional realities.

Human in the Loop Still Stands in the Forefront to Ensure Transparency

Thinking backward and making the human who stands at the end of the AI systems deployment process (end users), brings us back to address the significant role of the person who will be conducting the responsibility of “human in the loop”. In reflection of this, if we narrow down our talks to address the use of AI systems in the courtroom, then we will need to direct our attention to the litigant, the defendant and the claimant, who very often has no idea that an algorithm shapes how their case is processed, prioritised or assessed. Transparency, in this setting, is not only a technical requirement. It is a matter of dignity. People have a right to understand the basis on which justice is being administered to them.

I left the audience with a simple test. Before any AI tool enters a judicial process, there should be a clear answer to three questions: What does this system do? On what data was it trained? And what happens when it is wrong? If those questions cannot be answered, the system has no business being in the courtroom.

What Do I Bring Back with Me from Rio to Munich?

What stayed with me longest was not anything I said, but what came from the floor. One colleague, Justice Fabio, asked simply: “Where is the human in this digital revolution?” This question serves as a reminder that technology can blind and corrupt the judge as much as it can assist. Then, the task is to tame the algorithm in the service of fundamental rights, to use technology without being used by it. Justice Anderson raised the prospect of AI enabling a kind of global precedent system, courts drawing on rulings far beyond their own jurisdiction, and the strange new question of whether something like a “digital legal personality” might eventually need defining. Among the most highlighted and raised questions that are worth investigating was: “Who actually gets to define bias? Is it the judge, technologist or the practitioner who lives with the consequences?”

Another highly remarkable note which stayed with me the most was a comment raised regarding the findings of a UNESCO report on judicial use of AI which stated that the worrisome point, here, is AI is deskilling judges rather than upskilling them—some are beginning to delegate judgment itself, not just the research that supports it. This is automation bias in its most consequential form. And as one colleague put it, the greater danger is that AI tends to agree with you rather than question you. A good law clerk pushes back. This highlighted a point addressing the importance of AI literacy, as raised in the AI Act, since several of us agreed that judiciaries are simply not yet being trained—not in the tools, and not even in how to prompt them responsibly.

Why This Kept Me up at Night

I did not travel to Rio to deliver a verdict on AI in the justice system. I went to share a framework still very much under construction, and I came home with a room full of questions that will sharpen it. This, I think, is the real value of these conversations across regions—not exporting answers, but testing the questions against contexts very different from the one in which they were first asked.

AI will not destroy justice. But ungoverned AI in courts, deployed without transparency, accountability or the voices of those most affected, in the room where it is designed, will quietly erode the trust that makes justice possible in the first place. This is a problem worth solving. It is also, I suspect, a problem that will define a meaningful part of my research project.

Mohaned Bahr is an alignAI doctoral candidate at the Chair of Public Policy, Governance and Innovative Technology, Technical University of Munich, Chaired by Prof. Urs Gasser, Rector of the Munich School of Politics (HfP), as well as Dean of the TUM School of Social Sciences and Technology. Mohaned is also a practicing judge in Egypt. His research, conducted as part of the alignAI Marie Skłodowska-Curie Doctoral Network, focuses on the governance of large language models through the lens of explainability and fairness, including their use across various contexts, such as judicial and public administration settings.