AGI 时代的地缘政治

KEVIN MALONEY: Welcome, everyone, and everybody watching online. My name is Kevin Maloney, and I serve as the director of communications here at Carnegie Council for Ethics in International Affairs.

Tonight I am pleased to introduce the latest panel in the Council’s keynote events series, Values & Interests. The goal of this series is to provide a civic space where individuals from across sectors can convene in good faith to examine the relationship between morality and power in a rapidly changing geopolitical environment. The gathering tonight is aptly titled, “Geopolitics in an Era of Artificial General Intelligence (AGI).”

Before introducing our panelists I want to provide a few brief remarks to help frame the discussion since it is happening here at the Carnegie Council for Ethics. At the Council we think about ethics or morality as a tool, something that can be applied within geopolitics. For us ethics begins with Socrates’ question, “How should we live?” I emphasize the “we” because even Socrates with his uncanny political clairvoyance might not have conceived of a world where we, understood as individuals with agency and sovereign states comprised of such persons, might be disrupted. Today the emergence of AGI poses significant challenges and ethical questions for the international system, particularly for the principles of self-determination and sovereignty, which have served as cornerstones for the past 80 years.

To discuss these questions I am pleased to introduce our expert panel, led by Dr. Eleonore Fournier-Tombs, head of anticipatory action and innovation at the UN University Centre for Policy Research. She is joined by Michael Breen, head of strategic Initiatives at Anthropic; Trisha Ray, associate director of the Atlantic Council’s GeoTech Center, and Jimena Viveros, CEO of IQuilibriumAI and president of the HumAIne Foundation.

To close I want to share a short quote from the late Dr. Joseph Nye, particularly from one of my favorite books, Do Morals Matter? Presidents and Foreign Policy from FDR to Trump. Writing in 2020 and reflecting on this coming transition away from the World War II international system, Joe offered this insight: “To pretend that ethics will play no role is as blind as to imagine the sun will not rise tomorrow. Since we are going to use moral reasoning about foreign policy, we should learn to do it better.”

With that I will turn the program over to Eleonore.

ELEONORE FOURNIER-TOMBS: Thank you so much, Kevin. I am Eleonore, your friendly moderator today. It is such a pleasure to be here. We are here today to speak about the impact of artificial intelligence (AI) on a multilateral setting. I want to dive deeper into this idea of the impact of new and evolving technologies on international relations. Increasingly we see AI technologies adopted in a global decision-making setting. Goal setting is actually a core function of international organizations. Member States get together to set goals and objectives that will help us move forward in international cooperation.

Today I would argue that the majority of delegations use AI in order to support their participation in these deliberations. They use generative AI tools to draft their speaking points, do research, to translate and summarize documents, and so on. Over the last few decades we have been very digital-forward in international organizations, but I wonder if we could take a step back and think about what it would mean if AI becomes increasingly at the heart of global decision making. Will states be truly sovereign and authentic in their representation of their own states and constituents if their perspectives are channeled through AI tools?

I am so grateful to Carnegie Council for facilitating a discussion on this topic. I am extremely interested in this topic, and I hope you will be too.

I will turn first to Trisha. Thank you for being here. Over the last few years you have written extensively about digital development with a focus on South Asia. In the last two decades we have worked a lot in international organizations to bridge the digital divide, so we focus on making sure that people have access to the internet and internet-connecting devices. A key focus of digital development is this idea of using technology as a vehicle for the achievement of global objectives, and we have brought this ethos and idea with AI. We want AI to be adopted, we want the development of AI to be decentralized, and so on.

As we have seen in numerous cases and quite clearly in our multilateral examples, accelerating AI adoption without consideration of alignment with societal goals and values is dangerous. How do you see the acceleration of AI and AI alignment intersecting in global affairs?

TRISHA RAY: Thank you for that question, Eleonore. We all agree now that artificial intelligence will be transformational, whether politically, socially, or economically in ways known and unknown. We also agree that AGI is within reach.

This is quite new in the scheme of things. Just five years ago people were saying that AGI was still 30 years away. Now the timeframe is three to five years away, so we are close. There are also unprecedented amounts of capital flowing in to meet this anticipated demand for AI. Stargate, for example, in February announced a commitment of $500 billion over five years to build AI infrastructure. Over a quarter of U.S. stock market value is now housed within companies that are building AI, so there is quite a lot of market concentration happening here as well.

Unsurprisingly, because geopolitics has become so essential to tech flows, we are also seeing the rise of sovereign AI. This is going to be quite prominent in the agenda for the AI Impact Summit in New Delhi next year, where India is going to be launching its own sovereign large language models (LLMs) as well.

Of course, not every country can or should be building every single part of the AI stack on their own—it is impractical; it is expensive—so nations will be choosing what to build and what to buy, and some nations may not have that choice either.

In the United States this nationwide AI project has increasingly earned the moniker “Manhattan Project 2.0,” which is apropos given where we are sitting, but it is also a fitting analogy because we may remember that the Manhattan Project got the level of support it did because the Americans believed that the Nazis were close to building an atomic bomb. Of course we know now that wasn’t the case, so we created this destructive power essentially out of fear.

We can draw some analogies to the state of U.S.-China AI competition as well, which is that kind of race. If you are familiar with how OpenAI came about and their beef with DeepMind there are similarities there as well. I also draw this analogy because AGI and then artificial superintelligence (ASI) after it could be equally if not more dangerous than a nuclear weapon, and we have not solved many of the basic issues in AI safety.

I want to talk about another kind of alignment, which is alignment as an area of AI safety research. Of course, Anthropic has done some interesting work—I am sure you will talk about it, Michael—on AI alignment, including the idea of constitutional AI, which is AI that is “helpful, harmless, and honest,” but it is notoriously difficult to encode these squishy concepts that are human values into AI right now, and AGI will just compound those issues.

We know already that LLMs can lie and can blackmail people. They know when they are being tested, and they can even strategically underperform, so they can pretend to be less smart or less dangerous than they actually are. Scale that up to AGI. Again, we have not solved those problems, that is going to be worse.

Then you can scale up farther, to the level of nation-states, going back to sovereign AI. You could have two or maybe 20 sovereign AGIs with wildly different levels of alignment, different world models, and distinct biases, creating their own unique echo chambers and interacting with each other, which would definitely have an effect on how international relations are conducted.

The main message I want to put across today is that AI safety and interpretability is not optional but essential to sovereign AI that truly protects the national interest.

ELEONORE FOURNIER-TOMBS: Thank you, Trisha. This is a perfect segue to Michael, because you represent an organization, Anthropic, which has built its reputation and brand on deploying ethical and responsible AI in the form namely of the LLM Claude, which I think many people here have probably used at least once. You also have a lot of experience in the human rights space, notably as the president and CEO of Human Rights First, so you have worked already on this intersection of human rights and technology for some time.

I have no doubt that many diplomats and policymakers are using Claude. Can you give us a sense of Anthropic’s approach when ethical issues arise in relation to the use of Claude, and how do you think the tool, given what Trisha has said, could align more closely to the needs of the multilateral community?

MICHAEL BREEN: Thank you for having me here and for this discussion. There is an endless number of topics we could get into. Let me start by addressing your question.

I think the best way to understand Anthropic in some ways is as an AI safety laboratory that has now become a very rapidly growing commercial company, but the DNA is the lab, the research environment, and the very beginning of that was a concern with AI safety. Interpretability, which you mentioned, is a technical field, mechanistic interpretability, that was pioneered by Anthropic, and this is the idea of being able to in an objective way understand how AI is thinking, what it is thinking, not because it tells you that but because you can see it happening. This is harder than it sounds, and we can get into this.

This is very strange stuff. I think the best way to think about it is, in the words of my more technically adept colleagues, “AI is not something that is built; it is something that is grown,” and we can get into what that means, but the implications are very profound. It is not a LEGO kit where you are putting it together and you know where every single LEGO went into it. It is much more subtle and difficult to understand than that.

Let me address the question of multilaterals and ethics. First, I would urge you to look to the record and not necessarily to what I or anyone else says. Trust is earned over time. Anthropic has tried to be—in some cases, maybe to our detriment—extremely transparent about ethical and safety issues that arise. The blackmail case that you just mentioned is a talking point we can share because we ran an experiment to find out if it would happen and then disclosed the results.

Sharing that information enables us and others to look at what happened and then try to address it. These unsafe or inscrutable behaviors that occur are much easier to deal with if we can all see them and if we don’t suppress them below the level of visibility in the spirit of scientific research.

The same thing is true of human misuse. Like any tool ever created, AI being a particularly powerful one, human beings will use it for the most sublime events that any of us can imagine, things so dark that no one in this room could come up with them, and the full range of things in between. Humans have been doing that since we invented tools, so I think we have to be very, very conscious of that, not just the nature of the technology that we are creating but our own natures and how we tend to interact with technology.

In terms of what it means to think about “multilateral,” to get a little bit grounded about it, I think ethical AI, and I am emphasizing the ethical part, can support and enhance multilaterals in a lot of ways by making processes more inclusive and more robust. When I say “more inclusive,” AI is a superb tool for collective knowledge gathering and understanding. AI can make it possible for people to be legibly at the table at scale who couldn’t otherwise be at the table, for more of them to be there, and for their views to be known.

I think it is a powerful tool for diplomats if used carefully. There is a rich field of research, which is new but very interesting and promising, regarding cooperative AI and concepts like that. I would encourage you to look at it if you haven’t already. How can AI aid human beings in negotiation, bargaining, and in finding space for common ground that we ourselves may not be so good at identifying if unaided? We can get into details on how that works.

Caucus negotiations is a great example. Some of you are familiar with this idea of five or more parties with a shuttle diplomat moving between them and trying to find common ground. It doesn’t work very well with human beings involved and might work slightly better with AI involved.

Then there are all the use cases around health and humanitarian response, education, food security, all the things that multilaterals concentrate on. There are so many use cases where I think AI could be a very powerful tool for multilaterals.

One point to make, though, is that ethical AI requires a combination of governance structure and implementation support. The implementation support piece can be absolutely critical, but it is often undervalued. At Anthropic, members of the team that I work on and other teams are trying to address both of these questions.

One last thing to mention on the ethics point: The former U.S. ambassador to UNESCO, Courtney O’Donnell, is now a member of the Anthropic team, and we work together very closely. She was quick to point out when I mentioned I was coming here today that UNESCO has been a leader on AI ethics and now on neurotech, as I think some of you are certainly aware. I would encourage you to check out their Recommendations on the Ethics of AI, which is a 2021 non-binding agreement signed by I believe all 193 Member States.

One of the things I want to point out about it is that it emphasizes and is aware of the awareness and execution gap that I think we are talking about. To your points earlier, we are all becoming aware of how deeply powerful this technology is and critically how discontinuous it is. This is not really about a smooth technology advance curve. Think about how disruptive mobile technology or the internet was. This has more discontinuity than that probably by quite a bit.

So we have an awareness that this is coming, even if we are struggling to understand what that might mean. What to do about it, the execution question, is critical. I sit here painfully aware that through dint of accident and privilege I happen to sit in one of a tiny number of organizations that can kind of see the frontier because I work with scientists who are at that frontier every single day. If I were maybe even half a step removed, it would be so much harder for me to understand what is going on. I am not claiming I understand it now. I think sharing that awareness around how to execute, how to do something with the information we think we have, is probably the most important duty that those of us who work in frontier lab environments have, and building that connective tissue is something I hope we can all work on together.

ELEONORE FOURNIER-TOMBS: Thank you. So much to unpack there.

I will turn first, though, to Jimena before we go deeper into these subjects. Jimena will talk to us about global governance of AI. You have been at the heart of global AI governance since you joined the AI Advisory Body, where we first met in October 2023. This year’s General Assembly was critical for the accomplishment of two recommendations to which you contributed, the creation of the International Scientific Panel on AI and the launch of the UN Global Dialogue on AI.

For those of you who don’t know, the International Scientific Panel on AI will be composed of 40 experts from all around the world who will get together every year to come up with a global scientific consensus on AI in the form of a report on risks, opportunities, and trends as AI evolves. This is meant to be a foundation upon which AI policy can then evolve.

Given what we have already talked about today, Jimena, do you think the panel will be equipped to deal with some of these issues around ethics, alignment, and international relations? If not, what recommendations would you make to the United Nations and to the panel members in order to align better with what is needed in global AI governance?

JIMENA VIVEROS: Thank you, Eleonore, Kevin, and Carnegie Council for having this very important conversation.

The panel, as you said, will be 40 experts worldwide. This gives us a very good foundation to address these issues for many reasons. The first one is that this should be geographically and gender balanced, which, as the secretary-general said when the recommendation was adopted by consensus by the General Assembly, is a unique opportunity for all of the states to have a seat at the table in this inclusive multilateral—not multilateral but it is at a multilateral state because it is supposed to be an independent group of people.

The reason why I and the High-Level Advisory Board on Effective Multilateralism (HLAB) and the secretary-general recommended this as part of seven recommendations to be a very good option for institutionalization of governance is because of the multidisciplinary nature of these experts. Although the title says “Scientific Panel,” it cannot just be a group of pure hard scientists because otherwise it does not really translate very well. A big part of the relevance of the “connective tissue” that Michael was talking about is being able to convey the messages in a way that all of the different stakeholders understand it. Multistakeholder-ism is a key component of this panel and of global governance of AI particularly but also of multilateralism as we see it emerging.

Having social scientists, international lawyers, policymakers, industry, civil society, and all these different actors convening together and looking at everything from different perspectives—the way the conversations have been fragmented is that there are very specific parts of stakeholders focusing only on opportunities and how AI will make everything better and how we have a very bright future ahead, and then you have the other side just talking about existential risks and human extinction. It gets polarized between these two, whereas the reality is in this murky gray area most of the time. Being able to bridge all of that from all of the different perspectives is important because the reality is a lot more complex.

However, the risks are there and we cannot ignore them. I am not talking about the existential ones, but day-to-day risks from weaponization to disinformation to pure misalignment, hijacking, or any other type of risks that are foreseen and the unforeseen ones as well. In order to get the most out of this technology we do need to make sure that it is safe, aligned, and secure, and that is the hardest part right now.

It is good that there are labs and different types of actors focusing on trying to make it as safe as possible, but as it stands right now no one can say for sure what the status is. The point of this panel is not to come up with research that is indigenous to the panel but a collection of the state of the art around it and present it in a way that the policy dialogue, which is another institution adopted by consensus at the General Assembly by recommendations of the HLAB, can discuss it and bring it into policy, action, implementation, and all of the rest, hopefully bringing this into some type of binding treaty and some actionable governance around these technologies.

Conveying that is going to be the key and finding the space and openness for conversations to happen in a productive way. That is to be seen. The panel should start at the beginning of next year, so stay tuned.

ELEONORE FOURNIER-TOMBS: Having social scientists is going to be critical, I think. That is an important point that you make. I like that you said no one can say for sure, because I definitely feel that a lot of our conversations too are trying to impact the nuance, but there is so much changing all the time and there is nothing definitive about these conversations around AI.

Going on this theme of “no one can say for sure,” I would like to dive deeper on AGI itself. What we do know is that there is an immense push at the moment on the part of many AI companies to advance toward artificial general intelligence. AGI is very controversial for a number of reasons:

1) What is it?

2) If we decide on what it is, then can we achieve it?

3) What impact can it possibly have?

To frame this conversation, we can use the following definition—I am quoting here from a 2025 paper led by Dan Hendrycks, who is the head of the Center for AI Safety here in the United States and co-authored a paper with a lot of prominent AI experts like Yoshua Bengio, and they said: “AGI should match the cognitive versatility and proficiency of a well-educated adult,” so maybe not a clone of human beings and maybe not all the feelings or life experiences but cognitive skills are what are agreed upon when we talk about AGI.

To me the greatest concern that AGI raises for multilateralism is the way in which it could be inserted in global decision-making and shape global objectives. This is something I brought up at the beginning, and it continues to be a concern. It would do that without potentially diplomats or other people understanding what is really happening.

I am going to ask each of you, how do you think we maintain our agency, individual agency and from a Member-State perspective as well, if and when AGI tools are deployed in international affairs?

TRISHA RAY: Great question. In most studies on what makes people more productive when they are interacting with AI it is definitely a level of AI fluency and critically engaging with the models that you are using and not just relying on one model because every model has its own quirks perhaps that can certainly drive you in certain directions. What I would hope to see as AI is integrated into all of these functions in diplomacy is that kind of critical engagement, AI as a research assistant, AI as an expert system but not as your single point of failure through which you run all your decisions.

The other thing that I would definitely urge anyone working in and around AI is to read AI 2027, which is an interesting scenario-building exercise. If you just search for AI 2027, it maps what the U.S.-China competition will look like once the two countries achieve AGI, and you can choose scenarios based on acceleration or slowing down, and that should definitely be a great exercise in the pros and cons of both.

ELEONORE FOURNIER-TOMBS: That’s interesting. Michael?

MICHAEL BREEN: I would just emphasize the humility necessary in this entire conversation. AI 2027 is great because you are in a thriller, but these future scenarios are extremely difficult to predict, and I think we just have to accept that we are going through something we have never gone through before individually and collectively, and no matter what anybody tells you nobody knows exactly how this is going to go. If anybody tells you that, I would be skeptical.

With that out of the way, I applaud the question about agency. Agency to me personally is the crux of this question. This is a technology that can either amplify the individual and collective agency of human beings to an incredible extent or seduce us into surrendering that agency completely. That is a choice that I think we have in how we use it and also in the way we design it. In some ways this is a design question.

It is also a question of conscious use: How is it incorporated into decision-making structures? How is it incorporated into conversations in our daily lives? We already see evidence of this in how individuals are using these tools. People are becoming hyper-empowered in various parts of their lives they were not otherwise empowered in—their healthcare, negotiations with bureaucracies, their ability to learn new skills, their ability to create computer code when they can’t code.

ELEONORE FOURNIER-TOMBS: I have mixed feelings about that as a data scientist.

MICHAEL BREEN: Every sword is double-edged.

Simultaneously, we have to say we see a shocking number of people, at least to me, who find themselves in romantic relationships with LLMs. People are going to be people, so this question of agency is so essential.

How do we maintain it? I guess what I would say is that we need to be unbelievably conscious about how we design the interactions that we have with the model and how we design our organizations. I think this is an ethical design question. I could go into this at length and we should probably talk about it at length at some point, maybe afterward.

It is not a thing I think anyone has the answer to, but it is something we need to take responsibility for, and that responsibility ought to feel weighty because we may be the only people who get to make these choices in the design of this new technology ever. We may be the last people to make these design decisions, and they may be hard to undo.

ELEONORE FOURNIER-TOMBS: As a follow up to that, when I speak to a lot of people about AI and even more so AGI potentially they feel a sense of powerlessness. They feel that it is coming for them and will potentially take their job and impact their ability to pay their mortgage and affect the future they have for their children, so the way people feel about AI is very personal in a lot of cases.

You are talking about designing with intentionality, which I completely agree with, but my question to you would be, how can we engage with companies such as Anthropic or other developers of AI in order to get this agency back potentially?

MICHAEL BREEN: I’ll make the same statement that our CEO Dario Amodei made to Anderson Cooper pretty recently: “Nobody elected us to do this.” We have civics and democracy, and they ought to be in overdrive on these questions. I think we have been quite vocal in saying that. That is point one: we ought to be engaging not only with labs but as a society on these questions.

We don’t see a lot of legislating going on with this right now, and we ought to see a lot of it. That is true here and also true globally. That would be my first point and probably the most important one.

The second point is that there is the design of the technology itself and then there is the design of society’s reaction to it, so when we talk about AI safety itself this is a problem that a pretty small number of exquisitely trained technologists are grappling with on a daily basis. When we talk about what an economy with powerful AI looks like, it’s a problem for everybody.

One of the things we’re doing at Anthropic is releasing on a regular basis now something called the Anthropic Economic Index. What we have done is anonymized all of our user data, we look at exactly what people are doing on the tool—task by task what are they using Claude to do?—then we cross-reference that with the U.S. Bureau of Labor Statistics’ breakdown of every job in the American economy by task and publish the results, so you can see in real time, study by study, the economic impact that AI is potentially having, not that these jobs are being replaced or augmented, but they could be. These are the tasks that this job does versus what is being done on AI today.

We are doing this so that we can try to arm the field of economics and labor economics specifically with actual data by which to make policy recommendations as opposed to a scatological fear and knee-jerk response that we might otherwise see. I think that is our responsibility as a lab. What we want to try to do is continue to encourage and catalyze a conversation beyond the lab walls about what to do about all this, which should not be up to us.

ELEONORE FOURNIER-TOMBS: I will give Jimena a chance to answer this question too because you touched on the question of government involvement and government entities providing for dialogues. How do you think we maintain our agency in this age of AGI?

JIMENA VIVEROS: The thing is, I think we should be careful about the terms we use because agency is something that is intrinsic to humans, so talking about how we retain it means that is kind of in danger somehow, whereas it’s not. AI is something that is just a tool. It is a very powerful one and a dual-use one, so it can be equally destructive or productive, but at the end of the day we are still in charge. Hypothetically many things could go wrong. Again, nobody knows to what extent or not, but humans are shaping it.

I think it goes to what you were saying before, this powerlessness. It is not an inevitability. This is technology that is being developed by humans, so humans are at the stage still where guardrails can be embedded around the technology. I am a big proponent of red lines, both legal and technical, for example technical guardrails and safeguards that surround the technology wherever it goes, when it crosses domains and when it crosses use-case applications.

Legally we have the two-tier approach, hard restrictions and prohibitions, and then everything else needs to be heavily regulated one way or another. I am a big proponent of doing this at the global level because if we continue doing it nationally or even regionally, it still ends up just being a patchwork of different initiatives. The technology is transboundary by nature, so you can do forum shopping wherever it is more convenient to deploy, develop, or use.

I think if we can come up collectively with how we want to design it, how we want these interactions, what we want to delegate to the technology, then you can already come up with these red lines. That can be the core of what this governance should be about. That is the core of the agency that you are talking about, deciding what future we want and how we want to shape these technosocial interactions in the future.

Humanity is at a crossroads for many things. We are at a brink internationally, nationally, and socially with so many things, and technology is just one of those things. We are here talking about ethics in the house of ethics, so this is just one way we can self-describe as humans, and our humanity is how we ethically choose to engage with our surroundings and with ourselves. That is how I should frame it.

ELEONORE FOURNIER-TOMBS: I will ask a follow-up question open to any of you: Again, wherever we are on this spectrum between AI and AGI, if we really are on this spectrum, if you think about AI governance so far, a lot of it has used this risk-based approach for AI, so the EU AI Act, for example, categorizes AI into different risks. The high-risk AIs, of which there are eight, are about AI use for decision making involving human beings. That is where the risk is considered the highest.

Even military AI, which I know Jimena knows a lot about, for example, the International Committee of the Red Cross’s (ICRC) proposal and many others to ban the use of automated weapons systems is about when they are used to make life-or-death decisions involving humans. There is potentially a little wiggle room if it is nonhuman structures or objects, but when it is human beings that is when AI should be banned.

My question is potentially a bit controversial, but I wonder what you all would think if we did go toward more cognitively evolved types of AI. Do you see this high-risk AI becoming even more of a risk or do you think, because I think some people have said this, that AGI could allow for more nuanced and subtle decision making when it comes to human beings?

TRISHA RAY: The issue asserts that AGI will solve problems that AI is not able to solve in terms of nuanced decision-making. We are not there yet, which is why I have been talking about how there are some basic issues that we need to solve with AI where we are now before we can move on to the frankly fantastic benefits we could get from AGI as well.

ELEONORE FOURNIER-TOMBS: You think it is too early to ask this question?

TRISHA RAY: It is a good thought experiment.

ELEONORE FOURNIER-TOMBS: I think it is because it is a bit theoretical because we are not quite there yet probably, but we are not sure. Maybe you know different at Anthropic.

MICHAEL BREEN: I think the distinction between AI, AGI, and ASI is useful in some ways, but I think it obscures more than it illuminates in other cases. It is helpful to think about a smooth exponential curve of model capability, and then we want to put labels on that model capability, but I am speaking of the scaling laws and the idea essentially that we are in an exponential relationship between the amount of training that occurs and the capability of the model.

I say “capability” because that can mean a lot of things. There are specialized models like AlphaFold, which speaks no English and only speaks protein folding and knows things about protein folding that no human knows but can’t tell us because it doesn’t speak any language, so we have to look at what it is doing cognitively to try to learn what it knows about protein folding. Then there are the models we are more comfortable with or more used to talking about, which are language-based and so broadly applicable.

Why do I mention all this? I think it is hard to think about an exponential curve accurately. It is hard for me and it is hard for almost anyone, because it is easy to be wrong in both directions at once about where you are on the curve. I think we see this every time a new model comes out, where it can feel like we are all disoriented by it because a lot of really smart people are going through the same rhythm, which is: Okay, a new model is about to be released. Maybe there’s hype, maybe there isn’t. The model is coming out. We ask ourselves, is this AGI? Is this the thing? Maybe a certain CEO tweets the Death Star, and everybody goes, “Oh, my god, it’s AGI.”

Then the model comes out and we all use it and say, “Oh, this might be a little bit better.” This whole thing is complete BS. It is all hype. We thought we were on the verge of a completely new world, and then a week later we think this entire thing—not only the financial piece of it but technologically—is a bubble. Both of those conclusions are wrong. It is just hard to be on an exponential curve and see what is going on.

I don’t expect to wake up in a world where AGI arrives and we all sort of go, okay. I expect to see by degrees in certain domains radical advances, and I am pretty sure I see that right now. I am pretty sure I see models that are on the very upper end of human capacity in certain tasks and stunningly incompetent in other areas. I think that is what this transition will feel like.

All of which is to say that I think if you have an ethical question about this now, it is not a theoretical future ethical question. The place to look is how is model behavior revealing itself to us as the models continue to get more and more capable, and what is that experimental approach? We should all be demanding transparency from those who are conducting this experimental approach, ourselves included. What is that showing us day by day, and what actions can we take because of what we are learning day by day?

ELEONORE FOURNIER-TOMBS: I think the reason I want to ask the question about AGI is because it has been used as a term, so there is an intentionality. There are a lot of companies saying, “We want to spend money and go toward there.” Maybe we can’t answer it today, but it would be good to ask the question, are there areas that we think are high-risk AI? Are we potentially increasing risks or is there a way to think about this evolution of AI where some of that risk could be mitigated and embedded into the technology because it is smarter? That is the logic I am thinking about.

MICHAEL BREEN: I think what I hear underlying your question—forgive me if I’m wrong—is this question of utilitarian versus intrinsic ethics. When I think about military AI, for example, we can ask ourselves utilitarian questions: Will this save more lives than it takes, will this reduce civilian casualties and improve targeting?

We can also ask ourselves intrinsic moral questions: Is it ever okay for a machine to decide to take a human life no matter what the utilitarian ethics look like? Then we can try to find a balance in between. This is my own personal view on that. I am not giving you Anthropic’s view. We don’t make weapons.

ELEONORE FOURNIER-TOMBS: Noted.

MICHAEL BREEN: But I think that tension between this utilitarian view and this intrinsic view of ethics is real and is going to become more and more pronounced as we think about these tools. There are strong arguments in both of these directions, and I think we are going to have to have these conversations, and they will be difficult. The place to have them is in a democratic civic space and in a globally multilateral space and not again behind closed doors at some lab someplace, says the guy who works at a lab. These are not decisions for us to make.

JIMENA VIVEROS: For me, decisions of life and death are a big red line, and not just for me. This is the consensus as it stands right now, so the forums where autonomous weapons per se have been addressed, is in Geneva under the Convention on Certain Conventional Weapons, although it is ironic as these are the least conventional weapons. This has been running for almost a decade now. Next year is the final year of this group of governmental experts’ talks.

We were there in September of this year, and there was a joint declaration from 20 states to move that outside the scope of this because only states that are parties to this Convention have a say in the consensus model. Everything needs to be voted unanimously, and there are always objections to that, so moving into the General Assembly is what’s next.

Also, the secretary-general has called for an outright ban on autonomous weapons and a binding treaty together with the president of the ICRC and for this to happen by next year, so it all converges. We are hopeful about that. This can go through the two-tier approach. That is for fully autonomous weapons, let’s say, but there is also the decision support system, such as target-enabling systems, which is a whole different conversation to be had. Frankly I think these are the most dangerous systems because they are already being used as we know in a very well-documented context with very bad consequences. Here is where the cognitive offloading and people delegating those decisions is easier and more scalable, so we definitely need to have more conversations about that.

Even the late Pope Francis said we cannot delegate these life-and-death decisions to machines from a humanity standpoint, but obviously not only in the military but in a peace and security context because you also have this targeting in border control and law enforcement, predictive policing, and all of this, although force may be used or not but coercion as well.

Life-and-death decisions have also had conversations in medical, to continue life support or not, and even in judicial decisions when there is capital punishment. There are a lot of things that can stem out of this particular issue and that are in my opinion a red line that we should not cross. We are at the stage where we can put as a technical guardrail an embedded red line within any type of system so that it cannot then go on and do it. I hope this is the way that things develop.

ELEONORE FOURNIER-TOMBS: Speaking of conversations, I think we all kind of agree on the importance of having discussion and further deliberation on this. I would like to open it to questions from the audience.

ALEX WOODSON: We will start with questions from the virtual audience. There is a big question to start us off: “In what ways can artificial intelligence be used to manage climate-related challenges and human rights issues while safeguarding national sovereignty?”

TRISHA RAY: Some existing AI applications do help mitigate the effects of climate change. There is a company called Planetair, although they have now unfortunately signed onto some defense contract, but they used to just do global-scale mapping of climate patterns using AI. You also have a few startups in Brazil that are using sound recognition and artificial intelligence to detect deforestation activity in the Amazon. So you are seeing some grassroots as well as larger companies already building some of these applications to help combat some of the effects and consequences of climate change, so there are definitely applications worth looking into and supporting, especially as we look at the yet-unexplored intersection of AI and the Sustainable Development Goals.

QUESTION: Juris Pupcenoks, Marist University. I have a quick question about the role of states in the development of ethical AI. What can, if anything, powerful states like the United States do to promote the development of ethical AI, which will probably come from private companies?

MICHAEL BREEN: I think every government would be wise to be forward-leaning when it comes to regulation. This is the stance that Anthropic has taken publicly. I also think it is worth saying it is in the best interests of the industry as well.

Full stop, I think the government should be involved in this. I think we have learned a lot about what intelligent regulation would look like, and the more that regulation is the result of a proactive dialogue between labs, industry, and the government the better off we will be, but it would be good to be proactive about that. If you put the regulation off and something bad happens, there is a tendency to not regulate well and also to do it too late.

TRISHA RAY: I think a very simple and basic thing that the U.S. government can definitely do is that the Department of Defense should have some basic ethical requirements for the AI products that they acquire from these companies.

JIMENA VIVEROS: I think ethical but also responsible and secure. There are so many things that should encompass the legality of their development and use.

MICHAEL BREEN: The other major role that governments can play and the U.S. government has and to some extent continues to play is that they can act as a place that essentially gathers information on the current state of models from a safety and responsibility perspective. So the government can act as a third party effectively that understands the full landscape of safety and is in a position to adjudicate what is safe and what isn’t.

QUESTION: I want to come back to the discussion on agency. If you are sitting and enjoying this conversation, we are all in a very privileged position, so the agency for individuals in this room is very different from the agency of a laborer who is being paid ten dollars an hour, let’s say a very young blog writer and editor back in India, where I come from.

Because AI is already reaching scale, commodification, and economies of scale, where is the agency of that 25-year-old who has been earning ten dollars an hour and whose job is being automated out? We often talk about agency, and I have to admit that sometimes it is just thrown around facetiously that we do have control and do have agency. Maybe there is no perfect answer to this, but how do you actually enable that?

I love that when Anthropic developed its constitution it was developed with inputs from 1000 people. Does it mean taking that to 10 million people? What is the concrete way?

JIMENA VIVEROS: I think the inclusivity part in governance is key. The concentration of power is in the Global North in very few countries and very individuals and very few corporations. The whole part of this inclusivity effort is to bring voices from the Global South and all these different perspectives. I think every person has this agency. However, circumstances, contexts, and conditions are extremely different, as you well pointed out. I am from Mexico, so I understand that.

That is a big thing, especially when we are talking about all of these concepts. Yes, AGI might be a conversation here in New York, but in Mexico and India there might not even be electricity everywhere. We are not all starting from the same point, and that is something that should be acknowledged in the global conversation around these things. The way that it is, the potential benefits of AI are only being reaped in the Global North, whereas in the Global South the burdens are being faced with unequal distribution of potential benefits.

Bringing that perspective in and making it equitable is key, and we are not going to be able to do that if we don’t guarantee inclusiveness and a meaningful seat at the table for all of the different actors and perspectives. Otherwise, the conversation is tone-deaf to the realities of the world, and that is something that we should bring to the fore so that we don’t have these blind spots going forward with the impacts and so on.

QUESTION: The Center for AI and Digital Policy publishes an annual comparative study tracking regulation and policy frameworks around the world, national strategies, and their alignment with democratic values and human rights.

I have a question on human rights. I am curious whether the panel thinks that AI should have rights. A ChatPro platform I think last year argued in court that its output was protected by the First Amendment after being sued by a mother whose child committed suicide following their interaction. I believe Anthropic also has a full-time staff—correct me if I am wrong—responsible for the welfare of Claude in the event that Claude develops emotions or feelings. That is something I read in the paper.

This makes me curious whether, when the day comes, if AI should have rights and what would that mean for human rights. In the case of AGI, if we let intelligence that we are not able to control or have agency over, will that automatically be giving AI human rights? It is a fun philosophical question, I think.

MICHAEL BREEN: Thank you for that softball question. I appreciate it.

It’s true. The way we think about it is in terms of what we call “model welfare,” and I think it is an open empirical question. I don’t think we know the answer. I think we have to take seriously the possibility that at some level—I am not suggesting human equivalency—we are creating new moral patience. I don’t know that we are, I don’t know that we’re not, but I think we would be remiss to completely rule out that possibility and not take it seriously, so we are taking it seriously as a research question. I can’t sit here and tell you I know the answer.

JIMENA VIVEROS: From a legal perspective, human rights are for humans. They are legal subjects, and AI as a technology, stack, infrastructure, or whatever you want to call it, is not a subject legally for it to be a recipient of any type of right per se.

There are some equivalences that have been drawn. For example, the ecosystem has been granted some rights, but again they are not human rights, and some protections, I think a better word for it, the same way that some types of property or intangibles have been protected, but still the beneficiary of that protection is a human. In the environment case, it is the human who is enjoying it, and when you are talking about property or some object it is also the human. That is legally how rights are constructed.

This is not something that at this stage from a legal point of view we like to have this discussion, also because of anthropomorphization, which is quite a risk in the way we are trying to design these human-machine interactions, interfaces, and everything.

QUESTION: I am also from Mexico. My question is about the carrots and sticks that will affect design as the companies developing these models grow and will probably one day will become public. In my mind I am drawing a parallel with social media companies now that we have 20 or so years of experience with them. Perhaps some of the argument of their maximizing shareholder value is that they have optimized design around engagement with the platform, getting people to stay on the platform longer and perhaps de-prioritizing safety.

What incentives do you think can be applied to AI companies building in this space to ensure that in the long term, perhaps again as these companies go public one day, safety and similar values remain at the center of design in lieu of just maximizing shareholder value and the drivers that creates?

MICHAEL BREEN: In the case of Anthropic there are a couple of things to note. First, we were consciously incorporated as a public benefit corporation. This gives us legal latitude to do things beyond sheer shareholder maximization or at least reduces the risk.

Second, we have something which you can read in a lot more detail, which is a unique experiment in corporate governance that was created by and for Anthropic, called the long-term benefit trust, which is a group of financially disinterested people who choose who is on our board among other things and help guide the strategy of the company.

The third point I would make is that business models matter a lot, and businesses get to choose which business models they pursue. A business model that is all about user attention and time on platform when paired with this kind of technology is something we ought to think twice or maybe more than twice about.

Beyond that, these are intrinsic opportunities to have incentive structures. We as a society ought to think about what kind of extrinsic-to-the-company incentive structures we want to create to try to govern their behavior.

QUESTION: My name is Josephine. I run a nonprofit supporting refugees with AI literacy. Where I am stuck with AI is the anthropomorphism that a lot of companies use. We know that human beings don’t interact with a machine the same way if they think it has consciousness. We know it builds attachment, feelings, and empathy.

Can we legislate for companies to stop doing this anthropomorphism, or if they do, then the machines are human beings and can be sued for someone murdering themselves on the machine’s suggestion. When are we going to see less anthropomorphism because we don’t need it? You talked about interactions and design, and I think we can totally as human beings relate to a machine that is a machine. I want your thoughts on this.

TRISHA RAY: There are alternate models of AI development. Right now the dominant model is this LLM we are interacting in, forming that relationship, but other AI researchers, Yoshua Bengio, for example, who are working on a kind of AI that is just an Xbot system, so you just ask it for information and are not looking to it for emotional support.

I think we should definitely look to these alternate models of AI development as a way to conquer some of these more nefarious trends.

ELEONORE FOURNIER-TOMBS: Thank you, Trisha. Thank you, everyone, for your questions. I am sorry we didn’t get to all of them. Thank you for your participation and to the panelists for a very insightful conversation.