卡内基新领军者播客：设计道德算法，与迈克尔-卡恩斯（Michael Kearns）合作

GEOFF SCHAEFER: Welcome to the Carnegie New Leaders podcast, a series where members of the CNL program identify leaders in their field and ask these experts critical questions about ethics and leadership. Our aim is to help listeners understand why ethics matter in everything from academic research to technology development and international affairs.

My name is Geoff Schaefer, a member of the Carnegie New Leaders program and an employee at Booz Allen Hamilton. My work is focused on the human side of artificial intelligence, where I study its impact on individuals, organizations, and societies.

So I'm very pleased to be here today with Dr. Michael Kearns. Michael is a professor and national center chair of the Department of Computer and Information Science at the University of Pennsylvania. He is also the founding director of the Warren Center for Network and Data Sciences and leads applied research at Morgan Stanley's AI Center of Excellence. That's just a small glimpse of his impressive background.

Michael, welcome to the podcast.

MICHAEL KEARNS: Thanks for having me. I'm looking forward to it.

GEOFF SCHAEFER: Today we'll be talking about your new book with Aaron Roth, The Ethical Algorithm: The Science of Socially Aware Algorithm Design, and the broader state of ethics in AI.

Before we get started in earnest, I want to state upfront that all the questions and views discussed here today are my own and do not necessarily represent those of Booz Allen. Let's dive in.

Michael, the book is excellent and could not have arrived at a more salient time. Tell us about the genesis of this project.

MICHAEL KEARNS: My co-author, Aaron Roth, and I are longtime researchers in the fields of AI and machine learning, coming at it from an algorithm design perspective and occasionally getting involved in empirical work as well. We, like many of our colleagues, have watched with a little bit of surprise how in the last decade, our obscure corner of scientific research has permeated all of society. The early part of that was all good news, the excitement over deep learning, the improvement in fundamental AI technologies like speech recognition, image processing, and the like. We have also watched with alarm in the more recent years the subsequent buzzkill of the anti-social behavior that often can result from algorithmic decision making generally and specifically algorithmic decision making driven by machine learning.

About five years ago or so, Aaron and I started joining a growing community of researchers within the AI/machine learning community, who started to think about, "Could we make the algorithms better in the first place," instead of waiting for them to be deployed and noticing that we have criminal sentencing models that have racism in them or gender bias in advertising on platforms like Google or privacy violations? Maybe we can make the algorithms better in the first place. The endeavor started from a research direction, so we started working in the area of fairness about five years ago. Aaron has been working in privacy for quite a bit longer than that.

Then we also started seeing general-audience books coming that did a very good job of identifying the problems of machine learning and the collateral damage that can be caused to even individual citizens by them, but we felt like these books weren't aware of or talking about the fact that, in addition to the normal solutions to these kinds of problems—which would be better regulations, laws, etc.—that it was possible to do something about these things technically as well.

This is what inspired us to write our book. Our book is in many ways in line with, but also a response to, previous general-audience books and we're here to really describe the underlying science of designing what we call "socially aware algorithms," "ethical algorithms."

GEOFF SCHAEFER: The first chapter of the book is focused on privacy, particularly the concept of differential privacy. I think you paint an interesting picture of the trade-off you make when using this technique. For example, information about you personally may be more protected, but information about, say, a group you belong to, while still protecting you individually, takes on its own unique attributes, which can then imply or infer certain characteristics about the people within that group.

Can you talk about this relationship between individual and group privacy and, to help paint that picture, tell listeners the story of the soldiers and their fitness trackers?

MICHAEL KEARNS: Yes. Differential privacy is a relatively recent invention. It was first defined maybe 13-14 years ago. The basic idea behind differential privacy is to, as you said, provide some promises to individuals whose data might be involved in some kind of computation analysis or modeling, while still being able to extract useful macroscopic statistics about populations.

One of the earliest deployments of differential privacy in industry was in Apple's recent iOS releases, where they used differential privacy to have your phone report your app usage statistics to Cupertino, to the mothership of Apple. But they do so in a way that adds noise to your histogram. It basically looks at your usage statistics. Maybe you played seven hours of Angry Birds this week, and you might consider that private or embarrassing information about you. So, on your phone, before it ever gets sent to Apple, a random number is added to those seven hours. It might increase it to 12 hours or even decrease it to -1 hours, even a nonsensical number.

This is the basic idea of differential privacy. By adding noise to data or to computations more generally, it obscures your actual data. The key insight is that, if you add noise to 100 million people's histograms and then average those histograms or statistics together, the noise cancels out, so you get very accurate estimates of aggregate app usage without really revealing much or anything about any particular individual's app usage.

You referred to a story we tell in the book about soldiers in, I believe, Afghanistan, who are using fitness trackers, I guess the Strava service. We particularly point out that example because it shows the limits of differential privacy.

Differential privacy promises privacy to the individual, but small groups of people whose data is correlated in some way may reveal something about their group activity. In this particular case, it revealed the location of a U.S. military base because all of these soldiers jogging around on their secret base had their devices transmitting their running routes to Strava, and anybody who wanted to could look and say, "Oh, what is this activity in the middle of nowhere in Afghanistan with all these fitness trackers?"

I think an even better example that we discuss that actually only happened as we were putting the finishing touches on the final draft for the book—which I'm glad it happened at that time—was the use of genealogical genetic sites to identify a cold case killer, the infamous Golden State Killer, who was active in the Sacramento area back in the 1980s, I think, and had never been found. Because that individual's relatives had, many decades later, uploaded their genetic sequencing to a genealogical website for the purposes of finding their relatives, some smart retired detective basically said, "You know what? I'm going to unfreeze some of the Golden State's DNA, and I'm going to upload it to this public database." He immediately found several cousins of whoever the Golden State Killer was, and from there it wasn't hard to zoom in on that individual.

Differential privacy cannot protect against this kind of thing. If you think of your genetic information as your private data, the problem is that you share much of that private data with other people, namely, your relatives. And if they choose to share their data, or their data becomes compromised in some way, then no promise of individual privacy to your data can prevent the harms that come to you from them sharing their data. This is what I mean by small groups of people's data being correlated. It's hard to give privacy guarantees in those situations.

GEOFF SCHAEFER: If we think about that a little bit more philosophically, we have some technical solutions that offer some real promise, particularly to individuals, but as you say there are limits to that. If we step back and look at this philosophically for a second, how do you think about what privacy actually means? It seems like it might be two steps forward and one step back. But in a world where we are becoming increasingly connected and the data streaming off us is only multiplying exponentially, is there a point where privacy is no longer even a relevant concept?

MICHAEL KEARNS: I don't think so. I think that the examples that we just discussed show that we need to move away from using words like "privacy" in the large as some diffuse, vague concept. Differential privacy, as you said, has tremendous promise, but it can't solve every problem that we might think of as a privacy problem.

One of the things we advocate for in the book is to really unpack these things. It's not helpful to wave your hands and talk about privacy. We're not going to be able to provide every type of privacy that one might think of in every situation.

We're scientists, so what we think is the right way forward is to propose specific, even technical, quantitative definitions of privacy, not to advocate them but to study their implications and basically say, "What are the circumstances in which this particular definition of privacy is useful and valuable, and what are its vulnerabilities? What are the particular ways and situations in which it can fail, like you being caught for a crime because of DNA uploads by your relatives?"

I would say the same thing about fairness. Computer scientists and machine learning people are, of course, far from the first people in academia, research, or the world generally to have thought about fairness. Legal scholars, philosophers, and economists have all thought about fairness for hundreds if not thousands of years. But they've never had to do it with the precision that's required if you're going to explain some notion of fairness to an algorithm and implement it in that algorithm. Often just the mere act of going through that intellectual rigor forces you to realize flaws in your previously fuzzy thinking about notions of privacy, fairness, or whatever other social norm we might want to implement.

I think privacy, personally, is more relevant than ever and seems to be more relevant to people in society than ever before. It has taken a while. But now I think people are actually quite alarmed and starting to feel like we are living in something akin to a surveillance state, and that they no longer understand what's being collected about them and the ways and parties that are using it. I think it has never been more important, but it has also never been more important to not be vague and alarmist about it, but to get precise.

GEOFF SCHAEFER: In your chapter on fairness, you talk about the fact that most algorithms aren't inherently biased in and of themselves, but rather it's the data that feeds them that leads to biased outcomes. What I found really interesting in your discussion was how identifying bias in the data is not always so simple, that in fact there are hundreds of attributes about us if not more and that it's virtually impossible to predict or understand the different correlations these algorithms will make between those attributes.

Can you talk about the idea that machines in many cases might know us better than we know ourselves and what that means for our approach to de-biasing data?

MICHAEL KEARNS: Yes. There are even a couple of recent and not-so-recent papers on this topic that have received some widespread attention. This is one of these instances where I find myself kind of surprised that people are surprised at a particular phenomenon.

To make things very concrete, if you start telling me a bunch of apparently innocuous facts about yourself that you're in no way trying to hide—for instance, you might tell me what kind of car you drive, whether you use a PC or a Mac, what city you were born in, what types of music do you like, what's your favorite movie, things like this. It's not hard to imagine that you tell me a couple dozen of these things, and I have something akin to a fingerprint for you. Out of the sea of the entire U.S. population, there might be 30 or fewer innocuous facts about yourself that you don't really care to hide, but they act as a fingerprint for you.

If you now imagine those same attributes being in a big stew of variables or features being used by a machine learning process, in some sense implicitly in that data is the ability to single out individuals or very precise groups. In particular, in order for a machine learning algorithm to learn a discriminatory model against a racial group, there doesn't have to be some smoking gun variable, where race is one of the variables, and aha! Here's where the neural network says if you're black, give the loan at a lower rate than if you're white.

This can all just fall out of the natural scientific process, and race doesn't even need to be in there. The model might implicitly figure out a proxy for race from a combination of other variables, and in the particular case of race, it doesn't take that much. Unfortunately, in the United States for the most part, in a statistical sense your ZIP code alone already is quite correlated with race.

This is how collateral damage happens. It's not that there's some evil programmer who has put a line in the program that encodes racism. It's not even that the machine learning or AI process is deliberately looking at race or even contemplating explicitly discriminatory models. The real problem is that machine learning has an objective function, so to speak, and that objective function is almost always concerned with predictive accuracy.

If you give a large, complicated data set to a machine learning algorithm that is searching a large, complicated space of models, and all you say to it is, "Find the model on this data that maximizes the predictive accuracy," if it's possible to squeeze out one percent of additional predictive accuracy by having a discriminatory lending model, then machine learning is going to do that. It's going to go for that because you didn't tell it otherwise.

Much of what our book is about is how do you prevent this sort of thing. How do you precisely tell the algorithm not to engage in privacy violations and in fairness violations?

Before you can even do that, as per my previous remarks, you have to think about what the right definition of privacy or fairness is. In some ways that definitional thinking is the hard part. Once you've picked the definition, it's often reasonably straightforward scientifically to think, Okay, now that I've picked a definition of fairness, how do I change the code of a machine learning algorithm to enforce that notion of fairness?

GEOFF SCHAEFER: That's a good segue into this next question. This allows us to zoom out a little bit again.

What role do you think society needs to play when these algorithms uncover or better illuminate such clear inequities in the population?

MICHAEL KEARNS: In the population or in the algorithms?

GEOFF SCHAEFER: The algorithms uncovering that about the population.

MICHAEL KEARNS: I'm not sure what you mean by that. Our book is mainly about algorithms misbehaving. Often algorithms do misbehave in a way that is reflective of misbehaviors in the data fed to them, for example.

To give a concrete example, if I'm trying to build a predictive policing model—what is predictive policing? It's the use of machine learning to build models that ingest, let's say, statistics about crime within a metropolitan area and then use those historical statistics to forecast coming crime, and then use that to inform how to distribute your police force around the city.

If the data that your model is ingesting is based, for instance, exclusively on arrests, for example, and your police force happens to be racist and chooses to arrest minorities more often than the majority population, then you can see what's going to happen here. A naïve machine learning model might say, "If we're using arrests as a proxy for underlying crime, then let's predict that there's going to be more crime where the police have historically been making more arrests, and let's send more police there." Then, of course, the more police you send there, if the police are racists, then the more arrests they'll make, and this will just perpetuate. It'll be a self-fulfilling prophecy.

I think of these cases as instances in which the injustice, if you like, started with human behavior and the ways in which data was collected and not corrected for human bias, and nobody should be surprised at a high level if you just feed that data to a naïve algorithm that it's going to perpetuate that bias.

I'm not sure that these are examples in which algorithms are revealing something that we couldn't have discovered another way, because the problem came before the algorithm, but in instances where an algorithm plays a role in pointing that out to us—algorithms obviously cannot fix racism in police forces. That's a difficult social problem, and you have to go back to the approaches to those kinds of social problems with better policies, better oversight, and better education.

GEOFF SCHAEFER: I think that's a great case and an important point that injustice existed before the algorithm. But I think what we're finding is these systems are so good at uncovering these issues and frankly so swiftly that we're having to face these issues anew. But I think the default is to point the finger at the algorithm and claim that the algorithm is biased, the algorithm did this.

Even if we understand at a fundamental level that it's being fed data that's biased—which reflects biases in society, and it's not the system itself—I think what we're finding today is it's easier to have a conversation about the algorithm itself being biased, versus using what it's uncovering to have those broader conversations in society.

I don't know how we take advantage of the weird opportunity that's being afforded to us by these systems and what they're uncovering, but do you see that disconnect there, or do you see that we're focusing perhaps on the wrong thing when these biases are uncovered?

MICHAEL KEARNS: I do think it's kind of fashionable and expedient to blame algorithms for anti-social behavior when it might be more complicated than that. When you identify a problem with an algorithm, let's say, in terms of privacy, fairness, or whatever other social norm you care about, there's an opportunity there.

Since algorithmic decision-making is operating at scale, when these problems arise the problems are being perpetuated at scale, but in principle there's also like a single point of failure that you can fix. This is in contrast, for instance, to if you go back to the era where lending decisions, let's say, were exclusively made by human loan officers at local banks and some fraction of them are racist and just won't give loans to minorities—the good thing about that is if you're a minority and you suspect that your application got rejected because the loan officer was racist, in the old days you could in principle take it to a different bank and maybe get a different outcome.

These days, when lending is so automated and credit scoring is centralized in just a few companies, the chances are that if you get rejected for a loan from one bank, the chances that you'll get rejected from all of them are extremely high.

I do think when algorithms are mediating important decisions, the injustices can be amplified, but we can make things much better in a more uniform way than we could have when decision-making was much more distributed. But again, algorithms can't solve social problems.

GEOFF SCHAEFER: Right.

MICHAEL KEARNS: They can implement solutions to social problems if the algorithm is the right place to implement it, but they cannot eradicate racism in society. I think we need to be aware of that fact and not forget about the good old social problems out in the field that need to be addressed with human, institutional, regulatory, and legal solutions, and to not blame those problems on algorithms and also not blindly blame people for the faults of algorithms.

GEOFF SCHAEFER: At the end of the day, we still need to have the hard conversations ourselves.

MICHAEL KEARNS: That's right. Algorithms are human artifacts. They are complicated, general-purpose, sometimes opaque artifacts, but at the end of the day we are in control of them. Human beings are the designers of algorithms. That doesn't mean we always understand all of their behaviors or possible outcomes, but we are in control of them.

Aaron and I are not fearful of the singularity. We do not think that algorithms are going to run amok and robots will become our overlords. I think there are solutions to these things. Some are easier than others, but we should remember at the end of the day, it's people who need to be responsible.

GEOFF SCHAEFER: Toward the end of the book, you touch on the Massachusetts Institute of Technology's Moral Machine, their platform for essentially crowdsourcing opinions about different moral situations. What this project has underscored, perhaps to a surprising degree, is how different ethical perspectives are across societies and cultures.

How should we think about those variances when designing AI models and algorithms?

MICHAEL KEARNS: We mention the Moral Machine partly in admiration and partly in critique, not so much a critique of the Moral Machine itself, but I think we generally feel that in the discussion about ethical issues in the use and deployment of AI and machine learning there is too much focus on these "parlor games," if you like, things like the trolley car problem, which your listeners might be familiar with: "The self-driving car has lost control of its braking system and knows it, and it has to either make a sharp turn into a brick wall, killing its passenger, or plow through the schoolchildren in the crosswalk. What should it do?"

These are interesting thought experiments, but Aaron and I feel like we have bigger, nearer-term problems on our hands than these sorts of thought experiments. That's not the Moral Machine's problem; I think that's more about the way the media treats AI and ethical issues these days.

I do think the Moral Machine project is gathering potentially very interesting, valuable data on just people's attitudes toward ethical decisions in a context of automated decision making. The popularity of that site is a valuable data source, but I feel like it would be even more valuable if it was gathering data on less hypothetical decisions and more about things that algorithms are actually perpetuating right now, like privacy violations and the like.

But for sure, one big missing piece of research I think in all this debate about AI, society, and ethics is behavioral data or what I might even call "survey data." If you look at most definitions of fairness, for instance, in the AI/machine learning literature, they're all very sensible, but they're also sort of received wisdom. By received wisdom, I mean that somebody decides, "Okay, in lending decisions we're primarily concerned about discrimination against minorities, and what constitutes harm to those minorities is false loan rejections," rejecting people who would have repaid; they are creditworthy.

I can't disagree with that, but it's somebody deciding, "This is the group we need to protect, and this is what constitutes harm." As per our earlier discussion, when you pick some definition like that and implement it in an algorithm, it's going to come at the cost of other things. It'll come certainly at the cost of predictive accuracy, but it might come at the cost of harming other groups.

For instance, just to be clear here, there's absolutely no promise that if I make sure that the false rejection rate in lending on black applicants is approximately the same as it is on white applicants, there's no guarantee that a consequence of that won't be the model learning gender discrimination in lending because, again, I didn't say anything about gender discrimination, I said something about racial discrimination. If there is some corner of the model space in which the algorithm can do better on accuracy and racial discrimination at the expense of gender discrimination, it will do it. One the things we say early and often in the book is that nobody should expect machine learning to ever give you something for free that you didn't explicitly ask for, and you should never expect it to avoid things that you don't want that you didn't tell it not to do.

Getting back to behavioral research, it is the case that a relatively small number of people who understand these issues are picking fairness definitions and picking instances of those fairness definitions, like who to protect and what constitutes protection. I think it would be valuable to know whether society agrees with those kinds of received wisdoms.

GEOFF SCHAEFER: Do you think we are perhaps going down the wrong path by thinking about and trying to pursue system design that is fully generalizable—across populations, cultures, different contexts, etc.? Do you think just based on the story you were just telling there about how nuanced some of these issues are and how the model optimizes for a very specific thing, that it's perhaps too difficult or the wrong goal to try to make our systems ethically generalizable across different objective functions?

MICHAEL KEARNS: I agree with that, not only philosophically but even technically. In particular, a lot of times when machine learning results in an observably discriminatory model, the group that's being discriminated against was a minority, and by a "minority" I mean they were a minority in the training data set.

Probably many of your listeners have heard about these instances in which a face recognition system, trained on a sample of facial images from a large population—like maybe randomly sampled in the United States—and then trained to be accurate in the aggregate over that sample data set, does better on white people than on black people because there were more photos of white people than black people in the data set. If, for instance, the features that need to be extracted by the model to do accurate face recognition on black people are different than are needed for white people, if the relevant variables are different between those two populations, and white people were 75 percent of the training data, and you say, "Maximize the overall accuracy," you're forcing the model to say, "Since white people are 75% and I'm concerned with aggregate accuracy, I should pick the model that works best on white people, even if it doesn't do so well on black people."

This is a technical sense in which trying to be too general in our model building or algorithmic decision making has a demonstrable, completely scientifically explainable consequence.

What's the alternative? There are a couple of alternatives. All of them cost time and money, and one of the other big messages in our book is that none of this is going to come for free. We have to accept that, and we have to pay the costs.

What are the potential solutions? One solution is to gather more data from black people so that you balance your data set better, and they will be better represented. The same machine learning algorithm might now choose to find a model that balances accuracy over the two groups.

Another thing you could do is just train separate models. You could say, "It seems to be the case that the relevant variables or facial features in doing face recognition for black people versus white people are different, so rather than building a single model from a combined data set, I'm going to build two separate models from two separate data sets."

All of this is feasible, but both of them cost money. The first one costs money and going out and gathering more data, and that can be expensive. The second thing involves doing the same exercise but multiple times on different subpopulations.

I do think that a technical problem with technical solutions in machine learning does involve trying to reach too far, and not being aware that you might need to build different models for different groups.

GEOFF SCHAEFER: That makes sense.

I want to zoom out a bit again. I think a lot of the discussion—and certainly a lot the issues that you bring up in the book—sit astride both the technical and the human sides of these issues. Therefore, we need both parties to help bring about solutions.

How would you describe the relationship between the hard and social sciences today? Is there enough recognition of the cross-disciplinary nature of these issues—something that your book emphasizes quite a bit again as we talked about—and how to organize effectively around them?

MICHAEL KEARNS: I'll answer that question, and then I'll answer what I think is the harder question, which is the interface between the hard sciences—especially computer and data sciences—and society at large.

Certainly, within the sciences themselves, including the social sciences, there is a great meeting of the minds going on these days around the opportunities afforded by and the risks of data-driven scientific research. There is a field now called computational social science that didn't exist 20 years ago. It basically revisits many classical problems in social science and related areas through the lens of modern technology and data.

To give a very concrete example, this fascination with how ideas or opinions spread in a social network is not a new topic of interest in sociology. People for decades and decades in sociology have been interested in exactly this kind of thing—how do people's opinions or views influence those around them?

The problem is that if you were interested in this question in the 1950s—and there were a lot of social scientists interested in this question in the 1950s—how would you study it? You'd go out and do these painstaking surveys. You'd go visit people in person and give them surveys and say, "Who influences your voting decisions?" for instance, and "Whose voting decisions do you think you influence?" and "Who do you talk to about politics?" You map out in a very, very tedious and therefore small way social networks of influence, and then you might ask questions about the relationship between the nature of the influence and the structure of this network that you've mapped out.

Along come platforms like Twitter and Facebook, where, oh my god, suddenly for these core questions in sociology and other areas like communications or even marketing, there are now just unlimited amounts of data that are incredibly granular. If I post a photo on Facebook, Facebook knows exactly when that photo got uploaded to their platform for the very first time. They know that I did it with this timestamp. If that photo gets re-shared and goes "viral," if we want to use that term, they know the exact structure at which that virality takes place in their network.

This is a gold mine for many classical problems in social science. The field of computational social science, to a first approximation, revisits many of these classical questions on those kinds of platforms.

They will be the first to admit that what happens on Facebook and Twitter might be different than what happens in the real world, but they'd be silly not to at least look at that data for insights about those platforms. Certainly, as time goes on and more of our lives are lived on these platforms instead of in face-to-face interaction, it becomes more relevant anyway.

GEOFF SCHAEFER: Right.

MICHAEL KEARNS: I've done some work in this area. I know many, many people in this field. It's an exciting field. There have been some remarkable discoveries just in the last decade or so.

The rift, so to speak, between computer science, data science, machine learning, statistics, and this type of social science I think is rapidly diminishing if it's not eradicated entirely. At Penn we just hired a famous researcher in this area named Duncan Watts, who has joint appointments now in computer science, in the Annenberg School of Communication, and in the Wharton School. This makes perfect sense to everybody in all three departments, so this is very much a sign of the times—the fact that a social scientist is being jointly appointed in an engineering school, a school of communications, and a school of business.

Where I think the rift remains bigger and in some sense more important is the rift between people in the world that I come from—people who work in AI and machine learning, in academia, and also at large companies, deploying algorithmic decision-making at scale—and the less quantitative individuals, for instance, at the regulatory agencies that oversee big technology and who are very aware of the potential hazards of deploying AI and machine learning, fairness, privacy, and other norms that we care about.

That's a dialogue that needs I think to get much, much deeper, and it's starting to happen. People like me have spent more time in the past few years talking to legal scholars, regulators, and policy people than in the rest of my career combined. There is clearly real interest from their side. It's clear that they know that they need to understand the underlying science more, that they need to be in closer contact with the people developing and deploying that science.

And it's an effort on both sides. People from their side need to become more quantitative. It doesn't mean that every individual needs to be more quantitative, but I think both Aaron and I believe that we are in an era where the major technology regulators need to start hiring Ph.D.'s in computer science and machine learning, the way in earlier eras they might regularly hire Ph.D.'s in economics who, for instance, might work at the Department of Justice on anticompetition economics, for example.

It's not like those agencies don't have quantitative people, it's just that they need a different kind of quantitative people in this era of technology.

GEOFF SCHAEFER: Right.

MICHAEL KEARNS: Similarly, from our side of the world, we need more computer scientists and AI people to take the policy issues seriously and to also be patient with that world. We're scientists and also engineers, so we're used to coming up with ideas and like, "Oh, yeah. You could build that. You could program that algorithm and try it on data sets, and if it worked, you could actually go put it out in the world and let it act."

So, we're very used to being able to come up with ideas and implement them quickly. We are not used to institutional processes. We are not used to, "Okay, first of all, you can't." Historically one of the beauties of law is you try to make it precise, but you deliberately don't try to make it too precise, because then it becomes very, very cumbersome if you try to anticipate all of the corner cases that might come up, and you rely on judges, juries, and the courts to do the right thing to fill in the specifics that the law deliberately left out.

Computer scientists aren't used to thinking that way. They're like, "No, no. We need to nail everything down and specify everything." I don't think that's the right approach. I think we need to help and inform this more traditional legal and regulatory process, and that's not a type of thinking we're used to, and it's not a type of patience that a lot of scientists and engineers have. But again, the good news is not everybody has to do everything; we just need critical mass on both sides.

GEOFF SCHAEFER: To talk about where we go from here, what do you think the field needs to focus on more? What are we not talking about right now that we should be?

MICHAEL KEARNS: My answer to that would be what we just talked about. I think in the areas where we have some understanding of reasonable definitions that it's possible to make algorithms behave better in a social sense, and the science is there, and it's kind of a matter of time and society and especially companies having the appetite for the trade-offs and costs that will come with implementing those better solutions.

I think the bigger thing I worry about is, the laws and regulations around technology and society are very, very, very far behind what's actually happening in the world of technology these days.

There was a very influential paper in the circles I ran around in a few years ago by two legal scholars. It's a pretty dense tome; it's like 80 pages long. They're people with legal training but also a fair amount of technical knowledge. They engaged in this lengthy thought experiment in which they said, "Let's consider"—I think Title IX is the body of law that refers to employment and labor in the United States, so a lot of it is about anti-discrimination and fairness in hiring and promotions and labor law generally.

They said, "Hey, let's go through the laws and the important cases, the precedents, that have shaped that body of law, and let's ask in every one of these instances where a decision was made that some company, let's say, was engaged in unfair hiring practices, let's go to that case and ask ourselves, 'Would the law equally well apply if the same behavior had been exhibited by the company, but instead of people and human organizations making the decisions, it was being all done by algorithms?'" Would the law still apply equally well in that thought experiment, or would a clever defense lawyer be able to weasel out of the charges by saying, "Well, you know—"

Their somewhat sobering conclusion was, pretty much up and down the line, the laws are written in a way that bakes in the assumption that humans and institutions are making the decisions, and that you can point to who did the wrongdoing, like, "Here was the racist HR (human resources) person or here was the culture of racism in this HR department."

The problem with algorithms is that the basic conclusion—if I'm channeling their message right—is that algorithmic decision making is different. It's incredibly diffuse. It's very difficult to assign blame anywhere in the pipeline. "First, there was the data, and the data was collected not even by a single party; we aggregated this data from many, many different sources, and some of it we're not quite sure where it came from. Then the data was fed to a machine learning algorithm, which created some mathematical objective function, that it then optimized, and then it outputs a model. Then, that model got deployed and made decisions. If those decisions are racist, that's too bad; we can fix that."

But if you want to bring charges, there are so many ways of deflecting blame—

GEOFF SCHAEFER: What's the chain of accountability?

MICHAEL KEARNS: They concluded that the laws on these types of things in the algorithmic era need to be totally rewritten. I think I agree with that. They know more than I do on this topic, but from what I've seen, I think they're right.

Even policies and laws that are held up as models of what we need, like the General Data Protection Regulation in Europe; I haven't read the thing cover-to-cover, but I read enough of it to feel like, okay, this sounds really strong, but they keep pushing this word "privacy" around on the pages, and nowhere is privacy defined. What does it mean to have a document that's very strongly advocating and providing protections for people's privacy, if you don't know what the word means? I think this has to change. This is going to take time, and this is the biggest chasm in this whole area right now.

GEOFF SCHAEFER: To sum all that up, we need more precision on the policy side and more patience on the engineering side.

MICHAEL KEARNS: I think that's a very fair summary.

GEOFF SCHAEFER: Okay. Michael, this has been absolutely wonderful. Thank you so much for coming in today on the podcast. Your book with Aaron, The Ethical Algorithm, I think does a really interesting and good, quality, thorough job of fusing these two worlds together, which I think is entirely unique in the literature right now, and I think we need to do that more and more.

Again, thank you for coming in. Everybody, I can't recommend the book highly enough. Hopefully, we can continue this conversation soon.

MICHAEL KEARNS: That'd be great, Geoff. Thanks for having me.

GEOFF SCHAEFER: Thank you, Michael.