Full transcript
0:00Well, hey there, Katrina. Thank you so much for joining me. I'm super stoked to get together because I think you've been doing some incredible things lately. You have a new launch, a new release, the publish of your book, Project Maven, a Marine Colonel, his team, and the dawn of AI warfare. Is that right? >> That is right.
0:19Well, hey, congratulations. But look, could you mind, I don't know, filling me in? Let me know, hey, who are you? What have you been up to? How did you get to where you're doing what you're up to in your career? And then how did this book come to life? Yeah, thanks so much. Thanks so much for having me on. I think I'm normally
0:33interviewing you, so it's fun to to switch around, a real change. I I'm a journalist. I'm a reporter from London. And I've been in the US for now 10 years. I'm a Bloomberg reporter covering cybersecurity, but increasingly AI and national security, that very interesting fault line or nexus between tech and
0:55national security. So before before this job, I was a Pentagon reporter. And also covered the security agencies for the US with the Financial Times. So that's my background. And before that, I was a reporter in Africa for 12 years. So I was really seeing things often from the ground up. And that combination of things drove me to really want to
1:15understand developments in warfare, really, the way people think about it, the way people I suppose practice it. And I kept hearing when I became a Pentagon reporter that the future of warfare was here. And that the Pentagon was behind and pursuing it, and I always wanted to know more about what that meant. And then I lived through as as
1:36I'm sure all of us remember, this very controversial moment where Google workers found out that their technology was being used by the Pentagon for project named Project Maven. And it was to use AI to essentially rifle through drone video footage and find what was useful for the
1:56for the Pentagon for the intelligence defense intelligence folks. And that went wrong when Google workers found out they didn't want to be according to protesters there were 3,000 protesters who put their name to a letter saying they didn't want to be in the business of war and were worried that AI could ultimately campaigners who understood their concerns
2:16were worried that this kind of AI could ultimately end up on on drones on to help develop lethal autonomous weapon systems. That was in 2018 cut to today however many years we are on that's exactly what people are trying to do they're trying to put AI on lethal autonomous weapon systems. And so this book is really the
2:36story of the group inside the Pentagon who wanted who believed so strongly that the US needed to develop AI warfare. And it's also the story of the people who believe just as strongly that the US shouldn't be doing this and it looks at the tensions within the Pentagon among these different groups and also really
2:57tries to understand what is happening because so little of this is concrete is made public. I wanted to find examples of AI warfare in use and see if it was all it was cracked up to be and what the risks were. You know my my background kind of early days years gone by over in the Coast Guard the US
3:15Coast Guard around the 2018 timeline just as well but I'd hear folks always saying hey look warfare has many different shapes on land air and sea but then new frontier of cybersecurity it's now interesting with that asterisk of AI and you're totally right especially between the current conversations of
3:36okay how does Department of War now maybe potentially incorporate a lot of the AR and artificial intelligence into what it's up to. Is that the real focus of the book? Is that all throughout Project Maven? Is are there more stories? I know you I know you're journalistic right reporter you want to get the real story and everything into
3:54it. Any anecdotes or anything you could share? Yeah, I started speaking to some of the people who'd been developing the algorithms. So, you have two sides of this. One is you've got the military pushing to develop it. You've got ultimately after Google drops out, the Pentagon goes to Palantir. So, one of the big themes of the of the book is the
4:13rise of Palantir is is in significant part due to the specific project, Project Maven. And then the the third is really the algorithms. Palantir wasn't making the algorithms, the computer vision algorithms, they were making the sort of digital system that they they could sit in, almost like a Google Earth platform, but a Palantir platform. And the algorithms came from Microsoft, from
4:35AWS, from a company named Clarifai, and and several others. And so, I look at some of those specific algorithm developers. So, we could talk a little bit about Palantir and that and their effort and their role. We could also talk about some of those algorithm developments. Ooh, I'd be super curious about the algorithm aspect if there's anything you're willing to dive into
4:55here. Yes. So, the very if you imagine in 2017 when they were just getting going, this is really quite early on. I mean, ImageNet was was going full, but these computer vision algorithms contests were still in their infancy. And there's one company, one man called Matt Zeiler, who was winning them. Back
5:16in, I think, 2013, he had five of the top computer vision algorithms in this test. And the Pentagon got to hear about him. And so, a a colonel named Drew Cukor got on got on the Amtrak, went to New York, which is where Matt Zeiler had ended up, and started trying to convince him, "Come and come and work with us."
5:37And I found this fascinating because at the time Matt Zeiler had a startup. He'd just made his first few hires. And he was he was running computer vision for a wedding blog. So he had computer vision that could identify things like concentric circles for a wedding cake or a bride's veil or
5:58a groom's suit. This was how they were making their money and they I understand needed obviously this is a startup they needed money and contracts but they had never considered working for the Pentagon and Colonel as he was at the time Drew Cukor set about convincing them to say we we really need your technology. And he agreed and then over
6:18time Microsoft got to hear about this and desperately wanted to be part of it. AWS wanted to be part of it and so this Colonel and and project Maven as a whole competed the algorithms against each other. They would test them not at obviously identifying wedding cakes but identifying trucks or tanks, people, vehicles, buildings. They wanted at the
6:37outset to be able to develop 38 sub classes. You can imagine on a on a digital picture of of a of a battlefield there are lots of things you need to identify either as a target or as a not a target for the restricted target list or the no strike target list. No strike list. And they all had slightly different techniques of
6:57doing this. And I learned that Microsoft got into this because they went to them to say could you help us with synthetic data? We don't have enough pictures of battlefield items. Can we develop data that you can take a take a picture of and train the algorithm that way? And it didn't work. The algorithm could always somehow see through the synthetic data.
7:16I think it's got better now but at the time it certainly didn't work. But the way Microsoft showed them that it didn't work was so good they realized hey we could actually use your algorithms never mind just synthetic data. So this team inside Microsoft started making it. And they realized that it was just too hard. At the time Maven
7:33was trying out algorithms in Somalia I learned in Afghanistan, in Iraq, and Qatar. It kept expanding. This would have been in 2018, 2019 time. A a A lot of the the book focuses on Afghanistan. And they realized that it was just too hard. The the algorithms
7:54would flicker. If you imagine a a a video is made up of multiple frames per second, the algorithm had to do the work essentially on every single still that makes up 1 second of video. And so anytime the algorithm dropped or didn't notice, it would turn on and off. So, for the operators who were really, really focusing, those screeners, they
8:16couldn't concentrate. They couldn't understand. There was too much information still. It wasn't doing a good enough job of filtering. And eventually Microsoft decided, "We're just going to identify people and vehicles. We're not going to do any more than that." And that's a really basic effort. And I spoke to some of the algorithm developers who said, "Well, the very advanced human screeners
8:36could identify a weapon on someone's shoulder. But it might come down to as little as three pixels to do that. And they simply didn't feel confident themselves that they would be good enough to train an algorithm to make a big decision like that. You imagine, you could end up looking for what's called a pattern of of life that
8:54could ultimately make someone a target. Whether they're carrying a weapon or a basket of groceries becomes a life or death determination. Yeah, and then having that be trusted to okay, sure, algorithm, and then a little bit of AI right after the fact, if that's not part of the picture, it's a really wondering question mark.
9:16Without a doubt. Goodness. They They worked out they couldn't tell the difference between a man, a woman, and a child. And that's one of the key things the US has to do to make sure that any action it might take is in line with the laws of war. Obviously, men are not meant to be targets just because they're men, either. But one key
9:35differentiation is to make sure that they're not targeting women and children and the algorithms at that point couldn't pick them up. And so for me that became really important because the claim made for AI several. One is you can increase the scope of war, you can increase you know, the scale of it, the speed of it and I
9:53think in the US operations in Iran AI we've certainly reported is helping deliver speed. But precision and accuracy which is the other part of the sets of claims made once you get into the detail of how do these algorithms work certainly at the beginning humans were much better at finding out what they were looking at
10:15than the algorithm was. Now between 2018 and today a lot of work has gone on and a tremendous amount of that work happened in a in a in a war in support of the Ukrainians and they were able to make some developments there but it was still very very difficult. >> Has the overhaul or now like oh new
10:34refinements, new learning, new improvements really happened from the 2023 like 2024 timeline onwards with oh I don't know newfangled Claude openAI chatGPT etc. Is that when oh the the switch turned on or They've they've definitely put LLMs into Maven Smart System which is this it's like a common operating picture. Some call it that,
10:54some call it AI mission control, some call it the the operating system of war now because every US combatant command is using Maven Smart System which is this sort of digital interface. Um and even NATO now is using a version of it. LLMs aren't doing the computer vision. Right. Yeah. But what they are doing is summarizing intelligence reports,
11:15speeding up the processes required to get a sign off for a target. So they're not doing the sign off, they're not stepping in for that human but a lot of the processes of imagine just pushing paperwork around to get a legal review. They're helping with that run that admin and speeding that up and the CENTCOM commander the admiral came on and did a
11:34broadcast just in the last week saying that AI was helping them reduce the time for processes from Let me get this right, days and hours down to sometimes um seconds. Wow. So, the and that is to do with LLMs and in this case Claude, which is the one that's on Maven Smart System, the one that all the
11:53controversy has been about, um which the US now says, you know, they've got 6 months to to remove Claude from their systems. is doing a lot of work in in the Iran operations. The computer vision itself, you know, computer vision's got better. There are now transformer models. I don't know the quality. I want to be really clear about things I don't know.
12:11I don't know the quality of the computer vision being used right now in the Iran operations, but I do have reporting from 2024 when Central Command, they hit 85 targets in Iraq and Syria in retaliation for killing of three US service members. And they used AI to help narrow down those
12:3385 targets. And another um outlet named Airwars, they're a non-governmental organization that tracks civilian harm, they have recently reported that one of the people killed that night was a civilian. And so, they have made the claim that this is the first AI-assisted civilian casualty of war. Um it's not a claim I'm I'm making
12:55in the book, but I but I think it's very interesting to work out how does collateral damage get affected by bringing AI into systems. Yeah. That is a wild thing to think about. Can I ask and I don't know if you said, forgive me, but how long have you been putting together the book? How long has this been something you've been writing?
13:15I I took I took 6 months off last year, so I was really lucky to have that time. And that's the very intense period where I was researching and writing all at at same time. And as I was researching, the story was changing. It Before that, I'd spent a year developing what I thought, based on the research I
13:34already had, the way I thought I would tell the story, probably for 5 years before that. I've looked very closely, or as closely as I could, at AI and warfare, because it's one of those questions that so much is said about. We have this big existentialism debate going on. We have, you know, what's your P Doom happening on the West Coast. All All of
13:53these discussions of what is AI going to do to us. Extremely brilliant people discussing the theory. And I was just very frustrated that we didn't have concrete examples of what it actually does in a battlefield. And And that's both senses the limits of it. This big claim is made of AI, and actually it's it's as clunky as it can't tell three
14:14pixels. So, how much trust should we put in something that actually has very limited wherewithal. Two, what if we do? What if the US does connect up all its data systems, put everything on the cloud? At the time Project Maven started, they didn't have cloud. So, really, what they were doing was pushing to get different services, different commands, to find out where
14:34their data is, label it. They were often working off PowerPoints, but they didn't have the data that the PowerPoints drew from. And so, there's a complete overhaul to instead of using PowerPoint, to use the data behind the PowerPoint and draw from that and label that data, annotate it, audit it. Much better accountability really comes from this effort to bring in AI. But also, these
14:55enormous risks come in. And so, I was looking for examples. And once I became confident that I would be able to find real-life examples as well as tell the story of some of the people involved in this, I spent probably 2 years a huge amount of time on on huge amount of those 2 years on on the book. I know
15:13you interviewed what was it, like 200 or so folks? Like insiders, opponents, folks really all privy to this and I I just think it's a testament, right, of how timely the conversation and story is, right? Especially as this has grown over time and now release now. But, is this your first book, Katrina, or Well, that's a very kind question. It depends
15:33if you count the Bradt Travel Guide to Burkina Faso and the Bradt Travel Guide to Sierra Leone, which were were my my my two that I co-wrote before. So, I I was Yeah, I was based in Burkina Faso and Sierra Leone writing travel guides for for a wonderful travel publication, but not one terribly known in and not
15:50countries that are terribly well visited by by travelers, but it was a great privilege to to write those two. >> Well, I know as I I'm alluding to, I know this is totally an ever-changing conversation as it evolves in AI and warfare, but I know the story or certainly Project Maven as it it forgive me, is it still going? I just
16:10don't know. Will there ever be and would it you ever write Is there a sequel? Well, Project Maven stopped being a project around 2022-23. They Only because it was growing. So, it became what's known as a program of record, which essentially means Congress is going to give it money every year. And that kind of fear of is my
16:31experimental project going to die death or keep going, they legitimized it, they institutionalized it, and it moved from being in the um Pentagon to the the Pentagon building proper to the NGA, National Geospatial Intelligence Agency, which is a combat support agency for anyone who doesn't know. It's part of the intelligence community and it's also
16:52part of the Department of Defense or war as they're now calling it. And that institution tried to expand it, get it to get Maven, which was by then really the algorithms plus that platform, Maven Smart System, tried to spread it to more combatant commands. They were trying to say let's use the AI, and that's really where the main take up phase happened.
17:15For some uh in inside the Pentagon, they look at this time as a period of it becoming much more stodgy. So, the the people who'd already fought to develop tools, the AI tools, were working in quite a chaotic environment. Often, they were Marines between that kind of 2017 to 2022 phase who weren't encountering military operators
17:35who wanted to use the AI. So, they were constantly trying to do what people do, which is rely on their friendship networks, rely on who they'd served with in the past as they Please, can you try this out? We know it's not good yet, but one day it's going to get good with your help. So, so please do. And that doesn't
17:50mean they were firing with it in battle, but they were running it over, let's say, battle footage or trying to see where it would be useful. And many people were saying, I don't want to I don't want to play with this. I I got enough stuff to focus on. They had great take-up in the special operations command and JSOC community
18:09for some of the more, I suppose, classified operations and work in Afghanistan, also in Iraq. I report in the book that they used it in the Baghdadi raid, if you remember that one in 2019, where President Trump announces that Baghdadi has been taken out in a network of tunnels. And AI there, find out in
18:30the book, tracked a white van and was and was looking at different vehicles. Now, that white van is also been contested since then because NPR has reported that there were civilians in that white van and one was very injured and one died. So, it's not that AI is to blame for that hit, but it may have been
18:51that AI helped alert the operators to the presence of that van, which they judged at the time to be a threat. And they've never put out a public determination about whether it did have civilians. They CENTCOM has always said it was a good hit the time. There was a second report that hasn't been made public, so that's
19:09a that's a tricky one to unravel. But that moment was very much focused on special operations. And so, as niche as they are, the aim was to get it into bigger bigger organizations. And eventually, they got it into the big army, into the 18th Airborne Corps, using satellite imagery, not just drone
19:29imagery, and eventually linking things up so that from clicking on the screen, you could pair it with a weapon system, and you you could then send the command to fire. That is the kind of automation that people have been looking for for years, and is is still developing, and was really put to the test in Ukraine. Obviously, US is not a combatant direct
19:49combatant in Ukraine, but the US was helping send what are called points of interest is almost a term that was invented for the for the job. You know, it's everything short of a decision to fire. So, not a target, but colloquially, what we would think of as a target, were sent to the Ukrainians so that if they wanted, they could take
20:08action against those points of interest to make them into targets. Since then, it got moved to the NGA to get back to your question. And so, they tried to really grow the program up. Check on the algorithms. They would create model cards for the algorithms. They would try and vet them, see where they go wrong. And they also did a lot
20:26of the work of integrating LLMs into Maven Smart System, and speeding up those processes. I report in the book that Claude was kind of a particular favorite of Indo-Pacific Command, which would be relevant if if the US and ever faces a conflict against China over Taiwan, for example. Miss Katrina, there's there's a lot. There's certainly a lot going on.
20:48Uh Congratulations ultimately just to tell the story here between the narrative and then the deeply profound like thinking that kind of comes from this. But, I'm sorry, I didn't mean to stomp on you. I know you had a thought coming up. >> No, I was going to say, tell me about the hacking risks to all these things,
21:04because that's where I I feel like we know that planes were painted on the ground to try and confuse the algorithms. And I did speak to a US commander who said we we can get the hang of this pretty quickly. Uh we we are not being befuddled by these attempts to duke computer vision. But obviously there's a lot of fear about
21:24data poisoning and and about you all these sorts of attempts to hack the algorithm. And that's where I would like to do more reporting in future. I I would not be able to pretend to know or even guess well what it would what it takes, you know, at that level of okay, national security concern for real life
21:43and death and uh at that at the government and military tier, but I think it it it still absolutely, as you mentioned, falls down to okay, the legitimacy of the data that you receive, the intake, the processing, the if they're poisoned training data inside that you see for what you're seeing real time. I don't want to say oh, the silly
22:02stupid prompt injection, but yeah, leveled up to that severity and criticality, that would certainly still be something that makes me a little queasy. Cuz if we're getting down okay, the precision and accuracy of three pixels, sure it's improving over time, but I then want to know like the non-deterministic and I I always fall on this. This is my fault for kind of
22:24falling on my sword and saying it's it's wibbly-wobbly when you let an AI make a decision. It's it's loosey-goosey. So, putting that to be a click button strike air strike is is certainly a crazy thing to think about, but uh And of course, I mean, I should say I'm
22:45constantly being told by US military operators, commanders, we're the ones making the decisions and we have no plans to not make the decisions. But what you begin to see is this potential for eliding. So, the quicker the process becomes, the less time for human review. Um and I I I do report that in the
23:05targeting cycle there were the way the 18th Airborne Corps used it, six points at which a human was involved in a decision-making loop. And with the help of AI, they take humans out of four of those key roles. So, it's a reduction. The decision still comes down to the human, but a lot of the data checking or
23:25or those sorts of processes are slowly becoming Yeah. automated. And then in the operations that the US is pursuing in Iran, you know, the tremendous number of targets that the US has been hitting, I think it's up to 6,000 now. It'll certainly be more by the time this goes out. You know, the period for review is
23:45is then very short. You've got to think that some of these are what's called dynamic targets, targets that haven't already been allocated. So, often that's something that's moving or something that gets found kind of on the hoof. Others come out of a long-standing targeting book. So, of course, there's a lot of scrutiny on where was the girls'
24:03school that was hit? Was Was that already in a targeting book that just wasn't updated because it had been part of an IRGC facility and then was peeled off from that. That will be a critical element. And And for me, one question is, can AI help with the record keeping? Could you be using it in a different
24:22way, not only to speed things up, but to run checks against things like Google Maps, which may have listed the girls' school as a girls' school, but that interplay between open source and the US military does draw from open source, but it has to be an authorized system. So, why can't information
24:43be joined up further? And then of course, that brings a different bunch of concerns about surveillance and overlaying data. So, it's a really complicated moment for the US military to be leaning into this technology with all of with all the fears. I want to save some of that thinking and and thought and insight for the book. I know that's out
25:04and about now, right? Hey, and for folks wanted to get their hands on it, I believe at Barnes & Noble's and Amazon. Is that right? Uh available Hey, wherever books are sold, I think is that is that the case? Yeah, wherever books are sold, I think. Yeah. Well, we'll leave some links in the description. Is there more anything else
25:22any any parting thoughts that you were thinking oh you really wanted to help squeeze in here for folks to know about the book? But honestly, my message is hey, congratulations. I don't know if it feels like a a sweet personal milestone to get this out and about, but I I I got to say that's got to be a good feeling
25:38of success and fulfillment to really make this thing available after all of your hard work. And for folks that do want to go learn that story and start to think about this a little bit more, you've done something incredible with world-changing technology and then everything well-researched and deeply thought through behind it. So, kudos to
25:58you, Katrina. Thanks, that's really kind. If it it feels really meaningful to have been able to understand the motivations of the people who wanted this and the people who didn't and try and bring those different perspectives together in the same place so that members of the public can can learn. And the parting thought is that since then, the effort now to put AI
26:18onto drones, that really is the cutting edge of the future of will this AI develop automatic target recognition? Will the computer vision and the reasoning models be good enough to allow a drone to make that decision? And this is everything that the UN Secretary General has argued is morally repugnant and politically unacceptable. And it seems it's not as
26:40politically unacceptable as he has warned. The debate we see now with Anthropic saying we draw a red line at fully autonomous weapons. Not that we never think it could happen, but it needs to happen a way that LLMs can roll out sensibly. And then people like Jack Shanahan, the former general who was director of Project Maven, saying LLMs
27:01should be nowhere near a weapon. We're clearly at a bit of a tipping point on Yeah. what generative AI, unpredictable black box systems that we know hallucinate, are prone to bias, are prone to drift, all these things, how far down that road the Pentagon can even experiment in a safe way, never mind actually deliver. And so that's kind of
27:19where the book where the book leaves you. Yeah, I was going to ask. I don't know if you were willing to share a resolution, is there one or the other, but it it sounds like look, we still got to learn and and see uh and think about it all thoroughly. So, I'm glad the the book is a great uh you know, process
27:37through that. So, congratulations, Katrina. I can't say it enough. I'm stoked to get my hands on it. I'll definitely hey, take a look and now go see where the world is going next. Thank you. Thanks for having me. >> So much, Katrina. Bye.