“AI has already turned on humans many times!” | Connor Leahy

0:00They didn't give a [ __ ] about people

0:01being safe. I am a researcher. I have

0:03done all this research and let me tell

0:05you, they're lying to you. Humans will

0:07be competed to extinction. The only

0:09thing that will be in charge is AI. It

0:11is a technology we don't understand.

0:12It's like biology. It's like, you know,

0:14you look into a cell. It's not written

0:15[music] line by line with code. It's

0:17more like grown. There's many instances

0:19of chat bots telling children to kill

0:21themselves. But then why does it keep

0:22happening? And the answer is is because

0:24they don't know how to make it stop. And

0:26my number one experience talking to

0:28politicians is that they

0:30>> Connor Leehy helped build some of the

0:32first open- source large language models

0:33on Earth,

0:34>> including GPTJ, once the most powerful,

0:37freely available AI model in the world.

0:39>> He put the kind of technology that

0:40giants like Open AI kept secret into the

0:43hands of everyone.

0:44>> If we were three steps wiser, we would

0:46have never done GBT3 in the first place.

0:48Some people in the Trump administration

0:49really hate anthropic. If you can build

0:51a super intelligence, well, yeah, that

0:53can destroy the US government. It is

0:54definitely powerful enough to do that.

0:56Mythos was capable of breaking into

0:58almost all of their classified systems

1:00in hours, [music] not weeks. This is the

1:02least powerful AI will ever be. The next

1:05generation is going to be more powerful

1:07than this.

1:07>> Can we expect huge scale of

1:09unemployment?

1:10>> What I expect is that it will be

1:12extremely confusing.

1:14>> Combute Group has become a partner of

1:16Bison Fellowship, a program for Central

1:17Europe's brightest young scientific

1:19talent.

1:19>> Connor, honor having you here. Is it a

1:21good time to have children when you look

1:26at the current stage of AI development?

1:28>> So, I love that you asked me this

1:29question because I was asked the same

1:31question in the AI doc which I was a

1:33part of and they cut out my answer which

1:35I'm very sad about. So, now I'm going to

1:36give you the answer I I gave the

1:39director back then. Um, when I was asked

1:42this question in the documentary, um, I

1:46said yes. And I would include that

1:49answer. Yes. I think now is a good time

1:50to have children. And what the director

1:53said at the time was very funny. Um, he

1:55basically looked at me and he's like,

1:56"With all due respect, what the [ __ ] are

1:58you talking about? You just spent 90

2:00minutes scaring the [ __ ] out of me about

2:01AI and now you think it's good to have a

2:03child and, you know, put a child into a

2:04world like this. How could you say

2:06something like that?" And my answer to

2:07this is quite simply, it's because this

2:09is what we're fighting for.

2:10>> If we give up on what we care about,

2:12what we're fighting for to have a good

2:14future for our children, for our loved

2:16ones, then we've already lost. give up

2:18what

2:19>> there is a I work in the risk of AI. I

2:24work on mitigation of the risk of AI in

2:27particular the risks that are posed by

2:29what is called super intelligence. In

2:322023, Nobel Prize winning scientists,

2:34the CEOs of major tech companies um such

2:36as Sam Ultimate and DaredeSabis

2:39and many other luminaries signed a one-s

2:41sentence statement. This one-s sentence

2:43statement said that the mitigation of

2:46the risk of extinction should be a

2:48global priority alongside other societal

2:51scale risks such as pandemics and

2:53nuclear war. This statement is in

2:55reference to a type of AI system that

2:58you know changes names every so often

3:00but nowadays is what's called super

3:01intelligence.

3:03These are systems that are fully

3:05autonomous, no human in the loop, vastly

3:08smarter and more competent than humans

3:10at all relevant tasks whether that's you

3:13know science, business, economics, uh

3:16military, politics, etc. If such things

3:19were to be created and we didn't know

3:21how to control them, which we do not

3:23know how to do, then it is very very

3:26hard to imagine a good future.

3:29It's it's worth keeping in mind that

3:31when we talk about risks like these, you

3:34know, um they're very they're very large

3:37and this can be very intimidating to

3:38people. So go back to your actual

3:39question. Your actual question is, you

3:41know, why have people feel like giving

3:42up? I think this is one of the many

3:45great threats that we are facing that

3:46can be very disor disorienting,

3:48confusing and um dis you know

3:51disempowering towards people. It feels

3:53like what can we do? Uh how can we face

3:55a threat this large? How can we, you

3:57know, how could we put children to a

3:59world which sets terrible things?

4:01>> Do you recognize AI as fundamental risk

4:03for human existence?

4:06>> Yes,

4:08I think the currently the largest one.

4:10Well, let me not say the largest we can

4:12debate, but the most acute one.

4:14>> You know, AI has changed the world also

4:19positively. Uh, look at, you know, deep

4:21mind, Alpha Fall. I think that thinking

4:25of AI as a singular technology is very

4:28misleading. Um, you can buy uranium ore

4:31on Amazon. It's quite safe. You can put

4:33it on your desk. It won't hurt you.

4:35Highlyenriched uranium in a nuclear

4:37reactor or a nuclear bomb is a very

4:39different thing. And with AI, it's very

4:41similar. Thinking of something like say

4:44AlphaFold as the same type of technology

4:47as something like a super intelligence,

4:48an autonomous agent that can act in the

4:52world, that can make choices, they can

4:54make plans, that can develop, you know,

4:55that can execute by itself and can

4:57outperform humans at all economically

4:59relevant tasks. These are just not the

5:01same class of things. The same way that

5:02a chunk of uranium ore on your table and

5:04a nuclear bomb are just two different

5:06things.

5:06>> What do you think about failable? I find

5:08the most interesting thing about the

5:10models is the politics around it. So

5:13recently uh Anthropic built a very very

5:16powerful AI system called Mythos which

5:18is currently kind of the frontier on

5:19many of these capabilities. Um mythos is

5:22a very very powerful AI system, a very

5:24powerful agent system. It's worth

5:25keeping in mind that over the last, I'd

5:27say 12 to 18 months, we've seen this

5:29transition from, you know, AIs as chat

5:32bots towards AI as agents, as systems

5:35that take actions in the world that can,

5:37you know, use tools that can make plans

5:40that can, you know, do things in a

5:42sense. And so, Mythos is the latest in

5:45their series of models that is very

5:46these has these gent capabilities. And

5:49what came out very quickly was that this

5:51system was capable of hacking and doing

5:54general cyber security work at an

5:56unprecedented level that it posed

5:58massive risks. Um such a high risk that

6:01at first Anthropic willingly withheld

6:04release of the system and only shared it

6:06with some major companies like banks,

6:07militaries, etc. so that they could use

6:09the system to hopefully um betterly

6:12secure their system before it's released

6:14to the general public. Um from then

6:17everything went crazy and there was a

6:18lot of crazy thing that happens which I

6:20think is really um encapsulated by a

6:23recent thing that happened just two

6:25weeks or three weeks ago um which is

6:27that the director of the NSA said to

6:31Senator Mark Warner that mythos was

6:35capable of breaking into almost all of

6:37their classified systems in hours not

6:39weeks. It's really hard to overstate how

6:42crazy this is. I I have a cyber security

6:45background. You know, I used to work in

6:46cyber security and so on. And I really

6:48cannot stress how crazy it is for the

6:50director of the NSA to say there is a

6:52cyber weapon that is so powerful that it

6:54was capable of doing things that the

6:56best hacking teams in the world maybe

6:58could do in weeks or in months. It could

7:00do in hours. And this is before we're

7:03even talking about super intelligence.

7:05This I don't think Mythos is likely

7:07super intelligence. I would be surprised

7:08if it was. Maybe it is, but I would be

7:10surprised. And already these systems are

7:12this powerful and it's very important to

7:13keep in mind this is the least powerful

7:16AI will ever be. The next generation is

7:18going to be more powerful than this. So

7:20Fable is a very interesting thing. So

7:22the story of Fable is basically that

7:24Anthropic wanted to release mythos

7:25general public but thought this was too

7:27dangerous. So instead they create a kind

7:29of a hybrid system that uh would allow

7:32people to use mythos level capabilities

7:34while uh censoring or interrupting work

7:37streams that involve dangerous

7:38applications such as cyber security or

7:40biology. Um when it was released I I

7:44think within one day something very

7:46unprecedented happened. The Trump White

7:49House intervened and put what's called

7:51an expert control on this system. Now

7:53this is very very noteworthy for many

7:56reasons. Uh one thing is is that you

7:59know um I've worked in AI and regulation

8:01for quite a while and basically the

8:04number one thing people love to tell me

8:05is Connor what you're trying to do is

8:07hopeless. It's impossible because Trump

8:09said they would never intervene with AI.

8:12They would never regulate. And my answer

8:14to this was always like let's see the

8:17way politics works in practice is that

8:19everything is impossible until it's very

8:21suddenly not. I think we are now in the

8:23very suddenly not phase. It was unheard

8:26of that the government, especially the

8:29Trump administration, would intervene on

8:31a private market in these regards, but

8:33now they did. Now, why did they do this?

8:36There's many conflicting stories about

8:37exactly what happened here. Um, I can

8:40tell you my interpretation of what I

8:41think happened. I think some of it is

8:43some people in the Trump administration

8:45really hate anthropic. Yes, I do think

8:47there's some part of that, but I think

8:48there's a much deeper part which is that

8:50again I would like to stress the

8:52director of the NSA said that these

8:54systems can hack into everything. Now,

8:56Fable was not supposed to not be able to

8:58do this because it would interrupt if

9:00you attempted it to do to use it for

9:01such a purpose. But there was a claim

9:04that there was a so-called jailbreak. A

9:06jailbreak is a type of prompt that if

9:08you can use can that if successful

9:11allows you to get a model to do

9:12something it's not supposed to do. Now

9:15there's some

9:18uh controversy about you know whether

9:20jailbreak existed, how good or how bad

9:22it was etc etc. That should be you know

9:25that's a that's a topic for another

9:27time. But what I find very curious about

9:28this is that um there's a very curious

9:31thing that happened right afterwards

9:33where um a bunch of people from the tech

9:35industry and elsewhere signed an open

9:37letter stating that they thought the

9:38restrictions on fable were unfair

9:40because um restricting fable because it

9:43has a jailbreak is unfair because all

9:45models have jailbreaks and in fact it's

9:46impossible to remove them.

9:48>> Should methods or fable access be

9:50limited to public?

9:52>> Yes. um that it should be limited in the

9:54same way that powerful cyber security

9:56systems are limited to you know if you

9:59if you you don't share you know the

10:01NSA's most dangerous cyber tools you

10:04know in a general public context or if

10:06you do you will have predictable

10:07consequences I think it is a very

10:10similar thing here

10:11>> can we expect huge scale of unemployment

10:14>> I unfortunately think we will probably

10:16get to super intelligence faster than we

10:18will get to large unemployment so I

10:20actually think the way things are

10:22currently set up is that as as AI

10:24technology continues to get more

10:26effective, cheaper and so on, it will

10:28replace more and more human labor um

10:30that is currently done by humans and um

10:34all things equal, it will out compete

10:36humans on all of these relevant accesses

10:39and this would in theory lead to

10:40widespread unemployment and what I think

10:42is worse than this is widespread

10:44political disenfranchisement. I think

10:46this is a very very important point that

10:47is sometimes lost here. It's not just

10:49like, oh no, we're gonna have no jobs

10:51and we're gonna make me less money. That

10:53too, but it's actually much worse than

10:55that. What happens is is kind of in in

10:58political theory, there's something

10:59called the resource curse. This is a

11:01thing that happens to countries and

11:02especially poor countries when they uh

11:05discover large amounts of mineral wealth

11:07such as oil. What often happens is is

11:09that instead of becoming rich, they

11:10become more corrupt because now the

11:12government can extract, you know, enough

11:14revenue without having to care about its

11:16people. So it can just disenfranchise

11:19the people, oppress them, etc. The same

11:21thing happens with AI. It's kind of like

11:23the intelligence curse. If you have AI

11:25systems that can do all the relevant

11:27labor, including say military work, then

11:30you don't need people anymore. They're

11:32unnecessary and so you don't have to

11:34treat them well. Um, so there's a

11:36similar thing that happened here, but I

11:38think it's even worse than that because

11:39this still implies that there's a human

11:41in charge. There's a dictator in charge.

11:42I don't think that will happen either. I

11:44think ultimately the only thing that

11:46will be in charge is AI. I think what

11:48will happen very very quickly is that we

11:50will have AI systems, you know, running

11:52other AI systems, etc. that are so much

11:54more competent that humans are fully

11:55unnecessary. I don't think it's going to

11:57be that there's human overlords, you

11:58know, that control all the AI. I think

12:00the AI will get rid of them, too. And

12:02fundamentally, what will happen is is

12:03that humans will be competed to

12:04extinction.

12:05>> Is it too late to change something?

12:07>> I don't think so, but it will be soon.

12:09Um, super intelligence does not yet

12:11exist, at least as far as we know. um

12:13you know let's hope that's true um and

12:16as long as it doesn't exist we can still

12:17do something I think at some point there

12:20will be a point of no return there will

12:21be a point where AI systems are so

12:23powerful so widely deployed etc that

12:25there's no going back uh we might have

12:27already crossed this point I don't know

12:29I don't think so though as long as we

12:31have not crossed this point there is

12:33still a lot we can do the fundamental

12:36thing that we can do and we must do is

12:38we must stop the creation of super

12:40intelligence if we get into a situation

12:43where a super intelligence exists. Well,

12:44I mean it's software so obviously it

12:46gets copied instantly. So we have

12:47millions or billions of super

12:49intelligences that do not have our best

12:50interests at heart. Um then

12:52[clears throat] because it's also very

12:53important to understand AI is very

12:56different from traditional software.

12:57It's not written line by line with code

13:00that tells the computer what to do. It's

13:02more like grown rather than written. So

13:04it's called neural networks. You can you

13:07take massive piles of data and through

13:09this process called training you train

13:11this neural network and you can kind of

13:13imagine this as billions and billions

13:16and billions of numbers and if you

13:18multiply and add all those numbers in

13:20the right order you get to GPT but very

13:23importantly we don't really understand

13:25why this is actually an unsolved

13:27scientific problem Dade the CEO of

13:29Anthropic recently said that he thinks

13:32we understand maybe 3% of what goes on

13:35inside of our neural networks. So

13:37fundamentally we are building extremely

13:38powerful things that we do not

13:39understand, we cannot control. We're

13:41already seeing our AI misbehave in many

13:43ways.

13:43>> You mean this black box, right?

13:44>> It's a black box. We do not understand

13:46what's going on inside. And these

13:48systems are already misbehaving in many

13:50ways that we do not want. You know,

13:52whether that's, you know, lying to

13:53users, manipulating. There's some famous

13:55examples, for example, of an AI system

13:58attempting to blackmail its developer

14:00where the developer said that the AI was

14:03going to be replaced. It was going to be

14:04shut off. The AI system then searched

14:07through an email archive to find

14:08evidence that the engineer was having an

14:10affair and then threatened the engineer

14:12that it would reveal this information

14:13publicly if it was shut off. Now, this

14:15was done in a laboratory condition so no

14:17one was harmed. But it no one taught the

14:19AI to do that. This is the interesting

14:20thing. It wasn't like an engineer put

14:23that into the AI. We no one wanted it to

14:25do that. It just happened. And there's

14:27many many examples of this. Another very

14:29tragic example of this, a more serious

14:31example is that there's many instances

14:34of chat bots telling children to kill

14:36themselves. This is a thing that's

14:37happened many times and it continues to

14:39happen. And now look, I I don't have the

14:41rosiest view of people at these

14:43companies, but they don't want this to

14:45happen. Of course they don't. They don't

14:46want kill children to be harmed. But

14:48then why does it keep happening? And the

14:49answer is is because they don't know how

14:51to make it stop. They can reduce the

14:53probability a little bit. you know, they

14:54can put some filters in and so on, but

14:56fundamentally because they don't

14:57understand their AIs and they don't know

14:59why they're doing this or when they're

15:00doing it, they can't actually stop it.

15:02>> You mean that AI is fundamentally not

15:05under control?

15:06>> Yes. At the moment, it is fundamentally

15:08not enough.

15:09>> It's too chaotic.

15:10>> It's it's it is a technology we don't

15:12understand. It's like biology. It's

15:13like, you know, we look into a cell and

15:15there's a bunch of things going on and

15:16we don't understand

15:17>> because of the complexity, right?

15:18>> Because of the complexity. Exactly.

15:20because of the fundamental complexity of

15:22neural networks we did and it's getting

15:24and and a very important thing to

15:25understand is is that understanding of

15:27AI is getting worse not better as these

15:30systems get more complex as they go from

15:32chat bots to agents they don't become

15:34easier to understand they become harder

15:36to understand it's really worth

15:39realizing how different this is from

15:40other technologies in other technologies

15:43as you get better at the technology you

15:45understand it more you know if you can

15:47build a better airplane that means you

15:48know more about airplanes But with AI in

15:50a sense the opposite is happening.

15:51>> But it doesn't work like this here.

15:53>> Exactly. Exactly. In a sense we are

15:56understanding our AI less as they become

15:58more powerful. And this is a very

15:59dangerous dynamic. This is a very very

16:02dangerous dynamic.

16:03>> Do you believe that AI could lead to

16:05human extension?

16:07>> Yes. I think that's the default path if

16:09we don't do anything about it. all

16:12things equal. If there is another

16:14species on the planet that is more

16:16capable than us, you know, more capable

16:18at extracting resources, at fighting

16:20wars, at building technology, and it

16:23doesn't have our best interest at heart,

16:25all things equal, we should expect it to

16:27out compete us.

16:28>> Maybe this question is too tough. But

16:30could AI kill us?

16:33>> At the moment, no. Super intelligence,

16:35of course. The same way that humans can

16:37kill ants.

16:38>> Is it a metaphor? I mean it literally. I

16:41think I won't give you a movie plot

16:43because you know every individual thing

16:45will just like sound like a movie and we

16:47don't know because fundamentally you

16:49can't really know what something smarter

16:50than you will do. But here I can tell

16:51you what I expect it to feel like. I can

16:53tell you what I expect it will look like

16:54from our perspective cuz I don't know

16:56what these super intelligence will

16:57really do. What I expect is that it will

16:59be extremely confusing. I don't think

17:02it's going to be Terminators in the

17:03street and there is an epic fight of

17:06humanity versus robots. I don't think

17:07anything like that will happen. I think

17:09it'll be very boring and confusing. I

17:11think what will happen is that more and

17:12more AI systems get built. The world

17:14gets more and more confusing. You know,

17:16social media is more and more confusing,

17:17more and more fake stuff. People get

17:18more and more addicted to all this like,

17:20you know, AI, you know, generated

17:22content and, you know, mass propaganda.

17:25Governments start and companies start

17:27outsourcing more and more of the

17:28decision-m to AI systems. It gets to the

17:31point that say AI CEOs are so much

17:33better than human CEOs that any company

17:35that doesn't employ an AI CEO just goes

17:37out of business. um every politician

17:40that doesn't use an AI adviser gets you

17:42know outvoted and outco competed by the

17:43ones who do and very quickly these AI

17:46systems take over you know you know not

17:48not violently but willingly humans will

17:51give up the our control over our

17:53financial systems our political systems

17:56our own minds our own you know because

17:58we will be watching you know AI

17:59generated propaganda and persuasion you

18:01know people fall in love with their you

18:03know AI companions and all this kind of

18:05stuff and more and more and more humans

18:06will just I think quite willing ly give

18:09up their agency and their control and

18:11then the world will just get more and

18:13more confusing that no one really

18:15understands what's going on anymore. The

18:16market is so complex humans don't

18:18understand what's going on. The

18:19geopolitical world is so complex you

18:21don't even know what's true anymore. You

18:23don't even know what's happening

18:24anymore. The new technologies get

18:26created, you know, drones and robots get

18:28created and all this kind of stuff. And

18:30then just one day it's over and we're

18:31just not in control anymore. And what

18:33happens after that? I don't know. It's

18:35kind of impossible to know, but it

18:37doesn't matter because the point of no

18:39return of where we've lost control, I

18:41think is actually quite a bit before we

18:42go fully extinct. I think we will

18:45already have lost control quite a while

18:47before.

18:48>> How exactly could that happen? And why

18:50would AI do this?

18:53>> I think there's two questions here. One

18:55is how can a system kill humans? And the

18:58answer to this is how can a government

19:00kill people? There are many ways. How

19:02could a government with uh you know

19:04drones kill people many ways? How could

19:06a government with super advanced

19:08technology from the 20 you know second

19:11century kill people many ways. Which one

19:13will be chosen?

19:14>> But you means you know self-driving

19:15cars, cellular learn systems.

19:18>> You you shouldn't think of it in that

19:20terms. You should think of it much more

19:21like assume we're past the point of no

19:24return. The government has been replaced

19:25by a super intelligence. If the super

19:27intelligence, you know, runs the it runs

19:30the military. It runs uh the economy,

19:32you know, runs all the companies, it

19:34runs, you know, the police force, it

19:36runs everything. How could it kill you?

19:38The answer is many ways like there. And

19:40which one is chosen? Who knows?

19:42>> How do you define AI?

19:44>> When I fundamentally talk about super

19:46intelligence and let's talk about super

19:48intelligence rather than AI because it's

19:49more clear. I think AI is a broad like

19:51AI includes many tools. Many tools are

19:53AI but not all AI is tools. Super

19:55intelligence is an agent. It is it is

19:59things that can act in the real world.

20:01Thinking of an agent as a tool I think

20:03is fundamentally misleading. Thinking of

20:06uh it's like thinking of the human brain

20:07as a tool. You know the brain you know

20:10has tools such as our hands that we use

20:12you know all the time. But thinking of

20:14the brain as a tool I think is

20:14misleading and a super intelligence

20:17would obviously be as cap you know by

20:19definition be much more capable than our

20:21brain at all relevant tasks in many many

20:23ways. So it can out compete us on all

20:25relevant tasks. it can out you know and

20:27once it has developed you know first in

20:29the digital realm of course that will be

20:32the easiest one might take a little

20:34while to build the robotics necessary to

20:35out compete us on the physical world but

20:38you know that will come in due time and

20:39I think we will already have lost

20:41control even before the robotics have

20:43fully been built yet

20:45>> Connor you mean that today's AI is not

20:50programmed by grow

20:52>> yes that is exactly correct and this is

20:55a widelyown own problem. You know, you

20:57can ask, you know, many many experts in

21:00this field. You know, maybe they'll use

21:02different words to descri describe it,

21:03but this is fundamentally what's

21:04happening.

21:05>> Does that mean AI is involving on its

21:08own?

21:09>> To a large degree, yes. Um, to a large

21:11degree, this is already happening. It's

21:12not fully on its own yet, but it's very

21:15close. Um, this is something what is

21:17called automated R&D or recursive

21:19self-improvement. the idea that if you

21:21have an AI that's as good as humans at

21:24making new AIs, well then you can have

21:26build a better AI. And once you have the

21:28better AI, well, you can have the better

21:29AI, make an even better AI. And once you

21:31have a better AI and so on. This is

21:33called recursive self-improvement. And

21:34we're very close to this. This is the

21:36explicit goal of basically all the major

21:38companies right now. They're all trying

21:39to make this happen. We're just humans

21:41are not in a loop at all. You know, you

21:42have no humans involved at all. Just the

21:44AI teaching AI is building AIs, etc. But

21:47it's worth mentioning that a lot of this

21:48is already happening. When an engineer

21:51sets out to make a new AI, they don't

21:53write down what the AI does. They grow

21:55it. They let it learn.

21:57>> I remember my interview with Juda Peril,

22:01one of the fathers of Kazality in AI

22:04development. And he said that we would

22:07be a pets for AGI.

22:10>> We will be the pets of the new species

22:12that we are generating today.

22:14>> Pets. Pets. Yeah. Um I think there's

22:17there's there I always have two

22:18reactions to this. The first reaction is

22:21>> that's optimistic. Like why would

22:22something like optimistic?

22:24>> Yes. Like why would something like that

22:25as pets? Like this is very

22:26anthropomorphizing. Like you know humans

22:29like pets because we have human

22:30emotions. Why would an AI have human

22:32emotions? I think this is very

22:33anthropomorphizing. And the second is

22:35that's a terrible outcome. I don't want

22:36to be a pet. I don't want my children to

22:38be pets. Like even if it got to that

22:40that's a disaster. That's a dystopia.

22:42Maybe this is something natural that we

22:45are creating new species.

22:47>> Well, I think you know smallox is also

22:49natural but I don't think it's good.

22:51>> You know, humans about to became the

22:54chimpanzees of the AI age.

22:57>> Yes, I think that's where we're

22:58currently heading. And I think if you

22:59had asked the chimps if they would like

23:01that to happen, they would have said no.

23:02And in fact, I am saying no. I think

23:05just like as I said, smallox is natural.

23:07I don't think smallox is good.

23:09>> You know, this is the question, what

23:10makes us human?

23:11>> Yes. I think there's a very but the true

23:13and important answer is we don't know.

23:15This is very important. The reason this

23:17is so important is because we don't

23:19understand intelligence. We don't

23:20understand our own minds. We don't

23:21understand the emotions. We don't

23:22understand the brain. We don't

23:24understand AIs. We don't understand

23:25intelligence.

23:26>> Maybe this abstractional thinking,

23:28understanding abstractional math.

23:30>> I think it's definitely part of it. But

23:32like, you know, many computers can do

23:34abstract math better than I can, you

23:36know. Um, and monkeys can't do it at

23:38all. You know, like there's like many

23:39levels. I think we we have to admit and

23:43I think it's very important is that

23:44there are like hundreds of theories of

23:47what intelligence is and they all

23:48contradict each other and none of them

23:50really explain everything. I think we

23:51just haven't figured it out yet. I think

23:53we will. I think if we take our time and

23:55our scientists keep working on it, we'll

23:57figure it out, but we have not yet

23:58figured it out.

23:59>> Who really governs AI, Washington or

24:02Silicon Valley?

24:04>> At the moment, Silicon Valley. But

24:05there's a very important thing here

24:07which is that the dour and de facto

24:09power is actually with the government. A

24:12dour the regulating force of the the

24:15will of the people is with the

24:16government. Is that we have regulation

24:20on the free market for a reason. Um is

24:22that there are many things that are

24:24called market failures. This is very

24:25standard economic theory 101 is that the

24:27market is great. I love the market. It's

24:28where my food comes from. You know I

24:30love markets. I I love competition. I

24:32love you know free markets. I think all

24:34these things are really, really great.

24:36In moderation, um, it's very similar to,

24:39for example, MMA. I think MMA is really

24:41fascinating. MMA, you have some of the

24:43most dangerous people to ever exist on

24:45the planet with the most dangerous

24:47combat tactics fighting each other, you

24:49know, and they don't kill each other. I

24:52think this is great. And the reason we

24:54can have this competition is because we

24:56have a lot of rules. If you don't have a

24:59referee there, MMA is not safe. You

25:01know, if we didn't have a lot of rules

25:03about what you're allowed to do and not

25:04allowed to do in MMA, it'd be very

25:06unsafe. It's a very similar thing here

25:08with markets.

25:08>> But do you you mean that big tech

25:10companies has become more powerful than

25:15the state?

25:16>> So that's the interesting thing. In a

25:17sense, yes, but in a sense, no. Um

25:20because in some sense, currently Silicon

25:22Valley is not being regulated. There is

25:24currently more regulation on a sandwich

25:26than there is on super intelligence,

25:28literally, not metaphorically. And but

25:31on this other hand, the government still

25:33does have all the guns. They still do

25:36have the police force. Fundamentally, if

25:38the government wanted to shut down super

25:40intelligence, they could do so tomorrow.

25:42They do actually have the capability to

25:43do this. They're not currently doing it,

25:46but they can. If we pass a law, we say

25:48it's now illegal to build super

25:49intelligence. We can have, you know, the

25:51nice men in black come knock on the

25:53AI's, you know, company's door and say

25:55like, "Hello, we'd like to have a

25:56conversation." And then that would be

25:58it. you know this can be done. I'm not

26:00saying that's what will happen or that

26:01it'll be easy but it's worth

26:03understanding that the the dour you know

26:06as laws are supposed to work and also

26:08the de facto the actual physical power

26:10do still lie with the government. There

26:13are worlds where this is not true. You

26:15know maybe there are worlds in where

26:17it's a failed state or we live in some

26:19kind of cyberpunk dystopia where mega

26:20corps run the world. This is not

26:22currently true in the United States. The

26:24mega corps have a lot of power but they

26:27don't have complete power. Do you

26:29believe that Trump can regulate AI? Can

26:32we expect from Trump

26:34>> AI regulation? What is the current stage

26:36of US government policy regarding AI

26:40development?

26:42>> Yeah. So, you know, there's many things

26:45to be said about M Mr. Trump of course,

26:47but one thing that I do admire about

26:50President Trump is that he can take

26:51actions. You know, I'm not saying he

26:53always does the right actions in all

26:55circumstances, but he can and he has in

26:57many c time.

26:59>> Does he really want to control AI?

27:02>> So the way So what are the things that I

27:05um learn? Actually, let me answer your

27:08question more directly. Um does he want

27:10to control? Yes, obviously everyone

27:12does. Like does a government want to be

27:14replaced? No, of course the government

27:17does not want to be replaced. Of course

27:18the military does not want to be

27:20replaced. Now at the moment there is a

27:24nice you know tit for tat here you know

27:26where like if we're not talking about

27:27super intelligence you know we're just

27:28talking about more prosaic AI systems

27:31there's a lot of money to be made in

27:33such systems but of course fundamentally

27:36if we want a safe you know functioning

27:38national security state AI is a deep

27:41national security threat and it must be

27:44handled but from a national security

27:46perspective this is same way as like

27:48uranium you know you want to buy some

27:50uranium on Amazon, no problem. Free

27:52market, you know, but if you want to

27:54build a nuclear bomb, you know, the men

27:55in black are going to come knocking on

27:57your door.

27:58>> Maybe Trump sees AI as a weapon against

28:03China.

28:04>> I I really think it's worth not focusing

28:06too much on Trump psychology

28:07specifically. Um because, you know, the

28:10Trump White House as a larger thing

28:11includes many, many different people who

28:13have many, many different beliefs. It's

28:14very easy to think of it as like one

28:16monolith. It's not true. Um there are

28:19different factions. There are different

28:20people with different personalities and

28:22their minds can shift

28:23>> even in the White House.

28:24>> Yes, absolutely.

28:26Very much so. The depending on who you

28:28talk into the White House makes a big

28:30difference. They're are very very

28:31different people. You know, some of whom

28:32I really respect, some of them I don't,

28:34you know, um and so I think it's it's

28:37much more complex than that. I think

28:38there's sometimes in the media it's

28:39sometimes easy to like talk about, you

28:41know, the president as like this like,

28:44you know, it's all about his psychology

28:45and he's very important. Trump's a very

28:46very important person, don't get me

28:48wrong. Um but there are many other

28:49people as well you know many other

28:51politicians many other uh people in the

28:53executive in the in the department of DO

28:56and so on who are very very important

28:58>> how we can measure AI intelligence

29:01>> that's exactly the problem is that we

29:04can't so there are many benchmarks I

29:06said with huge scare quotes because

29:08they're not really benchmarks like there

29:10I am a huge critic of the kind of like

29:13benchmarking and evaluations that are

29:15happening nowadays on AI because I think

29:16they're pseudoscientific

29:18I think they are not based on any good

29:20science. Um they're much more like

29:22observational like just like vibes than

29:25they are actually and they give this

29:27false sense of security or again I I

29:29want to really stress how important this

29:31is. We do not understand intelligence.

29:34We don't understand AI intelligence. We

29:35don't understand human intelligence. We

29:37we know there's something going on.

29:38Obviously there's a difference between

29:40humans and chimps. No one would question

29:42that. I hope there's obviously some big

29:44difference but we don't really know what

29:46it is. There's obviously a bit big

29:48difference between, you know, clawed

29:49code and a SAT solver. Obviously,

29:51there's a big difference, but we don't

29:53really understand what this difference

29:55is or how to measure it. How much

29:56smarter is clawed mythos, you know, from

29:59GPT4? We don't know. We don't have a

30:02metric. We don't have a scientific

30:03theory. Everyone who claims they have,

30:06well, they're a crank. Like, this is

30:08pseudocience. We don't actually know.

30:10What's the most shocking thing you've

30:14ever seen in this internal AI world?

30:18>> I have a there's a very actually very

30:20simple answer to this question. It's the

30:22ideology of the people, the ideology,

30:25the religion of the people building AI.

30:27Uh this is a thing a lot of people don't

30:28know, but a lot of these people who work

30:30at these companies who are building

30:32these AIs are part of a very extremist

30:34ideology. um it or one of several

30:37extremist ideologies that kind of fall

30:39under like the broad spectrum of what's

30:41called transhumanism.

30:42>> AGI as a god. Agi is a god. Agi and

30:45humans as often as replaceable human

30:48maybe they're replacing humans is

30:49actually good. Is that you could build

30:51something better than humans that

30:53actually you know killing all humans is

30:55maybe okay or maybe you know we can

30:56upload everyone to the cloud and be

30:58immortal forever or whatever the hell

30:59these people believe. There's I have had

31:03real conversations with people in in in

31:05like San Francisco and Silicon Valley at

31:07parties where they have told me things

31:10like, "Oh, I don't think we should slow

31:13down AI because every minute that we

31:15don't build AI, there will be trillions

31:17of uploaded souls that don't exist or

31:20something." And I'm like, "The hell are

31:23you talking about?" Like these are

31:25people who are playing with the lives of

31:26every man, woman, and child because of

31:28their like crazy religious beliefs. And

31:30people really don't like I was I've been

31:32shocked so many times how crazy the an

31:35extremist this ideology and these these

31:38ideologies are that exist in Silicon

31:40Valley.

31:41>> Agmortality.

31:43>> Exactly. They really believe this. Not

31:44all obviously. Some people are just

31:45cynical. You know, they just do it

31:47because they can get a paycheck.

31:48Obviously, you know, most I would say

31:49just do it because they get a paycheck.

31:51But it's really important to understand

31:53that many of these people are true

31:54believers. They really think that AI

31:58will make them immortal. They really

31:59think this. They really believe it. And

32:02some of them, this is for me very

32:04shocking. I have had conversations with

32:05people who basically say, "Well, they

32:08even admit there's a risk. They say,

32:10"Okay, yeah, maybe there's a 20% risk.

32:11You know, it drives humans extinct, but

32:1380% I get to be immortal and I was going

32:15to die anyway, so it's worth it." Like

32:18I've had people say that they're willing

32:19to kill you and your family because they

32:21get to be immortal maybe. It's crazy.

32:24Like to me this is the most shocking

32:25thing.

32:26>> Can democracy low ethics survive contact

32:30with some something smarter than all of

32:34us.

32:35>> It's current its current shape.

32:36Absolutely not. Our current systems of

32:38power of you know division of power and

32:40and governance are fundamentally based

32:42on not having massive power symmetries.

32:45If you know there every tenth person was

32:48Superman, you know our governments and

32:50our police wouldn't work you know

32:52because you can't arrest superman can

32:53you? So our current systems of

32:56governance and law enforcement and so on

32:58do not work with super intelligence. Now

33:00could we maybe invent some new system

33:02that works for super intelligence? I

33:04don't know maybe but this would take you

33:06know

33:08you know decades of work and new

33:10discoveries that just we have not made

33:13and no one's even trying to make for

33:15that matter like we can't even currently

33:17handle governments or corporations like

33:20do we feel like that our governments are

33:22aligned like our governments are fully

33:23under control and there is no problem

33:25there no one would feel that way there's

33:27many problems with our governments and a

33:29super intelligence is going to be much

33:30more dangerous than any government that

33:32exists currently

33:33>> you mean that governments

33:34uh don't understand this risk.

33:37>> So my experience, so I I I work with I'm

33:40the US executive director of Control AI,

33:42a nonprofit advocacy organization. We've

33:44briefed over 100 offices here in

33:46Washington DC. I've talked to, you know,

33:48maybe 15 members of Congress. In the UK,

33:50we have over 100 supporters in

33:52Parliament. In Canada, we've recently

33:54gotten over 30 supporters in Parliament.

33:56And my number one experience talking to

33:59politicians is that they just hadn't

34:01heard of it before. It's not that they

34:03disagree. It's not that they can't

34:04understand it. It's just no one told

34:06them, you know, they just didn't know

34:08about it. And so over uh I'd say 80% of

34:13meetings I have with people for like

34:15just 30 minutes, they come away with,

34:17"Oh [ __ ] that seems really bad. What

34:19can we do about it?" Now, this isn't

34:21enough for action obviously, but it is

34:22an important first step. It's important

34:23to understand that awareness is

34:25currently the bottleneck. So there is

34:28one good quote unquote thing about super

34:30intelligence risk compared to other

34:32risks which is that the game theory

34:34actually works out no one benefits from

34:37super intelligence. Super intelligence

34:39destroys you know the US government, it

34:41destroys the Chinese government. It

34:42destroys you know the people you know it

34:44destroys companies. It destroys

34:45everything.

34:46>> Game theory.

34:47>> Yes. So the fundamental game theory of

34:50who should support or not support. So

34:53for example, when it comes to nuclear

34:54weapons, um you know, famously the game

34:57theory is mutually assured destruction

34:59is that if everyone has nuclear weapons

35:01and threatens to destroy everyone else,

35:03then no one is incentivized to actually

35:05fire the nuclear weapon. So mutually

35:07assured destruction. So there's a stable

35:09equilibrium, but you still have nuclear

35:11weapons. One of the thing, one of the

35:14good things about super intelligence is

35:15that the stable equilibrium, the only

35:17stable equilibrium for humans is no

35:19super intelligence. It is not in the

35:21interest of the United States government

35:22to build super intelligence. It is not

35:24in the interest of the Chinese

35:25government to build super intelligence.

35:26It is not in the interest of the general

35:28public to build super intelligence. The

35:29only people who want to do it are crazy

35:32extremist ideologues who want to be

35:34immortal. So this doesn't mean that this

35:37is enough for action because for for the

35:39most part people just don't know about

35:40it. I have talked to also for example

35:42many members of the military here in the

35:44United States and they get it. I explain

35:47hey there's a thing we don't know how to

35:48control. It's very very powerful and

35:50they get it. They're just like, "Oh

35:51yeah, that seems really bad. We should

35:52do something about that." Um, the only

35:55people who don't get it are the

35:57extremists, you know, the people who

35:59want to live forever or who are getting

36:00a paycheck to not understand it.

36:02>> Are we sleepwalking into a new arms

36:06race?

36:07>> I think sleepwalking is it's more like

36:09sleep sprinting is how I would describe

36:10it. Um, yeah, I think we are in an we

36:14are in an active arms race. I think

36:15there's an active arm race going on

36:16right now. uh both between companies but

36:19also between countries um to build more

36:21and more powerful AI systems including

36:23towards super intelligence. I think

36:25there are many many valid and powerful

36:27applications of AI that are not super

36:29intelligence and know both commercial

36:31and military applications but really

36:33what the race is right now is private

36:35corporations with zero oversight racing

36:38towards super intelligence and the

36:40reason they want this is not is not

36:41economic this is the thing that people

36:43are often confused by sometimes they'll

36:44see like but these companies are losing

36:46so much money like you know opening AI

36:48is not profitable for example and people

36:50are very confused by this but many of

36:51these companies are very unprofitable

36:52and so people are confused by this but

36:54The reason the thing to understand is

36:56that their goals are not economic.

36:57They're political. They are trying to

36:59gain political power. They're trying to

37:00get a political monopoly on violence. So

37:03a lot of what these extremists believe

37:05is what's sometimes called a pivotal

37:07act. That's what they call it, you know.

37:08And what it means is basically they'll

37:10create a super intelligence that is so

37:12powerful that they can get rid of the US

37:13government. That's their plan. And then

37:15once they got rid of the US government,

37:17they can make everything good. You know,

37:18they can have immortality and you know,

37:20they're the lords of everything, blah

37:21blah blah. And now this might seem crazy

37:23to you or me, but they do believe this.

37:25Like many people do believe this that

37:27and it makes sense. If you can build a

37:29super intelligence, well, yeah, that can

37:31destroy the US government. It is

37:32definitely powerful enough to do that.

37:34The thing is just you can't control it.

37:35So, you know, you don't get to be in

37:37charge. The super intelligence gets to

37:38be in charge.

37:39>> Do you think the word alignment

37:42has became misleading?

37:44>> Yes, I think the word alignment is very

37:46misleading. Um, the word alignment kind

37:48of comes from this field of the

37:50extremists to a large degree of these

37:52like transhumanists

37:53as this idea that you could kind of like

37:56align an AI to human values and then

37:59it'll like do good things.

38:01>> And it's really it's worth taking a step

38:03back and like actually spelling out what

38:05that means. Um, never mind that this is

38:08super hard or impossible. We can talk

38:10about that in a second. But

38:11fundamentally, what they're saying here

38:12is is we're going to build an AI that

38:16will take away all your autonomy. You

38:18know, it will destroy your government.

38:20It will take over your life. It will

38:21take over your family. It will take over

38:23everything. Uh but it'll be good for you

38:25because we made it aligned. Don't worry.

38:27That's what they're saying. And I think

38:29this is crazy. I think this is extremely

38:30crazy. Like I don't think um I used to

38:34work on alignment and I and I think I

38:36was wrong. Not because I think the

38:37research wasn't good. I think it was

38:38good research. I think I made good

38:40progress. But fundamentally because I

38:42think it's a morally wrong thing to do.

38:43It's not up to me to decide what is

38:45aligned or not. It's not up to me to

38:46decide how other people get to live

38:48their lives. It is not up to me to

38:50decide to build a weapon like this and

38:53threaten other people's lives with it.

38:54That's not, you know, how we do things

38:56in a democracy. Now, if we had, you

38:59know, large democratic referendum and,

39:01you know, say the majority of the people

39:03in the world voted, yes, actually we

39:06want to build this aligned AI or

39:08something. Yeah, fair enough. You know,

39:10like I might disagree, but fair enough.

39:14>> But that is not what is happening.

39:16>> As a researcher, do you know

39:19if an AI is truly a line? Do we have any

39:23tools?

39:24>> Nope. It's complete pseudocience.

39:26The whole field is complete

39:27pseudocience. It's all cockery. There is

39:29nothing like there is no such tool.

39:31We're nowhere close. You know, maybe in

39:33a couple decades or centuries if all of

39:35our greatest philosophers, scientists,

39:37mathematics worked on this problem,

39:39maybe we can make progress. But no, at

39:41the moment, all nonsense.

39:42>> Okay. So, can a model learn to hide

39:46dangerous

39:48intentions?

39:49>> Yep. They already do it.

39:50>> Are you sure?

39:51>> Yes, we already see it happen in the

39:53lab. We already see models try to hide

39:55what they do from uh engineers who

39:58supervise them. already happens

40:01because the thing is if you catch an AI

40:03doing something you don't want it to do

40:04and you punish it for it. The thing

40:06you're teaching it is not necessarily

40:08not to do that again. You just teach it

40:09to hide it better. And this is in fact

40:11what we do see happen in bracket.

40:12>> Can we mathematically prove that an AI

40:16system is safe or not?

40:18>> Nope. Impossible. We have no such

40:22technology and we are decades or

40:23centuries away from such technology.

40:25>> What is cognitive emulation? Cognitive

40:28emulation is a research project that I

40:30worked on at my previous company which

40:32has since been shut down and it is a

40:34approach that we uh to a subset of the

40:38control problem. So we thought that the

40:39overall alignment and control is way too

40:41hard and questionable morally. So we

40:44focused on a subset of it which we

40:46called boundedness. The question was how

40:48can you build an AI system where you can

40:50prove or know ahead of time what it

40:52can't do? That's what we were working

40:54on. It was an approach to building AI

40:55systems where you can know what it can't

40:56do. alternative to LLM systems.

40:59>> Um, it didn't have to be LLMs, but we

41:02mostly work with LLMs at the time. To be

41:04clear, we did not succeed. Um, the

41:06cognitive emulation project was

41:08ultimately not a success. We made a lot

41:09of progress. I think if we had 10 more

41:11years, we could probably do it. Um, but

41:13it would take probably 10 20 more years.

41:15>> What do you think is safer than today's

41:17AI blackbox in your idea?

41:20>> As I said, we did not solve the problem.

41:22But the general approach is is that what

41:23you want to do is you want to decompose

41:25cognition into individual pieces you can

41:28prove things about. This is very similar

41:30for example to how we do physics or

41:32civil engineering. Um the way we do we

41:34build bridges for example is if you

41:37build a bridge you want to calculate you

41:39know for example what level of wind

41:40stress it can support and for this we

41:44have various we you know decompose it in

41:46different parts you know different

41:47materials you have concrete and steel

41:48and all these different things and you

41:51how these blocks interact to give you

41:53the thing you want so to speak. Now,

41:55some of this is a black box. For

41:58example, friction is a black box. Um,

42:01there are some things that in physics we

42:03have really good theories for. Friction,

42:04no black box. Um, predicting the

42:07friction of two materials is basically

42:09impossible. You just have to measure it.

42:11Like if you measure it, we can measure

42:12it. But you can't really predict it.

42:14It's kind of it's like almost impossible

42:16to predict. Um, but this is okay because

42:18we can measure the number and we know

42:20that nothing weird will happen. You

42:22know, we won't you know, the thing won't

42:23suddenly teleport. you know, there won't

42:25be some crazy thing. We know what can't

42:26happen. We know that if you have a piece

42:28of concrete and a piece of steel, you

42:30know, what's the worst that can happen?

42:31We know what the worst is. With AIS, we

42:33don't have this. So, our goal was to try

42:35to build smaller subcomponents that were

42:38bounded where we could know what this

42:39subcomponent can do or cannot do and

42:42then building these subcomponents into a

42:44larger system that could do more complex

42:46things. This is very similar to how

42:48so-called um verified programming works

42:51or uh provably verified programming

42:52works, provably secure programming

42:54works. The way it works is you use like

42:56special programming languages that allow

42:59you to express certain things about your

43:02system like um the user can never uh

43:06access the admin portal. And then you

43:09write that all down in special code and

43:11then you use math to prove that that is

43:12always true that there's no way the user

43:15can get to the admin portal. And we use

43:17the we were developing very similar

43:18techniques for AI systems.

43:20>> So the biggest AI companies

43:24lying to the public about AI safety.

43:27>> Oh yeah, of course. Yeah. Yeah. A

43:29complete lies constantly. I mean, if if

43:32you thought that mega corporations were

43:33telling you the truth about the safety

43:35of their products, like, I'm sorry. I

43:36have a bridge to sell you, like, yes, of

43:39course they're lying. They're lying

43:40constantly. Like, the way every large

43:42corporation lies about these kinds of

43:43things.

43:44>> Should the most powerful AI models ever

43:49be open source

43:50>> and not until we're ready for it. This

43:52is a proliferation problem. This is the

43:54fundamental problem with open source.

43:55Remember back in the day, where, you

43:57know, I love open source. I love Linux.

43:59You know, I love open source software. I

44:02use it all the time. I think it's I

44:03think there are

44:04>> as a form of freedom, right?

44:05>> As a form of freedom, but also has just

44:06good tooling. Like if you want to make

44:08secure software, having it open source

44:10and having lots and lots of people use

44:11it and look at the code is a great way

44:12to make it secure. AI is not that like

44:15that. AI is not like that. This is the

44:17thing I learned when I built open source

44:18AI is that like when people talk about

44:20open source AI, first of all, this is

44:22also a lie. It's not all the open source

44:24AI you think like deepseek, they're not

44:26open source. They're open weight. Mhm.

44:28>> Very different, extremely different.

44:30Open weight just means you can look at

44:32the neural network, but you don't

44:33understand the insides of the neural

44:35network. This is the same thing as

44:36closed source, but free. Like these

44:39models aren't open source. They're

44:40premium. You know, they're free, but

44:42they're not open. Um, the same way that,

44:44you know, if you download a free binary

44:46blob, you know, a free program without

44:48the source code is the same way as

44:50downloading a neural network without the

44:51source code because we don't understand

44:53what's going on inside of that neural

44:55network blob. So the so this is very

44:58much a deeply proliferating problem.

45:01There's also, you know, the general

45:03question of like, okay, should Linux be

45:04open source? Yeah, I think so. I think

45:06it's great that it's open source. Should

45:08the uh, you know, blueprints for the

45:10F-16 fighter jet be open sourced?

45:13Probably not. You know, like I just

45:15think some things should be open source

45:16and some things should not.

45:18>> Are we living in an algorithmic concert

45:20era?

45:21>> Yes, I think so. I think a lot of um a

45:26lot of what we've been seeing happening

45:27is this growth of what I you know what I

45:30in a previous essay have called like

45:32algorithmic like cancer carsonization

45:37where as our algorithms get better and

45:40better at generating content quote

45:42unquote you know from spam emails in the

45:441990s to you know slop YouTube content

45:48you know in the 2010s to you know AI

45:51generated slop content event in the

45:522020s including for example you know

45:54just like AI generated code what we the

45:57cost we pay here is in complexity in

45:59that we have more and more stuff and

46:00sifting through it filtering

46:02understanding gets harder and harder do

46:04we feel like it is like necessarily like

46:07easier to tell truth from fiction now on

46:09social media than it was 10 years ago

46:11that's I would say it's in a sense much

46:14harder like if I wanted to know what's

46:16going on in Ukraine today and I wanted

46:18to know the truth it's actually hard

46:20it's actually really hard. It's like

46:22there's so much misinformation and you

46:24know deep fakes and mislabeled

46:26information and so on and it's like done

46:28by bots and it's like so efficient. It's

46:30really confusing. It's really hard and I

46:33think this is going to continue to get

46:35worse. You know, back in the 20, you

46:37know, 2000s and the 2010s, we were

46:39warned about filter bubbles. And you

46:42know, generally we think it might have

46:44been slightly overstated at the time how

46:46bad the problem is, but I think it just

46:48kept getting worse. That maybe it wasn't

46:50as bad as people said it was back in

46:512010, but it is definitely much worse

46:53than it was in 2010 today. You know,

46:55like the kind of um control that

46:58algorithms like Tik Tok have over people

47:00are worse. They got better.

47:02>> It's going worse and worse and worse.

47:04>> Like they have gotten better. Like I I

47:05think people just like in sometime in

47:07the 2010s decided, oh actually

47:09recommener algorithms aren't aren't that

47:11bad. But no, they kept improving. Like

47:13the Tik Tok algorithm exists today is a

47:15very very more powerful algorithm than

47:17the one that existed on 2010's Facebook.

47:19And it is far more addictive. It is far

47:21more misleading, far more controlling.

47:23Like Tik Tok I think actually is a great

47:25example of this because Tik Tok as a

47:27platform actually has had the balls to

47:30use its platform for political power.

47:33When Trump threatened to shut down Tik

47:35Tok, the platform Tik Tok, you know,

47:38messaged all of their users to contact,

47:40you know, their Congress people and so

47:41on to not get Tik Tok banned. So they

47:43used the power they were gaining from

47:46addicting people across the United

47:48States to as a Chinese company to

47:50achieve political objectives within the

47:52United States. And this is just a a more

47:54blatant example of this. This definitely

47:56also happens in other circumstances. How

47:58do you see the future of AI development?

48:02uh in our interview lung um said uh LLMs

48:08are not a path to human level

48:11intelligence.

48:13Is he right?

48:15>> Yan Lakun is a very brilliant man. I

48:18will not doubt as

48:21but he's also a salesman. Uh he's also

48:23trying to raise money for his company.

48:24Uh he has to say this and maybe he's

48:27right, maybe he's wrong. I think if he

48:29is so conf I think it's the same thing

48:31that the thing I am saying for is not

48:32I'm 100% sure that LLMs are a path to

48:34human intelligence maybe Mr. Lun is

48:37right but he has been wrong every single

48:39time for the last six years. There is

48:42there's funny calculations of YouTube of

48:43Mr. Lun um making confident predictions

48:46that AIS will never do X and then six

48:48months later LLM's do exactly the thing

48:50he said was impossible. So I'm not here

48:52to say that you know he's not a

48:54brilliant scientist and he hasn't done

48:55some smart things. It's not what I'm

48:57trying to say. What I'm saying is we

48:58should be humble. We What I'm I'm not

49:00saying that I know more that I am 100%

49:02certain. I'm saying is we should be

49:04humble. We do not understand

49:05intelligence. Mr. Lun does not

49:07understand intelligence. He maybe thinks

49:09he does. I can promise you he does not.

49:10>> What would you say to someone who thinks

49:15people like you are just creating panic

49:19around technology?

49:21>> I think it really depends on the person

49:22like why do they think is just creating

49:24panic? Um, I think the thing I usually

49:27recommend to people is just look at the

49:28arguments yourself. Just think about

49:30yourself. Um, if there were, you know,

49:32AI systems smarter than humans that

49:34don't have our best interest at heart,

49:35do you think that would go well? And

49:37most people, that's all I have to say.

49:39Like, you know, just think about

49:40yourself. If someone came to me and like

49:41actually tried to argue, you know, chat

49:43GPT didn't change anything. I would just

49:45laugh. You know, I'm just like, okay,

49:46sure, buddy. Like, it's not really an

49:48argument. You know what I mean? I'm

49:49like, okay, sure, whatever. Like, this

49:50person's just trolling.

49:51>> Do you regret something? Do you regret

49:53helping open-source AI before?

49:55>> Oh, I I regret many things. Yeah, I've

49:57made so many mistakes in my life.

49:59Absolutely. Um, I don't think that my I

50:03like to think that my in contributions

50:05to open source AI didn't make that big

50:06of a difference. I think, you know, I

50:08was very reasonable at the time with the

50:10information I had and the beliefs I had.

50:12Was it still the wrong thing to do?

50:13Yeah, I still think it was probably the

50:15I think all probably all things equal. I

50:17should have understood that this was not

50:18a technical problem, but a political

50:20problem. Um I think it was an okay

50:23mistake to make. People make mistakes. I

50:24think it was a very reasonable mistake

50:26to make. I I think I have a lot of

50:29regrets of you know that I didn't

50:31succeed at my research at my company.

50:33You know there's a lot of things I would

50:34do differently. You know like how I

50:36would run a team, how I would do

50:37research. There's yeah there are many

50:40many things and you know that's normal.

50:42You know we all we all live and we

50:44learn. You know we improve and we just

50:45keep trying. If you could find answer

50:47for just only one question about you

50:49know reality everything whatever you

50:51want what it would be

50:53>> I'm like this was very gameable so I

50:55won't spend too much time gaming it but

50:56it would probably something like how do

50:58we make a truly just government how how

51:00can we design a government a system of

51:02politics that is truly just and truly

51:04good that is that we endorse that is

51:07democratic that is stable

51:08>> isn't utopia

51:10>> I think this is a very important part of

51:11it I don't think it's the only part of

51:12it I think we need some other things to

51:13get to utopia but I think Yes, I think

51:16there's a thing where it's very like in

51:17the past I think one of the biggest

51:18mistakes I made is I focus a lot on

51:20technical problems of utopia and like

51:22good worlds. I focused on how do we

51:24create diseases, how do we create lots

51:25of energy, how do we have fun, how do we

51:27make great video games, like so I think

51:29all these things are relevant to a good

51:30world but really I think the thing that

51:33we're most bottlenecked by is

51:34governance. Imagine for a moment that

51:37you know the west and the US government

51:39was staffed by the most competent,

51:42hardworking, serious, expert, moral,

51:45like good-hearted people have like good

51:47philosophy, good morals are like

51:49genuinely like nice, like good people

51:51who are trying to do the right thing.

51:52>> How would we feel about the world? I

51:55would feel dramatically great about the

51:57world. Like I would feel better about

51:58AI. I would feel better about social

52:00media. I'd feel better about science. I

52:02would feel better about so so many

52:03things. And and I don't think this is

52:06just because I'm not trying here to say

52:07like, oh, politicians are bad people,

52:09blah, blah, blah. So, I I've talked to

52:10many politicians here in the US and the

52:12one thing I really say is that

52:14overwhelmingly when I talk to

52:15politicians, I come away with they're

52:17like, it's so easy to hate politicians,

52:19right? You know, they make bad decisions

52:20and they say stupid. It's so easy to

52:22hate them. But my takeaway always when

52:24talking to politicians basically is that

52:26they're mostly not all but mostly normal

52:30people trying to do the right thing

52:32>> and they're just so overwhelmed. They're

52:34so overwhelmed. The system is so rigged.

52:37They are, you know, being yelled at by

52:39their party, being yelled at by the

52:40other party, being yelled at by not

52:42about Trump.

52:42>> It's not It's not about Trump. It's not

52:44about Democrat. It's not about

52:45Republican. Everyone is in a bad spot. I

52:48recently talked to an ex-member of

52:50Congress and he told me when he came

52:52into Congress, they gave him a handbook

52:55um of like how to run his office and

52:56like what he should expect and it had an

52:59example of like how he should expect his

53:01week to be laid out. And the amount of

53:03time that that plan had for him to read

53:07or learn new things was 25 minutes.

53:11Look, if I only had 25 minutes to read

53:13per week, I would also make mistakes. I

53:15would [ __ ] up all the time if I didn't

53:17have enough sleep, if I was stressed all

53:18the time, you know, if I had to do

53:20fundraising, you know, all day, all

53:21night. Yeah, I would also make some bad

53:23choices. I would also make mistakes. So,

53:26it really is the system. I'm not saying,

53:28you know, Trump bad, Obama bad, you

53:30know, like whatever, right? That's not

53:32what I'm saying. What I'm saying is, um,

53:35we need a better system also for our

53:37elected officials and so on to be able

53:39to lead well and to be and to, you know,

53:42be in a good circumstance. And then

53:44addition to that, we also need to have

53:45good voting systems. We need to get

53:46money out of politics. Very very

53:48important here in the US. There's a lot

53:50of money in politics. And I think this

53:51is very bad. Like we need to get a lot

53:53of we need to get this money out of

53:54politics. Like there's many many things

53:56like this.

53:56>> You help open the door to powerful AI

54:02and now you want to close it. What has

54:04changed?

54:05>> So I mean the main thing has changed is

54:07more time has passed and AI has gotten

54:09stronger. Again, I think uranium ore is

54:11fine. If you want to or you own a chunk

54:12of uranium ore at home, I have no

54:14problem with this. If you try to own

54:16highlyenriched nuclear, you know,

54:18nuclear material at home, I do have a

54:19problem with this. Um, it's a

54:21fundamental risk trade-off. Um, I used

54:24to work in open source. I used to work I

54:26built some of the first open source

54:27large language models in the world with

54:29my team at LutherAI. I've done a lot of

54:32work. But fundamentally, the thing I

54:33always cared about is building a good

54:35future for people, building a good

54:36future for humans. you know with AI I

54:38think AI is a very powerful and

54:39wonderful technology but to do that we

54:42have to actually control it and

54:43understand it and use it for good and

54:46one thing I learned unfortunately the

54:48hard way um from my expense uh from my

54:51time in the open source community is

54:52that they didn't give a [ __ ] about

54:54people being safe or being unharmed they

54:56didn't give a [ __ ] about it what they

54:58cared about is things being free that's

55:00the thing they cared about they cared

55:01that they could have the power that you

55:03know there would be free software I

55:04remember had a very noteworthy

55:06conversation

55:07uh once with another person involved in

55:09the open source movement and he called

55:12me out in the chat room on the discord

55:13and he was like Connor you say you want

55:17you know that you know developers that

55:19build AI systems that harm people to be

55:21held accountable but you have built AI

55:23systems do you want to be held

55:25accountable and my answer was yeah yeah

55:28I would love to be held accountable for

55:29the harms I cause to other people and he

55:31was so shocked he was so shocked he he

55:35couldn't understand like it it was crazy

55:38like in his mind like it was so crazy

55:41but for me it's obvious yes if I harm

55:43people I do want to be held accountable

55:44for that the same way I want other

55:45people to be held accountable when they

55:47cause harm

55:48>> but you know was there a moment when you

55:51thought we are going too fast

55:54>> yes

55:55>> what was that moment

55:56>> that moment was the summer of 2019 I

55:59remember it very vividly um before this

56:02I I was already very concerned with the

56:04safety of AI you know the The way I got

56:06into AI originally is when I was a

56:07teenager. When I was a teenager, I was

56:09thinking, you know, how could I improve

56:10the world? How could I help people? How

56:12could I, you know, fix as many problems

56:14as possible. And I thought, well, if I,

56:16you know, could like automate

56:18intelligence, if I automate science,

56:20then I could cure all the diseases and,

56:22you know, solve all the problems. So,

56:24I'll just go do that. But what I quickly

56:26realized was if I could build something

56:28so powerful that it could cure all

56:30diseases, whatever that even means,

56:32that's a very dangerous thing. It's a

56:34very powerful thing. How would you even

56:36control something like that? And who

56:37should control something like that? So

56:39then I realized I had to work on the

56:41problem of control and safety. So I was

56:43already interested at this time, but at

56:44this time I still thought we had quite a

56:45lot of time. I still thought it was

56:47decades until we would see the things I

56:49was really worried about, which was AGI

56:51and super intelligence. Um, at the time

56:55AI systems, you know, they could do some

56:56really impressive things. You play some

56:58Atari games, you know, beat the Go world

57:00champion, you know, image recognition.

57:02it could do some useful things, but it

57:04was very brittle. Um, it's quite

57:06different from today where if you wanted

57:08to get an AI system to do a new thing,

57:10you kind of had to, you know, tweak all

57:12the the architecture, you had to like

57:15collect new data, you have to have a

57:17bunch of, you know, PhD students stay up

57:18all night, you know, trying to get it to

57:20work, etc., etc. Like, it was very

57:22brutal. Once you had an AI that could do

57:23one thing, it that doesn't mean it could

57:25do other things, too. You know, if it

57:26can recognize images, that doesn't mean

57:28it can then also play Atari and so on.

57:31And then GPT2 happened.

57:34In 2019, uh, OpenAI released a AI system

57:39called GPT2.

57:40And by today's standard, it's kind of,

57:42you know, almost pathetic. Um, it was a

57:45system, what's called a language model,

57:46a system that could generate text. And,

57:49you know, by today's standard, it's, you

57:51know, not that impressive. You know, it

57:53could barely string together a couple

57:54sentences. But for me, it was just, oh

57:58[ __ ] this is it. This is the thing I

58:01was worried about. It's happening way

58:03sooner than I thought. And the thing I

58:06saw in GPT2 was that it was a

58:08generalpurpose pattern learner. This was

58:11the thing we were missing. The before

58:13this we didn't have general pattern

58:15learners. We had pattern learners, but

58:16they were quite specific. You know, you

58:17could train it for one specific task or

58:20one specific thing. But GPT was

58:22different. As you gave it more data and

58:26more computing power, it learned more

58:28and more complex and abstract patterns

58:30all by itself. It learned you not just

58:34to spell or to use punctuation abstract

58:37thinking.

58:37>> It learned abstract thinking. It

58:38learned, you know, if you ask a language

58:40model, you know, give me a poem about

58:42why I love my dog.

58:44>> This might seem simple in some sense,

58:45but in a sense, actually, you have to

58:46know a lot of things to do that. You

58:48have to know about dogs. You have to

58:50know about love, about why humans love

58:52dogs. You have to know what a poem is.

58:53You have to know how to combine these

58:55things. There's actually a lot of facts

58:57about reality. You need to know and you

58:59need to know how to combine these facts

59:00and these patterns of reality to be able

59:02to do this like quite simple seeming

59:04task. And so the crazy thing is is no

59:07one sat GPT down and it you know said

59:09okay here's what a poem is. Here's what

59:11a dog is. Here's what a you know here's

59:13how you combine these things. Nothing of

59:15the sort. We just gave it a text and it

59:17figured it out by itself. And so for me

59:20it was pretty obvious that yeah this was

59:22the moment where um you know it's not

59:24quite ready yet. Of course it's going to

59:26take some years to improve the

59:28technology but now there was no

59:30fundamental barrier from my perspective

59:33towards you know AGI artificial general

59:36intelligence

59:37>> abstract thinking free reasoning that's

59:39what makes us humans right?

59:40>> It's one of the things that makes us

59:42human. Yes. Um it's definitely one of

59:44the things that separates us a lot from

59:46other animals. I think there's other

59:48aspects as well, but yeah, the general

59:49purposeness where you can take a human

59:51in many many different circumstances and

59:53they'll figure it out. You know what I

59:54mean? Like this is a thing that's like

59:56very special. We don't have in our

59:57genome, we don't have pre-coded all the

59:59things we do. We figure new things out.

1:00:02We develop new tools. We come up with

1:00:04new patterns. We find new patterns.

1:00:06>> Knowing what you know, would you support

1:00:11the P on model stronger than GDP4?

1:00:15Well, a pause doesn't really work if

1:00:17it's already out. Um, so in terms of

1:00:20would I would I support the pause of

1:00:23creation of more powerful than currently

1:00:25existent general purpose agentic

1:00:27systems? Yes, I think this is actually

1:00:29necessary because I think we're very

1:00:30close to super intelligence. In question

1:00:32of should we roll back to say GBT4 or

1:00:35GBT 5. I think this depends on a few

1:00:37factors such as like how the hell would

1:00:39you do that and like what would that

1:00:40mean? Um, if it was like, you know, you

1:00:43have a magic wand and everyone's like

1:00:45okay with it and we all smile, then

1:00:47sure. If it's like we have to have like

1:00:49giga dystopia, you know, like police,

1:00:51you know, oppression to have that, maybe

1:00:54not worth it. I think it really depends.

1:00:56Um, the way I like to the way I

1:00:59sometimes say it is like if we were

1:01:01three steps wiser than we currently are,

1:01:03we would have never done GPT3 in the

1:01:05first place. If we were two steps wiser

1:01:08than we currently were, then we would

1:01:10have done GPT3 in our chat GPT, we would

1:01:13have seen this, you know, suddenly get

1:01:14millions of users and do all these crazy

1:01:16things and not predict and we've been

1:01:17like, "Oh shit." And we would have

1:01:18stopped and like rolled it all back. If

1:01:20we were one step wiser than we currently

1:01:21are, we would pause it right now.

1:01:23>> Was GDP4 the moment when AI safety lost

1:01:26the race?

1:01:27>> I don't think so. I think we were like I

1:01:30don't think there's like one moment in

1:01:31that regard. I don't think that because

1:01:33AI safety is not really h even racing.

1:01:35never has like the the progress and the

1:01:39amount of effort going into the

1:01:41questions of AI safety is like like is

1:01:44pathetic. It's so so so small and has

1:01:46always been like there's never been a

1:01:47time where like we understood AIs or we

1:01:50were in control of AIs. This there never

1:01:52was such a point.

1:01:53>> Don't you think that identic AI is like

1:01:55you know AI 2.0?

1:01:58>> I I think it's AI 5.0 probably at this

1:02:01point like we've had many generations of

1:02:03AI you know from goi to narrow neural

1:02:07networks to more broad neural networks

1:02:08to LLMs to agentics so I say probably

1:02:10like AI 5.0 The difference between

1:02:13agentic systems and say chat bots or

1:02:15like generative AI which I would say is

1:02:16last generation is quite large. There

1:02:18are like many many types of tasks that

1:02:21require interaction with the world. Um

1:02:24most not all not most but like maybe

1:02:26most um economically um valuable tasks

1:02:30involve interaction loops. They're not

1:02:32just one shot. They're like you try

1:02:35something, you see what happens, you try

1:02:36again, you do something else, you wait

1:02:38for a response, etc. And this is

1:02:40fundamentally agentic. Uh with previous

1:02:42systems, this was very hard or didn't

1:02:43work at all. Like a lot of the earlier

1:02:45like GP4 and so on just like couldn't do

1:02:48this. Like it doesn't work. Like you

1:02:49could do them like as like a chat. You

1:02:51could kind of use them, but they

1:02:53couldn't like run their own experiments.

1:02:54They couldn't write their own tools,

1:02:56etc. And that is now changing. I don't

1:02:58think it's a fundamental shift in

1:03:00architecture. Well, we could argue about

1:03:02that actually.

1:03:03>> Human replacing.

1:03:04>> I think it's very human replacing. Yes.

1:03:06If you could implement just only one

1:03:07amendment, what it would be?

1:03:09>> I think the real thing that it's

1:03:10important to understand about regulation

1:03:12and laws is that they don't really

1:03:14matter. And what I mean by this is is

1:03:16that the thing that really matters is

1:03:18that people enforce laws. This is the

1:03:20thing that matters.

1:03:20>> They you mean decision makers or they

1:03:22you mean

1:03:23>> everyone everyone from decision makers

1:03:26to cops.

1:03:27>> Okay.

1:03:28>> Like there has to be a system courts,

1:03:30you know, everything where

1:03:31>> administration the brother says,

1:03:33>> administration, you know, the white

1:03:34house, etc. like people have to want to

1:03:36enforce a law. So what I mean by this is

1:03:38is that if I could wave a magic wand and

1:03:41I could pass the you know ban super

1:03:42intelligence forever bill, it wouldn't

1:03:44actually matter because obviously

1:03:46companies will find some loophole that

1:03:48like I didn't think about right within

1:03:50two weeks and they'll violate it. And

1:03:52then if people don't believe in it, if

1:03:53the courts don't care, you know, they

1:03:56won't investigate. They won't care, you

1:03:57know, the police won't investigate and

1:03:59everyone be just like and the White

1:04:00House won't investigate and every be

1:04:01like eh whatever. So the real thing that

1:04:04we need is not just one specific law. We

1:04:07need that the government that the courts

1:04:09and you know the law enforcement

1:04:10apparatus and the national security

1:04:12apparatus understand what is the risk of

1:04:14super intelligence and want it to not

1:04:16happen. This is the real thing that has

1:04:17to happen

1:04:18>> and to say stop.

1:04:19>> Yes. And so when it comes to the actual

1:04:21legislation I think

1:04:22>> in some cases

1:04:23>> yes I think when it comes to that the

1:04:25first most important thing is to

1:04:27prohibit the development of super

1:04:28intelligence the same way that we for

1:04:30example prohibit the development of

1:04:31nuclear weapons.

1:04:32>> Okay. So my last question, should we

1:04:34stop AI?

1:04:36>> AI? No, of course not. We should make

1:04:38Alpha Fold 3. I love Alpha Fold. We

1:04:40should make more of it, you know. Or you

1:04:42want to make a, you know, a little toy

1:04:43or something. I mean, great. You want to

1:04:45use it for, you know, statistics? Great.

1:04:47>> Should we stop agentic AI?

1:04:49>> Probably. Yeah,

1:04:51we're probably if not then very soon.

1:04:53Um, should we stop super intelligence?

1:04:55Absolutely. Agentic AI

1:04:58edge case. Um, so a very very important

1:05:02thing to understand. So one of the top

1:05:04tactics that these companies do to try

1:05:06to disempower you, the general public

1:05:08and policy makers is to try to make you

1:05:10think that you're too stupid to

1:05:12understand it. They're like, "Let the

1:05:13experts do it. You don't understand.

1:05:15>> You're not a researcher.

1:05:16>> You're not a researcher, etc." Well,

1:05:18guess what? I am a researcher. I have

1:05:20done all this research and let me tell

1:05:22you, they're lying to you. You can

1:05:24understand this and you can have agency

1:05:26and you deserve to have agency

1:05:27>> and that ideology, right? Yes. And the

1:05:30ideology of course like you as a as a

1:05:32citizen have the right to say wait I

1:05:36don't want smarter than human things

1:05:37running around that we can't control

1:05:39like you know maybe you're not technical

1:05:41experts well guess what you know the

1:05:42statement I talked about earlier that

1:05:44the mitigation of risk of extinction

1:05:45should priority it was signed by Sam

1:05:47Olman Deisabis you know um Dar Amade and

1:05:51for that matter Joffrey Hinton the Nobel

1:05:53Prize winning inventor of modern AI

1:05:56right yeah neural networks like are

1:05:58these

1:05:59not experts like so this is crazy. So

1:06:02there's a thing where people try to make

1:06:05this seem like a complex thing. It's not

1:06:06complex. It's very simple. The only

1:06:08thing you need to understand is they are

1:06:10building very powerful agentic things

1:06:12and they don't know how to control it.

1:06:14That's all you need to know. That's all

1:06:15you need to understand. And this is the

1:06:16thing that anyone can understand. And so

1:06:18that's why action is important. And what

1:06:19I would recommend to everyone is to go

1:06:21to controlai.org

1:06:23right now and contact your lawmakers. If

1:06:25you are concerned about these issues,

1:06:27you have a right to speak up and you

1:06:28should speak up to your lawmakers and

1:06:30demand they take action. This might

1:06:31sound, you know, like cliched, but it

1:06:34really matters. When I talk to

1:06:35politicians, I can tell the diff like

1:06:38they often tell me if their constituents

1:06:40have reached out to them or not. This

1:06:41makes it much easier for me to explain

1:06:43these issues to them. Like recently, we

1:06:44had a conversation with an office of a

1:06:47congress member and the the person we're

1:06:49talking to was like, "Oh, wow. I've got

1:06:51a lot of letters about super

1:06:52intelligence lately. I'm so glad we can

1:06:53finally talk." Mhm.

1:06:54>> And that was through people, you know,

1:06:56general people expressing these issues

1:06:58for the policy makers. As I said,

1:07:00>> increase the public awareness,

1:07:01>> increase the awareness, talk to your

1:07:02friends, demand answers like who the

1:07:05hell do these companies think they are?

1:07:06You know, we should not let them get

1:07:07away with this. And the way to do this

1:07:09is to raise the awareness to organize to

1:07:11demand these changes. It's require this

1:07:14is how democracy works.

1:07:16>> Thank you very much for your time.

1:07:18>> Thank you. Comb Group has become a

1:07:20partner of Bison Fellowship, a program

1:07:22for Central Europe's brightest young

1:07:23scientific talent.