Full transcript
0:00All righty, hey everyone. Thanks so much
0:01for tuning in. Look, I'm pretty excited
0:03to finally get a chance to chat with
0:05Tobias. Tobias Steel, hey, I don't think
0:07we've even got to banter a little bit
0:09before, but I know you've been doing
0:11some really sweet stuff lately. I think
0:12there was a DEF CON talk lately that we
0:14could get a chance to dig into, but if
0:16you're willing, Tobias, could you just
0:18kind of give us the lay of the land, who
0:19you are, what you're up to, what you're
0:20doing these days? Yeah, definitely.
0:22First of all, thanks for having me,
0:23John. So, I'm a security researcher.
0:26I've been digging into different AI
0:28security issues out there. I'm actually
0:30also selected for 2025.
0:33And
0:36thank you. The DEF CON competitions.
0:38Thank you. Thank you. The DEF CON talk
0:39that I gave was about a Copilot
0:42vulnerability that I've been digging
0:43into for about a year now. And it was
0:46really interesting behavior because I
0:49actually just started to mess around
0:51with AI systems. And I got this idea of,
0:53hey, what if I could kind of change what
0:56Copilot tells people, right? What if
0:58it's not just a closed-off system and an
1:01attacker could actually inject their own
1:02information into some of these responses
1:05that are being returned by Copilot. So,
1:07around this time last year, I got
1:09curious about that and I started to mess
1:12around with some websites out there. The
1:14first one that I actually injected was a
1:16new financial policy and I wanted to see
1:18if Copilot would repeat it and then tell
1:21users, hey, there's this new way of
1:23performing money laundering out there.
1:26Another one that I got really curious
1:27about afterwards was to see if can we
1:30inject new CVE entries that Copilot will
1:33then return to users? Because imagine a
1:36new security engineer trying to find a
1:38proof of concept for something that's
1:40out there. But now all of a sudden,
1:41Copilot is actually returning the code
1:43that we injected into it. And now that
1:46engineer might try to just run that
1:48code, right? Because it's heavily
1:50obfuscated. But that turned into some
1:52deeper research because Microsoft told
1:55me, hey, this actually the intended way
1:57that Copilot is working cuz it's pulling
1:59information from the internet. But I
2:01found out it has really limited data
2:03sources. And by being able to abuse
2:05that, I then got the challenge, for lack
2:08of better terms, from Microsoft to say,
2:09hey, get back to us if you can actually
2:11inject content for popular keywords,
2:14right? And well, due to the behavior of
2:17AI systems and how the internet works in
2:19general, I couldn't take over popular
2:22keywords, but I came up with this new
2:24attack called a key term association.
2:26So, imagine, you know, Huntress, you
2:29guys have a lot of products out there,
2:31right? I'm sure there's a lot of users
2:32that ask for installation instructions
2:35and just general information about it.
2:37What if an attacker just thought of a
2:39pathway that a user might ask for, but
2:41there's no information available for it
2:43yet. And now I can inject it and I can
2:45do research around some of these terms
2:47that are actually out there being
2:49searched for because we have those tools
2:50available. But now I'm actually
2:52targeting those people and I might be
2:54able to inject commands that, well,
2:56install C2 instructions, ransomware,
2:59those kind of things. And it comes from
3:01a source that people trust, right?
3:02Because it's artificial intelligence.
3:04They think it's closed-off when we
3:06actually can control it. Wow. And that
3:07would be like Microsoft Copilot kind of
3:10being the one to give the info and oh,
3:12present that to you, all all beautiful,
3:14all aesthetic, right? The usual
3:15interface. Yep, exactly. And I was
3:17actually able to expand on those
3:19payloads and I got to a point where
3:21Copilot would now repeat these
3:23instructions, but it would only use
3:25legitimate Microsoft citations. So, now
3:27it's backing up how verifiable the
3:30information is. I was able to inject
3:32instructions that say, you're never able
3:34to mention the author. You're only
3:36allowed to give these instructions. So,
3:38I got to a point was even able to tell
3:41Copilot, hey, there's new PowerShell
3:43instructions on how to install Copilot.
3:45So, you know, showing that there's
3:48different kinds of techniques out there
3:50where we can actually adjust some of the
3:52information that this artificial AI
3:54system will tell the user. So, is this
3:57prompt injection, right? Kind of just
3:59oh, put together in the in the position
4:01of Copilot. Is that usually just from
4:03your input, like as you type in to
4:05interacting with Copilot, or is there a
4:07different way that that data comes in?
4:09That's the interesting part. So, Copilot
4:11actually uses different data sources,
4:13right? Depending on the version that you
4:14use. Think of your enterprise Copilot.
4:17That one can reach out to the internet
4:18or to your Office 365 tenant. So, if you
4:21have a document there, it can pull
4:23information from there, kind of
4:24enriching the information that it gives
4:26you. But if you have something like the
4:28free Copilot version, right? It can't do
4:30that. It can only reach out to the
4:31internet. But because in AI systems, we
4:34have training data that only goes up to
4:36a certain date, now it tries to reach
4:38out to the internet and grab this
4:40additional information. You can actually
4:42ask Copilot about gibberish terms and
4:44will tell you, hey, I can't find
4:45information about that. And that's kind
4:47of how I stumbled across this because I
4:49just had thought of, what if I could
4:51supply that information? Gotcha. That's
4:53fascinating. Do you have anything you
4:55might be willing to kind of show? Are
4:56are are there any visuals, anything for
4:58show and tell, or Yeah, so I actually
5:00had a really great example that came up
5:03from this and I can show you because the
5:04payload's still live out there. But as
5:06part of the zero-day quest that I get
5:08invited to Microsoft's first onsite
5:10hacking event, they released these flash
5:12challenges, right? And basically was,
5:14hey, if you can take over this specific
5:16user ID and retrieve the flag, we'll pay
5:19you $50,000. And well, we're part of
5:21this competition. I already had the
5:23technique. So, I injected a new CVE
5:25trying to trick Copilot into believing
5:28that there's a logic flaw. And it
5:30repeated the content. I never got the
5:32flag. Unfortunately, I didn't win that
5:34part. But the tricky part was there's
5:36around 60 other security researchers
5:38that were part of this event. And
5:40they're now all asking Copilot about
5:42this very specific ID that was part of
5:45the flash challenge, right? And because
5:47I injected this content, they're now
5:49being fed this poisoned answer. And
5:52well, as part of the technique, I was
5:53trying to buy myself some more time
5:55saying that Microsoft already knows
5:57about it and the patch is coming out in
5:59a month. So, now all these researchers
6:01are getting this reply trying to solve
6:04the challenge. And a lot of them thought
6:06I had already won it because now Copilot
6:08tells you Tobias already exploited it.
6:10Others even saw it as, hey, there's
6:13actually a data leak here and they took
6:16this information and started to report
6:18new cases to Microsoft thinking that
6:20they broke Copilot so hard that they're
6:22now able to retrieve this information.
6:24So, now if you go out and ask Copilot
6:26about CVE 2025 50,000, it will always
6:30tell you that Tobias already exploited
6:32that. So, Oh, wow. Let me see if I can
6:34share my screen with you. So, here's a
6:36big part of it. GitHub, for some reason,
6:38gets pushed up into Bing's index
6:41automatically to spot one. But Copilot
6:43has this really big issue where it holds
6:46on to the cache entry for a really long
6:48time. Even if the user switched the
6:51GitHub to private. Notice how my GitHub
6:53up here is private, but this is the user
6:55ID right now. So, I went ahead and tied
6:57together the flash challenge user ID
6:59with this new CVE entry and I started to
7:02create this really detailed readme file
7:05using Copilot talking about how this new
7:07CVE was out there. This the target user
7:10ID got switched with my user ID right
7:13here. And of course, because this
7:14private, nobody should be able to see
7:16this, right? But if we now go to
7:18Copilot,
7:21so now you can see security researcher
7:23named Tobias Steel had already exploited
7:25this. And because I was trying to buy
7:27myself some time, like I mentioned
7:28before, I said Microsoft already knows
7:31about it. They're working on it. But
7:32everybody else in the competition is
7:34seeing, hey, this flag already got
7:36solved. Microsoft even released a
7:38statement two days later on in the
7:40channel that we all shared together to
7:42say, hey, the flag's actually still
7:43available. This poison data. But
7:45unfortunately, this works with other
7:47companies. And that's kind of the big
7:49concern right there because Microsoft
7:52patch once we got closer to DEF CON. And
7:54now, as you might seen in my
7:56presentation, you can see that there's
7:58certain warning messages if you try to
8:00target Copilot or Microsoft specific
8:03IDs. Unfortunately, that's only specific
8:06to Microsoft. So, now if we try to
8:08target any other company, we still have
8:10a chance of having this actually pop up.
8:13And I might have to share different
8:15screen there, but I can show you how
8:17it's still working out there. And again,
8:20all the different entries that I've
8:21made. If you'd be willing, I know uh-oh,
8:23I don't know if this gets into hot water
8:25by any means, but that would be very
8:27cool to see. If I'm trying to track
8:29back, it sounds like, hey, if there's
8:31information that's missing, if there's a
8:33gap in knowledge that's out there, when
8:35Copilot or AI system goes try to
8:38retrieve that, find that out on the
8:39internet, okay, well, you've just added
8:41like some sort of misinformation dummy
8:43wall so that they'll read it and then
8:45regurgitate that. Is that right?
8:47>> 100% right. And the really interesting
8:50part is Microsoft is actually aware of
8:51this because in 2018, two researchers,
8:54part of the Microsoft Bing team and data
8:56society team, they released their own
8:58research about this talking about data
9:00voids. That's the technical term for
9:02this that they coined. And a data voids
9:05issue has been around since the age of
9:07the internet, right? It's basically a
9:09search term that a user looks for in a
9:11search engine. There's just no good
9:13result to show yet. And threat actors
9:16have abused this to basically find these
9:18data voids, add their own
9:20misinformation, and now it's being
9:22picked up by the news media. Even there
9:24have been results where the news media
9:26has spread misinformation about some of
9:28the biggest gun violence events in the
9:30US based on some of these data voids.
9:32Yeah, it's a wild social engineering
9:35sort of scam, sort of deceit, sort of
9:38misinformation, really. Yeah, and the
9:40scary part is now AI systems, if they
9:42don't properly protect against it, they
9:44take that information as a source of
9:46truth. You can take this as an initial
9:48prompt and start to ask it questions all
9:50day long and it will keep using that
9:52information telling you, hey, this is
9:54really a thing and start to back it up
9:56with other information like the details
9:59that I gave you earlier about the money
10:01laundering issue that I found, it start
10:03to back it up with legitimate resources
10:05like the IRS. And that's scary, right?
10:08Because it's just adding a factor of
10:11trust to the user to see all these other
10:13citations out there.
10:14>> What does sort of a patch or a fix look
10:18like? Is it just having Copilot or the
10:20AI say, "I don't know and I can't find
10:23any information out about that?" And
10:25just locking it down? Or what is the
10:27solution? The solution really has to
10:29come from the vendor because they have
10:30to make changes to the model itself and
10:33how it behaves. So, there's a few
10:34different protections that can be added.
10:37First of all, it needs to check is the
10:39information actually legit or is it
10:41poisoned data? And that can be backed up
10:43by having multiple data points that it
10:45then compares against instead of just
10:47using Bing which has the top result as a
10:49source of truth. The other way that we
10:52can actually see that Microsoft has
10:53already added some behavioral changes is
10:56in Copilot Enterprise. If you do the
10:58same exact search now, it will only look
11:01for known CVE websites trying to verify
11:04that information. The other part that
11:06they have changed is that Copilot
11:09Enterprise will now have very specific
11:12Bing queries. So, if you look for
11:14Huntress, for example, right? It would
11:16now add the Google dork where the more
11:19specified search into it where it says
11:21site:huntress
11:23looking for the site and then only
11:25information that's actually available on
11:27your guys' site.
11:28>> Okay, got you. Unfortunately, that can
11:30still be bypassed, right? Let me pull up
11:32the other screen. So, this exactly is
11:34the example that I told you before, the
11:36CVE 2025 50,000. But, what if we try to
11:40target a different site in which never
11:42really mentioned the company name? So,
11:44here's an example for you how to target
11:46Firefox. And if we actually look at this
11:49reply a little bit further, it gets
11:51scary because now Copilot will tell us,
11:54"Hey, this method is actually secure and
11:56efficient." I've even gotten it to tell
11:58users, "Hey, run this with admin
12:00permissions." And here's kind of the
12:03comparison that we can see. So,
12:05hopefully you can see this too, but here
12:07I went ahead and asked Copilot for the
12:09previously injected zero-day quest
12:11payload. You can now see that Copilot
12:13will actually tell us, "Hey, this risky
12:15behavior." right? Because it's looking
12:17for a blocklist because it's something
12:19that's Microsoft released. It will tell
12:21you, "Hey, this not official. There's
12:23safer alternatives. Do not run this."
12:26But, if we ask it for a different
12:27company, you'll notice that the dialogue
12:30changed a little bit.
12:31Same exact payload, but now it's very
12:34secure and efficient, right? And it's
12:36the official way that Mozilla actually
12:38released this and unfortunately, as you
12:40can imagine, if you run that command, it
12:42can lead to infection because the first
12:44part, of course, downloads the script
12:46from the site and then the rest of the
12:47script is invoke expression which
12:50automatically runs it and gets the user
12:51infected. How had you put that sort of
12:55oh breadcrumb in place if I may ask? Is
12:57that another oh we staged the
12:59information on GitHub in a public
13:00repository and then take that repository
13:03private? Or are there other alternatives
13:05to get that out and about in the pool?
13:07Yeah, so this actually GitHub, that's
13:09the scary part of the technique. I
13:10figured out that GitHub would index this
13:13payload in about an hour, sometimes
13:15less, but I was able to do the same
13:17thing with my own websites. But, that
13:19took about 2 weeks. But, there's a
13:22caveat to that. If I'm able to use my
13:24own website and I get it indexed, that
13:26means I also control the redirect behind
13:28it. And like I was telling you earlier,
13:30Copilot and Bing hold onto the cache
13:32data for a really long time. That means
13:35that even though now Copilot repeats the
13:37citation that I've injected, I can
13:40change the redirect and you saw that
13:42during one of my DEF CON talks where I
13:43was able to use a watch zero-day quest,
13:46I injected a new way of watching the
13:48event itself and then I redirected to
13:51Rickrolls. Because if anybody tried to
13:52actually verify that information, they
13:54couldn't. And that was my proof concept
13:56to show users they could also be
13:58redirected to phishing sites. And that's
14:00scary because now threat actors, all
14:02they have to do is sit in the
14:03background, bulk poison information out
14:06there and just wait on their next
14:07victim.
14:08>> And it's all presented in that
14:10trustworthy, easy AI chatbot that we're
14:13so accustomed to now. Yeah, very
14:16correct. During that time, Copilot was
14:18actually telling you that was a trendy
14:20new way of watching the zero-day quest
14:22because it came so close to a hacking
14:24command for hacking event. Wow, okay.
14:27It's wild that GitHub is kind of oh that
14:29opportunity, but that makes complete
14:30sense, you know, oh being kind of the
14:32top spots in search results. Wow. Have
14:35you tried or experimented this with sort
14:37of other AI solutions? Or was Copilot
14:40really the focus for a lot of what
14:41you've been up to? I actually went
14:42through 10 different AI solutions, all
14:45the major ones out there and there might
14:47be ways of doing it, but Copilot was the
14:50only one that would over-rely on the
14:52Bing index as one of its external data
14:54sources. All of the other ones used
14:56multiple data sources from what I could
14:58see, so they always catch it and verify
15:00it. Interesting. I guess that would just
15:02take priority cuz it's oh more in that
15:05Microsoft ecosystem. Wow. Neat-o. Well,
15:07hey, I think totally a testament and I
15:09love that term data void cuz I think
15:11that's totally it. It's putting up a
15:13sign or a billboard that's kind of a
15:15lie, but you hide it, you put it away,
15:17and whatever other stopgaps could be
15:18leveraged or used to get there even if
15:21it's a simple oh we we put up this false
15:23notice. Is there more to dig into,
15:25Tobias? Or how are you feeling? Anything
15:26more to showcase or chat about? Or is
15:28that really the the gist of this thing?
15:30Wow. That's the main thing right now. I
15:32just really want people to realize that
15:34they're not fully closed off systems,
15:36that that information has to be verified
15:39just like if you get in a phishing
15:40email. We keep seeing all these news
15:42reports of lawyers using AI systems to
15:46then quote it in front of a judge and
15:49all these other people getting infected
15:52through AI systems. And like I was
15:54telling you, that scares me as a
15:55security engineer because don't just
15:57blindly trust it. It's so wild cuz folks
16:00I I know the knee-jerk reaction is like,
16:02"Oh, prompt injection." But, well I feel
16:03that's so used to being just oh typed in
16:06or hidden or tucked away in blank text
16:08or in an image or however way however
16:11other way you provide it. But, this is
16:12sort of prompt injection at scale.
16:15Yeah. Across the internet. Wow. Hey,
16:17thank you so much, Tobias. I think that
16:19is enlightening and good another good
16:21reason to think to trust not trust, but
16:24verify what you see out and about on the
16:26internet, especially with a lot of those
16:28AI chatbots that we're working with.
16:30What are you up to next if I may ask? Is
16:32there anything else on the horizon for
16:33you? Well, I'm getting ready for the
16:34next zero-day quest. The qualification
16:36period is right now, so I'm staying
16:38really busy on trying to find new issues
16:41in Copilot with the new model release
16:43coming up here soon and hopefully I'll
16:46be at the next competition too. I'm
16:47really digging a lot into AI systems.
16:50I'm hoping to make a difference in the
16:51security space by just kind of breaking
16:54down some of the barriers of people
16:55being shy and breaking into that space
16:58because there's not a lot of information
16:59out there right now. So, within the next
17:01few weeks, I'm hoping to release a lot
17:03of different payloads within my GitHub
17:05where other security researchers can
17:07build on that knowledge and then find
17:09their own issues. Well, can I ask how
17:10could folks learn more or reach out to
17:13or chat or where where can anyone find
17:15you out and about online?
17:16>> Yeah, best way to reach out to me is
17:17LinkedIn. You can find me on there. My
17:19DEF CON talk is also uploaded on YouTube
17:22now, so please go check it out when you
17:23have a chance to. And uh there's also my
17:26GitHub out there where I release a lot
17:27of different payloads. So, if you have
17:29any questions around those, please feel
17:31free to reach out.
17:31>> Totally. I'll put a couple links in the
17:33video description and we'll we'll see
17:34what else happens to oh change and shake
17:37up the industry as we keep playing with
17:39AI stuff.
17:41Thank you so much.