Full transcript
0:11Hi, my name's Wade Wells. Uh, like Jason said, I am on the Talking About News
0:14podcast. I'm also on a couple other things, and I teach CTI 101 here at
0:18Antisyphon, so I'm all over the place.
0:21I, I did wanna do this talk about augmented detection engineering. Uh, it's
0:25a little bit more
0:27how to actually get detection engineering using AI off the ground rather than using
0:33it completely. I, I... There's a lot of talks out there on how the different ways
0:37to utilize it, but no one really tells you how to set it up.
0:40And
0:41I also wanted to gear it towards more towards people who only have access to
0:44lower-end AI products, where anything that has, like, you could upload documents to,
0:49so like RAG, or you can give a prompt to. Nothing in command line or anything like
0:52that. So this is usually, like, the bare level you could do with detection
0:55engineering. Um, I do have a quote on here, which I was just talking about. It
0:58says, "Be wary of shortcuts to knowledge." So this is, uh, directly related to this
1:04talks and making sure you have enough context. The quote is actually from Kratos
1:08from God of War. Luckily, I was playing it yesterday, and he actually said that,
1:11and I was like, "Oh, that works perfect for my talk, so I'm gonna throw it in." So
1:16hopefully you guys enjoy it, and here we go.
1:18So up very first, I wanna really dive into what is detection engineering, because
1:23I'm seeing different definitions, and it's gonna be different at different
1:26organizations, right? So I wanted to put my definition out there. And for me, I see
1:32it in three different steps. Uh, a lot of times you have, like, threat
1:35intelligence, some type of software and system engineering, and then, of course,
1:38security operations. Uh, so to give a little bit of an example, I myself am
1:43looking at the news constantly, and I have an intel team that's actually feeding
1:46stuff to me, so I'm doing a little bit of both. And I actually have to know what
1:50internally I have, 'cause I have to build these detections. So I also have to know a
1:53little bit of threat modeling to know which logs to go to, right? I have to
1:57actually write the code or build log parsers or log pipelines in order to get
2:01them into SEM, and then I also have to figure out if the security tools are
2:05properly configured in order to make... get, like, the most out of them and to
2:09make sure my detections are valid through them. And lastly, I also have to build
2:13detection, detections for it. So it's a little bit all over the place on exactly
2:18what detection e- detection engineering is. Most managed security providers are
2:22gonna have people who all they do is write detections, but maybe solo shops with
2:26four or five people, you're gonna be wearing a bunch of different hats, and
2:28you're gonna have to make sure the logs are parsed correctly, that they're
2:31properly, uh, put into the right data models, their, uh, the timing is set
2:36correctly. So it's a lot of work, and it's a lot of different aspects, and it's
2:39kinda hard. Uh, very much like the DevSecOps, if you've ever heard that term,
2:44is what detection engineering is to me.
2:47So before we get into this talk about AI, I wanted to go over at least a little bit
2:50of security considerations, 'cause you are gonna be pulling in important information
2:53and pumping it into a third-party platform. So data exposure, right? If
2:58you're gonna be putting this into any type of AI platform, make sure that the proper
3:02controls are in place that others can't read this. If you have all of your
3:06detections, uh, being able to read by some type of AI platform and a threat actor
3:11gets to that, they can easily query the detections and see possibly how to move
3:15around your system so they don't get detected. Prompt injection is one way they
3:19would do that if they do get to that AI. If they have access to the AI, they may be
3:23able to just download the RAG, the files you have uploaded. And the last one is
3:27output trust, right? So if you're asking the, the AI questions,
3:32don't always take it to 100%. Like, most of the time, AIs to me are always 80%, and
3:37then I have to do the work a little bit there. At least when I'm messing with
3:41anything in production or anything at an actual organization, I don't never take it
3:46as face value. I try to always do a little bit of research or make it provide
3:50research for me. "Hey, here's, here's some information. Give me back information and
3:55provide links to or cite your sources," that type of thing.
3:59Um, all three of these are very important. Make sure your AIs are locked down and
4:02not, uh, not just anyone has access to any, uh, bots that you make, right?
4:09So the next one is augmentation. So for me, augmentation is the acceleration, uh,
4:14detection engineering task, like writing detection logic, mapping coverage gaps,
4:19triaging alerts or quality. Uh, you're making... You're enhancing the person and
4:24not necessarily taking over the person. You're trying to free up little things
4:30here and there. So to give an example, my... Out of all the AI stuff I've
4:34written, the most useful thing I have written is most likely a bot that will
4:40open tickets for me. I don't know how many meetings I'm in where a topic comes up or
4:45someone makes a request, and I then have to go open a ticket. And usually, like,
4:50even when you open a ticket in Jira or ServiceNow, there's a bunch of different
4:54tasks that go through it. Usually, you're gonna have some type of standard in order
4:57to follow that, expectations. And so having an actual AI where I just say,
5:02"Hey, so and so wants, uh, this detection done in this platform," and then it
5:07completely spits it out for me, and I say, "Yeah, that's perfect," open the ticket
5:10and assign it to me. That has probably saved me
5:13tens, tens if not hundreds of hours of work and also made it, made me, made it
5:18look a lot like I can actually write really good tickets. So that's just one
5:22way that I've been augmenting myself, right? Brainstorming, ghostwriting,
5:27building out guide, guiding you in particular ways or just having another set
5:31of eyes on the detections you're writing can all really push m- to move you
5:35forward. Um, my biggest one is I do not like writing. Uh, it's kinda really weird
5:40because now as I'm aging into my career, I'm finding that documentation is one of
5:44my more fun things, but I actually hate writing, so ghostwriting is helping it,
5:49uh, with my grammar, helping it with just making stuff sound better, has really been
5:54a big thing for me. Uh,
5:56all of this, you can see some, like, made-up stats I have on here. Forty-two
5:59percent less tabs, zero percent reg- regex learning, and ninety per-- ninety-six
6:03percent less spelling errors is probably, like, right on point for me and augmenting
6:07my stuff.So when we get into this, uh, with that, uh, that knowledge quote I had
6:13at the fr-- at the very beginning, the more context, the better, right? What I've
6:17noticed with a lot of the AI is the good teams who have all of this documentation
6:22written out already are just getting better.
6:25If you have everything written down, say a detection library, right? A centralized
6:29location that has all of your detections either mapped to a framework, tagged, and
6:33everything like that, it's easy if you already have that, but honestly, getting
6:36that actually done is a hard task. Usually, you're gonna do it with like
6:40detection as code or something, write a couple scripts that hit a c- several
6:44different platforms, maybe your SEM, your EDR, your cloud security tool. It's not an
6:49easy step. And then making sure that they're all properly parsed into a single
6:52format that you can read, like that, that's what takes a while. Logging
6:56inventory,
6:58it... I don't know how many times I've talked to people and asked them, "Do you
7:00have an exact inventory of every single tool you're logging, the prioritization
7:04behind it, and who's in charge of it in case it goes down?" Sometimes the
7:09detection engineering isn't, but usually you're gonna have to hit up somebody else
7:13if it breaks down one step before that. So this documentation all fuels the context
7:18for the AIs you're building. So these mature teams that usually have this can
7:24easily pivot to AI and have it help out more. Um, so I highly suggest looking into
7:29building out all of this documentation. So like company tooling, exactly what
7:33tool- tools your entire organization uses, or at least organizations who you're
7:37gonna be monitoring more or... Right? And how they use it, like which team uses
7:42which tool is very important. And then entity information, having possibly, uh,
7:48who, who is on your team, different teams, what they do. Usually, uh, if you can
7:53have some type of definition of every team, maybe a charter, and have that built
7:57in as well as possibly, uh, asset names, like these are the critical assets. If you
8:02see anything like this,
8:04ma- either write it in there that this is critical or
8:08mention it in some way, whether depending on which AI you're using it, right? Some
8:12other stuff would be network data, like network ranges, domains you're using, and
8:16that sort of thing.
8:20So I wanted to give clear examples of what I expect when -- with some of these
8:24documentations, because there are things out there for logging inventory, but I
8:29really wanted to show you what I really wanna do. So this is... Okay, this is all
8:33dummy data. This isn't really from the organization I work with, and I threw it
8:37in Notion because one thing, it's really easy to have AI write me dummy data that's
8:41in a CSV. So I did this and dumped into the Notion 'cause it also looks pretty.
8:47Uh, so if you do notice, I have the name, uh, the category, what type it is, the way
8:52it's being ingested,
8:53the, uh, status of it, the log types, what priority it is, right? Uh, the review
9:00cadence of that, and then what security tool that the final destination is in. All
9:04this stuff is really important for when you're having AI help you out with certain
9:09log types or, say, helping you build a detection, it will know which log type to
9:13go to and maybe query it if you're that far, that through that much further.
9:18Building this out is not an easy thing. Uh, every -- the last couple organizations
9:23I've had been at, we've had something like this, but it, it's a grueling task to
9:27go in and understand exactly which log source and what it's doing and what's
9:31exactly in it. So usually for every single log source, I'm actually writing a
9:35full-on description of what's inside of it, what events are in it, maybe a way to
9:40use it, and of course, some third-party resources to go to, to go after and go
9:45look at. So if I need to know Windows event IDs, right, I know exactly where to
9:49go to get that, and I can easily pivot. And if I can use these notes, AI can then
9:54use exactly the same thing to enhance its knowledge and move forward. Some other
9:59very important things, like I said, are just priority. If this log source goes
10:02down, what is the priority of it that needs to be back up, right? You can
10:07usually judge that based on the type of detections that are already written for
10:10that or just the importance that your organization feels that this holds, right?
10:15If I need to contact someone at midnight, who's the proper person to contact in
10:19order to get these logs running because a, a hundred or so of my detections are down
10:23because this log source is down. Very important, not just for you, but once
10:27again, building more context for your AI, so when you ask it questions, it knows as
10:33well.
10:35Next one I want to show is the detection library, right? This is actually pretty
10:39hard to build in my, in my, like, overall judgment of things. Uh, of course, this is
10:44huge. You c- probably can't read it. This is, uh, the one small text that I told
10:47Jason I had on here. I have every organization I've been at, I quickly write
10:51some scripts to hit every security dep- tool to pull every single detection and
10:56then map them to MITRE ATT&CK if they're not already mapped, right? Mapping them
10:59already is a hard thing to do. And if you can do this and have a source, it's not
11:04just great for the context of the AI, which it is gonna be great, but you, when
11:08you're building the detection... Like there was a question earlier, um, how do I
11:11know if a vendor has a detection if I'm writing custom detections? Well, if you
11:16have a detection library, you can go look it up and say, "Hey, show me every
11:19detection for who am I," and just quick Control + F, who am I? And then there you
11:24go. Here's several detections. The, the vendor actually has it, right? And then
11:27you can actually take that and modify it further. Having this information for the
11:31AI can then provide more value, which we'll see.
11:36Uh, here's like a quick overall definition that I write for each one of these
11:40detections too. So if I am a big on Palantir's alerting detection strategy
11:44framework, um, so I have one of these for every single detection that I have, so
11:50then I have more context as well as the AI. So some other lists that I would
11:55expect you to see and expect you to possibly get, this is actually being
11:58pulled straight from my CTI 101 class. Uh, so geolocation data, digital footprints,
12:04VIP tracking, industry partners, supply chains, employees, applications, right?
12:09All this type of data, if you can make these lists and build out different AI
12:13bots surrounding this list to give them context, it's gonna help you a lot. A lot
12:17of the times, uh, your CTI team is gonna -- already gonna have stuff like this, or
12:21there's gonna be tools to pull this data for you.
12:23Um, so these are all different types of things that I would highly suggest looking
12:27into in order to enhancing the bot and building out more context around itSome
12:33other things that, uh, that I didn't put on here that I think are team charters,
12:37which I've talked about, a description for every team and what they do. So if you're
12:40building anything out and you ask a question to the AI, it knows what team to
12:44talk to. The other one is IR plans.
12:47Uh, so if you're building out, uh, what we'll see here... we'll see a little bit
12:51further, if you're building out any detections, it knows the exact IR plans if
12:54that detection were to fire or how to escalate, right, and, uh, properly
12:59prioritize it.
13:01So the first big thing I think that you should do with detection engineering in
13:06order to enhance yourself with AI is, is really pushing how can it help you
13:11transfer knowledge from yourself to others, right?
13:14Uh, I-I'm, I, of course, have my one bot that is an... it, it writes those AD
13:19detec- ADS,
13:20uh, attack detection, s- whatever, ADS Palantir, uh, framework. I have a AI that
13:26writes those for me. I literally will give it, give it a description or give it the
13:30actual detection logic, and it writes that whole thing out for me. I look at it, I
13:34say, "Okay, it's around eighty to ninety percent right. Maybe I want some different
13:37sources on it," and I'll add it onto it, mess it around, and bam, there you go.
13:42It's great. Uh, s- and one thing, like we know, I am very bad at grammar and very
13:46bad at spelling, so then I also look better as well. Some other things you can
13:50also use it for is we talked about building out the detection library and
13:53building out the logging inventory and all those lists. You can have AI help you
13:57build those, right? Have it help you across this entire thing. Create a format
14:02for all of the documents once you're building something like this and say,
14:05"Hey, here's a format for all the lists. Here are the fields I want. Here's all
14:08this raw data. Make this list for me." And then you would regurgitate that and use
14:12it elsewhere, right?
14:14Um, the other thing to properly give it to build this documentation is examples. So
14:19you create one first,
14:21then tell it, "Hey, here's an outline and here's an example. Create something like
14:26this in the same format." Works out really well. Detection playbooks are really big,
14:30right? So usually for detection playbooks, you're gonna need a tool
14:34inventory of all the tools you have, and then a IR playbook, right, and logging
14:40inventory. So as you're writing these detection playbooks, it knows, okay, we
14:45have these type of logs, we know this person, uh, this is how we escalate an IR
14:49and what the priority of this probably should be, as well as, hey, here are the
14:53other tools. So to give an example,
14:55um, someone gets promoted to root admin of, of a particular, uh, root or admin of
15:01a particular tool. What's the first place you're gonna go? It's usually service
15:05desk. Whatever ticketing system they're using, you're gonna go say, "Hey, did
15:09anyone put in a ticket for this to get promoted?" If the, if the AI has the
15:14proper context to know, as you write out the playbooks, it'll tell you, "Hey, go
15:18check ServiceNow to see if a ticket is open for this particular user, and maybe
15:23here's a quick link to go do it." Um, stuff like that. Some other great things
15:27for the detection playbooks, of course, is other detections, examples that you have
15:31in there, right? Um, and, uh, I usually provided a couple resources online to go
15:37look at as well.
15:39Ticket and task creation. I talked about that a little bit earlier, which is one of
15:42my favorite bots that I've ever made.
15:45Uh, being able to quickly pivot to a task and not have to worry about the proper
15:49fields,
15:50uh, if everything is correct, if the format is correct, if my spelling is
15:54right, and just bam, it's done with a single second saves me such amount of
15:59time. The last one is really big is case summary and for target audience. So we
16:03talk about case summaries for SOC analysts all the time. It reads out the case,
16:07writes it for you as you close the ticket. But sometimes you wanna give a case to
16:11somebody else, maybe a manager, maybe someone who's not in security, leadership,
16:16maybe someone that has a more auditing focus. Having different case summary bots
16:20that are gonna tailor the reports in a different way is a quick pivot and an easy
16:24win for you that you don't have to rewrite everything and in a different
16:27format or with just different context. Um, one of the big things is have an outline
16:32for this. Have the team charters and a description of each one of their teams so
16:35then AI knows what to write and, of course, at least one example.
16:40Uh, logging. So I have four things here in logging and things that I would expect
16:44you to have in order to give the proper context for logging. So gap analysis,
16:48right? If you want a gap analysis bot, you're gonna want a logging inventory.
16:52You're gonna want a tool inventory. A lot of the times that logging inventory is
16:55gonna compare to that tool inventory. If you have this tool, why don't you have the
16:59logs for it, right? The last one is a network diagram. Maybe there's a
17:03particular part of the network it realizes you're not getting logs from, or you
17:06don't have any proper security tools around. Super important. The other one is
17:10ingesting and parsing scripts. So like I've said before, I actually have to write
17:14the scripts or have to figure out how to get logs from A to B, whe-whether it be
17:20from that tool to my SIEM or that tool to AWS, then to my SIEM or vice versa. Having
17:25AI help me write those scripts is, of course, gonna be a fundamental thing that
17:29everyone's using it for, as well as having... writing the documentation for it
17:33and commenting on the code. Usually for me,
17:37a lot of the times I'm coding, I'm writing out the functions and then saying, "Hey,
17:41here are all my functions. Put these together." I don't want AI to give me a
17:44script that I have no clue what it's gonna do, 'cause I'm most likely gonna have to
17:47go either teach someone how to use it or look at it myself. So me writing all the
17:51functions that I know, I know how the code works, I just let it put it together and
17:55make sure that all the Legos properly connect.
17:58Uh, data quality, dumping logs into it to make sure that they're properly parsed,
18:02maybe formatting changes, maybe a tool upgrades and breaks everything. Having AI
18:07constantly looking at that in some form, some way is s- another important. And then
18:11documentation and normalization and schema creation, right? This is going back
18:14to creating that logging inventory and creating that definition of each log, as
18:18well as normalizing the logs into a certain format and then helping you act-
18:22when I say schema creation, like sometimes like, yeah, there's data models out there
18:26for whatever tool you're using, but sometimes they don't make sense and you
18:29gotta customize them a little bit more. I have found rifting with the AI about what
18:33I expect it and what, uh, where certain data sets or certain fields, which fields
18:38should go in certain data model fields, has been pretty, pretty, uh, strong and
18:42helped me out a lotUh, last one, right, is detections. So what can we use detections
18:47for? So like Hayden before me has talked about intel. Detections for intel is
18:52probably the big-- one of the bigger things I've seen. I do have, uh, intel
18:55analysts who help me out a lot, but having a bot that goes out there and looks at a
19:00list of trusted websites, and then if something happens, and it maybe picks up
19:05on some type of keyword that I have in a list,
19:08it'll go and look like, "Hey, do we have logs for this?" And then boom, "Do we have
19:12a detection for this?" And then if it-- if you don't, it'll actually pump out
19:16detection or notify you in some way. Um, I, I have a bot right now that actually
19:21I'll run once a day. It's actually just like, it's a GPT that I'll say, "Hey, go
19:26look at all the news sites. Let me know if there's anything based off of my tools
19:29and logging that of a detection you think we should write in the next couple weeks,"
19:32and it will pop out. So I could definitely put that on some type of
19:36scheduled task and use Cloud Code, but this is a low-level thing that, uh, I
19:40really wanted to do.
19:42Um, detection writing, I have found mixed results with detection writing, but it is
19:46health... it is healthy to do it. And if you have all the proper detection
19:50inventory as well as some samples of detections and the security tool in which
19:55you're using, having their documentation, at least a link out to it, in order to go
19:59pull it and for it to notify. With certain security tools, this is gonna be a little
20:03bit harder, right? Like Splunk with custom fields, you're gonna have to give
20:06it some logs to understand. But if you're using data models, you should be good to
20:10go if you tell it data models. Um, one thing that I was gonna put that Hayden
20:13touched on was, uh,
20:16uh, capitalization, right? Certain, certain algorithms or certain detections
20:20will mess around with capitalizations within CLI and not either say lowercase
20:25and uppercase. Uh, I-I've actually had a GPT ca-catch me on that, be like, "Hey,
20:30this is capitalized. You should probably like look for both ca-- this isn't case
20:33sensitive or this is case sensitive. You should be looking for... You should write
20:37the detection in a different way because this isn't gonna work if they change it."
20:40Um, super fun thing. The other one is tuning recommendation. Looking across all
20:44your detections, right? Um, having log samples and a log schema and security tool
20:48knowledge. I will, uh, pump maybe false positives in there and say, "Hey, how
20:52would you expect me to tune this? Here's what's going on." Um, and that usually
20:57helps out a lot too.
21:00So I'm going a little quick, but should be good. Building the foundation first,
21:04that's the big thing here, right? Having all these documentations written in order
21:08to provide the context for the LLMs is super important. Uh, just having an LLM up
21:14there and said, "Hey, write this detection for me," without the proper
21:17context, it's not gonna give you the best thing. It's not gonna give you the best
21:20kno-- doesn't have the best knowledge and not gonna have the best information. The
21:23LLM doesn't know your stack. It's useless to you pretty much. You could make it so
21:27much better with a l-a little bit more effort with... that'll also not just help
21:33the, the AI itself, but help your team. A lot of the documentation I just talked
21:37about is great when a new person is starting. Be like, "Hey, here's all of our
21:41log sources. Here's all of our detections. Just go look at this and
21:45you'll be caught up instantly." As well as, "Hey, here's an AI if you have any
21:49questions," 'cause it has all that information too.
21:52Uh, docu-- document what you have before you automate what you don't, right? H-have
21:57all this information ready to go. It's usually gonna be a lot harder once
22:02everything is going and you don't have this documented. Like even with the AIs,
22:06have... making sure you know which AI is what, what documentations live within it,
22:10when the last time they were updated, right? Um, sometimes you could have some
22:14type of, uh, change control with a lot of these prompts or RAG, uh, if, if you're
22:19that advanced. But I f- I find that's a little bit more than most people. But at
22:23least having it documented on how you made it, what you did, and a quick link to
22:27maybe a overall summary for it is great.
22:30Uh,
22:31so if you're b- just now thinking about starting to build this, I would highly
22:35suggest starting with log inventory. That's usually the bare bones of anything.
22:39The detection library and the tool inventory are probably not as important,
22:45but still are important. Tool inventory is probably a little bit easier if you're a
22:48larger organization that can go around looking at stuff, but there is still gonna
22:51be shadow IT out there, right? There's still gonna be tools you've never heard
22:54about. And detection library stuff sometimes is a little bit harder writing
22:57that code in order to extract all those detections from a several different
23:01security tools. Um, to give an example, I think, uh, there's a certain EDR out there
23:05that would not give me access to their homegrown detections, so theoretically I
23:11couldn't map them and I had to scrape them in a different way. Luckily, uh, ChatGPT
23:16helped me out there too, right? And remember, the better context you put in,
23:20the better quality you're gonna get out. Make sure that stuff is all well-written
23:23and it's good to go, and you're gonna be getting better resources on the way out.
23:28All right, so thank you guys very much. Uh, that was my talk. Uh, if you wanna
23:33hear more about me, you can watch "Talking about News," usually on it every time.
23:37I'm also on another podcast called "Random but Memorable," which is for 1Password,
23:41which is where I work. I teach the Cyber 101 course, uh, and I do have an upcoming
23:46webcast that isn't, uh, officially announced yet called the Turn
23:51Cybersecurity Headlines into Action, which will be talking about biases and how to
23:56read the news in order to get the most out of it.
23:59And thank you very much.