Lesser Known Linux Persistence Mechanisms

0:00Hey, quick context as to what this video

0:02is. This is the recording of my talk in

0:05presentation from Black Hat USA titled

0:08Lesser Known Linux Persistence

0:10Mechanisms. Now, I had given this

0:12presentation with my day job, Huntress.

0:14And let me please say this is not a

0:16sponsored video. Huntress did not

0:17sponsor me or pay or anything to get

0:19this video uploaded. I guess you know

0:20they pay my salary. But I just genuinely

0:23thought, hey, this is a cool

0:24presentation. I hope the technical work,

0:26the concepts, the material, I hope

0:27that's educational and valuable for you.

0:30But the inspiration from this talk is

0:31just, hey, you know what? We've got a

0:33new Linux agent out in beta. We have a

0:35Linux agent for our EDR platform. That's

0:38super cool. This is just off the tales

0:39of us announcing the uh now Jen

0:41Easterly, the former director of SIZA,

0:43the cyber security and infrastructure

0:45security agency coming to work with us

0:47on our team, on our board, and even a

0:49super sweet partnership with Microsoft.

0:51So, a lot of cool stuff. Check it out if

0:53you want. But look, let me get to the

0:54actual presentation. Lesser known Linux

0:57persistence mechanisms. All right. Hey,

0:59thanks so much everyone. Look, my name

1:01is John Hammond. I am a principal

1:03security researcher over at Huntress. Uh

1:05Huntress is a managed security platform.

1:08Hey, proiding providing security for all

1:09companies, EDR, Seam, SAT, U, ITDR, all

1:13the cool buzzword bingo acronym stuff.

1:16But what I'd like to talk with you about

1:18today is Linux and lesserk known Linux

1:23persistence mechanisms. Uh, and if it's

1:25all right with you, I guess we can just

1:27kind of dive in. Yeah. Sweet. I know

1:29we've only got like 20 minutes, so we're

1:31going to try and speedrun and have some

1:32fun. Look, so we're talking about

1:34persistence, and I've got the obligatory

1:37MITER attack framework screenshot here,

1:39so you know what I mean. persistence.

1:42When an adversary, actor, hacker will

1:45maintain access. They'll keep a

1:47foothold. They'll keep a back door.

1:49They'll keep some means of getting back

1:51in to that host or that environment to

1:54do more damage and wreak havoc. Uh now,

1:56the reason that we wanted to do this

1:58thing, why we're chatting about this is

2:00cuz if you way way back in Huntress

2:02history, foothold and persistence was

2:04exactly the first thing that we kind of

2:06started with. Um, it was originally in

2:09the Windows world, but now we've

2:10expanded that to Mac and now Linux. So,

2:13we're getting pretty excited about it,

2:14but I wanted to chat more about those

2:17Linux persistence mechanisms. So, if you

2:19were to hop over into the Huntress

2:21portal, if you were one, oh, checking

2:22out what we're up to, whether you

2:24started a trial, Kickstart, kick the

2:25tires, whatever, you can download and

2:27you'll see an Linux agent on the way.

2:31All that to say, let's dive into the

2:34persistence mechanisms that you probably

2:38already know just to get this stuff out

2:40of the way because this is the boring

2:42boilerplate crap that we probably just

2:44like, oh yeah, okay, this this again

2:46sort of thing. So, we'll start with that

2:47and then hopefully I can give you some

2:49new good nuggets. But if you all think

2:52of persistence, some of you might think

2:54of, oh, we probably want to do this with

2:58root privileges as the admin in the

3:02Linux world. Root user, super user. So,

3:04I'm going to use a little bit of

3:05connotation here for like a user prompt

3:07with that dollar sign in the command

3:09line and the root prompt with the

3:11hashtag pound symbol. Fun fact, the

3:14scientific term for that is octothorp.

3:17And now you know that's a good little

3:18nugget for you. All right. So, obviously

3:21boring yawn. We know you could add a

3:24backdoor user. That's not very sexy.

3:28That's not very elite, not very cool,

3:29but it would be a backdoor user. That is

3:32a persistence mechanism to get back in

3:34if you give it a password you could use.

3:37Uh, but folks tend to know that, right?

3:40It's not sophisticated or stealthy or

3:42neat. It's just another opportunity.

3:44Maybe you'll add SSH keys. Secure shell.

3:47Folks are familiar with SSH, correct?

3:49Yes. All okay. If not, that's okay. This

3:52is a safe space. SSH keys are saying,

3:55okay, a private and public key pair that

3:57I might use to gain access back into

4:00that machine. Granted, this relies on

4:03SSH, the secure shell being open on port

4:0622. That would need to be the access

4:08vector if you were to do that. But

4:10again, eh, whatever. We kind of shrug it

4:13off. We've seen that kind of thing

4:14before. And with that, maybe you'll do

4:18cron jobs. Okay, cool. Folks are

4:20familiar with cron jobs. These are these

4:22scheduled automated timing things like

4:24an alarm clock you set to wake up every

4:26morning. But you could toggle this not

4:28to be a specific time of day or every

4:30couple of minutes. You could do this on

4:32reboot or whatever. So what I'm doing is

4:35you might be able to see as I echo that

4:37into a cron tab. I pipe it in just oh so

4:39we have that input going into the

4:40standard output there. I'm running a

4:42binary user or local bin backdoor quote

4:46unquote. And that could be

4:48whatever. That could be a payload. That

4:50could be some other malware. That could

4:52be our call back to command and control.

4:55Blah blah blah. Kind of get the gist on

4:57that, right? Again, we're still in our

4:59boring basics, but maybe we have systemd

5:03services. Very similar to services in

5:05the Windows world. Maybe systemd or

5:07cisv. Some of the old school stuff.

5:09System V in a different way. you would

5:11use either systemctl or the service

5:13command on the command line. These are

5:15all things you define as text, right?

5:18And that's always been the core of Linux

5:20is plain text. You can pipe that in and

5:23out of the terminal really easily and

5:26that might be something that either

5:27isn't all that fancy because it's plain

5:30text. it could be easily manipulated or

5:32we might be actually able to do some

5:34sinister stuff with it when we get into

5:36other more sophisticated air quotes root

5:39kits or other persistence mechanisms

5:41there.

5:43Another one. Oh, maybe you're working

5:45with those uh remote uh I believe what

5:47is it? Runtime control or run control

5:49files, bash RC, Etsy profile, other

5:52spots that will automatically trigger as

5:55you either log in to an interactive

5:57environment or spawn a new terminal,

5:58etc., etc., every time you open bash or

6:01zshell, zsh, etc., etc., these are all

6:04again kind of like, okay, we sort of

6:06know these. Yeah, basically, right? Is

6:08this still boring and dumb so far?

6:10>> Cool. I appreciate boring and dumb. You

6:13could do the same for other programs,

6:14right? Vim, command line, text editor.

6:16We could start the holy wars, Vim versus

6:18nano, Emacs, whatever. But all those

6:21things could be done in those other

6:23scenarios.

6:24Questions though, when I was alluding to

6:26the user local bin payload or whatever

6:29we actually try to execute or run, um,

6:32again, that could be whatever you want.

6:35That's a variable that's dynamic. That

6:37could be whatever you're most interested

6:38in. Maybe it's a compiled binary or a

6:41shell script, whatever the case may be.

6:44What I would probably be the easy button

6:46as folks would again just be like, "Oh

6:48yeah, okay. Uh, what about a shell? What

6:50about a reverse shell or a bind shell?"

6:53If folks weren't tracking, obviously the

6:54bind shell is listening on that victim

6:57on the target that you've compromised.

7:00Easy opportunity with that is with

7:02netcat. Granted, I'm using netcat nc

7:06with the old I think traditional is

7:08probably the right vernacular for that

7:10one where you have the dash e flag e to

7:13be able to run a command or a program.

7:14Here we run bin bash listening on port

7:1790001 because it's over 9,000. That's a

7:20joke. Um DNS is removed with tac nv for

7:24verbose p for that port number over

7:269,000. And then we can connect back to

7:28it as the attacker. Now, reverse shells

7:32normally we tend to see in oh that CTF

7:34world capture the flag, try hack me,

7:36hack the box, blah blah blah, whatever.

7:38Bind shells might give you a little bit

7:40quote unquote more stealth if you maybe

7:43sneak it in some way and make it sleep,

7:45make it dormant. You got a lot of

7:47creativity there, but reverse shells

7:48could probably stick out like a sore

7:50thumb. Granted, it is one opportunity to

7:54get your connection back, to get your

7:56call back. Here we flip the script. We

7:58just make a connection from the victim

8:02calling back home to the attacker. We

8:05use tac e as the opportunity there. Once

8:08again, I won't belittle that too much,

8:11but I think it's more fun to do anything

8:14else you want. Curl of course being an

8:17opportunity to pull down a payload from

8:19a remote resource like a website, HTTP,

8:22HTML, whatever. uh consider adversary

8:24website their external C2 domain and you

8:27pipe that to bash or you pipe it to your

8:29shell amperand here to format to put

8:32that in the background and now okay

8:34you're off and running you got code but

8:37again maybe you do something a little

8:39bit more flashy or fancy from GTF opens

8:42are folks familiar with that resource

8:44there's a fun acronym of course GT get

8:46the f out um a lot like the lens living

8:50off the land binaries in the Windows

8:51world a Linux equivalent But now that

8:54I've wasted five minutes, let's talk

8:56about maybe hopefully some persistence

8:59mechanisms that you maybe haven't heard

9:01of or aren't as familiar, just a little

9:03bit more esoteric than add a new user,

9:06put another service in. There are a lot

9:09of opportunities here. Um, are folks

9:12familiar with the prompt command

9:13variable? Yeah. Okay. A couple these are

9:16ones that'll actually execute, I

9:17believe, after a command will run. Uh, I

9:20will need to double check because

9:21there's another opportunity if you were

9:23to use a trap. When you have a trap

9:26configured inside of your terminal, that

9:28debug will allow you to run something

9:30before a command would run. That will

9:34again, every single time there is

9:36interaction on that terminal in the

9:38command line, fire off something else.

9:40That could be your payload, that could

9:41be your back door, other code you would

9:43want to run. That's an opportunity

9:45that's a little bit sneaky. Maybe you

9:47cram that into your bash RC or your etc

9:50profile or something that could trigger

9:52this, but on its own that might not look

9:55as malicious as it sticking out outright

9:58in another patch d

10:01one opportunity there. And of course,

10:03you've got other options for aliases.

10:05Those are simple small things like, oh,

10:06we just gave a nickname to some other

10:09command that you might naturally run.

10:10Think like ls, think cat, think history,

10:15who am I, pwd, cd, whatever you want.

10:18You of course have a creative canvas

10:20there. Same thing with functions. You

10:21can do that very, very same thing. Maybe

10:24that's another one that we're like, oh

10:25yeah, okay, we've seen that before.

10:28But let's talk about SSH again. Let's

10:31kind of go back and revisit secure shell

10:33if that's an opportunity for us. Usually

10:36you'll have a user configured SSH config

10:39file right out of your home directory.

10:41So that's that till day right squiggly

10:43line for your home directory or the

10:45victim the target who you're impacting

10:47putting our hacker hat on acting as the

10:49adversary in that SSH file or if you did

10:53this even in that root global all

10:56everywhere across the system file

10:59there's a proxy command feature which I

11:02don't know if you've ever seen but this

11:04is as you make a connection with SSH you

11:07have a command that runs something else

11:09that executes as a program as another

11:12opportunity as a proxy, quote unquote.

11:15So if you wanted to, if you were

11:17stealthy a little bit, oh, you could

11:19make the capability and functionality to

11:21actually complete the transport and do a

11:23SSH communication. Or if you wanted to

11:26just break stuff, hey, you could have

11:28that binary, your payload, just run your

11:30malware, run whatever commands or

11:32execution you would want. But that's one

11:34opportunity with SSH. Proxy command in

11:37the configuration files or even

11:39sometimes in the command line arguments.

11:41You can pass tac O to be able to do

11:43that. Proxy command is what we're

11:45zooming in on there. The host actually

11:47lets you configure that. You'll notice I

11:49have a star or an asterisk kind of as a

11:51wild card. That could be any host, but

11:53you could tweak that to be any specific

11:56host name or port. Uh if you wanted to

11:59get a little shady and stealthy with

12:00that. This one I really like actually uh

12:04another SSH alternative where you no

12:06longer use the authorized keys file, but

12:09you cram it in to another file, whatever

12:13you want. You could name it, you could

12:14hide it, it could have the dot prefix so

12:16that it's hidden. And if you just

12:18include that or run any other command,

12:22that could work just as well.

12:25Interesting thing, if you configure that

12:27in the SSHD, SSHD server, not just the

12:32client, but the server configuration,

12:34that will actually allow you to specify

12:36a thing that runs every time someone

12:39would invoke an authorized key

12:41connection with that. So, your opt

12:44hidden keys or any other folder or file

12:45you wanted to hide something in, smuggle

12:47a payload in or do some damage there.

12:51Now, this one is honestly one of my

12:54favorites, but it is kind of tough to

12:57find. Uh, folks heard of TCP wrapped

13:00services, and it's okay. Some it's okay,

13:02some have, some not. Um, so if you run

13:06ldd, one command to list all the like

13:08the loaded libraries oro files that some

13:11other Linux binaries pull in, you could

13:13see lib wrap or lib TCP wrap. If you

13:17find that you get to configure any other

13:19file you might like and slash it's

13:21setost.allow

13:23or slashitsetrahost.deny

13:26and you could say look if you connect

13:29either with ssh something that could

13:32pull in that lib wrap library or even

13:35some other capabilities uh and I'll tell

13:37you about that one in just a moment. you

13:39could use and run and execute your back

13:42door, your payload, user local bin,

13:44whatever.

13:46This is very cool. I've seen this and I

13:48got to play with this and I'll admit it.

13:50It was in Cali when I got to test it.

13:52So, I don't know how often you're going

13:53up against a Kali Linux machine, but

13:56SNMP like SNMP community strings, SNMP,

14:00that UDP protocol includes lib TCP wrap.

14:05So from a remote machine, if you added

14:08this back door on a victim, if you tried

14:10to run SNMP just to walk the community

14:12strings of another host, this would

14:14fire. And I thought that's kind of cool.

14:17Like you can remotely trigger your back

14:19door, your payload. So uh would

14:21definitely recommend keeping an eye out

14:23for those TCP wrapped services. I think

14:25that's a good one personally. Now this

14:28one, PAM degradation attacks folks are

14:30probably familiar with. Maybe you'll nod

14:32and follow along with this one. As you

14:34log in to Linux, as you enter your

14:36command line, oh, you log in Ubuntu

14:38root, whatever the case may be, that's

14:40all handled by the pluggable

14:42authentication module or PAM. And you

14:46might have seen inside of your /var or

14:48SL, etc. directories, you can see a lot

14:51of the PAM configuration files. Some of

14:53those are shared object files, so things

14:55that you could determine, okay, what is

14:57the correct authentication? When do I

14:59allow someone to log in or we deny it

15:02one way or the other? Now you can craft

15:05your own PAM login modules, your

15:09pluggable authentication modules. You

15:11can make your own. So uh over on the

15:13left hand side, it might be kind of hard

15:14to read, but there's a back door that we

15:17had put together some some time ago for

15:19a cyber force competition or capture the

15:22flag training environment. We were up

15:23against a blue team and a red team. So

15:25as a red team, we were thinking, you

15:27know what? Let's backdoor Pam and enter

15:29so that no matter what login credential,

15:32password, whatever a user actually used,

15:35we'd let them in and authenticate if

15:37they entered sneaky password. But you

15:40could configure it to be whatever you

15:41want. You can make it Apple, you can

15:43make it microphone, you can make it

15:45anything cuz you get to control that

15:47once you embed and put that in. Of

15:49course, you'd need the root privileges

15:50to do that, but that's persistence.

15:53There's a back door. you define a

15:55password for any and all user allow them

15:58in. There was a very recent bleeping

16:00computer article. You might be able to

16:01see it on the right hand side there. Uh

16:03this was August 4th, so two days ago. Uh

16:07there's a recent Linux malware payload

16:10that was uh what was it called? The

16:12plague. And Nexttron Systems was

16:14covering it. So Florian Roth and some of

16:15his cool folks, they were saying, "Look,

16:17this did exactly that. This used a PAM

16:20degradation attack and had a backdoor

16:22pluggable authentication module to log

16:25in with whatever credentials they

16:26wanted. Neat. Have folks seen that one

16:30before? Kind of familiar with a little

16:31bit with Pam. Sort of. Kind of. Now

16:34we'll get to the fun sexy stuff, right?

16:36Everyone kind of gets pretty hyped up

16:38and excited about like, whoa, a rootkit.

16:40Wang, it sounds so cool. Now we're

16:42thinking about userland rootkits to

16:44start. What I'm going to show you, I

16:46hope it give you a decent resource. Uh,

16:48I'll make this bigger and hopefully give

16:49you that link in the next slide, but

16:51there's one of those GitHub awesome

16:53links or GitHub awesome resources that

16:55just lists a ton of these. One that I

16:57really like is called Prism. This is a

17:00userland rootkit. So, hey, not at the

17:03kernel, nothing elite, nothing lowlevel

17:05quite yet, but it operates over

17:09MP.

17:10Folks know what ICMP is. Yeah, ping.

17:13Cool. Easy. So, normally when you're

17:14looking for malicious traffic, you're

17:16looking for whoa HTTP, webshell

17:18communication, DNS, X-filling data. How

17:21often do you look at ping traffic? Some

17:24do, some might. Kind of varies, but that

17:27one was pretty neat and pretty stealthy.

17:29Another competition event we we've seen

17:31was a pros versus Joe's. They do that

17:34often at Bides Las Vegas. Uh, so just a

17:36couple days here before that is, uh,

17:39their red team would use this. They use

17:41Prism as the userland rootkit to be able

17:43to communicate get their C2 over ICMP.

17:46Kind of cool, I thought.

17:49Now, um, a next one here. I know we're

17:52getting to probably the end of things

17:53here, but if you wanted to take a look

17:55at that awesome Linux root kits, let me

17:58bear in mind this was strictly rootkits

18:00in that resources, but it's another

18:01GitHub awesome link. One of those

18:04references a project and utility called

18:06Reptile. Um

18:10happen to know does anyone know another

18:12content creator or YouTube fellow? His

18:14uh his name is IPSC or Nick Ipo. He

18:17covers a lot of the hack the box

18:18training material etc. Um I learned

18:21Reptile from Nick from IPSC. uh he

18:24covers this in some other previous old

18:26videos, but one of the coolest things

18:28about Reptile because this is an LKM

18:30like Linux kernel module, but if you

18:33wanted to modify a lot of the C code,

18:35like the low-level source code that it

18:36uses, you could actually specify what

18:40the kernel will respond to any user or

18:43input or programmer interface reading a

18:46file from your Linux system.

18:48So,

18:50say we were playing some King of the

18:52Hill game, quote unquote, but maybe

18:55nerds and geeks are familiar with like

18:56Hack the Box Battlegrounds or try hack

18:58me king of the hill. Again, silly, oh,

19:00practice playgrounds cyber crime

19:02shenanigans. But what they'd get to do

19:04is say, look, if I planted a flag in the

19:06ground that said John is the one that

19:09owns this server, if another individual

19:11were to come and go beat it up, they

19:13compromised the box. Maybe Ben is now

19:15the next one that wants to claim their

19:17ownership.

19:18Well, we've got Reptile or some other

19:21Linux kernel module that says like,

19:23"Sure, that's fine. You can modify. You

19:25can write to the file. You can make the

19:27change. I don't care. But every time

19:30that the server, the computer, any

19:31program, anything else reads that file,

19:34it will always tell you no matter what,

19:37John is the of this file, actual output

19:40or contents you receive."

19:43I think that's neat because when you

19:45think back to a whole lot of those Linux

19:46persistence mechanisms that are like SSH

19:49configuration files, authorized keys,

19:51etc. password, it's et

19:54files. So you could force the operating

19:57system and the kernel to make that give

19:59you the output that you want. The back

20:01door, the persistence payload, etc.,

20:03etc. Keep an eye on those if that's okay

20:06with you. I think that's a good

20:08opportunity. Let me uh speedrun for the

20:10last couple moments here. I think

20:12there's another really cool utility that

20:13would be helpful for you all if you are

20:15playing with this. How could we automate

20:17and test a lot of these Linux

20:19persistence mechanisms? Let me give

20:21credit where credit is due. This is a

20:23project called Panics. Are folks

20:25familiar with this? Have you seen this

20:26before

20:28on GitHub? Customizable Linux

20:30persistence tool for research and

20:32detection engineering. Let me give

20:34credit where credit is due and I'll let

20:36a couple more folks take a quick

20:37picture.

20:39But it's these persistence mechanisms

20:41against Unix systems. And I got to say,

20:44hey, this was put together by a lot of

20:46the sweet researchers and smart folks

20:48over at Elastic. I do want to give them

20:51everything, all the blessing, all the

20:52kudos. But it's very, very cool cuz it's

20:54just a ton of bash script, super

20:57modular, super customizable, but that

20:59has almost all of the things that we got

21:02to touch on and talk about here. maybe a

21:04couple that oh aren't in that list that

21:06we covered, some that are in this list

21:08that we didn't cover, etc. But that is a

21:12really really sweet utility and it's

21:13just a whole lot of command line

21:15opportunities to be able to test those.

21:17So I hope panics is a good resource for

21:20you if you haven't seen it before. But

21:22big shout out to Ruben, big shout out to

21:24the developer. Again, I want to give

21:25them that credit. Credit where credit is

21:26due. Elastic's doing smart stuff. Not

21:29afraid to say it.

21:32All right. Thanks everybody. That was

21:34like a 20-minute crash course on Linux

21:36persistence mechanisms, but my goodness,

21:38there's still a heck of a lot of stuff

21:39that we didn't get a chance to talk

21:40about. Backdooring GitHub, GitHub

21:42utilities and Git hooks, set UID

21:44binaries, LD preload hijacking for other

21:46libraries you load. So, so much stuff

21:51and it's all the stuff that we get to

21:52chase over Huntress while we're building

21:54out this new Linux agent and we're

21:56excited about that as the opportunities.

21:58Thank you so much everybody. Look, I

22:00will say that back at our booth, if

22:03you're willing to join us, we're back

22:04over there just a little bit. You might

22:06be able to see the Huntress logo from

22:07here. We have a super special guest that

22:09I'm really excited about. I don't know

22:10if folks saw the news, but uh Jen

22:12Easterly, the uh previous director of

22:15SIZA, the Cyber Security Infrastructure

22:17and Security Agency, just came on board

22:18with us as a strategic adviser. So, she

22:21has a uh awesome new presentation coming

22:23up. So, I'd love for you all to come

22:24hang out, follow me to the booth, and

22:26we'll see you there. Thank you

22:27everybody.