How Forward Deployed Engineering is done at Factory — Eno Reyes

0:01[music]

0:12>> This is the forward deployed engineering

0:14track in case you're in the wrong room.

0:16Um as you already know, forward deployed

0:18engineering is one of the hottest topics

0:19in AI. The most important companies on

0:22the planet are building out massive FTE

0:24teams. So, think OpenAI, Anthropic,

0:26Google DeepMind, you get the idea.

0:28Forward deployed engineering was

0:30pioneered by Palantir many years ago to

0:31embed really strong software engineers

0:33directly into their customers' orgs

0:35uh to implement customize their

0:37platforms around the nuances of the real

0:39world. So, today we brought in some

0:42amazing speakers from Anthropic, Cursor,

0:44Factory, Ramp, Decagon, and many more uh

0:47to talk about the current state of

0:48forward deployed engineering, how it

0:50works at their companies, and where it's

0:52going. Our first speaker

0:54Our first speaker is Eno Reyes. He's the

0:56co-founder and CTO at Factory, which is

0:58building autonomous software engineering

0:59agents for enterprise teams. Previously,

1:02he worked in machine learning and

1:03software engineering roles at Hugging

1:04Face and Microsoft. Let's give it up for

1:06Eno.

1:08>> [applause]

1:11>> Yeah, hey everyone. Excited to chat

1:13today. Um and you know, basically I I my

1:17hope is that at the end of this you guys

1:19get a sense of some of the work that

1:21we're doing on behalf of our customers

1:23and with our customers, and the role of

1:25what we call a deployed engineer should

1:28hopefully be a little bit clearer since

1:30I think that there are honestly tons of

1:32different models um for

1:34uh for how this should actually operate

1:36inside of an org. Um and so

1:40I think that when when we start I I I do

1:42think that there are some nuances in

1:44sort of like the Palantir era playbook.

1:47Um and I generally the way that forward

1:50deployed goes is I I see that there are

1:52lots of different takes on sort of where

1:55forward deployed sits within the org,

1:57how much it interfaces with the actual

1:59product team or the engineering team,

2:01how much work is done on behalf of the

2:03customers versus with them, and how much

2:05work is done on code itself or basically

2:08like in the software system versus with

2:10the humans and sort of strategizing,

2:12right? And so, um generally, I think the

2:15there's um

2:16in this older model, a lot of the way

2:18that software needed to be built was you

2:20needed to go and access that codebase.

2:23You needed to integrate directly in to

2:25data streams or software or products

2:27that basically you could only access

2:30behind the curtain of the customer. And

2:32so, if you were building something that

2:33was heavily integrated into their

2:34environment, yeah, you kind of had the

2:36need to send and sort of parachute in

2:39individuals into the org. Um

2:42but really that has transformed over

2:45time into a role that sort of forks out,

2:49and you see a lot of people who are sort

2:51of quote-unquote forward deployed

2:52engineers or deployed engineers or

2:55applied AI engineers, and it's always

2:57it's always a little bit unclear. Are

2:59they doing maybe professional services

3:01work on behalf of their customer? Are

3:03they transforming like the product

3:06around an individual customer? Are they

3:08just building entirely net new things in

3:11the customer's environment, maybe on top

3:13of your product? Uh and I think that the

3:15the at least at Factory, we definitely

3:18do not want to be doing professional

3:21services work on behalf of a customer.

3:23So, if a customer says, "I want to do a

3:26uh modernization of a codebase, uh and

3:29it's you know, I just got quoted from

3:30all of the big consulting firms, it's

3:32going to cost this much. Could you do

3:34this consulting work for us?" Uh our

3:36goal is not to go and actually do that

3:39migration on their behalf, even if we

3:41happen to be using our product, right?

3:44Um and that is because we don't think

3:45that that actually makes our product

3:47that much better. Uh and ultimately,

3:49that is a great way way get I'd say a

3:51decent amount of revenue, but I don't

3:52think that that's the way that you can

3:54scale a business out enormously, right?

3:56And so, what we've done is we've instead

3:58said, we need deployed engineers to be

4:01the tip of the spear of the product. And

4:04I'm going to do this and then go back.

4:06But but really when we say the tip of

4:08the spear, what we mean is that deployed

4:11engineers are basically the stream of

4:14information from our largest and most

4:16critical customers of the engineering

4:19leadership in that org. The

4:21on-the-ground tactical engineers, their

4:23thought process about how software

4:24development and AI is actually happening

4:26at that org, and then flowing all of

4:28that information back into our product

4:31to then rapidly adjust our product in

4:34order to then fit into the customer's

4:35environment better, right? And so,

4:38factory really should be when it gets

4:39deployed, and I'll talk about what

4:40factory is in a second, but we want that

4:43to be effectively self-assembled inside

4:45of our customer's environment, right?

4:47And then there's a lot of work that goes

4:50into understanding that customer's

4:51environment, what the flows that happen,

4:54and ultimately the ROI story.

4:57And what factory really is to our

4:59customers is a set of building blocks

5:02for building a software

5:04software factory, right? And so, when we

5:06say software factory, what we mean is

5:09there's this implicit process that every

5:11organization in the world sits on top

5:13of, where signals from the outside world

5:16flow in on one side, and those signals

5:18could be a lot of different things. It

5:19could be customer conversations, it

5:21could be bug reports, it could be

5:22internal Slack or Teams conversations,

5:25it could be an executive saying, "We're

5:26going to build this thing," right? All

5:28of these are signals. Some of them have

5:31higher weight than others, and those

5:32signals flow in, and humans implicitly

5:35or explicitly then choose to then

5:38prioritize, triage, and build plans

5:41around those signals. Those plans are

5:43then converted, typically by software

5:45developers, into changes into some

5:48source of truth, a code base, an

5:49engineering system. Um and as those

5:51changes are actually executed on, they

5:54flow through a validation stage where

5:57people maybe review the code, they QA,

5:59they assess the security implications,

6:02they uh pass it through automated

6:04validation like SAST tools, linters,

6:06type checkers, and ultimately when

6:08everything passes, they then ship and

6:10deploy. And what do you do with deployed

6:12monitored software? Well, it it

6:15generates more signals, right? So, this

6:16implicit feedback loop is instrumented

6:20very poorly, to be honest, at most

6:21organizations. And if you're able to

6:23take AI and actually transform each of

6:26these stages of the pipeline and build

6:28an understanding of what the workflow

6:30looks like at your org from each stage

6:32to each stage, then you actually can get

6:34to the point where you have a a flow

6:37through from signal to deploy that has

6:40no human intervention. Now, importantly,

6:42that does not mean that humans are not a

6:43part of engineering this system, right?

6:46But it is that the flow of signal to

6:48deploy is uninterrupted by a human. Um

6:50and that software factory concept is

6:53obviously not something that can just

6:55snap your fingers and it appears, right?

6:57Instead, it requires an investment from

6:59the organization. We we like to say this

7:01is built, not bought, right? But what

7:03the platform that we've built basically

7:05provides to people are the canonical one

7:08model independent agent harness that you

7:11need to do this, because if you want to

7:13build a software factory, if you choose

7:15to build that software factory in a

7:17vendor locked solution that has like one

7:19model available to it, uh that is going

7:22to not only be expensive, but two, uh

7:25there's open questions about model

7:27independence and like what is the role

7:29of the model provider in dictating what

7:31you can or cannot build with your

7:32software factory, right? Um and if you

7:34also don't own the traces, the data,

7:37everything that flows through your

7:38software factory, um then you're

7:40probably going to be in trouble as you

7:42start to want to evolve your software

7:44factory, right? And so with Droid, the

7:46hardness that we build, you not only

7:48have model independence, but you also

7:49have access to every piece of data that

7:51flows in in and out of Droid, alongside

7:54centralized governance and control at

7:56the enterprise layer to be able to

7:57dictate where what information flows

8:00where. Um you can air gap Droid if you

8:02want. Some of our partners um in, you

8:05know, the most secure uh environments,

8:08uh think finance, health care, uh gov,

8:11uh they air gap Droid and they run their

8:13software factories entirely contained.

8:15Um

8:16One of our deployed engineers jokes that

8:18you could run Droid in a submarine if

8:19you wanted to. And that's that's

8:20honestly true. And so when we think

8:23about what the role of this deployed

8:25engineer is in that context, which I

8:27probably should have started with, um

8:29you know, you really need somebody who

8:31can go in and say, I understand this new

8:34model of building software and I

8:37understand the building blocks and the

8:39pieces. I can help enable building and

8:42constructing these software factories

8:44with your team, but I ultimately would

8:46like to one, make it so that our product

8:49effectively, you know, one click

8:50self-assembled into your environment,

8:52which is needed when you have 45,000

8:54people, maybe hundreds of thousands of

8:57uh of of engineers, maybe you have tens

8:59of thousands of code bases. Uh you

9:01you've got to self-assemble, right? You

9:02just can't manually install this level

9:05of of complexity. Um and and also on the

9:09sort of like end loop,

9:10why do all of this, right? I I would

9:12argue that there needs to be an ROI or

9:15an outcome story that is extremely clear

9:17from the beginning so that you can say,

9:19well, we know every code change that

9:21flows through that gets AI code review,

9:24AI QA, AI security analysis is maybe 87%

9:28less likely to hit a bug. And what that

9:31means is that we can reduce our our bug

9:33rate by X, that increases our customer

9:35satisfaction by Y, and that leads to

9:38revenue or growth or new business,

9:41right? Something needs to flow from this

9:43software factory process to core

9:45business goals. And that often is a

9:47complex story that requires engineering

9:50knowledge, it requires business

9:51knowledge. And so, if those are the

9:53types of things that you think are

9:55interesting, that is what deployed

9:57engineers today are doing for us.

9:59Um

10:00I've sort of outlined it a little bit

10:01here, but that teach the model step is

10:04super important because most

10:06organizations do not have an autonomy

10:08maturity model. They do not have a road

10:10map, they don't have a conception of

10:12what it means to truly build an

10:14autonomous software organization, right?

10:17I think a lot of people ask the

10:18question, what do the humans do in this

10:20world, right? For us, we see an

10:22extremely clear role for humans in

10:24evolving, refining, and scaling software

10:28factories, right? So, you basically the

10:30engineers at a company go from directly

10:32manipulating software to directly

10:35maintaining and managing a system that

10:37builds software. And that sort of like

10:39upgrade in the level of abstraction that

10:41you operate at is actually very

10:43difficult. And a lot of people

10:46find it extremely challenging. I would

10:48argue that in fact most people, even

10:51very thoughtful software engineers, will

10:53have a learning curve in trying to

10:54shift. The people who I think are are

10:56well suited for this are DevEx people

10:59who have already been thinking about

11:00enablement of other developers. I think

11:03product managers who want to become very

11:06technical very quick can become really

11:08great at doing this. And I think that

11:10generally like people who are used to

11:13to working on teams where

11:15>> [clears throat]

11:16>> high quality dev environments were a

11:18priority, you will you will get some of

11:21the canonical things necessary to enable

11:23these agents to succeed.

11:25Um

11:26I haven't really talked about this last

11:28one, which is design the workflows.

11:31And I will get to that in a sec, but I I

11:33think that when I say tip of the spear

11:35of the product, like keep in mind I

11:37really do mean everything that is

11:40happening inside of factory. So, our

11:41product encompasses enterprise controls,

11:44the droid harness, the workflows that

11:46run on top of it, the observability

11:48tools, the cost controls, the auto model

11:50routing, the quality of the harness.

11:52Like, all of these are pro- potential

11:54opportunities of improvement that you

11:57will discover when you work very closely

11:59in these varied or diverse orgs like how

12:01to solve. Um,

12:04so

12:05making a code base agent ready, right?

12:07This is a very challenging thing to do.

12:10Uh, most organizations have some degree

12:12of consistency in how they've chosen to

12:14build deterministic validation loops

12:16inside of their company, right? So, your

12:18code base runs linters, type checkers,

12:20uh, it might run some security scans,

12:23and it's like check mark. Like, it

12:25passes or it doesn't. The end end test,

12:27they pass or it doesn't, right? Or they

12:28don't. Um, what agent readiness really

12:31is is it's a measure of how many of

12:33these deterministic validation loops are

12:35present inside of your code base. Uh,

12:37when you have a huge volume of these

12:39feedback loops, uh, agents are able to

12:41operate for greater periods of time on

12:42more complex tasks without human

12:44intervention. So, we have like a product

12:47that we call missions, which I'll also

12:48touch on in a sec. But, missions is

12:50basically an extremely elaborate harness

12:53built around the concept of working on

12:55extremely difficult knowledge work

12:57problems that are validatable, right?

13:00And so, the quality of the output of

13:02these very long-running harnesses of

13:04advanced agents is directly proportional

13:07to the degree to which you can validate

13:09their work. And so, if you introduce the

13:11ability to validate at scale, then you

13:13introduce increasing autonomy to the

13:15org. So, what we'll look at is we have

13:18tools that help scan all of these

13:19things, but often times, uh, the change

13:22is not so simple. Uh, for I'd say maybe

13:2430 to 40% of the low-hanging fruit, you

13:26click droid, please fix all of this and

13:28it'll go in and it'll fix it, right? But

13:30for the other 60% some of them involve

13:32workflow changes. Sometimes humans are

13:34not used to the degree of I would say

13:37like nitpickiness of these automated

13:39systems. And so you have to sort of be

13:41aware of the

13:42concerns, the the humans, you have to

13:44think about like the way that people are

13:46currently developing systems and say,

13:48"How do we introduce some of these more

13:50extreme validation strategies without

13:52interrupting the dev flow of the humans

13:54who are involved in the work?"

13:57Um and and I mentioned missions because

13:59really I think this is one of the more

14:01end game of the agent era at least,

14:04pre-software factory era. But the more

14:07end game of the agent era style

14:09harnesses where it's simply a long

14:12running harness that has almost no human

14:14intervention except for the planning

14:15stage, right? Where you go in and you

14:17say, "I would like to have this very

14:18bounded task. I know that I want to

14:21solve this task and here is what solving

14:23this task means. I will now basically

14:26push a lever of inference until the task

14:28is complete, right? And so

14:30that is actually unbelievably competent

14:33at solving problems where like is

14:36complete is verifiable. So if you can

14:39frame any problem as the set of

14:41verification

14:43uh systems that need to validate it,

14:45then you can solve that problem with AI

14:47today. Uh and we've seen this work on

14:50some pretty insane problem spaces like

14:53migrating, you know, 30, 40, 50 million

14:55plus line code bases uh fully

14:57autonomously, um working on advanced uh

15:01like deep learning strategies around

15:03biomed, health care uh sort of problems,

15:06uh financial institutions that optimize

15:09equity research where you can actually

15:11build models of different equities and

15:14sort of analyze and compare and and

15:16build sort of a system that can then

15:18back prop and or trade on top of the

15:21those equities. Um like it it's

15:23mind-blowing to me every day what I what

15:25I hear people are using with these

15:26tools, but it is not something that you

15:29can just download, install, and hit

15:31play, right? It does require agent

15:33readiness. So, if your code base isn't

15:35agent ready, you won't see any of the

15:37success of the most capable AI systems

15:39in the world today, right? So, this is

15:41why we want people to go in and help our

15:43customers and say, "Hey, look, you can

15:45solve this actually very difficult

15:47problem, but it is going to require a

15:49different form of investment than you

15:51were thinking. Less so solving the

15:53problem, more so preparing the

15:55environment for verification of the

15:56problem."

15:58And by the way, if you're familiar with

15:59how these models are actually trained,

16:01like this makes total sense, right? They

16:03they get dense reward when they get post

16:05trained on all these complex tasks.

16:07Models need dense reward. These

16:09verification signals form the basis of

16:11that reward that they use to keep them

16:13on track over a long-term goal-directed

16:15problem.

16:17Um

16:18So,

16:19I sort of mentioned this earlier, but

16:20but I think that the the core goal for

16:22us really is to say, if we can hand over

16:25a model to you of how this should this

16:28transformation should go, then we should

16:30theoretically be able to say, "Let's do

16:32this in a couple of different places,

16:34and then let your team actually scale

16:36this out across the company."

16:38I always use the analogy of if you're

16:40familiar with Walt Disney's Epcot,

16:42the the theme park. Like basically that

16:45theme park was created originally Disney

16:48wanted to create like a master planned

16:51exemplary city. He said, "Look, if I can

16:54create a city that is the future city,

16:57then I can use that as a model to the

16:59rest of the world cities, and they can

17:00develop entirely new forms of

17:02transportation and flourishing." And it

17:05became a theme park. But, what's

17:06interesting is that in that small

17:08example, a lot of other cities actually

17:10did cite some of the ideas that he was

17:13writing down and sharing about what like

17:15centralized urban transit should look

17:16like. And now you have like some more

17:19contemporary cities built in the last 50

17:21years that basically modeled after that

17:23toy example. Um what we want to do is we

17:26want to make sure that we get some of

17:28that lesson that if you have a working

17:30example of a city of the future, of a

17:32code base of the future, um people are

17:35smart. They're clever. Humans will look

17:37at that and they'll say, "Man, that's

17:38really cool. Let's bring that to my part

17:40of the code base, right?"

17:42But if you build too much of an advanced

17:45example, then people will say, "That's a

17:47theme park. That is not at all how the

17:49rest of the world works. I just can't

17:51see how that would apply to the way that

17:53we currently work today, right?" So it's

17:54kind of a delicate balance that you have

17:56to walk of building something that

17:58demonstrates the future is achievable

18:00enough, but ultimately does not scare

18:03away uh an org who is thinking, "Man,

18:06what is going to be the cost of

18:07transforming at this pace, right?" Um

18:11I always think about that quote, you

18:12know, the the future is here, it's just

18:13not evenly distributed. Um there are

18:15some code bases, and I I say code bases,

18:18not even companies, that are truly

18:20remarkable. They are effectively uh

18:22beginning to run on autopilot. Uh we

18:24ourselves have roughly 15 to 20% of what

18:27we call like autonomy, and our autonomy

18:30ratio is like in the upper 80%, which

18:33means the ratio of actions done by

18:34humans to AI systems before

18:36interruption, right? So our own code

18:38base is fairly agent-ready, pretty

18:40autonomous, um but uh the code bases of

18:43some of our customers are actually even

18:45more autonomous because they operate in

18:47more constrained uh ways, right? So it's

18:49it's sort of like a uh it is not obvious

18:52like who gets 100% autonomy first. I

18:55would argue it's probably very contained

18:57internal tools. Like we have something

18:59we call like legal droid, which is our

19:01legal workflow. That is effectively 100%

19:03autonomously maintained, but our like

19:06core harness, uh we do not yet have

19:08validators that can validate some of the

19:10hard visual problems of a like terminal

19:13based harness. Uh Things like flickering

19:15are really hard to catch in a verif- in

19:18a verifiable way. So, we're unable to

19:20close the loop on some of those

19:21challenges. It's an engineering task to

19:23build the system that can verify some of

19:26those very hard problems. And that might

19:29give you a picture into sort of like the

19:30weird world of the future where humans

19:32are sort of visually our advantages in

19:34being visual, our advantages in having

19:36context of the outside world provide us

19:39a lot of work to do in order to build

19:41these systems. So, who's great at this?

19:44I If you are a former founder, for sure

19:46you should do this. I think it's like

19:48the quickest way to basically build out

19:51I mean each like stage of the SDLC that

19:54Droid has, we think is a billion-dollar

19:57business. Like just code review, just

19:59incident response, just QA, just

20:01testing. Like each of these

20:03you will help define basically the

20:05nature of these products.

20:07If you are someone who is used to tech

20:10communication, right? If you are fluent

20:12in AI, you understand how to speak to

20:15every level, you have business acumen,

20:17you have executive presence, that is

20:19another great example of someone who

20:20should do this. And if you are a systems

20:22thinker, if you love designing systems,

20:24if you love closing loops, modeling

20:26data, and understanding how the flow

20:28through a potentially extremely complex

20:30org should look, then you are also

20:32someone who would thrive at doing this.

20:35So, if all of this seems interesting,

20:38hopefully it does. Please do reach out.

20:42And you can reach out to me directly.

20:44I'm just

20:45Yeah, I'll say it. It's on the slide.

20:47I'm eno@factory.ai.

20:49And so, you can just email me directly

20:52or you can apply on our careers page.

20:53It's called engineer, {comma} deployed.

20:57So, that's the role. Hopefully this is

20:59interesting and gives you a taste of

21:01what we're doing at Factory.

21:19>> [music]

21:20>> Woo!