Full transcript
0:14All right, I get I have the green light
0:16so we can get it started. Hello
0:19everyone. You might have noticed that
0:21the the titles changed a bit from the
0:23one that we have in on the schedule.
0:25That's because when I submitted the
0:27talk,
0:28there's the MCP versus skill debate was
0:30still going on was a topic. I think now
0:32we settled on they're both different.
0:35They all
0:36have their own roles and now the I think
0:38the debate is more on the MCP versus
0:40CLI. So I thought it would be more
0:41useful to come and explain how we've
0:46wrote our Superbase skill
0:48and the lessons that we got from writing
0:50the this document cuz I've never spent
0:53more time writing a single document
0:55since I've wrote my master thesis, okay?
0:58So I know that writing a skill sounds
1:00simple, but it can be very complex
1:02especially when you have complex product
1:04like Superbase.
1:07For starters,
1:09I'm Pedro. I'm an AI tooling engineer at
1:12Superbase. I'm an MCP enthusiast AI in
1:15general.
1:16Feel free to connect me on LinkedIn and
1:18I'm also a co-founder of the Lisbon AI
1:21Week in Lisbon. We'll be on
1:24late October this year. And if we're
1:27talking about me at the moment, I
1:29usually prefer
1:31doing this on
1:33dark darker
1:34darker mode. I don't know how many of
1:36you prefer dark mode over light mode.
1:39The majority. I thought so. So let's do
1:42this presentation on dark mode instead.
1:45So we I think we can all agree that
1:48agents are already smart enough, right?
1:51They can
1:52do very they're very capable of doing
1:54mundane tasks by themselves. But
1:58when you present a task about something
2:00that they've haven't seen yet or you've
2:02updated since they were trained like
2:04your product for example, they need the
2:06right guidance.
2:08For for example, Superbase we noticed
2:10that they would usually either miss some
2:13some security pitfalls that we have like
2:15row level security instructions that
2:18they have to set to
2:20to basically not expose your app.
2:23They could just usually operate on stale
2:26knowledge on their training data and
2:28they're very lazy to and very stubborn
2:31to admit that they don't know and they
2:33do need to find fresh information.
2:36And also
2:37we would like to guide them on specific
2:40workflows that we think are the most
2:41optimized for agents
2:44on our products. So for starters, how
2:46many of you know or have written a skill
2:49before?
2:51Okay, so what I'm going to say is
2:52probably not new for the most of you,
2:55but
2:55just to get an introduction on skills.
2:57So skills
2:59are folders containing instructions,
3:01scripts and resources that agents
3:03discover, right?
3:05Progressively they discover. This is the
3:06main selling point of skills and they
3:09they have this envelope called front
3:11matter where they have the name and the
3:12description. This is how the agent is
3:13going to decide when to load the skill.
3:16Then they have the actual instructions
3:17inside the file the main file called
3:19skill.md and then optional bundled
3:22resources like scripts to perform
3:24actions or reference files that the main
3:28file can reference for more information
3:31that doesn't have to be loaded
3:32immediately to context.
3:35So
3:37we tested out at Superbase we
3:39experienced giving the same agent
3:42in this case was Claude Sonnet 4.6
3:46the same prompt for a simple task. Like
3:48we had an app like a collaboration
3:50collaborative app and we wanted to
3:52create this new SQL view, right? On top
3:55of table that already had row level
3:57security enabled. So the users could
3:59only see the information that belonged
4:02to them.
4:03We gave we gave them just the the MCP on
4:08one condition and the MCP plus the the
4:12agents the agent skills. And the result
4:15was well the expected.
4:17If you don't know in Postgres, if you
4:19create a skill over
4:21a table on top of a table that has RLS
4:24enabled, if you don't explicitly pass
4:27that
4:28that flag over there the security
4:30invoker equals true, it will bypass the
4:32RLS. So basically the view will expose
4:35data that is not this exposed by default
4:38on the
4:39on the table. So the agent with the
4:42skill with the knowledge was able to get
4:44this
4:45this information implemented correctly
4:47and safely while the one that only had
4:50access to the integration to the
4:53to the MCP tool did not. So
4:57for this we decided well just like this
5:00we wanted to enable agents to know how
5:03to work correctly with
5:06with Superbase. So we decided to we're
5:08announced actually announcing today
5:11this
5:12this Superbase agent skill that I've
5:14been working on on this couple couple
5:16past of months and to make things
5:18official
5:19I'm actually going to try something.
5:22Wait.
5:23I'm going to
5:25live tweet it on stage. So
5:28it's live.
5:36All right. So
5:38what exactly is this skill about and
5:41what lessons can I share with you? So if
5:43you're building a skill for your
5:44product, you can build one like we did
5:47and happy to discuss the all details
5:49like this is free text text and we
5:51haven't achieved a a standard for it. So
5:54happy to to chat about it later.
5:56So to break down on some principles that
5:58we
6:00converged to. The first one is that
6:02don't duplicate information. Yeah, like
6:05treat skills as a document documentation
6:08for yourself and you will not
6:10duplicate your documentation, right? You
6:12already have documentation on your
6:13product. Just point the agent to it to
6:16the most up-to-date. You have to be very
6:18stubborn with the model to go to ask
6:21them to go
6:23search the the web or your
6:25documentation. Provide the guidance.
6:28Tell where to and how to find the
6:30documentation, but be very persistent
6:33with it to go for it.
6:36We also did a run a little experiments
6:39and this is still
6:41well as I said
6:43an experiment and yet to be
6:47it's
6:49we're still figuring out the details,
6:50but we also announced quite recently and
6:52you can see it can read it on our blog.
6:55We're we basically exposing our
6:57documentation through SSH. The main
7:00reason behind it is that the agents can
7:02now look for the documentation like it
7:04was a file system. So they're very
7:06familiar with file systems in general
7:09are navigating them finding files and
7:11information on
7:12using Linux based
7:14tools. If we expose if you give them
7:17this interface to also do this but
7:19remotely,
7:21our premise is that they will have easy
7:24ease to to navigate the the agent. So
7:27would also would love after the talk or
7:28during the the conference to see and to
7:31hear your opinions on this idea.
7:34The second principle that I get that I
7:36get I have for you is that if something
7:40can get skipped
7:42it will be skipped. What I mean by this
7:44is that besides newer information on
7:48searching online, so agents like
7:51fetching information online or tool
7:53calling it's expensive for for agents.
7:55So they mostly
7:56default to their training data. The same
7:59is true for reference files. We've
8:02noticed that
8:04even if when the the agent loaded the
8:07skill,
8:08it will even if it had reference files
8:11there,
8:12it will be very lazy to load them. And
8:14even if it loaded one reference file,
8:17if your problem requires more than one
8:20the information that it's more than in
8:22more than one file,
8:24it will most likely will not load two
8:26files, right? It's almost impossible and
8:27not not even starting on three or four.
8:30So
8:32you have to be
8:34very critical about what you put on your
8:36skill.md file from the beginning. You
8:38put information that is likely not to
8:40change like in our case a security
8:42checklist about Superbase
8:44that we didn't really wanted the agent
8:47to miss at all. So we decided this
8:50cannot be on a reference file. We
8:51actually started by putting on a
8:53reference file and he usually missed it.
8:55So we put it on the skill itself. So if
8:57you have any
8:59information that the agent can just not
9:02miss and like defines your product, go
9:06to the skill.md file. Do not afford it
9:08to put on on a reference file.
9:10And lastly,
9:12the third principle that I have for you
9:14when writing a product skill is to be
9:17opinionated. You know your product the
9:19best. You know how to work with it. You
9:21know how your users or you should know
9:23how your users are using it. Don't be
9:25afraid of guiding the agents on what on
9:29workflows that you think are the most
9:32effective when working with your
9:34product. In our case for example,
9:37managing a database schema, right? I
9:39have actually haven't asked this and
9:41should have in the the beginning. How
9:42many of you know Superbase and what
9:44Superbase does?
9:46Okay, for So for the ones who who don't
9:49know, we're basically a backend as a
9:51service. We provide a storage database,
9:54authentication and other uh features
9:56that you would need to create a back end
9:59out of the box. So, we provide as I said
10:02we provide a database and the agent can
10:04interact and manipulate your your
10:06schema, right?
10:07We found that this for our platform was
10:11the best workflow for the agents to
10:13efficiently manage the schema. So, in
10:16this case run uh
10:18direct DDL operations like change the
10:21schema freely on your um
10:23development or staging database. Once
10:26you're happy about it, we provide an
10:29advisor to our basically linked to give
10:32any
10:33uh security or performance issues that
10:35the database could have, fix them,
10:38and only then create the migration file.
10:41So, this will prevent the agent to
10:43create a migration file every time he
10:45changes the schema. We found this
10:48to be the best um
10:50workflow to manage the schema. So, for
10:52us it should be in the
10:55um in the skill when working with with
10:57Superbase.
10:59How we tested this skill? So, we're
11:00living
11:01a very interesting times where we now
11:03can test free tests. We we can now test
11:07um documents. We can now test
11:09documentation, right? Uh this would be
11:12uh completely bonkers to think uh years
11:15ago. Now, I've basically been testing a
11:17markdown file.
11:19And how did we do this? Uh through
11:21evals. Uh so, for those of you who don't
11:22know evals, evaluations for a short for
11:25evaluations, are basically uh tests that
11:28you can run
11:29mostly like you would run uh on your CI,
11:33uh but now instead of evaluating code
11:35you're evaluating an agent and LLM and
11:37its behavior, what tools it's it's
11:39calling, what's the it's reasoning,
11:41right? And so, we ran on an early stage
11:44we ran a set of six specific scenarios
11:47for Superbase. So, ongoing Superbase
11:49projects in different uh
11:52in different scenarios, right? And we
11:54ran it against four different uh agents
11:57from two vendors
11:59um in three test conditions. So, we
12:00wanted to test on a baseline, so no MCP,
12:03no skills, with just the MCP server, and
12:07with the MCP server plus the skill. All
12:09this was done based on a test
12:12completeness score
12:13four graded score uh
12:15the run on Braintrust. If you don't know
12:17them, they're around they're sponsors of
12:19this uh conference. Go um go to their
12:22booth and and talk about it. It's a very
12:24cool product. So, the results are uh are
12:27out here.
12:28Um
12:29the skills plus the MCP uh outperform
12:32any other conditions on every model that
12:35we we tested. We tested on on Claude
12:37Code for Opus 4.6 and Sonnet 4.6 and
12:40also on Codex for GPT 5.4 and GPT 5.4
12:45mini.
12:46Uh so,
12:47I think we can
12:49conclude that it's pretty unanimous that
12:52the skills were actually improving the
12:54performance the test completeness score
12:57uh because they were providing the right
12:58guidance to to the agent. So, we had
13:00already the tools. We already had an MCP
13:01server. We just needed the right
13:03guidance so on how to operate with with
13:06Superbase. Uh it's agent agnostic. Uh
13:09most more and more agents are adopting
13:12this open standard on on skills. Um and
13:16currently as I said in the beginning the
13:19bottom line is not the the context, it's
13:22the guidance.
13:24So, if you can take something from this
13:26talk uh when building a skill for your
13:29product is that points for your single
13:32source of truth.
13:34Point to your documentation basically.
13:36Be opinionated. You know your product.
13:38Don't be afraid to to show it. And start
13:41minimal. Any model vendor, let's say
13:44whether it's that's Anthropic, OpenAI,
13:46on any blog post about skills, you will
13:49read start uh start minimal, start slow,
13:53and then iterate, expand, don't be
13:56afraid to create new versions for it.
14:00So, if you want to know more about it,
14:02we
14:03we actually wrote a a blog post. Uh it's
14:06live today. You can check it on my
14:07Twitter account on on Superbase um
14:11blog
14:12uh or you can just run this uh this
14:15commands and install your
14:17um
14:18install it on your project and start to
14:20to to to use them now.
14:22That's all. I'll be around and once
14:24again, thank you very much.
14:32I do have one more thing to show you.
14:34So, we're running a a giveaway. If you
14:38want to to have a chance to win a Mac
14:39mini, uh just scan the the QR code,
14:43uh sign up for Superbase, and good luck.
14:49Do I have time for any questions if
14:51there's any?
14:54Yeah. Okay.
14:55I I just had one thing I wanted to ask.
14:57So, since we're all moving into ranks,
14:59right? I mean
15:00I'm just wondering how much of a demand
15:03do you guys see on vector databases
15:06Superbase right now?
15:07Hm.
15:08Uh it's a it's an interesting
15:10interesting question. So,
15:12uh
15:13I mean it depends the the you you're
15:15asking about like uh
15:17how many customers are
15:22have been more and more
15:24as the as the days pass. Like um
15:27the use cases for for vectors are mainly
15:29for embeddings, right? Yeah. And
15:32semantic
15:34uh there are many use cases for for
15:35embeddings. The one that I'm most
15:37interested about is uh
15:39semantic search, right? That you could
15:41use to provide even more context. Uh for
15:44example, through the SSH
15:46uh exposing the docs through the SSH,
15:49you can now instead of naively uh let
15:52them navigate with the the bash tools,
15:55you can augment the tools already known
15:58bash tools into all providing some sort
16:00of uh semantic search. So, I would do
16:03see a very big potential on vectors.
16:07On our data, definitely customers have
16:10been exploring more this uh
16:12this solution. Thank you for the
16:13question.
16:16Yeah.
16:17Thank you for the talk. Very
16:18interesting.
16:19How do you
16:20distribute the skills inside your
16:22organization?
16:25Do you just pass them? Do you have a
16:27repository? Do you have a repository or
16:28do you have
16:29some package manager?
16:31That's That's an amazing That's an
16:33amazing question because
16:35uh currently one of the downsides or the
16:37I would say the the constraints of using
16:40skills is their distribution
16:41distribution system, right? Uh we're
16:43still finding the there's still some
16:45players trying to reclaim
16:47uh the
16:48either the registry or the way to
16:50distribute it. So, Vercel came up with
16:51the skills package. Uh we're seeing
16:54plugins, right, that you can bundle with
16:56MCP servers and other things. Uh but
16:58they are model specific. So,
17:00this is to distribute in general, skills
17:03in general. It's already a a problem for
17:05itself that we haven't solved it yet.
17:07Um
17:08Internally, we have uh we are packaging
17:11the skills on the the repos themselves.
17:14So, if you want to
17:16uh create a plugin, you just uh create a
17:18dot Claude
17:20uh plugin or a dot cursor plugin or
17:22whatever in that repo, right?
17:24Uh and then it's available or or it's
17:27discoverable uh if the if the
17:30the repo it's it's open sourced or you
17:33have access to it, uh you can use the
17:35skills package to to fetch it.
17:37So, yeah, this and this is how I've seen
17:40the other uh companies
17:42distributing the skills like a repo, a
17:45skill, trying to package the the skills
17:47into the the knowledge.
17:49Thank you.
17:50Thank you.
17:52Are there any more? I think we have time
17:54for one more.
17:55Yeah.
17:57Yes. I recently built self-improving
17:59skills.
18:00Should we do a collab?
18:02Sure. Let's talk after.
18:05All right. So, once again, thank you
18:07very much. It was a pleasure to be here.
18:09I'll be around.
18:10Thank you.