Full transcript
0:00Hi everyone!
0:00My name is Anton and I'm a Developer Advocate at Chainstack.
0:03In today's video, we are going to figure out how Yellowstone gRPC
0:07Geyser plugin works and how to use it.
0:10Let's go.
0:11First, let's discuss the naming.
0:14Let's start from the end.
0:15Geyser means that some time ago validator nodes have been introduced
0:18to so-called Geyser plugins.
0:21This interface allows validators to decrease the workload and stream data
0:28to external storages or services.
0:30gRPC means that this Geyser plugin interface has been developed based on gRPC
0:37protocol and gRPC service, it means that the data is transmitted to gRPC service.
0:44Yellowstone here means that this concrete implementation has been
0:48delivered by Triton One team within the project Yellowstone.
0:53So this is where this long name came from.
0:58Now let's talk about gRPC protocol.
1:01From user perspective, the gRPC protocol means that you have to compile client
1:08code using so-called proto and then use that compiled code in your modules.
1:15Why we do that?
1:16Because gRPC by default uses protocol buffers as the
1:21interface definition language.
1:23These protocol buffers contain description of services, messages
1:29and other data structures which are used in the gRPC protocol.
1:35The advantage of this protocol is that you can compile that specification
1:41into your desired language.
1:43In our example, we'll use Python, but the list of supported languages is quite huge.
1:49repo with Yellowstone Geyser plugin, to get more
1:58before, for gRPC service, we need proto files with a description
2:05of service and messages.
2:08In case of Yellowstone, we can find those files in the folder yellowstone-grpc-proto
2:18and then proto subfolder.
2:22So this is the files which we need to download to our machine
2:28and then compile client code.
2:31Moving on to a practical example.
2:34In this example, we will listen to new tokens on pump dot fun.
2:40This repo will be published as a supporting material for this video.
2:44So no worries.
2:46You can either just clone it and run it or if you like diving into details, you
2:52can follow this tutorial step by step.
2:55So, the first step for this example will be installing dependencies.
3:01We need a few modules here and GRPC library for Python, proto compiler,
3:10modules for Solana , and a module for environment variables.
3:14Then as we discussed before, we need to download those proto
3:19files to generate client code.
3:21To keep our project clean, will create separate folders for proto
3:25files and for generated code.
3:29We'll download both files from GitHub, from the Yellowstone repo.
3:36And then, since we use uv, we'll run uv run to generate code
3:43based on downloaded proto files.
3:46After that, you will see in the generated folder, four files and,
3:52one note here, proto compiler by default applies absolute paths.
3:58Since we place our generated code into a subfolder, we need to make
4:05sure that those absolute paths are replaced with relative ones.
4:10So here you need to make sure that you have from dot import geyser pb
4:17two, and the same in the second file.
4:21So after this step, you should have downloaded proto files, generated
4:27client code, and we can proceed.
4:30Let's create our main script.
4:34Don't forget to put your credentials and endpoint to
4:38the environment variables file.
4:42First of all, we import required modules, load the environment
4:51variables, and assign our constants.
4:55Since we want to listen to new pump dot fun tokens, we need to have the address
5:03of pump dot fun program and to have the discriminator for a create instruction.
5:11just copy-paste it into the main dot py, and the next part of the
5:18script is creating Geyser connection.
5:21In the majority of cases, you will have x-token authentication type.
5:27So you can either use only this one or you can copy both parts of this function.
5:37Also, important thing is to have these keep alive options because gRPC
5:44channel can be closed or you can lose a connection without pinging a stream
5:50which doesn't produce a lot of events.
5:52The most interesting part in this script is here because this is the
5:58part where we define our subscription.
6:03So, to listen to new tokens, we have to listen to transactions
6:09which mention our pump program id.
6:14We don't need failed transactions since they don't create anything, and we want
6:20to get our results as soon as possible.
6:23That's why we have commitment level processed.
6:27To understand why we have this structure or not the other one, you can go to the
6:36geyser proto file and see this service description and messages structure.
6:43In this case, we use a subscribe method which accepts a subscribe
6:47request and produces subscribe update.
6:50So we can go to subscribe request message, right, and see that
6:56it has transactions argument.
7:00This is what we need in the request.
7:04So we use transactions.
7:08We name this subscription pump filter, and we use this account
7:14include argument of transactions.
7:17So this is the object, which we need to find here.
7:21Subscribe request filter transactions, and it has account include property, so
7:30here we use it, account include and we are appending our pump dot fun program.
7:38Also, failed false.
7:40You can see it here.
7:42Failed.
7:43False.
7:44You can see other types.
7:47We will look at them in our other examples.
7:51For now, it's enough to have these three parameters to get updates
7:58on new tokens as soon as possible.
8:03If you're familiar with our pump dot fun repository, then you already
8:06know how to parse those instructions.
8:10Here we have pretty much the same logic.
8:14We parse received bytes into structured data, so we can read and understand it.
8:22One more thing regarding those instructions.
8:25Since we listen to all transactions which mentioned pump dot fun
8:30program, we will receive not only new tokens, but everything including
8:36swaps and other transactions.
8:39So when we receive a new message from our channel, we need to filter
8:45out those instructions, which are not create instructions, and we
8:51can do that using the discriminator we defined at the top of our file.
8:58If you don't know how to use those discriminators to generate those
9:03discriminators, you can also check our pump dot fun bot repository, or
9:09you can check the learning examples in our repository for this tutorial.
9:16One of the files will contain the example for bonding curves.
9:21That's it.
9:22If you did everything right, you will have this file with all constants defined , a
9:29function for establishing connection, for creating a subscription for decoding
9:35a create instruction, and for monitoring the Solana blockchain for new tokens.
9:44Now let's run the script to check the results.
9:47uv run main.py.
9:52So the script has been launched, and we can see the results in the terminal
9:59all new tokens will be displayed here.
10:02So our listener is working fine.
10:07Let's stop it.
10:10In the learning examples of our repository, you can check other
10:14examples of how you can use different subscriptions available in the
10:21Yellowstone gRPC Geyser plugin.
10:23Let's check them now.
10:27So let's check the accounts subscription.
10:34In this example, we are listening to the updates of Raydium market address.
10:40This is Wrapped SOL RAY pool, and the core part of the subscription is here.
10:49In the subscriber request, we define which filter we would like to use, name
10:56it accounts filter, and then pass the Raydium market address to the account
11:05and processed commitment level.
11:11Let's run this file
11:17to check how many updates we'll see.
11:20In this example, we're getting account updates.
11:23So since this market is quite active, then you receive a lot
11:31of updates on that program.
11:40Okay, let's stop it.
11:41Let's check the advanced example.
11:44In the advanced example, we are listening to pump dot fun program
11:54and updates to all bonding curves, which are owned by that program.
12:01To do that, we're using the same accounts argument, the name of
12:07the filter, instead of account we use owner property and filters.
12:13In the filters, we specify the filter for account data.
12:18So we want to get updates only for bonding curves.
12:23That means that we will filter using bonding curve discriminator.
12:28In this example, you can check how that discriminator is calculated.
12:35Let's run it too.
12:38So as you can see, huge amount of updates coming in.
12:45This is because we are subscribed to all bonding curves owned
12:49by pump dot fun program.
12:54Let's stop it.
12:58Okay.
12:59Also, interesting examples for transactions.
13:04We already know how to use transactions subscription thanks to our main dot py
13:11example where we listen to new tokens.
13:17Let's check the advanced filter.
13:20In the advanced filter.
13:21We're listening to potentially arbitrage transactions.
13:25This can be achieved using account required.
13:29We specify here accounts which we want to see in each transaction.
13:36All these programs must be in the transaction in order
13:40to appear in our terminal.
13:44Let's run it.
13:48As you can see, a huge amount of transactions.
13:53Probably, if you want to work on it, you would add some additional filters
14:00to filter out noise from this channel.
14:04In our repository, you can find even more examples for other types of subscriptions.
14:10And one bonus is a more practical example with added parsing of
14:16instructions for Meteora dynamic liquidity market maker pool.
14:22Thank you very much for your time and attention.
14:24Feel free to submit your questions, concerns to our issue
14:29section to this repository.
14:31I hope this tutorial was useful for you and have a nice day and see you later.
14:37Bye.