Full transcript
Introduction
0:00Solana maintenance blazing speed and
0:02quick block time makes accessing
0:03realtime data both crucial and
0:06challenging. So choosing a right
0:08approach is essential for running
0:10reliable and scalable applications. In
0:13this video we will explore three
0:15effective approaches to access realtime
0:18Solana data each tailored for a
0:21different use case. So without any ado
0:24let's jump into it.
0:29Quicknote offers three approaches for
0:30accessing real-time Salana blockchain
0:32data. The first one is websockets where
0:36you can create websocket subscriptions.
0:38Second one is Yellowstone gRPC and the
0:41third one is quicknode streams. So
What is Solana WebSocket, and how do you use it?
0:44websockets is the traditional and most
0:46straightforward way to get real-time
0:49data from the Salana blockchain where
0:50what you do is you establish a
0:52birectional connection with the Salana
0:55blockchain via a node and you get the
0:58data back. So what happens is you send a
1:00request via websocket. The node gives
1:02you back a handshake and then a
1:04birectional connection is set up which
1:07is a TCP connection. So you can set up a
1:10websocket for getting blog data, for
1:12getting slot data, account updates,
1:14events etc. And whenever that occurs or
1:18whenever there is a update on the
1:20information for whatever you are
1:21subscribing for, you get the data back
1:24via this birectional connection you have
1:26established with websockets. So to get
1:28started with websockets you need to go
1:30to your quick node dashboard create a
1:32Salana endpoint and once it's created go
1:34to the Salana endpoint and copy the
1:37websocket provider URL. So how it works
1:40is first you need to establish the
1:42connection with the websocket and then
1:44you need to put a filter for whatever
1:46change you want to create the websocket
1:49connection for. Over here we are
1:51establishing a websocket connection for
1:53onslaught change websocket method. So if
1:57we run
1:58it now what happened is the connection
2:01got established and whenever there is a
2:03new slot we will get the data back from
2:06the websocket as you are seeing on the
2:08screen. Now the following are the
2:09considerations or characteristics of
2:12websockets. So the setup complexity of
2:14websockets is low because it has native
2:17Solana JSON RPC support. The infra
2:20management for websocket is self-managed
2:23because you'll have to manage the
2:25websocket connections. When it comes to
2:27historical data, websocket connections
2:29or websocket endpoints themselves do not
2:32have historical data support. But you
2:35can get historical data via a HTTP RPC
2:39endpoint by sending HTTP RPC requests.
2:42Then the latency on websockets is low.
2:46Data transformation in a websocket setup
2:48happens on the client side where you get
2:50the data from the blockchain via
2:52websocket and you will have to do the
2:54transformation on your end which is the
2:57client side and for websockets it's a
3:00single connection destination where you
3:03can just set up a websocket a particular
3:05websocket for a single destination.
3:08Reliability of websockets is not that
3:11great because websockets tend to
3:15disconnect a lot and you will have to
3:18manage manually in your script the
3:20reconnection of those websockets with
3:22the blockchain. When talking about
3:24suitable use cases, websockets are most
3:26suitable for development and web based
3:30applications. And now let's talk about
3:32gRPC which is available via Yellowstone
3:36gRPC endpoint on quick node. So the
What is Solana Yellowstone gRPC, and how do you use it?
3:39Yellowstone gRPC API uses the Geyser
3:41plug-in to provide enterprisegrade data
3:44and enterprisegrade volume of data
3:47without burdening the validators. So in
3:50the Yellowstone gRPC API, what needs to
3:52happen is there needs to be a Geyser
3:54plug-in framework setup on the validator
3:56side plus on the client side or on the
3:59customer side, you need to make a setup
4:01to handle the gRPC request. So what
4:04happens is you set a request or you set
4:08up a gRPC connection to the validator
4:11with the Geyser plug-in framework. For
4:12example, let's say you want to subscribe
4:14for a particular account updates. So you
4:18send a request and whenever there is an
4:21upgrade to that account, the Yellowstone
4:24gRPC API will keep on streaming that
4:26data to you. So this setup has very much
4:28low latency compared to all the other
4:30setups and approaches. So this is very
4:33much suitable for high frequency
4:35tradings. Now to get started with
4:37Yellowstone gRPC API what you need to do
4:39is you need to again go to your quick
4:41node endpoint go to the add-on section
4:45look for the Yellowstone gRPC
4:47add-on so to install that what you will
4:50need to do is you'll need to click on
4:52explore as you can see over here and
4:55then just click on install and once
4:58installed you will be able to find it
5:00under the installed add-ons of your
5:04endpoint in the add-ons tab. Setup for
5:07Yellowstone GRPC looks like this where
5:09you will first have to provide your
5:10endpoint URL from quick node and then
5:12your token of your endpoint URL and then
5:16you will have to initialize the function
5:19where you will have to initialize the
5:21client with the URL and token. Then you
5:24will need to set up your data handlers
5:28on your side where you will need to
5:30handle the incoming data. You will need
5:32to create the subscription request, send
5:34the subscription request to the gRPC API
5:37and then you will need some functions to
5:40handle and transform the data on your
5:43end. So if we run
5:48this as you can see the subscription has
5:50been established and we are getting
5:52realtime Salana slot data as the slots
5:56are coming in or being created on the
5:58blockchain. And you can see the speed is
6:02so fast because uh this is the fastest
6:05way with lowest latency to get the data
6:08from the Solana blockchain. Now let's
6:10talk about Yellowstone gpc API's
6:12considerations and characteristics. So
6:15the setup complexity of Yellowstone gRPC
6:18API which uses Kaiser plug-in is medium
6:21because it requires gRPC setup on the
6:25client side where you need to handle the
6:27data plus it also requires additional
6:30GIS plug-in on the node side or the
6:32validator side. The infra management
6:35part is uh self-managed in terms of
6:38historical data. Yellowstone gRPC API
6:40cannot get you historical data in terms
6:44of latency. It's the lowest data
6:47transformation needs to happen on the
6:50customer server side via gRPC. As we saw
6:54that we need to handle the data coming
6:57in from the gRPC API. So that needs to
6:59be taken care by the customer and
7:03destination it's a single end point. So
7:06you get single destination. Then in
7:10terms of reliability, it has some
7:12builtin reconnection logic so that you
7:16don't have to manually handle the
7:18reconnection every time and it's best
7:21suited for high frequency dating systems
7:25or custom indexers and in cases where
7:29you want or need direct validator
7:32access. Now let's talk about quicknote
7:36streams which helps you build your data
7:40pipelines all on quicknote platform. You
7:43do not need to set up anything. So
What is QuickNode Streams, and how do you use it?
7:45Quicknote streams is a new kind of tool
7:48using which you can set up your entire
7:51data pipeline on just one platform. So
7:54as you can see in this diagram, Vignno
7:56stream can stream to multiple
7:59destinations using the HTTP protocol and
8:03everything from data handling, data
8:07transformation, sending the data,
8:09retries happens on the Quicknote
8:11platform itself. Now let's see how we
8:13can set it up in the Quicknote platform.
8:15So once you're logged into your
8:17Quicknote dashboard, you'll need to go
8:18to streams, then click on create
8:20streams. Then of course you'll need to
8:23select Salana Salana
8:27mainet and then what you can do is you
8:30can add a starting block and a end
8:34block. You can also set a block delays
8:36and reorg handling. Then click on next.
8:39Then for Salana you can choose from two
8:43data sets which is block and block with
8:46locks. So streams also lets you do
8:50backfilling of historical data. So if
8:53you're doing that you can enter batch
8:56sizes. For example, let's say that you
8:58want five blocks to be delivered to you
8:59at one time. You can do that. You can do
9:01up till 10 for Salana. So since we will
9:04be getting latest blocks, we will have
9:07this unchecked because it's irrelevant
9:09for latest blocks. Then this is how the
9:12raw payload would look like. But the
9:15best part with streams is that you can
9:18customize the payload on the quick node
9:21server side. You can also see some
9:24templated streams filters over here. So
9:27the customization which you do on the
9:29stream side is via filters. You can add
9:31a filter on your stream and get the
9:34transform fully
9:35customized payload on your end or on
9:38your destination's end. So you can do
9:42pump.fund fund trades, new tokens,
9:45filter transactions by addresses and
9:47programs, tracks, solve token transfers.
9:50The possibilities are endless. You can
9:52basically filter anything. You just have
9:53to write a filter code and you will get
9:55the filter data on your end. So if you
9:57want to write your own filter code, you
9:59will be able to do that as well where
10:01you can write your own filter logic and
10:03run the test and see the data on your
10:07end. So if you save and close click on
10:10next then you can set up a destination.
10:14So you can set up S3 bucket you can set
10:16up Postgress and snowflake of course a
10:20web hook as well but you need to be on a
10:22specific plan for those for now I'll
10:24just set up web hook. So I'll just get a
10:26temporary web hook from this website and
10:29send the data over there. You can also
10:33check the connection where it should
10:35send a ping to the web hook. as you can
10:38see here. And you can also send a
10:42payload example on your destination site
10:45so that you can be prepared or so that
10:49you can be aware beforehand what your
10:52payload would look like. So this is how
10:54our payload would look like. Now if we
10:58create the stream, it should send a
11:01payload to the web hook every time a new
11:04block is created. So as you can see we
11:06have already started getting new blocks.
11:09So this is how streams work. Now let's
11:11look at some considerations and
11:13characteristics for quicknote streams.
11:15So in terms of setup complexity, it's
11:17very low as you have UI based
11:19configuration. You can also set up
11:21streams via rest API but there is a UI
11:24as well. In terms of infra management,
11:26it's fully managed by quick node. So you
11:28do not need to worry about anything.
11:30Streams does have support for historical
11:31data because it has builtin backfilling
11:34support. In terms of latency, it's low
11:36to medium. For data transformation, it
11:39occurs on QuickNote server side via
11:41filters. As we saw that we can transform
11:43data on Quicknotes server side using
11:46filters. So you do not need to worry
11:48about transforming the data on your end
11:50or client side. In terms of
11:52destinations, you can set up multiple
11:54destinations like web hook, S3 bucket,
11:56Postgre, SQL, Snowflake.
11:59Then in terms of reliability, it's uh
12:02guaranteed delivery with automatic
12:05retries. You do not need to configure
12:07your own retry logic or you do not need
12:09to worry about dropping ping connections
12:12because the data will be delivered to
12:14you guaranteed and you can also set up
12:16number of retries. Stream is best
12:18suitable for anyone who is trying to
12:20build a data pipeline for their project.
12:24For example, analytic platforms, block
12:26explorers, anyone who needs data at a
12:29very large scale or who need to index
12:32data. Then anyone who has multi-
12:34destination needs, they can use streams
12:36as well. So basically streams can be
12:38used for multiple use cases without the
12:40need for building your own data
Outro
12:42pipelines. So these were the three
12:43approaches using which you can get
12:45realtime Salana blockchain data. Hope
12:48you learned something from this video.
12:49Give this video a thumbs up. Subscribe
12:51to the BookNA YouTube channel and I'll
12:53see you in the next one. Bye-bye.