Full transcript
0:15All righty, let's dive into it.
0:17So, uh in the previous session, I I
0:19don't know how many of you were here,
0:20but I talked about how MCP gives access
0:23to LLMs to websites that are behind uh
0:26CAPTCHA, bot detection systems, and so
0:28on.
0:30And in this session, I want to talk
0:32about um how do you actually collect
0:34data on scales with LLM? Because a lot
0:37of times I'm seeing on Reddit and other
0:38social media is like, "Oh, I need to
0:40scan 10,000 products, but that's so many
0:41tokens if I need to parse everything
0:43with LLM." So, obviously, you don't do
0:45that.
0:46Right? So,
0:48the whole session is how do you build
0:51pipelines, right? With LLM. Instead of
0:53telling, "Hey, LLM, can you go and uh
0:55parse this for me?" build a scraper
0:57that's going to parse it for me, and I
0:58will demonstrate how easy it is
1:01uh with our skills, right? So, Bright
1:03Data has skill sets that actually
1:05teaches your LLM on how to build a
1:07pipeline. With our MCP, it can actually
1:10extract HTML so that it knows what are
1:12the selectors that it needs to parse,
1:14and so on.
1:16So, um
1:17the scrape tags, right? Write the
1:19scraper.
1:20Uh I don't know how many of you actually
1:22wrote scrapers before.
1:25Okay, so you know the headache when all
1:26of a sudden things are missing, data is
1:28missing, you got to wake up. I don't
1:30know if you guys did it for clients, and
1:31clients are like, "Oh my god, there's no
1:33So, basically, that's what used to be.
1:35You write a scraper, and you maintain
1:37it. As a matter of fact, sometimes you
1:38maintain it more than it it it takes you
1:40to write it, right? So, everything,
1:41especially if the website is constantly
1:43changing
1:44uh its selectors, or if it's a React
1:45website, it gets really complicated,
1:47right?
1:49So,
1:50an agent it it it solves all that
1:53headache, right? It can explore with our
1:55MCP, it understands what the data is
1:57needed. It writes the scraper and it
1:59actually runs it and executes it and
2:01maintains it. As a matter of fact, I do
2:03collections on the daily basis and I
2:05have every 30 minutes I have an LLM
2:07class pool spools up. It checks what the
2:10data is collected, make sure that
2:11everything is fine. Everything is fine,
2:13it shuts down. If for this example,
2:15there's a
2:16always set a validation for data, right?
2:18Let's say the data point is uh missing
2:20some something.
2:22Your agent fix it. 5 minutes. You don't
2:24have to wake up in the middle of the
2:25night.
2:27Uh
2:29So, let me demonstrate. So, for this
2:31demo, I want to use Cloud Code. I don't
2:33know how many of you use Cloud. If
2:35you're using Codex or whatever it is, I
2:36prefer Cloud Code. I like how it
2:40I like it. It does a great job.
2:42So, the way that you build scrapers now
2:45and it's ridiculous because, you know,
2:47back in the day it took weeks to set it
2:48up. We created a Bright Data like GitHub
2:51page, right? Bright Data skills. Here
2:53basically, your agent has everything
2:55that it might need, okay? Um scrape
2:59builders back practices, how to
3:01everything. Everything that it needs in
3:03order to build a scraper.
3:05And what I want to show is
3:07uh first of all,
3:09let's
3:11I'm going to start simple. Go to
3:15build me a scraper and I wanted to
3:17record it because I thought it's going
3:18to take a long time, but now it takes a
3:20little like 3 minutes. Build me a
3:21scraper
3:23uh two inputs
3:25keyword search
3:27max pages for walmart.com.
3:32Everybody's familiar with Walmart.
3:33Walmart has a very aggressive anti-bot
3:35systems. If anybody tried to scrape it,
3:37um it just doesn't work. But, through
3:40Bright Data,
3:48it will figure it out, but you're you're
3:51all right. Let's just correct that.
3:54walmart.com
3:56Um
3:59run a search for headphones, collect
4:03three pages.
4:08Uh so, what it's going to do right now
4:10is going to go to our GitHub, it's going
4:11to get all the skill set that it needs,
4:13how to build scrapers, then it's going
4:14to go uh with the MCP that's already
4:16connected to the cloud, it's going to
4:18extract the HTML from the page.
4:21Uh it's going to find all the selectors
4:23that it needs, and it's going to build
4:24you a scraper, and uh
4:29So, right now you can see a scraper as a
4:30markdown, it's basically extracting the
4:32the the the text, right? So, for the
4:35skills, we don't need the HTML, we just
4:36need the text.
4:38Uh scraper as a markdown is a part of
4:40the MCP of Bright Data MCP.
4:42Uh the MCP has 5,000 requests for free,
4:45so if anybody wants to try it, you guys
4:47can try it for free. Um
4:50You will need to just open an account
4:51with Bright Data, which doesn't cost you
4:53anything.
4:54And
4:57let it do its thing. So,
5:01why is this better than having an LLM
5:04parse every single page, every HTML?
5:07It's I'm I'm literally afterwards I'm
5:09going to ask it to tell you how many how
5:11much tokens it saves.
5:12So, for the three pages, I've done this
5:14before, it saves about a million tokens
5:16just to from building the scraper
5:19uh and using a script to parse the HTML.
5:23And um
5:25even if you are not doing any scraping
5:28for production, but let's say that you
5:29want to find the best best headphones
5:32for the money, right? It used to be you
5:34go to Google, you go to some CNET where
5:36they uh compare everything, but here you
5:38can use the the marketplace reviews.
5:42So, I can tell it here like hey hey, um
5:45scan the five pages, find me the best uh
5:47review to headphones. So, it could be
5:49even useful for you as a personal in in
5:51a personal way, right? So, instead of it
5:54getting blocked, it can actually find
5:56you things that you need on websites
5:57that are behind Cloudflare.
6:00Oh, I forgot to delete the old one.
6:04No. No, stop. Stop. Stop. Hold on a
6:07second.
6:09Delete the old scraper.
6:12Scraper.
6:14I totally I was testing it and I totally
6:16forgot to clean up.
6:19Uh, let's do a different website. Pick a
6:21website. What's a popular marketplace in
6:23uh that's aggressive in blocking in UK?
6:28>> Very.
6:28>> Amazon? It's not that aggressive. You'll
6:31be surprised. I can scrape it with data
6:32center IPs.
6:34What's a popular website that everybody
6:36uses here?
6:36>> Very.
6:38Very.
6:39>> V-
6:39>> v e r y.com.
6:43>> V-
6:46Very?
6:47>> Very.
6:47>> Like that?
6:49>> y
6:49>> y.com?
6:50>> Yeah.
6:53>> Oh, that's the UK?
6:55Okay, perfect.
6:58So, this way I don't I don't want to
7:00Stop it.
7:01New session.
7:03Uh
7:06Let's do that.
7:09Let's do that.
7:11Where's the new session? Oh, there it
7:12is.
7:19Really?
7:28What is wrong with my paste?
7:31Okay.
7:33Uh, it's clothing store?
7:36>> Uh, it's category.
7:38>> Okay, so let's do headphones again, the
7:40same thing, right? So, I don't know.
7:42I've never used this website. I've
7:46Look.
7:47Clean test. No cheating.
7:54Um
7:57So So, what else does this give you? So,
7:59again, market research.
8:01Um
8:02For example, I was looking for a new
8:04apartment, right? I would want to move a
8:06house. So,
8:08with a CloudCrawl, with BrightData, I
8:10set up a listener. Literally just
8:11telling Hey, listen. Build me a scraper
8:13that will run every half an hour when in
8:16this area there's a a house, private
8:18house, under this price, notify me.
8:20That's all I did. In a few days, I got a
8:22notification and now I live there.
8:24So, these things are not only useful on
8:26a scale where, you know, oh, I need to
8:28scrape millions of records, but even for
8:30your personal use.
8:32It's so easy these days.
8:35CloudCrawl does an amazing job, right?
8:39I know Crawlera does a great job as
8:40well, but I prefer CloudCrawl because I
8:42don't know. It just
8:43gives me less headache.
8:45Crawlera sometimes takes me on a wild
8:47goose chase.
8:49Um how many of you use VibeCode?
8:54It's It's amazing. Why not, right?
8:56Anybody can build anything now.
8:58Um
9:00So, did you build it? I've got the
9:01structure, the search patterns. I don't
9:02know what third party I'm going to
9:03build. Oh, okay. It's building a
9:04scraper.
9:05Um
9:07This used to take,
9:09I don't know. If you If you're having a
9:10good day, maybe a full day, maybe a day
9:12and a half, right? You need to go check
9:14out the selectors, figure it out. It
9:16used to be interesting and fun,
9:18but you don't need to do that anymore.
9:20So, what I'm trying to show you guys is
9:22that
9:24with our
9:25MCP, with our
9:28infrastructure, we have over 150 million
9:30IPs.
9:31Uh with our unlocking technology where
9:33we even though if it needs to run a
9:35remote browser, so if you want to It can
9:37actually write you a browser automation.
9:39Uh Uh the browsers are running on our
9:41system. So, I can literally right now
9:42open a thousand browsers on this laptop
9:44that are running on our servers.
9:46And they can go do things whatever you
9:48needed to.
9:50So, it already got 90 products.
9:55Okay, it just needs to fix the Unicode
9:58because it's in pounds.
10:06Almost done.
10:08>> Can you also
10:10go into details about the MCP or is that
10:12it?
10:12>> Sure, I did it in the previous session,
10:14but while it's loading, let me tell you.
10:15So, basically, the MCP gives you a
10:17agent 66 tools.
10:20Uh some of them is basically uh we have
10:22a system where it can send a curl to any
10:24URL and our system will literally get
10:26the HTML back. It will solve a capture
10:28if needed and send it with a token. It
10:31will It knows exactly what headers and
10:33cookies the website needs. So,
10:34basically, it will make sure that the
10:36server thinks it's a browser and serve
10:38the HTML back. So, your agent can
10:39literally send curls, pull data without
10:42any questions.
10:43It can pull a full HTML. It can pull
10:45just a scrape as a markdown. Markdown is
10:47It's to save tokens, right? You just
10:49want the text of the page. You don't
10:50care about the HTML tags.
10:53It doesn't like the pound.
10:57Come on, you can do it.
11:00Um so, that's number one. Second, we
11:03have about 500 different APIs pre-built
11:06for different domains.
11:07So, instead of actually getting
11:08markdown, you can actually get a JSON of
11:10the product. For example, for Amazon, we
11:12have pre-built API. You can listen when
11:14you add your agent, it can be like,
11:16"Okay, go and check on Amazon
11:17something." It doesn't even need to
11:19build a scraper. It can literally just
11:20send the the request and get the data
11:22back.
11:23Um on top of that, remote browser
11:25infrastructure and the anything that
11:28needs
11:29anything that your agent needs to access
11:31the web.
11:32It has, right? So, you don't get
11:34blocked.
11:35While this is running, I want to show
11:36you for example
11:38uh tell
11:40one
11:41to Walmart
11:43do a search for headphones without
11:48MCP one
11:51So, I'm telling it go to Walmart, do a
11:53search for headphones, and tell me what
11:54is the first result.
11:56Without the Bright Data MCP.
11:59Uh
12:01It's going to do a fetch, which is going
12:02to get blocked.
12:07Product security verification screen,
12:08robot or human, obviously, right? That's
12:10That's the first thing.
12:13Now, do the same with Bright
12:17Um
12:18Oh my god.
12:27So, now it's using Scrape as a markdown.
12:29It's doing a search. It's only pulling
12:31the text. It's not pulling the HTML,
12:32just the text itself.
12:45Always when you do a live, it's slows
12:47down.
12:49The time is like
12:55Oh, actually, it could be. It could be
12:56opening a browser and actually holding
12:58the button. You know, the Walmart has
12:59like one of those click and hold, and it
13:01needs to hold it for like 30 seconds.
13:02So, it could be doing that right now.
13:05So, is it
13:07We have built to capture solving
13:09solutions. Like, we literally have
13:11in-house We have a AI solving capture.
13:14And
13:15moving, clicking things, and like what?
13:17It's perfect.
13:18So, it already got
13:22first result, blue headphones. Oh,
13:24basically, there you go. It already has
13:26the results of the headphones without
13:29robot or human with full product listing
13:31name.
13:33Uh
13:33so, we here already have the output.
13:36Tell me
13:38if you were to do this manually,
13:42how many how much more tokens have you
13:46used?
13:47How many did you save? Let's just give
13:50it a breakdown, and this is probably the
13:51biggest issue, right? Tokens is
13:53expensive.
13:55I'm going through millions and millions
13:56of tokens a day.
13:58I used to go more, but I I optimized it.
14:01So, um
14:02all of you are looking how do we save
14:03money on LM? How do we waste less tokens
14:05of web access and um
14:08Bright Data has the solution. Instead of
14:10parsing the full HTML, create a scraper.
14:13Uh instead of using
14:15um you know, okay, it pulls the HTML and
14:17then it extracts the data, it builds the
14:19the parser.
14:22But, it's working very slow right now.
14:25I feel like everybody's white coding.
14:26So, basically, here's the breakdown,
14:27right?
14:28What do we have here?
14:31The big swings output
14:33parsing 90 products with
14:37input tokens,
14:39output tokens,
14:41total save.
14:43So, it's about a 62% save of tokens.
14:46This is not a high number. This website,
14:47I guess, uh has um maybe a structured um
14:51HTML. I'm not sure.
14:54But,
14:55this is what I wanted to show you. And
14:57the best thing about it is that it will
14:59maintain it. If it breaks it, it will
15:01fix it. I can set up a loop, right? So,
15:03for example, in Cloud, I can set a
15:04schedule every 30 minutes you do and run
15:06and check something.
15:08And this is
15:09we are giving you guys, okay? With our
15:11MCP, your agent has access to all the
15:13web
15:14uh without getting blocked. No capture
15:17will stop it. No robots antibot systems
15:19will stop it. And uh you have no
15:22headaches of that. I don't know how many
15:24of you have
15:25You guys probably dealt with blocking,
15:27right? When you build a scraper, it gets
15:29really complicated. So, this kind of
15:30removes all the headache for it.
15:32Um any questions?
15:37>> Where is the code?
15:38>> Excuse me?
15:39>> The code the code you generate a
15:40scraper, right?
15:42>> I can I If you you want to see the code?
15:45I mean, it's here. We have the scraper.
15:47Let's check it out.
16:03So, it's using API web unlocker. This is
16:07the same thing as a scraper as a
16:08markdown with an MCP.
16:10It sends a simple request and gets the
16:12data back. This is the parser that it
16:13built.
16:17I mean,
16:21it looks pretty good. This is the the
16:23schema that it built for the output.
16:28It's doing a keyword search. It has the
16:30max page, right? So, I asked for two
16:31inputs, the keyword search and how many
16:33pages to scrape.
16:38And hold on. Let's see Let's see We can
16:40run it
16:41through here as well. Where is the
16:43variables? API key is on key.
16:52Let's run it.
16:54Do another run
16:56on
16:57laptops.
16:59What are the two pages results?
17:04And now you kind of control it. Like you
17:06don't even need to run it, right? It
17:08will trigger it for you. It will
17:09maintain it for you. Um you can create a
17:12script that will automatically do all of
17:14this. I prefer the visual interface for
17:15the demonstration. Sometimes they you
17:17like looking at our code it's a little
17:19more complicated. But right now
17:21basically I can ask it any questions. It
17:23will actually run the script. So it's
17:24not wasting tokens to execute the script
17:26maybe like 60 tokens, right right now.
17:28So it's going to literally get all this
17:30data for 100 tokens.
17:32And then you can do whatever you can ask
17:34it questions about it. But now we're
17:35working with a JSON and JSON is much
17:37more token efficient than any markdown
17:39or any other file, right?
17:44Still the usual.
17:51>> Can I ask can you can you do this on all
17:53the authorization sites as well?
17:55>> Uh no, we only talk we only deal with
17:57public data. So behind login it's um
18:00it's private data. And and uh just you
18:02know just a heads-up.
18:04If you create an account, you accept
18:06terms and conditions of the website.
18:08Always check terms and conditions of the
18:09website. If they say don't scrape, don't
18:11use robots, and you do, then that
18:14company can actually sue you. And this
18:15is happening
18:16you probably heard it in the news all
18:18the time. LinkedIn is suing them.
18:20Everybody's suing each other because
18:22they the data right now is like the new
18:24gold the oil, right? Everybody's trying
18:26to like oh this is my data. Elon Musk
18:28took over Twitter, locked it down.
18:30That's it. Literally like uh the it used
18:32to be so open and now it's all locked in
18:34a few accounts that you can actually
18:35scrape. So the we're seeing this on
18:38every single level.
18:40Very slow today.
18:42Um so yes, also same thing if you accept
18:45uh there's like terms and like if you do
18:47a search and there's a checkpoint, I
18:49accept terms and conditions, always be
18:50careful with that. So we only deal with
18:53public data. So nothing behind login. We
18:55don't accept terms and conditions.
18:57And we actually won a few lawsuits. We
19:00were sued by Meta. We were so sued by
19:02Elon Musk a month after he took over.
19:05And uh the judge said it's very simple.
19:07Public data is public data.
19:09It doesn't matter how you collect it.
19:10Doesn't matter what you do with it. It's
19:12public. You know, it's like walking on
19:14the street, you write down the prices on
19:15the counter and then you sell it to
19:17somebody.
19:18It's public data. It's available. You
19:20can do whatever you want with it. Okay?
19:21So,
19:24as usual, it gets stuck on a live demo,
19:26but
19:28we can
19:31we can run this manually.
19:48Oh my god.
20:00What the
20:02I'm setting a variable. It's doing this.
20:04No, it's not going to work.
20:17What am I missing?
20:21Still stuck? Oh, okay. It's done.
20:24Okay, so here we got we got
20:28Let's see. How many tokens?
20:31How many tokens did you spend on getting
20:36the these?
20:42So, even if you're not be able to
20:44building pipelines for any
20:46you know, big companies and you don't
20:47need to get for personal use, I always
20:49have the MCP connected. I don't ask the
20:51you know, LLM to go do I always ask
20:54build a script that can later on be used
20:57by it. It's using its own script to save
20:59the tokens.
21:10>> Literally like we're talking about 1,000
21:11tokens where if you know if if it needs
21:14to go through the JSON, it's like 10,000
21:16tokens. We're talking about like
21:17literally pennies compared to what it
21:19would be to actually scrape it.
21:21Any more questions?
21:24How about this? Anybody has problems
21:25with actually getting data?
21:30No, no question. Everybody is like
21:32I'm not sharing my problems. I have no
21:34problems. And they go home and like oh I
21:35do have a problem. But in any case,
21:37listen, we have a booth. If you don't if
21:39you want to talk in private and you have
21:40questions about data or access of LLM,
21:44feel free. I'm always willing to help.
21:46Um
21:48we can connect
21:50on LinkedIn by the way, guys, if you
21:51want. Let me just summarize a few things
21:53what we did here.
21:54So um self-healing pipeline. When you
21:56have when your agent actually has access
21:59to a blocked website and you're working
22:01I mean, the MCP is mostly useful in
22:04about 20% of the of the domains, right?
22:06The ones that have Akamai, Data Dome,
22:07Cloudflare, those heavy protected
22:09domains. They're the usually the most
22:11juiciest one, right? Like real estate or
22:14uh big e-commerce places.
22:16Uh with our MCP, it has access to it. If
22:19it has access to it, it can build a
22:20scraper. If it can build a scraper, it
22:21can maintain the scraper. This is
22:23basically the whole thing that I wanted
22:24to show you. That in Bright Data, we
22:26have all the tools that your agent might
22:28need to explore, to build and maintain
22:32uh a pipeline, right? So, if you want to
22:35set up a listener for an apartment and
22:37you're looking to move into a cheaper
22:38place or maybe you want to book a table
22:41in a restaurant where it's always
22:43packed. And literally I I I have a
22:45listener right now. I'm waiting for 2
22:46months already. Everybody is like
22:47booking right away. So, as soon as the
22:49spot opens, it will automatically book a
22:51spot for me. It can be very useful in
22:53just kind of like even even the small
22:55things, personal things, right? I'm not
22:57talking about just enterprise scale
22:58where I need to download a million
23:00records. Okay? But having access to all
23:02the websites
23:04is this is important.
23:06>> So, apart from scraping
23:08can you also do actions on the website?
23:10>> Yes, of course.
23:13Fill up a form, submit it as well. Yes.
23:15The only thing you can't do is log in.
23:19Right? So, if you have let's say you
23:21need to perform a search, right? You
23:23cannot generate the URL, you need to
23:25click buttons. Let's say flights. You
23:27want to check flights availability,
23:28Skyscanner or something like that,
23:29right? The URL is usually a hash and you
23:32can't do anything with it. So, yes, LLM
23:35can spool a browser, remote browser,
23:37even if it's a geo-restricted site. It
23:39can be like, "Okay, I want IP from the
23:40United States." So, it's open a browser
23:42with the United States IP and then it
23:43goes and clicks things, fills it up, and
23:45so on. The beauty of it is that our
23:47browser will mimic real human behavior.
23:50So, when your agent clicks, it's not a
23:52teleportation. There's a mouse
23:53pre-recorded like a real human being
23:55moving it. When it types, it will type a
23:58little slower, speed up, like maybe even
23:59mistake, and so on. So, we have
24:01pre-recorded typing, we have
24:02pre-recorded mouse movements. So, if the
24:04website has a tracker that's constantly
24:06sending to the server what the user is
24:07doing, it will look like a real human
24:09being.
24:10Doesn't matter what your agent, even if
24:11it's a low like for browsing agents, I'm
24:14never used the top models, right? For
24:16example, if with Claude Haiku model is
24:18more than enough for browsing. If it's
24:21being masked that it's a real human, it
24:24works just fine.
24:27And
24:28so, that's pretty much it. If you guys
24:29want to connect on LinkedIn, feel free.
24:31I'm always willing to help. If you have
24:32any questions regarding how to get data,
24:34if you have problems with accessing
24:35anything, feel free to message me.
24:40I feel like there's been so much more
24:41than 15 minutes. I love it. It's the
24:43second time I'm doing a speech and I got
24:44like half an hour instead of 15 minutes.
24:46It's amazing.
24:47Um
24:50Now's the time if you have any
24:50questions,
24:52discussions.
24:55No, nothing? Great. Okay, guys, I guess
24:57I will conclude this session in this
24:58Feel free to come to the booth on the
25:00third floor if you have more questions
25:01and um
25:03let's keep the public data public.
25:07Thank you.