Data Engineer Full Course in 10 Hours [2024] | Data Engineer Course For Beginners | Edureka

0:03[Music]

0:08have you ever played with building

0:09blocks data Engineers are like expert

0:11Builders but instead of blocks they work

0:14with information they put together

0:16structures that make data neat and

0:18organized like creating a super cool

0:20puzzle where everything fits

0:23perfectly a data engineer is someone who

0:26designs and builds system to gather and

0:28store and manage lots of

0:31information on that note hello everyone

0:33welcome to this session to the data

0:35engineer full course join us on the

0:38journey and discover how data Engineers

0:40shape the future on decision making and

0:43Innovation we'll Begin by exploring the

0:46data engineering and the steps to become

0:48data engineer next we'll provide the

0:50overview of Big Data followed by the

0:53guidance on becoming the data engineer

0:55and insights into the associated salary

0:59asure is essential for data engineer

1:01because it offers a range of specialized

1:03tools and service designed to manage

1:06analyze and scale data effectively so as

1:09we will cover Azure topics such as Azure

1:12data Factory Azure database services and

1:14Azure SQL database and aure data link

1:18advancing further will explore Advanced

1:20Data modeling using PBI and aure

1:22including assure data bricks as we

1:25progress to Advanced topics we'll

1:27explore Haro Essentials covering HD

1:30Yan map reduce spark Pig hi and EDP for

1:34its

1:35ecosystem following this will progress

1:38to understanding Kafka streams to

1:40conclude the course we'll discuss the

1:42essential her interview question and

1:44answers to advance your career in data

1:46engineering interviews before we begin

1:49please consider subscribing to our

1:50YouTube channel and hit the Bell icon to

1:53stay updated on the latest tech content

1:55from edura also visit the Eda website

1:58for the data engineer master master

2:00program the link to which is given in

2:02the description box

2:04[Music]

2:08below today I'll share you an engaging

2:11story about two friends Alex and Bob hey

2:15have I told you about the incredible

2:16data engineering journey of a

2:18multinational e-commerce company no I

2:21don't think so what happened Alex well

2:24let me share this story with you Bob

2:26this company had lots of valuable data

2:28but struggled with scattered systems and

2:31outdated databases oh that's a tough

2:34situation what did they do about it Alex

2:37they embarked on a data engineering

2:38initiative B it was quite a journey I

2:41can imagine so what changes did they

2:43make Alex they integrated their data

2:46into a centralized system and automated

2:48data cleaning and transformation it made

2:51a huge difference Bob that sounds so

2:54promising did it impact their operations

2:56Alex definitely Bob they gained faster

2:59access to ACC data and generated

3:01realtime analytical reports impressive

3:04did they address data governance as well

3:06Alex yes they implemented data

3:08governance practices for data quality

3:10privacy and compilance that's

3:13commendable it's a great example of how

3:15data engineering can transform a

3:17business what an inspiring story Alex

3:20yeah I thought you would find it

3:22fascinating Bob data engineering truly

3:24has the power to drive success and

3:25Innovation together yeah thanks for

3:28sharing the story Alex it reinforces the

3:30importance of data Engineering in

3:32today's data driven World hello everyone

3:35this is Saia from edua and in this video

3:37we will be diving into the fascinating

3:39world of data engineering so data

3:42engineering is all about harnessing the

3:44power of data to drive meaningful

3:46insights in today's rapidly evolving

3:48digital landscape organizations are

3:51grappling with massive amounts of data

3:53and that's where data and sharing comes

3:55into play now let's take a quick look at

3:57the agenda for this video we will start

4:00by the introduction of what data

4:01insuring is and why it is important for

4:03so many businesses outside then we will

4:06delve into the key components of data

4:07engineering such as data inje data

4:10integration data transformation and data

4:12storage next we will discuss some of the

4:15responsibilities of data engineers and

4:17the comparison between data engineer

4:19data analyst and data scientist after

4:22that we will delve into the installation

4:24process of popular tools and

4:26Technologies which is used in data

4:28engineering finally we will wrap up with

4:30a glimpse of what data pipelines are so

4:33let's understand what is data

4:35engineering First Data engineering

4:37involves the design development and

4:39maintenance of system and infrastructure

4:41to handle large volumes of data

4:43effectively it focuses on the extraction

4:46transformation loading and storage of

4:48data which ensure its quality and

4:49scalability of analysis now let's

4:52understand what are the key components

4:54of data engineering data engineering

4:56helps to collect data from the disparate

4:58sources and integrate into a unified

5:01format which allows organization to have

5:03a comprehensive view of their data it

5:06also involves processing and

5:08transforming data into a suitable format

5:11which ensures data quality consistency

5:13and applications it includes the

5:16development of data pipelines and

5:18processing system that enable the

5:19efficient processing and Analysis of

5:21data this includes batch processing for

5:24large scale data transformation and

5:26realtime processing for immediate

5:28insights and action

5:30data engineering plays a crucial role in

5:32enabling data driven World by providing

5:35clean integrated and accessible data

5:37data insuring empowers organizations to

5:40make an informed decision based on

5:42accurate insights and Analysis overall

5:45it is used to handle the complexity of

5:46data processing storage and integration

5:49which ensures that the organization can

5:51leverage the full potential of their

5:53data assets for strategic operations and

5:56Innovations now after the thorough

5:59understanding of what data engineering

6:00is and its key features also we will Del

6:03into the importance of data

6:05engineering so here are some of the key

6:08reasons why data engineering should be

6:10important or why data engineering is

6:12important so our first reason is it

6:15focuses on building scalable and

6:18efficient data processing systems by

6:20optimizing data pipelines leveraging

6:23distributed computing Technologies and

6:25employing performance tuning techniques

6:27data Engineers ensured did organization

6:30can handle large volumes of data and

6:33Achieve faster processing times it

6:35provides the foundation for advanced

6:38analytics and machine learning

6:40initiatives by structuring and preparing

6:42data in a switchable format data

6:44Engineers unable data scientists and

6:46analyst to extract valuable insights

6:49build predictive models and develop

6:51machine learning algorithms data

6:54insuring enables organization to process

6:56and analyze data as it arrives this is

6:59crucial in scenarios such as fraud

7:02detection recommendation systems iot

7:05applications and monitoring systems that

7:07require immediate insights and actions

7:10as we already know the importance of

7:12data engineering we will cover the real

7:14word applications based on this so our

7:17first application is e-commerce sites so

7:20data insuring is used to collect and

7:21process large volumes of customer data

7:24transaction data and product data this

7:27enables e-commerce companies to predict

7:29recommendations optimize pricing

7:31strategies and improve Inventory

7:33management our second example is social

7:36media sites like data insuring plays a

7:39crucial role in collecting processing

7:41and analyzing social media data which

7:43allows companies to monitor brand

7:45sentiment track user interactions and

7:48derive insights for targeted marketing

7:51campaigns so our third example is in

7:53finance and banking sector data insuring

7:56is used to handle financial data

7:58including trans action records customer

8:00information and Market data it enables

8:03fraud detection risk assessment

8:05algorithmic trading and personalized

8:08Financial Services also okay so now the

8:11fourth application is in the healthcare

8:13sector data insuring is employed to

8:15manage and analyze patient records

8:17Medical Imaging data and clinical child

8:20data it enables Healthcare Providers and

8:22researchers to gain insights improve

8:24patient outcomes and develop predictive

8:27models so this examples highlight the

8:29diverse application and the importance

8:31of data Engineering in various

8:33Industries demonstrating how it enables

8:35organization to Leverage The Power of

8:37data for operation and Innovation now

8:40the data insuring process involves

8:42several key steps including data

8:44injection transformation storage

8:47processing and integration let's discuss

8:50each step in more detail so our first

8:52step is data inje data inje refers to

8:56the process of collecting and importing

8:58data from various sources into a data

9:00system or data pipeline this can involve

9:03extracting data from databases files

9:06apis streaming platforms or the other

9:08resources the goal is to GA relevant

9:11data and make it available for the

9:12further processing and Analysis Second

9:15Step would be the data transformation

9:18once the data is ingested it often needs

9:20to be transformed into a suitable format

9:22for analysis or storage this data

9:25transformation involves cleaning

9:27validating and restructuring the data to

9:29ensure consistency and usability this

9:32step may include tasks such as data

9:34filtering aggregation normalization data

9:37type conversion or the application of

9:39business rules so our third step would

9:41be data storage so after the data is

9:44transformed it needs to be stored in a

9:46structured manner right so this

9:48typically involves using databases or

9:50data storage system that provide

9:52efficient storage and retrial

9:54capabilities popular choices for data

9:56storage include relational databases

9:58data data warehouses data laks or

10:01distributed file systems the selection

10:03depends on the specific requirements of

10:05the project such as data volume SS

10:07patterns and the analytical records so

10:10our next step would be data processing

10:13data processing involves performing

10:15computations and analysis on the stored

10:17data this STP can include tasks such as

10:20data aggregation data Improvement data

10:22summarization statistical calculation or

10:25machine learning algorithms data

10:27processing can be done through various

10:28to tools such as the SQL queries data

10:31processing engines or the custom scripts

10:34Next Step would be the data integration

10:36so in this step data integration

10:39involves combining data from multiple

10:41sources to create a unified and

10:43comprehensive view this step is crucial

10:45when dealing with heterogeneous data

10:47sources or when different style produced

10:49data this needs to be Consolidated data

10:52integration can be achieved through data

10:54consolidation or by using extract

10:56transform load process to combine and

10:59merge data from various sources next our

11:02last step would be the data governance

11:04so data governance is nothing but a

11:06framework in a set of process which

11:08ensures the effective management and

11:10utilization of data assets within an

11:14organization it involves establishing

11:16policies procedures and guidelines for

11:18data management data quality and data

11:21usage all the steps in data Engineering

11:24Process are typically iterative and may

11:26require continuous monitoring optimizing

11:29ation and maintenance to ensure data

11:31quality reliability and performance data

11:35Engineers play a crucial role in

11:37designing and implementing efficient and

11:39scalable data pipelines to support

11:41datadriven applications and

11:43analytics Now we move ahead with the key

11:46responsibilities of data Engineers which

11:48typically includes designing and

11:51developing scalable and efficient data

11:53pipelines which extract transform and

11:56load data from various sources to

11:59targeted systems next would be the

12:01managing and optimizing data storage

12:03infrastructure for performance

12:05scalability and reliability selecting

12:07and configuring appropriate data storage

12:10technology such as relational databases

12:12data warehouses or no SQL databases to

12:14meet data storage and retrial

12:16requirements data Engineers are also

12:19responsible for applying data

12:20transformation techniques to clean

12:22filter Aggregate and structure data for

12:25analysis or consumption by Downstream

12:27systems they're also responsible for

12:30implementing data processing task using

12:32programming languages or data processing

12:34Frameworks to manipulate and transform

12:37data efficiently now after analyzing the

12:40key responsibilities of data insuring we

12:42will proceed with our next topic and

12:44understand the key differences between

12:46these three roles that is data engineer

12:48data analyst and data scientist these

12:51three are the distinct role within the

12:53field of data science each with its own

12:55set of responsibilities and skill

12:57requirements so our our first profile is

12:59of data engineer so data engineer are

13:02responsible for Designing building and

13:04maintaining the infrastructure and

13:06system that enable data storage

13:08processing and retrial as we all know

13:11they focus on creating and managing the

13:12data pipelines and architecture

13:15necessary for efficient data collection

13:17transformation and storage data

13:19Engineers also work closely with

13:21software engineers and database

13:23administrator to ensure data is

13:25accessible reliable and scalable they

13:28typically work work with tools like

13:29Hardo spark SQL ETL Frameworks and

13:33cloud-based platforms for data

13:34processing and storage then our next

13:37profile would be the data analyst so

13:40data analysts are focused on analyzing

13:41and interpreting data to derive

13:43meaningful insights they work with

13:46structured and unstructured data to

13:47identify patterns strengths and

13:49correlations data analysts are

13:51proficient in statistical analysis data

13:54visualization tools and data quaring

13:56techniques as well they often use tools

13:58tools like SQL Excel table or powerbi to

14:02analyze data and create reports and

14:04dashboards after dat our next profile

14:07would be the data scientist data

14:09scientist possess a blend of skills from

14:11mathematics statistics programming and

14:13domain knowledge they leverage their

14:15expertise to develop and Implement

14:17complex algorithms and models to solve

14:20integrate data patterns or extract

14:22insights from large data sets data

14:25scientists employ techniques like

14:26machine learning predictive modeling and

14:28statistical analysis to build predictive

14:30models uncover patterns and make

14:32predictions also they also collaborate

14:34with stakeholders to Define business

14:36problems and design experiments to

14:38together their data so to summarize this

14:41data Engineers focus on the

14:43infrastructure and data pipelines data

14:46analysts work on analyzing and Reporting

14:48data while data scientists concentrate

14:51on Advanced modeling and extracting

14:54insights so to make all this happen data

14:57Engineers rely on power powerful tools

14:59and Technologies platforms like Apache

15:02spark Apache Kafka SQL and no SQL

15:05databases and cloud services such as AWS

15:08and gcp provides the building blocks for

15:10efficient data engineering this tools

15:12help process vast amounts of data

15:15facilate realtime data streaming and

15:17ensure secure and scalable data storage

15:20now let's understand what are the data

15:23pipelines are so here is the glimpse of

15:25what data pipelines are and why we use

15:27data Pipelines

15:29so data pipelines are the series of

15:31steps that extract transform and load

15:34data from source to destination itself

15:36okay so they enable the efficient and

15:38automated flow of data through different

15:41stages ensuring data quality consistency

15:43and avability

15:45let's understand each step one by one so

15:48the first step is extraction the

15:50extraction phase of ETL involves

15:53retrieving data from various sources

15:55such as databases files apis or

15:57streaming platform platforms the goal is

16:00to extract the relevant data needed for

16:02further processing and Analysis this

16:05process typically includes establishing

16:07connection to the data sources

16:09performing data queries or using data

16:11extraction tools to pull the required

16:14data into the data pipeline Next Step

16:16would be the transformation the

16:18transformation phase of ETL focuses on

16:20cleaning validating and reshaping the

16:23exctic data to ensure its quality and

16:26consistency this step involves applying

16:28various operations and rules to the data

16:30such as data filtering data type

16:32conversions data aggregation data

16:35enrichment or data

16:36normalization the transformation process

16:39aims to make the data suitable for

16:41analysis storage or integration into the

16:43destination system Next Step would be

16:45the loading the loading pH of ETL

16:48involves storing the transform data into

16:50the targeted system such as databases

16:52data warehouses or data Lakes then the

16:55data is loaded in a structured format

16:57that aligns with the schema or format of

17:00the destination system this phase may

17:03include tasks such as data mapping

17:05schema matching data passing or indexing

17:08to optimize data storage and retrieval

17:11after that our next step would be the or

17:13maybe we call this our last step would

17:15be the monitoring and handling

17:17monitoring and handling refer to the

17:19ongoing monitoring management and the

17:21maintenance of data pipelines on okay so

17:24now let's discuss about batch processing

17:27and realtime streaming pipeline lines so

17:29in a batch processing pipeline there is

17:31a delay between time data is collected

17:33and when it is processed this delay can

17:36range from minutes to hours or even days

17:38depending on the schedule intervals on

17:41the other hand realtime streaming

17:43pipeline aim to process data as it

17:45arrives which results in a minimal

17:48latency data is processed analyzed in

17:50near real time or within a very low

17:52delay the second point is batch

17:55processing pipelines are designed to

17:56handle large volumes of data efficiently

17:59they can process and analyze massive

18:01amounts of historical data in a batch

18:03mode on the other hand realtime

18:05streaming pipelines focus on processing

18:07data as it arrives making them more

18:10suitable for handling data streams with

18:12continuous High Velocity data updates

18:15now the third point is batch processing

18:17pipelines typically utilize batch

18:19processing Frameworks like Apache spark

18:22or Hadoop map redu this Frameworks

18:25process data in chunks or batches which

18:27allows for parall processing and

18:29optimize resource utilizations also

18:32while realtime streaming pipelines often

18:34use streaming Frameworks like Apache

18:36Kafka Apache Flink or Apache stom this

18:39Frameworks enable continuous processing

18:41of data streams supporting low latency

18:43operations and realtime

18:45analytics now the fourth point is batch

18:48processing pipelines are commonly used

18:50for tasks that involve historical

18:52analysis generating periodic reports or

18:55data preparation for machine learning

18:56models they are Well Suited for

18:58scenarios where processing time is not

19:00critical but analyzing large volumes of

19:02data is essential on the other hand

19:05realtime streaming pipelines are ideal

19:07for application that require realtime

19:09monitoring immediate response or instant

19:12insights based on live data use cases

19:14include like fraud detection realtime

19:16recommendation systems network

19:18monitoring or iot sensor data analysis

19:21so our last point is batch processing

19:24pipelines often require significant

19:25Computing resources during the

19:27processing phase as they process large

19:29volumes of data in a batch mode whereas

19:32realtime streaming pipelines also

19:34requires Computing resources but are

19:36more focused on low latency processing

19:38and continuous data streams requiring

19:41efficient resources allocation and

19:43management so it's important to note

19:45that there can be the overlap between

19:47batch processing and realtime streaming

19:49pipelines and hybrid architectures

19:51combining both approaches are common so

19:54that wraps up our Deep dive into the

19:56world of data engineering we have

19:57covered the essential aspects of data

19:59pipelines governance and security giving

20:02you a comprehensive understanding of how

20:04data is ingested transformed stored and

20:07[Music]

20:12processed now how to become a data

20:15engineer the road map to become a data

20:18engineer can go like from being

20:21proficient in programming languages to

20:23learning Automation and scripting then

20:26understanding your databases

20:28and mastering data processing techniques

20:31to studying cloud computing and

20:34internalizing

20:35infrastructure now we'll see how to get

20:38started with becoming a data engineer to

20:41get started with the learning process

20:43you can look into our Eda YouTube

20:45channel to start with even with no prior

20:48knowledge of data engineering one can

20:50just go through the videos and get an

20:53understanding of the whole

20:55subject we even have edureka blogs to to

20:58help you with detailed information on

21:00the topics and can give you a clearer

21:03picture apart from these we even have

21:06premium courses which can help you

21:10understand the topic at ease with the

21:13personal trainer and 24 hours access to

21:17Lifetime content you can learn here at

21:19your own pace with a live trainer who's

21:22extremely efficient and knowledgeable

21:24and experienc in the particular field

21:27these courses can even get you

21:28certificates which you can add in your

21:30CV for better opportunity in your job

21:33market so that's it for today I hope

21:36this video helped you and will help you

21:38decide how to become a data

21:42[Music]

21:46engineer now I feel Sor of it's the best

21:49time to tell the story about how data

21:51evolved and how big data came fine RMA

21:54so we'll move forward so sort of what

21:57can you notice here RMA I see how

22:00technology has evolved earlier we had

22:02landline phones but now we have

22:04smartphones we have Android we have IOS

22:06that are making our lives smarter as

22:08well as our phone smarter apart from

22:10that we were also using bulky desktops

22:12for processing MBS of data now if you

22:14can remember we were using floppies and

22:16you know how much data it can store

22:18right then came hard dis for storing TBS

22:20of data and now we can store data on

22:22cloud as well and similarly nowadays

22:25even self-driving cars have come up and

22:28I know you must be thinking why are we

22:29telling that now if you notice due to

22:32this enhancement of Technology we're

22:34generating a lot of data so let's take

22:36the example of your phones have you ever

22:39noticed how much data is generated due

22:41to your fancy smartphones your every

22:44action even one video that you send

22:46through WhatsApp or any other messenger

22:48app that generates data now this is just

22:50an example you have no idea how much

22:53data you're generating because of every

22:55action you do now the deal is this data

22:57is not in a format that our relational

22:59database can handle and apart from that

23:02even the volume of data has also

23:04increased exponentially now I was

23:06talking about self-driving cars so

23:08basically these cars have sensors that

23:10records every minute details like the

23:12size of the obstacle the distance from

23:14the obstacle and many more and then it

23:17decides how to react now you can imagine

23:19how much data is generated for each

23:21kilometer that you drive on that car I

23:24completely agree with you RMA so let's

23:26move forward and focus on ious other

23:28factors behind the evolution of data I

23:31think you guys must have heard about iot

23:33if you can recall in the previous slide

23:35we were discussing about self-driving

23:36cars it is nothing but an example of iot

23:39let me tell you what exactly it is iot

23:42connects your physical device with

23:44internet and makes the device smarter so

23:46nowadays if you have noticed we have

23:48Smart ACS TVs Etc so we'll take the

23:51example of smart air conditioners so

23:53this device actually monitors your body

23:55temperature and the outside temperature

23:57and accordingly decides what should be

23:59the temperature of the room now in order

24:01to do this it has to First accumulate

24:04data from where it can accumulate data

24:06from internet through sensors that are

24:08monitoring your body temperature and the

24:10surroundings so basically from various

24:13sources that you might not even know

24:14about it is actually fetching that data

24:17and accordingly it decides what should

24:19be the temperature of your room now we

24:20can actually see that because of iot we

24:22are generating huge amount of data now

24:25there's one startat also that is there

24:26in front of your screen so if you notice

24:28by 2020 we'll have 50 billion iot

24:31devices so I don't think so I need to

24:34explain much that how iot is generating

24:36huge amount of data so we'll move

24:38forward and focus on one more factor

24:39that is social media now when we talk

24:42about social media I think RMA can

24:44explain this better right RMA yeah s but

24:47I'm pretty sure that even you use it so

24:50let me tell you that social media is

24:52actually one of the most important

24:54factor in the evolution of big data so

24:57nowadays everyone is using Facebook

24:59Instagram YouTube and a lot of other

25:02social media websites so these social

25:04media sites have so much data for

25:07example it will have your personal

25:09details like your name age and apart

25:11from that even each picture that you

25:14like or react to also generates data and

25:16even the Facebook pages that you go

25:18around liking that is also generating

25:20data and nowadays you can see that most

25:23people are sharing videos on Facebook so

25:25that is also generating huge amount of

25:28data and the most challenging part here

25:30is that the data is not present in a

25:32structured Manner and at the same time

25:35it is huge in size isn't that right sort

25:38of can't agree more the point you made

25:40about the form of data is actually one

25:42of the biggest factor for the evolution

25:44of big data so due to all these reasons

25:46that we have discussed have not only

25:48increased the amount of data but it has

25:50also shown us that data is actually

25:52getting generated in various formats for

25:54example data is generated with videos

25:56that is actually unstructured same goes

25:58for images as well so there are numerous

26:00or you can say millions of ways in which

26:02data is getting generated nowadays

26:05absolutely and these are just few

26:07examples that we have given you there

26:09are many other driving factors for the

26:11evolution of data so these are few more

26:14examples because of which data is

26:17evolving and converting to Big Data

26:19we'll discuss about the retail part I'm

26:21pretty sure that all of you must have

26:22visited websites like Amazon flip cart

26:24Etc and rishma I know you visited a lot

26:27of times

26:28yeah I do and suppose RMA wants to buy

26:30shoes so she won't just directly go buy

26:32shoes she'll search for a lot of shoes

26:35so somewhere her search history will be

26:36stored and I know for sure that this

26:39won't be the first time that she's

26:40buying something so there will be her

26:42purchase history as well along with her

26:44personal details and there are numerous

26:46ways in which she might not even know

26:47that she's generating data and obviously

26:49Amazon was not present earlier so at

26:52that time there is no way that such huge

26:53amount of data was generated similarly

26:56the data has evolved due to other

26:57reasons as well like Banking and finance

26:59media and entertainment etc etc so now

27:02the deal is what exactly is Big Data how

27:05do we consider data as big data so let's

27:07move forward and understand what exactly

27:10it

27:11is okay now let us look at the proper

27:14definition of Big Data even though we've

27:16put forward our own definitions already

27:19so sort of why don't you take us through

27:21it yes RMA sure so big data is a term

27:24for collection of data sets so large and

27:26complex that it becomes difficult to

27:28process using onhand database system

27:31tools or traditional data processing

27:34applications okay so what I understand

27:37from this is that our traditional

27:39systems are a problem because they're

27:41too oldfashioned to process this data or

27:44something no RMA the real problem is

27:47there is too much data to process when

27:49the traditional systems were invented in

27:51the beginning we never anticipated that

27:53we would have to deal with such enormous

27:55amount of the data it's like like a

27:57disease infected on you you don't change

27:59your body orientation when you get

28:01infected with a disease right RMA you

28:03cure it with medicines couldn't agree

28:06more sort of now the question is how do

28:09we consider some data as Big Data how do

28:11we classify some data as Big Data how do

28:14we know which kind of data is going to

28:16be hard for us to process well Sor of we

28:19have the five vs to tell us

28:21that so let's take a closer look at what

28:24are those so starting with the first V

28:27it's the volume of data it's

28:29tremendously large so if you look at the

28:31stats here you can see the volume of

28:33data is rising exponentially so now

28:35we're dealing with just 4.4 zettabytes

28:38of data and by 2020 just in three years

28:41is expected that the data will rise up

28:44to 44 zettabytes which is like equal to

28:4644 trillion gigabytes so that's really

28:50really

28:51huge it is because all these humongous

28:54all this humongous data is coming from

28:56multiple sources and that is the second

28:59V which is nothing but variety we deal

29:01with so many different kinds of files at

29:03all once there are MP3 files videos Json

29:06CSV tsv and many more now these are all

29:09structured unstructured and

29:10semi-structured all together now let me

29:12explain you this with the diagram that

29:14is there on your screen so over here we

29:16have audio we have video files we have

29:18PNG files we have Json log files emails

29:22various formats of data now this data is

29:24classified into three forms one is

29:26structured format now in structure

29:27format you have a proper schema for your

29:29data so you know what all columns will

29:31be there and basically you know the

29:33schema about your data so it is

29:35structured it is in a structured format

29:37or you can say in a tabular format now

29:39when we talk about semi-structured files

29:41these are nothing but Json XML and CSV

29:43files where schema is not defined

29:45properly now when I go to unstructured

29:47format we have log files here audio

29:50files videos and images so these are all

29:52considered as unstructured files and Sor

29:56it is also because of of the speed of

29:58accumulation of all this variety of data

30:00alog together which brings us to our

30:02third V which is velocity so if you look

30:05here earlier we were using Mainframe

30:07systems huge computers but less data

30:10because there were less people working

30:11with computers at that time but as

30:14computers evolved and we came to the

30:15client server model the time came for

30:18the web applications and the internet

30:20boomed and as it grew among the masses

30:22the web applications got increased over

30:24the internet and everyone started using

30:26all this applications and not only from

30:29their computers and also from mobile

30:31devices so more users more appliances

30:34more apps and H hands a lot of data and

30:37when you talk about people generating

30:39data or Internet RMA the one kind of

30:42application that strikes first in my

30:43mind is social media so you tell me how

30:46much data you generate alone with your

30:47Instagram posts and stories uh it will

30:50be quite a boast if I only talk about

30:52myself here so let's talk including

30:55every social media user so if you see

30:57the stats in front of your screen you

31:00can see that for every 60 seconds there

31:03are 100,000 tweets actually more than

31:06100,000 tweets generated in Twitter

31:08every minute similarly there are 695,000

31:12status updates on Facebook when you talk

31:15about messaging there are 11 million

31:17messages generated every minute and

31:20similarly there are

31:25698,000 email and that equals to almost

31:301,820 terabytes of data and obviously

31:34the number of mobile users are also

31:36increasing every minute and there are

31:382117 plus new mobile users every 60

31:42seconds gez that's a lot of data I don't

31:45even want to go ahead and calculate the

31:46total it would actually scare me yeah

31:49that's a lot now the bigger problem is

31:52how to extract the useful data from here

31:55and that's when we come to our next week

31:57that is value so over here what happens

32:00first you need to mine the useful

32:02content from your data basically you

32:04need to make sure that you have only

32:05useful fields in your data set after

32:07that you perform certain analytics or

32:09you say you you perform certain analysis

32:11on that data that you have cleaned and

32:13you need to make sure that whatever

32:15analysis you have done it is of some

32:17value that is it will help you in your

32:19business to grow it can basically find

32:21out certain insights which were not

32:23possible earlier so you need to make

32:25sure that whatever big data that has

32:26been generated or whatever data that has

32:28been generated it makes sense it will

32:31actually help your business to grow and

32:32it has some value to it now getting the

32:34value of this data is one big challenge

32:37let me tell you why and that brings us

32:39to our next V which is veracity now this

32:42big data has a lot of

32:43inconsistencies obviously when you're

32:45dumping such huge amount of data some

32:48data packets are bound to lose in the

32:50process now what we need to do we need

32:52to fill up these missing data and then

32:54start mining again and then process it

32:56and then come up with a good Insight if

32:59possible so if you can notice there's a

33:01diagram in front of your screen so over

33:03here we have this field which is not

33:04defined similarly this field and if you

33:06can notice here when we talk about this

33:08minimum value you see the other minimum

33:10values and when you talk about this it

33:12is it is way more than the other fields

33:15present in this particular column

33:16similarly goes for this particular

33:18element as well okay so obviously

33:21processing data like this is one

33:23problematic thing and now I get it why

33:26Big Data is a problem statement well we

33:29have only five vs now but maybe later on

33:31we'll have more so there are good

33:33chances that big data might be even more

33:36big okay so there are a lot of problems

33:39in dealing with big data but there are

33:41always different ways to look at some

33:43things so let us get some positivity in

33:45the environment now and let us

33:48understand how can we use Big Data as an

33:50opportunity yes RMA and I would say the

33:53situation is similar to the proverb when

33:55life throws you lemons make

33:57lemonade yeah so let us go through the

34:00fields where we can use Big Data as a

34:02boon and there are certain unknown

34:05problems solved only because we started

34:07dealing with big data and the Boon that

34:09you're talking about RMA is big data

34:11analytics first thing with big data we

34:13figured out how to store our data cost

34:16effectively we were spending too much

34:18money on storage before until Big Data

34:20came into the picture we never thought

34:22of using commodity Hardware to store and

34:25manage a data which is both reliable and

34:27feasible as compared to the costly

34:29servers now let me give you a few

34:31examples in order to show you how

34:33important big data analytics is nowadays

34:35so when you go to a website like Amazon

34:37or YouTube or Pandora Netflix any other

34:40website so they'll actually provide you

34:42certain fields in which they'll

34:43recommend some products or some videos

34:45or some movies or some songs for you

34:47right so how do you think they do that

34:49so basically whatever data that you are

34:51generating on these kind of websites

34:53they make sure that they analyze it

34:54properly and let me tell you guys that

34:57data is not small it is actually big

34:59data now they analyze that big data and

35:02they make sure that whatever you like or

35:04whatever your preferences are

35:05accordingly they'll generate

35:06recommendations for you and when I go to

35:09YouTube I don't know if you guys have

35:10noticed it but I'm pretty sure you must

35:12have done that so when I go to YouTube

35:14YouTube knows what song or what video

35:16that I want to watch next similarly

35:18Netflix knows what kind of movies are

35:20like and when I go to Amazon it actually

35:23shows me what all products that I would

35:25prefer to buy right so so how do you

35:27think it happens it happens only because

35:28of big data analytics okay so there is

35:31one more example that just popped into

35:33my mind I'll share with you guys so

35:36there was this time when the Hurricane

35:38Sandy was about to hit on New Jersey in

35:41United States so what happened then the

35:44Walmart used big data analytics to

35:47profit from it now I'll tell you how

35:49they did it so what Walmart did is that

35:52they studied the purchase patterns of

35:55different customers when a Hur hurri is

35:57about to strike or any kind of natural

35:58Calamity is about to strike on a

36:00particular area and when they made an

36:02analysis of it so they found out that

36:05people tend to buy emergency stuff like

36:08flashlight life jackets and a little bit

36:11of other stuff and interestingly people

36:13also buy a lot of strawberry poptart

36:17strawberry poptarts are you serious yeah

36:20now I didn't do that analysis so I

36:22Walmart did that and apparently it is

36:25true so what they did is so they stuffed

36:28all their stores with a lot of

36:29strawberry Pop-Tarts and emergency stuff

36:32and obviously it was sold out and they

36:34earned a lot of money during that time

36:37but my question here RMA is people want

36:39to die eating strawberry poptarts like

36:42what was the idea behind strawberry

36:43poptarts I'm pretty unsure about it but

36:45yeah since you have given us a very

36:47interesting example and Walmart did that

36:49analysis we didn't do it so yeah so it

36:51is a very good example in order to

36:53understand how big data analytics can

36:55help your business to grow and find

36:57better insights from the data that you

36:59have yeah and also if you want to know

37:01why strawberry poptarts maybe later on

37:04we can start making an analysis by

37:05gathering some more data also yeah that

37:08can be possible okay so now let's move

37:11ahead and take a look at a case study by

37:14IBM how they have used big data

37:16analytics to profit their company so if

37:19you have noticed that earlier the data

37:21that was collected from The Meters that

37:23you have in your home that measures the

37:24electricity consumed it is actually

37:26sending data after one month but

37:29nowadays what IBM did they came up with

37:31this thing called smart meter and that

37:33smart meter used to collect data after

37:35every 15 minutes so whatever energy that

37:38you have consumed after every 15 minutes

37:40it will send that data and because of it

37:42big data was generated so we have some

37:44stats here which says that we have 96

37:47million reads per day for every million

37:50meters which is pretty huge this data

37:52the amount of data that is generated is

37:54pretty huge now IBM actually realize the

37:57data that they're generating it is very

37:59important for them to gain something

38:01from that data so for that what they

38:03need to for that what they need to do

38:04they need to make sure that they analyze

38:06this data so they realize that big data

38:08analytics can solve a lot of problems

38:11and they can get better business Insight

38:13through that so let us move forward and

38:14see what type of analysis they did on

38:16that data so before analyzing that data

38:19they came to know that energy

38:20utilization and billing was only

38:23increasing now after analyzing Big Data

38:25they came to know that during Peak load

38:27the users require more energy and during

38:30off peak times the users require less

38:32energy so what advantage they must have

38:34got from this analysis one thing that I

38:36can think of right now is they can tell

38:39the industries to use their Machinery

38:41only during the off peak times so that

38:43the load will be pretty much balanced

38:45and you can even say that time of use

38:47pricing encourages cost savy retail like

38:50industrial heavy machines to be used off

38:52peak time so yeah they can save money as

38:55well because off peak times pricing will

38:58be less than the peak time prices right

39:01so this is just one analysis now let us

39:03move forward and see the IBM Suite that

39:05they developed so over here what happens

39:09you first dump all your data that you

39:11get in this data warehouse after that it

39:13is very important to make sure that your

39:15user data is secure then what happens

39:18you need to clean that data as I've told

39:20you earlier as well there might be many

39:21feeds that you don't require so you need

39:23to make sure that you have only useful

39:24material or useful data in your data set

39:27and then you perform certain analysis

39:30and in order to use this Suite that IBM

39:32offered you efficiently you have to take

39:34care of a few things the first thing is

39:36that you have to be able to manage the

39:38smart meter data now there is a lot of

39:41data coming from all this million Smart

39:43Meters so you have to be able to manage

39:45that large volume of data and also be

39:48able to retain it because maybe later on

39:50you might need it for some kind of

39:52regulatory requirements or something and

39:55next thing you should keep in mind is is

39:56to monitor the distribution grid so that

39:59you can improve and optimize the overall

40:01grid reliability so that you can

40:03identify the abnormal conditions which

40:05are causing any kind of problem and then

40:07you also have to take care of optimizing

40:10the unit commitment so by optimizing the

40:12unit commitment the companies can

40:14satisfy their customers even more they

40:16can reduce the power outages that is the

40:19they can reduce the power outages so

40:21that their customers don't get angry

40:23more identify problems and then reduce

40:25it obviously

40:27and then you have also to optimize the

40:29energy trading so it means that you can

40:31advise your customers when they should

40:33use their appliances in order to

40:35maintain that balance in the power load

40:38and then you also have to forecast and

40:40schedule loads so companies must be able

40:43to predict when they can profitably sell

40:45the Excess power and when they need to

40:48hedge the supply and continuing from

40:51this now let's talk about how Encore

40:53have made use of the IBM solution so

40:56Encore is an electric delivery company

40:59and it is the largest electrical

41:01distribution and transmission company in

41:03Texas and it is one of the six largest

41:06in the United States they have more than

41:08three million customers and their

41:10service area covers almost

41:13117,000 square miles and they began the

41:17advanced speeder program in 2008 and

41:20they have deployed almost 3.25 million

41:23meters serving customers of North and

41:27Central Texas so when they were

41:29implementing it they kept three things

41:31in mind the first thing was that it

41:34should be instrumented so this solution

41:36utilizes the smart electricity meters so

41:39that they can accurately measure the

41:41electricity usage of a household in

41:44every 15 minutes because like we

41:46discussed that the smart meters were

41:47sending out data every 15 minutes and it

41:50provided data inputs that is essential

41:52for consumption insights next thing is

41:55that it should be interconnected so now

41:58the customers have access to the

41:59detailed information about the

42:01electricity they are consuming and it

42:03creates a very Enterprise wide view of

42:06all the meter assets and it helped them

42:08to improve the service delivery the next

42:11thing is to make your customers

42:13intelligent now since it is getting

42:15monitored already about how each of the

42:17household or each customer is consuming

42:19the power so now they're able to advise

42:22the customers about maybe to tell them

42:24to wash their clothes at night because

42:27they're using a lot of appliances during

42:28the daytime so maybe they could divide

42:30it up so that they can use some

42:32appliances at off peak hours so that

42:34they can even save more money and this

42:37is beneficial for both of them for both

42:39the customers and the company as well

42:42and they have gained a lot of benefits

42:44by using the IBM solution so what are

42:47the benefits they got is that it enabled

42:50oncore to identify and fix outages

42:53before the customers get inconvenience

42:55that means they were able to ident

42:56identify the problem before it even

42:57occurred and it also improved the

42:59emergency response on events of severe

43:02weather events and views of outages and

43:05it also provides the customers the data

43:07needed to become a active participant in

43:09the power consumption management and it

43:12enabled every individual household to

43:14reduce their electrical consumption by

43:16almost 5 to 10% and this is how oncore

43:20used the IBM solution and made huge

43:23benefits out of it just by using big

43:25data analytics that IBM performed but

43:27let me just interrupt right now so since

43:30RMA told us in the beginning as well

43:32that there are no free lunches in life

43:34right so this is an opportunity but

43:36there are many problems to encase this

43:38opportunity right so let us focus on

43:40those problems one by

43:42one so the first problem is storing

43:44colossal amount of data so let's discuss

43:47few starts that are there in front of

43:49your screen so data generated in the

43:51past 2 years is more than the previous

43:53history in total so guys what are we

43:55doing toop generating so much amount of

43:58data and it said that by 2020 total

44:01Digital Data will grow to 44 Zab bytes

44:04approximately and there's one more stat

44:06that amazes me is about 1.7 MB of new

44:11information will be created every second

44:13for every person by 2020 so storing this

44:16huge data in traditional system is not

44:19possible the reason is obvious the

44:21storage will be limited for one system

44:24for example you have a with a storage

44:26limit of 10 terab but your company is

44:29growing really fast and data is

44:31exponentially increasing now what you'll

44:33do now at one point you'll exhaust all

44:35the storage so investing in huge servers

44:38is definitely not a cost effective

44:41solution so RMA what do you think what

44:43can be the solution to this problem uh

44:45according to me a distributed file

44:48system will be a better way to store

44:50this huge data because with this we'll

44:52be uh saving a lot of money let me tell

44:54you how because due to this distributed

44:57system you can actually store your data

45:00in commodity Hardware instead of

45:02spending money on high-end servers don't

45:04you agree sort of completely now we know

45:07storing is a problem but let me tell you

45:09guys it is just one part of the problem

45:11let's see few more okay so since we saw

45:15that the data is not only huge but it is

45:18present in various formats as well like

45:21unstructured semi-structured and

45:23structured so you not only need to store

45:26this huge data but you also need to make

45:28sure that a system is present to store

45:31this varieties of data generated from

45:33various sources and now let's focus on

45:35the next problem now let's focus on the

45:38diagram so over here you can notice that

45:40the hard disk capacity is increasing but

45:43the disc transfer performance or speed

45:45is not increasing at that rate let me

45:47explain you this with an example if you

45:50have only 100 MVPs input output Channel

45:54and you are processing say 1 ter of data

45:56now how much time will it take maybe

45:59calculate it'll be somewhere around 2.91

46:03hours right so it'll be somewhere around

46:062.91 hours and I have taken an example

46:09of 1 terabytes what if you're processing

46:11some Zab bytes of data so you can

46:13imagine how much time will it take now

46:15what if you have four input output

46:17channels for the same amount of data

46:20then it'll take

46:22approximately 72 hours or converted to

46:25minutes so it be around 43 minutes

46:27approximately right and now imagine

46:30instead of 1 TB you have zabt of data

46:32for me more than storage accessing and

46:34processing speed for huge data is a

46:37bigger problem okay so RMA has a very

46:39good example to discuss yeah so since

46:41you were talking about accessing the

46:43data and you told us already about how

46:46Amazon at different websites and YouTube

46:48they make those recommendations so if

46:51there was no solution for it if it would

46:53take so much time to access the data the

46:55recommendation system won't work at all

46:58and they make a lot of money just for

47:00recommendation system because a lot of

47:02people go there and click over there and

47:04buy that product right so let's consider

47:06that that it is taking like hours or

47:08maybe years of time in order to process

47:10my that big amount of data so let's say

47:13that at one time I purchased an iPhone

47:165s from Amazon and after two years I'm

47:20again browsing onto Amazon and since it

47:23took so much time to access the data and

47:25I already switched over to a new iPhone

47:29and they are recommending me the old

47:31iPhone case for 5S so obviously that

47:33won't work I won't go there and click it

47:35because I've already changed my phone

47:37right so that will be a huge problem for

47:40Amazon the recommendation system won't

47:42work anymore and I know that RMA changes

47:45her phone every year so if she has

47:47bought a phone and people are

47:50recommending if she has bought a phone

47:51now and someone's recommending the case

47:54for that phone after 2 years years

47:56doesn't make sense to me at all yeah

47:58only it will work if I have both the two

48:01phones at the same time but yeah I don't

48:03want to waste money on purchasing new

48:05iPhone case for my old phone so

48:08basically it won't be fair if we don't

48:10discuss the solution to these problems

48:12RMA we can't leave our viewers with just

48:15the problems right it won't be fair what

48:17is the solution had doop Hadoop is a

48:20solution so let's introduce Hadoop now

48:24okay so now what is had

48:26so Hadoop is a framework that allows you

48:29to first store big data in a distributed

48:31environment so that you can process it

48:34parallel there are basically two parts

48:37one is hdfs that is Hadoop distributed

48:39file system for storage it allows you to

48:42store data of various formats across a

48:45cluster and the second part is map

48:47reduce now it is nothing but a

48:49processing unit of Hadoop it allows

48:51parallel processing of data that is

48:54stored across the hdfs now let us dig

48:57deep in hdfs and understand it better

49:01yeah so hdfs creates an abstraction of

49:04resources um let me simplify it for you

49:07so similar to virtualization you can see

49:09hdfs logically as a single unit for

49:11storing big data but actually you're

49:14storing your data across multiple

49:16systems or you can say in a distributed

49:18fashion so here you have a Master Slave

49:21architecture in which the name node is a

49:23master node and the data node are slaves

49:26and the name note contains the metadata

49:28about the data that is stored in the

49:30data nodes like which data block is

49:33stored in which data node where are the

49:36replications of the data block kept and

49:38etc etc so the actual data is stored in

49:42the data nodes and I also want to add

49:44that we actually replicate the data

49:46blocks that is present in the data nodes

49:49and by default the replication factor is

49:51three so it means that there are three

49:53copies of each file so s of can you tell

49:56us why do we need that replication sure

49:59RMA since we are using commodity

50:01Hardwares right and we know failure rate

50:03of these Hardwares are pretty high so if

50:06one of the data notes fail I won't have

50:08that data block and that's the reason we

50:10need to replicate the data block now

50:12this replication Factor depends on your

50:14requirements right now let us understand

50:17how actually Hadoop provided the

50:18solution to the big data problems that

50:20we have discussed so RMA can you

50:23remember what was the first problem yeah

50:25it was storing the big data so how hdfs

50:29solved it let's discuss it so hdfs

50:32provides a distributed way to store Big

50:34Data we've already told you that so your

50:37data is stored in blocks in data nodes

50:39and you then specify the size of each

50:42block so basically if you have a 512 MB

50:45of data and you have configured hdf as

50:48such that it will create 128 megabytes

50:51of data block so hdfs will so hdfs will

50:54divide the data in four blocks because

50:5652 divid by 128 is four and it will

51:00store it across different data nodes and

51:02it will also replicate the data blogs on

51:05the different data nodes so now we are

51:07using commodity hardware and storing is

51:09not a challenge so what are your

51:11thoughts on it sort of I will also add

51:13one thing RMA it also solves the scaling

51:16problem it focuses on horizontal scaling

51:18instead of vertical now you can always

51:20add some extra data nodes to your hdfs

51:22cluster ads and when required instead of

51:25scaling the resources of your data nodes

51:27so you're not actually increasing the

51:29resources of your data nodes you're just

51:30adding few more data nodes when you

51:32require let me summarize it for you so

51:35basically for storing one TB of data I

51:37don't need a one TB system I can instead

51:40do it on multiple 128 GB systems or even

51:43less now RMA what was the second

51:46challenge with big data so the next

51:49problem was storing variety of data and

51:52that problem was also addressed by hdfs

51:55so with htfs you can store all kinds of

51:58data whether it's structured

51:59semi-structured or unstructured it is

52:02because in htfs there is no pre- dumping

52:04scheme of validation so you can just

52:06dump all the kinds of data that you have

52:08in one place and it also follows a write

52:11once and read many model and due to this

52:14you can just write the data once and you

52:16can read it multiple times for finding

52:18out insights and if you can recall the

52:21third challenge was accessing the data

52:23faster and this is one of the major

52:26challenge with big data and in order to

52:29solve it we're moving processing to data

52:31and not data to processing so what it

52:34means sort of just go ahead and explain

52:37it yes RMA I will so over here let me

52:40explain you what do you mean by actually

52:42moving process to data so consider this

52:45as our master and these are our slaves

52:48so the data is stored in these slaves so

52:50what happens one way of processing this

52:52data is what I can do is I can send this

52:54data to my Master node and I can process

52:57it over here but what will happen if all

53:00of my slaves will send the data to my

53:02master node it'll cause Network

53:04congestion plus input output Channel

53:06congestion and at the same time my

53:08master node will take a lot of time in

53:10order to process this huge amount of

53:12data so what I can do I can send this

53:14process to data that means I can send

53:17the logic to all these slaves which

53:19actually contain the data and perform

53:22processing in the slaves itself so after

53:24that what will happen the small chunks

53:26of the result that will come out will be

53:28sent to our name node so in that way

53:30there won't be any network congestion or

53:32input output congestion and it will take

53:34comparatively very less time so this is

53:37what actually means sending process to

53:41[Music]

53:44data so who's a big data engineer now

53:47every data driven business needs to have

53:49a framework in place for the data

53:51science and data analytics Pipeline and

53:54a data engineer is the one who's

53:56responsible for building and maintaining

53:58this framework now these Engineers must

54:00ensure that there is an uninterrupted

54:02flow of data between servers and

54:04applications so in simple words a data

54:07engineer builds tests maintains data

54:10structures and architectures for data

54:12ingestion processing and deployment of

54:15large-scale data intensive applications

54:18now data Engineers work in tandem with

54:20data Architects data analysts and data

54:22scientists so they must all share comp

54:25these insights to other stakeholders in

54:28the company through data visualization

54:30and storytelling but what does a big

54:32data engineer do exactly now the most

54:35crucial part of a big data engineer is

54:37to design develop construct install test

54:40and maintain the complete data

54:42management and processing systems they

54:45are basically the ones who handle the

54:47complete endtoend infrastructure for

54:49data management and processing they

54:51build a pipeline for data collection and

54:54storage and funnel the data to data

54:56analysts and scientists so basically

54:59what they do is they create the

55:00framework to make data consumable for

55:03data scientists and analysts so they can

55:06use the data to derive insights from it

55:09know that the data Engineers are the

55:11Builders of data systems and not those

55:14who mine for insights so the data

55:16engineer Works more behind the scenes

55:19and must be comfortable with other

55:20members of the team producing Business

55:22Solutions from this data now all there

55:25responsibilities revolve around this

55:27they need to take care of a lot of

55:28things while performing these activities

55:31hence one of the most sought after

55:33skills in data engineering is the

55:35ability to design and build data

55:37warehouses this is where all the raw

55:39data is collected stored and retrieve

55:41from without data warehouses all the

55:44tasks that a data scientist does will

55:46become obsolete it is either going to

55:48get too expensive or very very large to

55:51scale now data Engineers should always

55:53keep in mind that the system which he or

55:56she builds needs to be scalable robust

55:59and fault tolerant so that the system

56:02can be scaled up without increasing the

56:04number of data sources and can handle a

56:06huge amount of heterogeneous data

56:09without any failure now imagine a

56:11situation wherein the source of data is

56:13doubled or tripled but the system cannot

56:15scale up will it not cost a lot more

56:18time and resources to build the same

56:20system again which is suitable for this

56:22kind of intake exactly this is why the

56:26Big Data Engineers have a role here next

56:29he or she is the one that handles the

56:32extract transform and load process which

56:35is basically the blueprint for how the

56:38collected raw data is processed and

56:40transformed into Data ready for analysis

56:43now you're going to acquire a lot of

56:45data from different sources how do you

56:48bring them together to one platform ETL

56:51is your answer apart from all this a

56:54data engine engineer should always aim

56:57at deriving insights by acquiring data

57:00from new sources some of the

57:02responsibilities of a data engineer also

57:04include improving data foundational

57:07procedures integrating new data

57:09management Technologies and the software

57:12into existing systems and building data

57:14collection pipelines and finally one of

57:17the major roles of a data engineer is to

57:20include performance tuning and make the

57:22whole system way more efficient which is

57:24pretty self-explanatory if you ask me

57:27now most of us have some idea about who

57:30a big data engineer is but there's still

57:33some confusion about their

57:36responsibilities now this ambiguity

57:38further increases when we gain more

57:41information about the role now let me

57:43help you debunk all your queries about

57:46it so let's talk about some big data

57:48engineer

57:49responsibilities first up we have data

57:52ingestion now this is associated with

57:54the task of getting data out of the

57:56source systems and ingesting it into a

57:58data Lake now a data engineer would need

58:01to know how to efficiently extract the

58:03data from a source including multiple

58:05approaches for both batch and real-time

58:07extraction as well as needing to know

58:10about the incremental data loading

58:12fitting within small Source windows and

58:14parallelization of data loading as well

58:17now another small subtask of data

58:19ingestion is data

58:21synchronization but because it's such a

58:23big issue in the Big Data world we are

58:26going to talk about it now since Hadoop

58:28and other big data platforms don't

58:30support incremental loading of data a

58:33data engineer would need to know how to

58:34deal with detecting changes in the data

58:37source merge and sync change data from

58:40sources into the Big Data environment

58:42next we have data transformation this is

58:45basically the T in the extract transform

58:48and load that we had discussed earlier

58:50it is basically focused on integration

58:52and transformation of data for a

58:54specific use case now a major skill set

58:56here is the knowledge of SQL as it turns

58:59out not much has changed in terms of the

59:01type of data Transformations that people

59:03are doing now compared to purely

59:05relational environments now imagine all

59:08this data that you've acquired from

59:10various sources what would you have to

59:12do to make them all palatable in the

59:14same platform you need to transform that

59:17data and this is what a data engineer

59:19does here and finally we have

59:22performance optimization which is one of

59:24the tougher areas because anyone can

59:26build a slow performing system the

59:28challenge is to build data pipelines

59:30that are both scalable and efficient so

59:33the ability and understanding of how to

59:35optimize the performance of an

59:37individual data Pipeline and the overall

59:39systems are a higher level of data

59:41engineering skill now for example Big

59:44Data platforms continue to be

59:45challenging with regard to query

59:47performance and have added complexity to

59:49a data engineer's job in order to

59:51optimize performance of queries and

59:53creation of reports BS the data engineer

59:56needs to know how to denormalize

59:58partition and index data models he also

1:00:01needs to understand tools and Concepts

1:00:04regarding in-memory models and olap

1:00:06cubes now let's quickly move ahead and

1:00:09look at the required skills to fulfill

1:00:11these

1:00:12responsibilities now we'll be going

1:00:14through these skills in a clockwise

1:00:16order so starting with big data

1:00:18Frameworks now with the rise of big data

1:00:20in the early 21st century a new

1:00:22framework was born and that is Hadoop

1:00:24all thanks to Doug cutting for

1:00:26introducing this framework it not only

1:00:28stores big data in a distributed manner

1:00:31but also processes the data parallell

1:00:33there are several tools in the Hadoop

1:00:35ecosystem which cater differently for

1:00:37different purposes and Professionals for

1:00:39a big data engineer mastering Big Data

1:00:42tools is a must some of the tools which

1:00:45you will need to Master first of all you

1:00:47have hdfs which is the storage part of

1:00:49Hadoop being the foundation of Hadoop

1:00:52knowledge of hdfs is a must to start

1:00:54working working with Hadoop framework

1:00:56next we have yarn which performs

1:00:58resource management by allocating

1:01:00resources to different applications and

1:01:02scheduling jobs Now map ruce is a

1:01:05parallel processing Paradigm which

1:01:08allows data to be processed parallely on

1:01:10top of the hdfs next we have pig and

1:01:13Hive now Hive is a data warehousing tool

1:01:16on top of hdfs which caters to

1:01:18professional from an SQL background to

1:01:21perform analytics on top of hdfs whereas

1:01:23apachi pig is a high level platform

1:01:26which is used for data transformation on

1:01:28top of Hado now Hive is generally used

1:01:31by data analyst for creating reports

1:01:33whereas pig is used by researchers for

1:01:35programming both are pretty easy to

1:01:37learn if you already familiar with SQL

1:01:40next we have Flume and scoop Flume is a

1:01:43tool which is used to import

1:01:45unstructured data to hdfs and scoop is

1:01:48used to Import and Export structured

1:01:50data from our dbms now next we have

1:01:53zookeeper which acts as a coordinator

1:01:56among the distributed Services running

1:01:58in a Hadoop environment it basically

1:02:00helps to configure management and

1:02:02synchronized services and finally we

1:02:05have Uzi which is basically a scheduler

1:02:07which binds multiple logical jobs

1:02:10together and helps in accomplishing a

1:02:12complete task next up we have realtime

1:02:15processing Frameworks now real-time

1:02:17processing with quick actions is the

1:02:20need of R either it is a credit card

1:02:22fraud detection system or a Rec

1:02:24recommendation system now imagine if you

1:02:26wanted a red dress today and Amazon

1:02:29decides to suggest it to you a month

1:02:31later now wouldn't that be completely

1:02:33useless for you in this case you need

1:02:36realtime processing it is very important

1:02:38for a data engineer to have knowledge of

1:02:41realtime processing Frameworks now

1:02:43Apachi spark is one of the distributed

1:02:46real-time processing Frameworks which is

1:02:48used in the industry rigorously it can

1:02:50be easily integrated with Hadoop

1:02:52leveraging hdfs as well next we have

1:02:55dbms now a database management system

1:02:58stores organizes and manages a large

1:03:01amount of information within a single

1:03:03software application now data Engineers

1:03:06need to understand the database

1:03:07management system to manage data

1:03:09efficiently and allow users to perform

1:03:11multiple tasks with ease this will help

1:03:14data engineers in improve data sharing

1:03:16data security data access and better

1:03:19data integration with minimized data

1:03:22inconsistencies these are the fun

1:03:24mentals that data Engineers should know

1:03:27prior to building a scalable robust and

1:03:29fall tolerance system next we have SQL

1:03:32based Technologies now there are various

1:03:34relational databases that are used in

1:03:36the industry such as Oracle DB Microsoft

1:03:39SQL Server Etc now data Engineers must

1:03:43have at least the knowledge of one such

1:03:45database now knowing SQL is also a must

1:03:49this structured query language as SQL is

1:03:52also known as used to structure

1:03:54manipulate and manage data stored in

1:03:57relational databases as data Engineers

1:03:59work closely with

1:04:01rdbms's they need to have a strong

1:04:03command on SQL now next we have no SQL

1:04:07Technologies as the requirements of

1:04:09organizations have grown Beyond

1:04:11structured data no SQL databases have

1:04:15been introduced into this environment it

1:04:17can store large volumes of structured

1:04:19semi-structured or unstructured data

1:04:21with quick iteration and agile structure

1:04:24as per application requirements some of

1:04:26the most prominently used databases are

1:04:29hbas Cassandra and mongodb now hbase is

1:04:33a column oriented nosql database on top

1:04:36of hdfs which is great for scalable and

1:04:39distributed Big Data stores it is also

1:04:41great for applications with optimized

1:04:43read and range based scan and it

1:04:46provides consistency and partitioning

1:04:48out of capap now Cassandra is a highly

1:04:51scalable database with incremental

1:04:53scalability and the best part about

1:04:55Cassandra is the minimal Administration

1:04:58and no single point of failure it's good

1:05:01for applications with fast and random

1:05:03read and writs it provides available and

1:05:06partitioning out of capap and finally we

1:05:09have mongodb which is basically a

1:05:12document oriented nosql database which

1:05:15is a schema free database it gives full

1:05:17index support for high performance and

1:05:20replication for fall tolerance it has a

1:05:23Master Slave sort of architecture and

1:05:25provides CP out of capap it is

1:05:28rigorously used by web applications and

1:05:31semi-structured data handling next we're

1:05:34going to discuss programming and

1:05:35scripting languages so various

1:05:37programming languages can serve for the

1:05:39same purpose so knowledge of one

1:05:41programming language is enough I'm

1:05:43saying this because the flavor of

1:05:44language may change but the logic

1:05:46Remains the Same if you're a beginner

1:05:48you can go ahead with python as it is an

1:05:51easy language to learn due to its syntax

1:05:53and good good Community Support whereas

1:05:55R has a steep learning curve which is

1:05:58developed by statisticians and it is

1:06:00mostly used by analysts and data

1:06:02scientists the next skill we're going to

1:06:05discuss is an important one it is ETL or

1:06:08data warehousing now data warehousing is

1:06:11very important when it comes to managing

1:06:13a huge amount of data coming in from

1:06:15heterogeneous sources where you need to

1:06:17apply extract transform and load now

1:06:20data warehousing is used for analytics

1:06:23and Reporting and is a very very crucial

1:06:25part of every business intelligence

1:06:27solution because this is the part which

1:06:30is going to take you most time now it is

1:06:32very important for a big data engineer

1:06:34to Master One data warehousing or ETL

1:06:36tool after mastering one it becomes

1:06:39pretty easy to learn new tools and as

1:06:41the fundamentals remain the same now

1:06:43Informatica click View and talent are

1:06:46very well-known tools used in the

1:06:48industry Informatica and talent Open

1:06:50studio are data integration tools with

1:06:53ETL architecture

1:06:54the major benefit of talent is its

1:06:57support from the Big Data Frameworks if

1:06:59you're new to data warehousing and ETL

1:07:01tools I would definitely recommend you

1:07:03start with talent because after learning

1:07:06this any data warehousing tools will

1:07:08become a piece of cake in finally we

1:07:10have our operating systems now intimate

1:07:13knowledge of Unix Linux and Solaris is

1:07:17very helpful as many mathematical tools

1:07:19are going to be based off of these

1:07:21systems due to their unique demands for

1:07:24root access to hardware and operating

1:07:26system functionality above and beyond

1:07:28that of Microsoft's Windows or Mac OS

1:07:31now some level of understanding of how

1:07:33to act upon this data is also very

1:07:35valuable for data Engineers for this

1:07:38reason some knowledge of statistical

1:07:39analysis and the basics of data modeling

1:07:42are also hugely valuable knowledge of

1:07:45machine learning in Cloud also will

1:07:46serve as a big plus while machine

1:07:48learning is technically something

1:07:50relegated to a data scientist knowledge

1:07:52in this area is helpful to construct

1:07:54Solutions usable by your cohorts now

1:07:57this knowledge has the added benefit of

1:07:59making you extremely marketable in this

1:08:02space as being able to put on both hats

1:08:05in which case makes you a really

1:08:06formidable

1:08:07[Music]

1:08:13tool here you can see the job

1:08:16distribution per salary range in India

1:08:19for a data engineer as we can see people

1:08:22who get paid more than5 inom are about

1:08:2533% 730 perom are 26% 8 lak 70,000 about

1:08:3220% Which are very high salary brackets

1:08:36apart from that the average salary for

1:08:38data engineer is almost 8 lakhs inim and

1:08:41for a senior data engineer is almost 16

1:08:44lakhs in anim if we look at the same

1:08:47numbers in the US there are 32% of

1:08:50professionals who make more than $90,000

1:08:53a a year and 27% professionals who make

1:08:57$105,000 a year the average salary in

1:09:00the US for a data engineer is way more

1:09:03than

1:09:04$90,000 and for a senior data engineer

1:09:07it is

1:09:09$124,000 per anom now as we have

1:09:12discussed the salary of a big data

1:09:14engineer let's look at a few factors in

1:09:17the form of skills and technology that

1:09:19they know on which their salary depends

1:09:23here we've carefully curated a table

1:09:25which lists out the skills and the

1:09:28average salary which can be encashed

1:09:30through them you can see services such

1:09:33as AWS data analysis data mining

1:09:36warehousing machine learning and even

1:09:39programming languages like Java and R

1:09:43apart from that you can see bi tools and

1:09:46statistical tools like Tablo database

1:09:49architecture ETL and structured query

1:09:52languages now another another influence

1:09:54on the salary is experience because

1:09:57experience is also a very important

1:10:00factor in deciding the Big Data engineer

1:10:03salary the distribution of salary is

1:10:06like so an entry-level data engineer

1:10:09makes about

1:10:11$85,000 a year people who have 5 to8

1:10:14years of work experience bag nearly

1:10:18$113,000 a year and people who are

1:10:20experienced I'm talking like 10 years of

1:10:22Industry experience experience get over

1:10:26$118,000 a year now this salary must be

1:10:30coming from somewhere presenting to you

1:10:33the companies that hire in this job role

1:10:36as you can see there are some very big

1:10:38names like Amazon Google Bosch Microsoft

1:10:42and IBM who hire Big Data Engineers now

1:10:46companies that hire Big Data

1:10:48professionals are companies that are

1:10:50invested in the future and the worldwide

1:10:53big data market revenues for software

1:10:55and services are projected to increase

1:10:58from $42 billion in 2018 to $13 billion

1:11:03in

1:11:042027 attaining a compound annual growth

1:11:07rate of about

1:11:0910.5% that sort of a growth needs some

1:11:12kind of work after going through

1:11:15multiple job descriptions we found that

1:11:17the Big Data engineer salary has many

1:11:21[Music]

1:11:22variables

1:11:27so let's go ahead and also understand

1:11:29the average salary of a data engineer so

1:11:31in the US the average salary of a data

1:11:34engineer is

1:11:36$133,000 but remember guys this is

1:11:39basically an average salary which

1:11:41basically means that there are people

1:11:42who are above this salary or can even be

1:11:44a person who is below this salary okay

1:11:47but later in this session I'm also going

1:11:49to talk about some of the job

1:11:51description which can tell you the kind

1:11:52of salary that you can earn once you

1:11:55start applying for a data engineer

1:11:56profile in India the salary is around

1:11:596.5 lakhs per anom or a 7 lakh perom and

1:12:03the same goes for an Indian jobs as well

1:12:05that the salary can be higher than this

1:12:07average package or it can be lower than

1:12:10the package as well now that you have

1:12:12seen the basic or an average scale of a

1:12:15aor data engineer let's go ahead and see

1:12:18the job description of an aor data

1:12:20engineer all right guys so thinking

1:12:22again about what a data engineer does

1:12:25and aor data engineer is responsible for

1:12:28Designing implementing and maintaining

1:12:30data management and data processing

1:12:32system on the Microsoft Azor Cloud

1:12:35platform now they work with large and

1:12:37complex data sets and are responsible

1:12:39for ensuring that data is stored

1:12:41processed and secure efficiently and

1:12:43effectively now when you basically

1:12:45Define a job role like that you have

1:12:47many jobs description which are floating

1:12:49in the market right now how can you

1:12:52identify which job description you have

1:12:55to apply to now let's go ahead and

1:12:57understand that so talking about the job

1:13:00description which basically exists in

1:13:02the market guys you will see a job

1:13:04description which is an entry-level job

1:13:06description and then on the next slide I

1:13:09will show a job description which is a

1:13:11mid or a senior level job description

1:13:13all right so heading back to the entry

1:13:15level so if you see the first job

1:13:17description which is an aor data

1:13:19engineer the salary is anywhere around 6

1:13:22lakh perom to4 14 lakhs perom and these

1:13:26are the skills that you require now as I

1:13:29mentioned before these are the expected

1:13:31skills required for an aor data engineer

1:13:35now what are the skills which are

1:13:37expected now you are expected to know no

1:13:39SQL or a cosmos database skill now they

1:13:42are expected to know data Lake data

1:13:45factory data warehouse and strong

1:13:47experience in building pipelines in Azor

1:13:49data or in aor data Lake then you should

1:13:52be able to analyze and understand

1:13:54complex data you should be able to

1:13:56understand business requirement and

1:13:58actively provides input from data

1:14:00perspective now at the same time if you

1:14:02look at these skill sets these skill

1:14:04sets are all the scales at which you

1:14:06will basically know after you study for

1:14:09or clear the aor data engineer

1:14:11certification so once you're done with

1:14:13the certification once you are done with

1:14:14the skill sets which is just there in

1:14:16the certification you can easily go

1:14:18ahead and apply for a job that lies in

1:14:20the salary range which is for an zor

1:14:22data engineer

1:14:23now talking about a mid senior level

1:14:25data engineer profile the list is quite

1:14:28long as you can see now you can see that

1:14:31over here apart from all these skill

1:14:32sets a lot of other things are also

1:14:34mentioned here as well for example you

1:14:36should have some four or five plus years

1:14:38of experience in implementing or

1:14:41designing solution using Azor Big Data

1:14:43Technologies then you should have an

1:14:45experience with an Hands-On in Azor data

1:14:47factory aor devops aor data Lake storage

1:14:50Etc now you should have a knowledge of

1:14:53big data pipeline then design and build

1:14:55modern data pipelines and maintain the

1:14:57data warehouse schematics layouts

1:14:59architecture and relational or

1:15:01non-relational database for data access

1:15:03and advanced analytics now you should

1:15:06also know Java jQuery SQL or Scala or

1:15:10any preferred programming language right

1:15:12so over here what we recommend to our

1:15:14Learners is you should go ahead and

1:15:15Learn Python because although they have

1:15:17mentioned only these programming

1:15:19language but companies are very much

1:15:21flexible if the target profile has any

1:15:23programming experience but a strong one

1:15:25in any of the programming languages

1:15:27right next thing that they expect you to

1:15:29know is Advanced skill using one or most

1:15:32common language for example like python

1:15:34batch Etc so this python will basically

1:15:38serve as a dual purpose that is all a

1:15:40scripting language and a programming

1:15:42language as well the next thing that

1:15:44they expect you to know is the ETL

1:15:46process using big data Technologies such

1:15:49as spark Kafka Hadoop and others now the

1:15:52next thing that they they expect you to

1:15:53know is the ETL process using big data

1:15:56technology such as spark Kafka doops and

1:16:00others and then you need to understand

1:16:01the data visualization experience using

1:16:04python python here is a plus guys so you

1:16:07need to understand this and then you

1:16:08actually need to learn tableu or a

1:16:10powerbi any one of the tools or

1:16:13technology is a plus point so you either

1:16:15have to have a skill on tblo or a

1:16:19powerbi so this again is an important

1:16:21skill to have and this again coincide

1:16:24with what a data analyst does right

1:16:27because he is also responsible for data

1:16:29visualization to some extent then you

1:16:31have a solid understanding and

1:16:33experience implementing cloud data

1:16:34platform in Microsoft aor devops so if

1:16:38you're a guy who wants to start off you

1:16:39can start off with the freshia profile

1:16:42and after having experience in the

1:16:43freshia profile and learn all the skill

1:16:45sets like big data zor these are the

1:16:48skill sets that if you gain you can

1:16:50actually apply to the senior or midlevel

1:16:52in particular particular okay so Guys

1:16:54these are the few job description which

1:16:56are related to the data engineer profile

1:16:59now that we are clear with why who and

1:17:01what are the career opportunities and

1:17:03salary of an aor data engineer we shall

1:17:05see the path towards becoming an aor

1:17:08data engineer so first of all you will

1:17:10have to talk about the different aor

1:17:12storage which are out there now you'll

1:17:14have to learn about these like the blob

1:17:16storage table storage file storage and

1:17:19the que storage now you will have to

1:17:21learn about the relational database

1:17:22option options as well which are there

1:17:24in the market such as the SQL database

1:17:27SQL DB Warehouse analysis Services now

1:17:30you will also have to learn about no SQL

1:17:33you will have to learn about big data

1:17:35services in Azor like data Lake

1:17:37analytics data Lake storage now you also

1:17:40have to learn about the data factories

1:17:42aor function stream analytics iot hubs

1:17:45even hubs Etc now apart from that you

1:17:48will also need to learn redist cache and

1:17:50aor search so these are the servic that

1:17:53are basically required for you to

1:17:55understand in order to clear the

1:17:56certification and go ahead and become an

1:17:59aor data engineer now these are also

1:18:01Conde with the job description that we

1:18:03had a look earlier right now all the

1:18:05services which are mentioned there is

1:18:07basically a part of what you have

1:18:08learned in order to correct the

1:18:10certification now apart from this we

1:18:12also recommend going through the open

1:18:14source Services of a doop such as spark

1:18:18hi and the Hadoop itself right now let

1:18:21us go step by step to reach our goal in

1:18:23becoming an aor data engineer so in

1:18:26order to become an aor data engineer you

1:18:28will also need to have a strong

1:18:29foundation in data engineering and cloud

1:18:32computing here are some steps you can

1:18:34take to develop the skills and knowledge

1:18:36needed for a career as an aor data

1:18:38engineer so the first one is learn the

1:18:41basics of data engineering now in order

1:18:44to become an Azor data engineer you

1:18:46should first develop a strong foundation

1:18:48in data engineering Concepts such as

1:18:50data modeling data pipelines data

1:18:52processing and data storage now you can

1:18:55learn these Concepts through online

1:18:57courses or books or by working on

1:18:59practical projects now the second is

1:19:02learning or programming language now as

1:19:05an Reser data engineer you will need to

1:19:07proficient in at least one programming

1:19:09languages as I've already mentioned

1:19:11earlier now python is a very popular

1:19:13choice for data engineering but you can

1:19:15also consider learning languages such as

1:19:17Java C or Scala now the next step is you

1:19:21need to learn an SQL now as a data

1:19:23engineer you will be working with large

1:19:25amount of data you've already known that

1:19:27since the name itself as an aor data

1:19:30engineer so you will need to be a

1:19:31proficient in SQL to extract and

1:19:34transform data now the next thing you

1:19:36need to learn is Azor data storage

1:19:38options now as you know Azor offers a

1:19:41range of data storage options such as

1:19:43Azor blob storage Azor data L Storage

1:19:46Azor Cosmos DB Azor SQL database and

1:19:49Azor signups analytics formally SQL

1:19:52database warehouse now you should

1:19:54familiarize yourself with the features

1:19:56and capabilities of these storage

1:19:58options now the next thing you need to

1:20:00learn is Azor data processing

1:20:02Technologies now Azor provides several

1:20:04Technologies for processing data such as

1:20:06Azor stream analytics Azor data bricks

1:20:09Azor data Factory and Azor HD insights

1:20:12you should learn how to use these

1:20:14Technologies to build data pipelines for

1:20:16ingestion transforming and processing

1:20:19data so the next thing you need to learn

1:20:21is aor data management and and security

1:20:24so you as an Azor data engineer should

1:20:26learn about the Azor tools for managing

1:20:28and securing data such as Azor data

1:20:31catalog aor data link security and aor

1:20:34private link so what is the next step

1:20:37the next step is get certified as you

1:20:40know getting certified is very crucial

1:20:42and it's very important in today's

1:20:45generation because all the organization

1:20:48as I mentioned earlier if I just have to

1:20:49go back in my slide I will show you the

1:20:51job description where they have actually

1:20:53mentioned that you need to actually pass

1:20:56the certification all right this is the

1:20:59certification you required that is a dp2

1:21:01200 and a DP 2011 but as for now you

1:21:04don't need dp2 200 or DP 2001 you only

1:21:08have to give one exam which I'll be

1:21:10further talking about it that is the

1:21:12dp23 all right guys so moving ahead

1:21:15again so you need to earn an aor data

1:21:18engineer associate certification by

1:21:20passing the dp23 exam

1:21:23right now what is the next thing you

1:21:25need to do is you need to gain practical

1:21:27experience now the best way to learn aor

1:21:29data engineering is by working on

1:21:31practical projects now we all know this

1:21:34right now you can find data engineering

1:21:36projects on online platform such as

1:21:38keigle or you can work on projects in

1:21:40your own organization or you can enroll

1:21:43with edura and they provide you a tons

1:21:45and tons of Life projects which are

1:21:47developed by the instructor who have

1:21:50already worked as a data engineer in

1:21:52their their organization now the next

1:21:54thing you need to learn is continue

1:21:56learning now as a data engineer you will

1:21:58need to keep your skills and knowledge

1:22:00up to date as Technologies and best

1:22:02practices evolve right now making sure

1:22:05to stay current by learning about new

1:22:06aor data engineering features and

1:22:08participating in professionals

1:22:10development activities all right guys so

1:22:13now let us understand the things that

1:22:14you need to know about this

1:22:15certification that I've just talked

1:22:17about that is your dp23 so guys if

1:22:20you're applying for an aor data engineer

1:22:22associate

1:22:23there used to be two exam that you have

1:22:25to give which I've just mentioned now on

1:22:27the screen in the previous slide right

1:22:29one is implementing an Azor data

1:22:31solution and the next exam is design an

1:22:34Azor data solution now after clearing

1:22:36both these exam it will give you the aor

1:22:38data engineering associate certification

1:22:41and these two exam basically have the

1:22:42code like I've mentioned dp2 200 and DP

1:22:462011 right but now this exam have been

1:22:49retired on 23rd February 2021 now you

1:22:53guys just have to give one exam to clear

1:22:55the data engineer certification and this

1:22:58exam is

1:22:59dp23 now what they have done is they

1:23:02have Club both these exam and they have

1:23:04now included its syllabus in just one

1:23:06examine they're asking questions from it

1:23:09right so earlier what you have to do is

1:23:11you have to pay for two exams then

1:23:13prepare for two exams and give them and

1:23:15then only you can get the certification

1:23:18but now just by passing one exam you can

1:23:20clear out the data engineer

1:23:22certification ation right now as part of

1:23:24the new exams now the skill sets have

1:23:26been updated as you can see on the

1:23:28screen so basically these are the

1:23:30distribution of topics and they'll be

1:23:32covered in this exam now first of all

1:23:34you will be asked most of the questions

1:23:36on design and Implement data storage now

1:23:39this is going to have 40 to 45% of

1:23:42weightage okay now I'm going to tell you

1:23:45what are the topics that comes under the

1:23:47design and Implement data storage so

1:23:49make sure you write it down okay guys so

1:23:51the first is design a data storage

1:23:53structure then the second is design a

1:23:55partitioning strategy then the next

1:23:57question is design the serving layer

1:24:00then you have the Implement physical

1:24:02data storage structures then Implement

1:24:04logical data structure and lastly

1:24:06implement the serving layer now after

1:24:09this you will have design and develop

1:24:12data processing now this is of 25 to 30%

1:24:15weightage now there are four points

1:24:17coming under this design and develop

1:24:19data processing now the first is you

1:24:20need to learn about the inest and

1:24:21transform data second is design and

1:24:24develop a batch processing solution the

1:24:27third is design and develop a stream

1:24:29processing solution and lastly we have

1:24:31the manage batches and pipelines all

1:24:34right so coming to the third is design

1:24:37and Implement data security which is of

1:24:4010 to 15% so there are two main topics

1:24:43here that is design security for data

1:24:45policies and standards and the second is

1:24:47Implement data security Now talking

1:24:50about the last is you have the Monitor

1:24:52and optimize data storage and data

1:24:54processing which is again of 10 to 15%

1:24:57now even here also we have two main

1:25:00topics that you need to be covered that

1:25:01is the monitor data storage and data

1:25:03processing and the second is optimize

1:25:05and troubleshoot data storage and data

1:25:08processing now if you are with me till

1:25:10at this point you shall have understood

1:25:12what are all the things that you have to

1:25:13learn in order to clear the exams and

1:25:16become an aor data engineer right since

1:25:19we have already discussed what are the

1:25:20path and what are the skills to learn

1:25:22learn to prepare ourself so like I said

1:25:24earlier you now just have to give this

1:25:26dp23 exam to get the Microsoft

1:25:29certification associate data engineering

1:25:31certification okay so just one exam and

1:25:34now you will get the certification for

1:25:36it all right so now let's move on guys

1:25:39and now let's talk about how you guys

1:25:41can get started in clearing this exam

1:25:43and developing those skills and go ahead

1:25:46and apply for the job and become a

1:25:47successful aor data engineer so we have

1:25:51mentioned a lot of things that you have

1:25:52to learn but how exactly you should go

1:25:55forward and start learning these things

1:25:57let's go ahead and clear that out so

1:26:00first of all guys what we can do for you

1:26:02is you can basically refer to a lot of

1:26:04blocks that we have written on or we

1:26:06frequently updat videos on YouTube as

1:26:09well such as this video which has info

1:26:11about the engineering certification or

1:26:13how to become an Azor data engineer now

1:26:16we frequently put more videos for such

1:26:18topics now you can go through them and

1:26:20basically get a jump start into how you

1:26:22you can prepare now my recommendation to

1:26:25you will be to plan out your working

1:26:27plan ask after or before working shift

1:26:30of yours now you should at least spend 3

1:26:32to four hours every day for the next two

1:26:34or 3 months in order to clear this

1:26:37certification exam otherwise if you

1:26:39don't invest this much amount of time

1:26:41guys it is going to be very difficult to

1:26:44crack the exam because there are a lot

1:26:46of things to learn especially if you're

1:26:48not from a data engineer domain now it

1:26:50is going to be a little difficult

1:26:52because you will have to read

1:26:53documentation you'll have to do Hands-On

1:26:56right now at some point you will get

1:26:58stuck and you'll have some issues you

1:27:00have to figure out what went wrong or

1:27:02what's wrong with the hands on I know

1:27:04this because I've also been there guys

1:27:06but getting stuck is the most beautiful

1:27:09part of learning anything because that

1:27:11is where you start the actual research

1:27:13and about how things work right so spend

1:27:17at least 2 to three hours every day

1:27:18either after your workshift or before

1:27:21your workshift to basic learn these

1:27:22Technologies right and now for those

1:27:25people who feel like they do not have

1:27:26the time or they do not want to invest

1:27:28time in researching and they want

1:27:30someone to help them out in getting this

1:27:32exam cleared and become a successful AER

1:27:35data engineer so guys we at a directa

1:27:38also offer a course of Microsoft Azor

1:27:40certification training of the Azor data

1:27:42engineer associate certification course

1:27:45and we also have a master program as

1:27:47well all right which will basically help

1:27:49you in clearing the aor data engineer

1:27:51associat certification right now if you

1:27:54need a helping hand and you need someone

1:27:56or you need to be taught by someone

1:27:57who's already cleared this examination

1:27:59and is already working as a data

1:28:01engineer in the industry then this is

1:28:04the right course for you

1:28:06[Music]

1:28:10guys so why as dat Factory because again

1:28:13we know that we have been generating

1:28:16data at an exponential rate especially

1:28:18since last 5 years and in 2015 we were

1:28:22generating data at the rate of almost

1:28:233.4 exabytes per month and now we are

1:28:27generating at the rate of almost more

1:28:29than 44 exabytes of data per month but

1:28:33that's almost 15 times increase in the

1:28:35amount of of data in just a span of 5

1:28:39years and it is going to be increased by

1:28:41almost 30% more because of the current

1:28:44lockdown fa by all the entire globe and

1:28:46that's why the entire consumption of

1:28:48data is again has been increased

1:28:51tremendously right that is that exactly

1:28:53is what we have the that's why we need

1:28:56to have the most optimized solution for

1:28:58data driven Solutions out there

1:28:59especially on the cloud computing

1:29:01platforms right and mod and modern data

1:29:04handling requires us to move from on

1:29:06promise to database to Cloud database

1:29:09services and that to quickly and then

1:29:11here we have to make sure again and

1:29:13that's why the data needs processing and

1:29:16goes through a series of steps making

1:29:18the process tedious because again when

1:29:20we are trying to load the data if we are

1:29:22trying to migrate other data from our

1:29:24database servers from our on premise

1:29:26storage Services then that has to go

1:29:28through a series of steps and that makes

1:29:30the entire process much more complicated

1:29:32it makes it much more slower as well and

1:29:35plus since it requires a good amount of

1:29:37investment both in times of both in

1:29:39terms of time and money it becomes we

1:29:41can see not feasible at all and data

1:29:44Factory simply help us in automating

1:29:46this entire process and does serve the

1:29:48cost that exactly is why we have data

1:29:51Factory

1:29:53now what exactly is data Factory what

1:29:55exactly is data Factory here so data

1:29:57Factory is basically a cloud-based

1:29:59integration service through which we can

1:30:03Define the entire we can design the data

1:30:05the entire workflow of data in cloud and

1:30:08making sure that we we can use for

1:30:11orchestration and for the automation

1:30:13purposes so for example suppose if we

1:30:15have now if you want we can create a

1:30:17complete pipeline we can Define what is

1:30:19a source and how the pipeline should be

1:30:21structured how the data is coming in

1:30:24where to where it should be stored and

1:30:26that to on a regular manner so we can

1:30:28Define the entire Pipeline and we can

1:30:30automate the entire data movement that

1:30:32means if we make any changes to any

1:30:34particular file or any objects in the

1:30:36oromis that will be automatically

1:30:38replicated or we can say moved into the

1:30:41cloud services as well by the help of

1:30:43data

1:30:44Factory right and using data Factory

1:30:47here we can create and schedule the

1:30:48entire data workflows called as

1:30:50pipelines that can inest data from

1:30:52disparate s data source for example if

1:30:54we have multiple data source defined

1:30:56here we can simply injest data from

1:30:58multiple sources and then we can process

1:31:00by simple a single pipeline it is

1:31:03basically used for processing large

1:31:05amount of large volume of data and these

1:31:09are done by using by integrating it with

1:31:11services that we have SEO SG inside Hado

1:31:14spark SEO data L analytics and is your

1:31:17machine

1:31:18learning so basically if we are looking

1:31:20to make sure that we have we have the

1:31:23optimized data available and optimized

1:31:25stream of data available for analytics

1:31:28and for machine learning then we have to

1:31:30do that by using data Factory to

1:31:31maintain that

1:31:33consistency and if you talk about the

1:31:35entire workflow here first of all the

1:31:37pipelines are all datadriven workflows

1:31:40in SEO data Factory typically performs

1:31:42the following four steps it simply help

1:31:44us in connecting and collecting the

1:31:46entire data set so first of all if we

1:31:48have multiple sources if you have wide

1:31:51variety of sources here we have to make

1:31:53sure we are able to set the source for

1:31:55each and every each and every

1:31:58connector we have to make sure that we

1:32:00are able to connect you to multiple

1:32:03services and then we can through

1:32:05multiple apis or if you have multiple

1:32:07sources of data says for example if we

1:32:09have data from our own CRM coming in if

1:32:11we have data from our own Erp tool from

1:32:13multiple social media analytic PL

1:32:15dashboard here we have to connect and

1:32:16collect all the different data sets then

1:32:19we have to transform and enrich because

1:32:21data always consists of multiple

1:32:23anomalities right they may be multiple

1:32:26missing values they will be some

1:32:28incorrect data formats available they

1:32:30the the data may be incomplete or it may

1:32:33be maybe multiple mistakes in that data

1:32:35corre so you have to make sure that we

1:32:37take care of entire data transformation

1:32:39that means if you want to convert the

1:32:40data format from one to the other like

1:32:42we have the ETL so again we can take

1:32:44care of the transformation and the

1:32:47enrichement of data that means data

1:32:49pre-processing and making sure it is

1:32:51much clear for and it is usable by the

1:32:54antical platforms for which we want to

1:32:55use it then we have to publish it to

1:32:58make it usable and then we have to

1:33:01continuously monitor it so that the

1:33:03entire process can be monitored in case

1:33:05there have been some breakdowns in any

1:33:07of the Clusters from which we are

1:33:09collecting data then that can be

1:33:10monitored and if something can be done

1:33:12it can be it can be processed as well we

1:33:15can do that now let's understand each

1:33:18and every concepts for data Factory here

1:33:20so if you talk about the enti Concepts

1:33:22when first of all we do have pipeline we

1:33:24do have something as pipeline so

1:33:26pipeline is a logical grouping of

1:33:28activities for example Suppose there are

1:33:3110 different sequence like we discussed

1:33:33first of all we have to connect then we

1:33:35can transform then we have to process it

1:33:37then we have to make it available for

1:33:39the analytical platforms so there if

1:33:41there are multiple sequences that needs

1:33:43to be followed again that then that is

1:33:45something that we Define as a part of

1:33:47pipeline alog together and then we have

1:33:49data sets so obviously without data data

1:33:51sets the entire data pipeline is of is

1:33:54of no use so data set simply represent

1:33:56the data structure within the available

1:33:59data store so there have multiple data

1:34:01stores available what exactly those are

1:34:03we are going to discuss step by step and

1:34:06then we have activities for activity

1:34:08simply represents a processing step in a

1:34:10pipeline for example there are 10

1:34:12different steps here for example we have

1:34:14to first of all connect to the source

1:34:16then we have to work on transforming

1:34:18data Exel then we have to work on

1:34:20pre-processing then we have to work on

1:34:22setting up the connection so each now

1:34:24this entire process itself is called as

1:34:27pipeline where we it consists of

1:34:28multiple sequence steps and each and

1:34:31every individual step here each and

1:34:32every individual step these are termed

1:34:34as activities so a pipeline is what we

1:34:36can say pipeline is simply a collection

1:34:38of different activities so we can so

1:34:41that activity is focused on completing

1:34:43one different task and pipeline simply

1:34:46defines the structure or we can say

1:34:48sequence of those STS and it simply

1:34:50makes sure that that sequence is

1:34:51followed whenever the entire pipeline is

1:34:53being

1:34:56implemented so let's erase this

1:34:59up and next we have Link services so it

1:35:03simply the information needed to connect

1:35:04to the external sources like for example

1:35:06we have apis if you have third party

1:35:08vendors and we have third party sources

1:35:10then again we do need an active APS for

1:35:12that and that's why these are all termed

1:35:14as link sources from which we can Source

1:35:16the entire assets here these are all

1:35:18additional links

1:35:20available and as we know data Factory is

1:35:23basically used for on premise itself if

1:35:25we are looking to to connect this if we

1:35:27are looking to connect this to multiple

1:35:29on premise data sets here then that

1:35:30exactly is what we use it for let's

1:35:33understand what exactly is data L

1:35:35service so data L as we know data l so

1:35:38data L as you know is simply an

1:35:40Enterprise wide hyperscale repository so

1:35:43here okay it is simply an Enterprise

1:35:44wide hyperscale repository for big data

1:35:46analytics workload and now it simply

1:35:49holds now it has a capability of

1:35:52petabyte so it can hold data for any

1:35:55size it will allow us to do again using

1:35:57that we can we can do multiple

1:36:00operational and exploratory analytics as

1:36:02well for example if we have multiple

1:36:04sources of data so for example if we

1:36:06have data sources like from on premise

1:36:08we have sensors data like for example if

1:36:10we when we talking about aviation

1:36:12industry then we have multiple we have

1:36:14tons of data sets coming it from

1:36:15different sensors especially for and

1:36:17same way for M for any manaing sectors

1:36:20as well same way if we have any data

1:36:22connected for any websites for example

1:36:24we are talking about any so any uh any

1:36:27stream of data coming in for analytics

1:36:29for any special websites for example

1:36:31suppose we have any big e-commerce

1:36:32solution for Amazon flip cart Airbnb so

1:36:35they have tons of data available on the

1:36:37platforms if we had data coming in from

1:36:39different devices from be it can also be

1:36:42a part of the I complete iot networks

1:36:45they can be data in the format of videos

1:36:47multiple social social media streams

1:36:49coming in for example we have streams of

1:36:51Facebook on Twitter redit on Reddit on

1:36:55multiple platforms so if you have

1:36:56multiple social media streams coming in

1:36:58in the formats of post or if you or

1:37:00let's say if you want to understand the

1:37:02real time I can say if you use case is

1:37:05to work on studying the user sentiments

1:37:09right then that in that case we have to

1:37:10make sure we are we are pitching in the

1:37:13multiple social media streams coming in

1:37:15if we have data in the format of images

1:37:17on application then these all Concepts

1:37:19have to be these all sources needs to be

1:37:21connected as a part of data Factory

1:37:24right and that's why here we can in now

1:37:26once we have these data sources

1:37:28available then we can connect it to ad

1:37:30analytics we can connect this to HG

1:37:32insight to R spark and machine learning

1:37:35purposes now if you have been aware if

1:37:38we have been aware of the fact such as

1:37:41now we can say data warehousing we can

1:37:43say data Lake works like a data

1:37:45warehouse for example if you're working

1:37:47on the realtime analytics right and if

1:37:50you have data sources from multiple

1:37:51platforms so instead of connecting each

1:37:53and every Source manually or we can say

1:37:55one by one to spark we can store the

1:37:57entire data and a send a centralized

1:37:59location and then we can we only need to

1:38:01connect a center location with a single

1:38:04connection to spark that's it so it

1:38:06simply improvises the entire entire

1:38:09performance as well just like we have

1:38:10red shift available in awss same way

1:38:14here we have data Le now let's also

1:38:17understand multiple data L Concepts

1:38:19let's understand multiple data Le

1:38:20Concepts here so data Lake as we know

1:38:23again has multiple components inside it

1:38:25like we have analytics now in data Lake

1:38:28components we have anal we have

1:38:29components for analytics such as a

1:38:31Insight we have AO data L available we

1:38:35can use data store for the complete

1:38:37storage purposes or for analytics we can

1:38:40indicate this with a inside and then we

1:38:42have hard inside and then we have AO

1:38:45data

1:38:46available and then when we are trying

1:38:49again in terms of Analytics we have to

1:38:51make sure we are we do remember these

1:38:53three main key points analytics we can

1:38:55do on data of any size there is no

1:38:58limitation users all users are

1:39:01productive on day one and then we have

1:39:04to make sure that ready it is all scale

1:39:07up exactly as per our Enterprise

1:39:09requirement we have to make sure of that

1:39:11part in terms of type of data stored

1:39:14here it supports all data types we have

1:39:16structured semi-structured and

1:39:18unstructured data types supported we

1:39:21have CSC files we have XML files right

1:39:23we have the emails we have Jon's files

1:39:25right so again these all are a part of

1:39:28sem instructure data when we don't have

1:39:30a direct format available on which we

1:39:32can start and perform the entire sorting

1:39:35and that is example for semi structure

1:39:36data set and then we have unstructured

1:39:39data set like we have images videos

1:39:42audio clips these all are part of semi

1:39:44structured data

1:39:46sets and then if you compare data League

1:39:49to Data Warehouse if we compare data

1:39:52leak to Data Warehouse here again data

1:39:54leak as you know is simply complimentary

1:39:57to the data warehouse whereas data where

1:40:00if you talk about the data warehousing

1:40:02service it may be soured to data leag

1:40:04data leag is basically used for detail

1:40:06data whereas as we can see data

1:40:08warehouse is basically used for filter

1:40:11summarize and for refinement of data all

1:40:13together data L offers schema on read

1:40:16whereas data house off for schema on

1:40:18right write and data L has one language

1:40:20to process data of any format whereas in

1:40:23data warehouse we can process it using

1:40:25the SQL complying it all together now

1:40:28let's move into the handson here and see

1:40:30how exactly a this is implemented on top

1:40:33of Edo data Factory portal here data

1:40:36warehouse and dat let's understand this

1:40:38by simple use case here and for doing

1:40:41that let's open up our notepad here for

1:40:45example let's say we have multiple data

1:40:47sources for example we have data sources

1:40:49available from our CRM correct for

1:40:51example our main use cases we want to

1:40:53perform the analysis for sales report

1:40:55for sales for any company correct we are

1:40:57here to perform analysis or multiple St

1:40:59data here right or suppose if we have

1:41:01data coming from from CR and we have

1:41:03data from Erp tools as well right and

1:41:06then we have data from the old data set

1:41:09available for example if we have the own

1:41:11CC file also stored here so if you have

1:41:13multiple sources of data coming in then

1:41:16in in if you want to Club it if you want

1:41:18to use this on top of any iCal platform

1:41:22so there are two ways of doing that

1:41:23either we have to connect each and every

1:41:25source with the antical platform and

1:41:28then only we can start working on it

1:41:30correct using the concept of data

1:41:33warehouse what we can do we can connect

1:41:34each and every Source we can store the

1:41:36the data from each of source to a

1:41:38centralized location as in called as a

1:41:41data house right and once the data is

1:41:43available in data house then we can

1:41:46connect that data warehous directly to

1:41:48us through a single collection to our

1:41:50artical platform so that whatever

1:41:51analyst we are trying to perform the

1:41:53entire process can be streamlined as a

1:41:55part of data warehousing

1:41:57service let's erase this

1:42:02up for getting started on data Factory

1:42:05here we can open we can come back to a

1:42:08portal so here we can come back to a

1:42:10portal in case you have not signed up on

1:42:12asure we can open this entire portal

1:42:14here where we can start working on top

1:42:17of data Factory one one by one now for

1:42:20getting started here what we can do is

1:42:21we can simply come back we can simply

1:42:23come back

1:42:25and first of all for having the for

1:42:28having the entire database that we can

1:42:30install and can interpret locally we can

1:42:32use this we can use entire platform

1:42:35called as SS SMS in case you don't have

1:42:38the enre setup done at your end then we

1:42:40can simply go ahead and download the SQL

1:42:42Server management studio in case you

1:42:44don't have the setup done now once we

1:42:46have configured these now what we can do

1:42:47we can come back to our a portal now in

1:42:49this s portal what what we can do we can

1:42:51simply first of all once we are into

1:42:53dashboard here we have to work around

1:42:55with first of all setting up the entire

1:42:57data warehousing service so what we can

1:42:59do is here we can set up the entire dat

1:43:01housing service and for setting it up we

1:43:03can come back to our entire dashboard

1:43:05now for creating a new resource we here

1:43:07we have to click on this option which

1:43:08says create a resource and then we can

1:43:11choose resource type here so from

1:43:13databases here we can choose now yeah

1:43:15these are the most commonly used

1:43:17resources here for example if you want

1:43:18to go for web application for functional

1:43:20applications for SQL databases we can

1:43:23choose accordingly and if you want to

1:43:25start with SQL database we can open up

1:43:27SQL database and then we can set up the

1:43:29entire database platform but again in

1:43:31here our main goal is not to create a

1:43:33SQL database but to set up the entire

1:43:35data with housing first and then we can

1:43:38saish in data services and then get

1:43:41started on top of it right so here what

1:43:43we can do here e either we can go ahead

1:43:45and search for the services using these

1:43:47categories or we can use it directly we

1:43:50can use the the service bar to search

1:43:52for the services directly for example

1:43:55here we can use data

1:43:59L as you can see currently we are

1:44:01planning to use data l so here we can

1:44:03choose data L

1:44:06here now if you want to create it now

1:44:08here we can simply click on create here

1:44:10we can Define the name of the current

1:44:12data storage service that we are going

1:44:13to create we can Define the entire

1:44:18name for example let's say we have we