Full transcript
0:00AI models can't cross this boundary and
0:02we don't know why as we train an AI
0:05model its error rate generally drops off
0:07quickly and then levels off if we train
0:09a larger model it will achieve a lower
0:11error rate but requires more compute
0:14scaling to larger and larger models we
0:16end up with a family of Curves like this
0:19switching our axis to logarithmic scales
0:21a clear Trend emerges where no model can
0:24cross this line known as the compute
0:26optimal or compute efficient Frontier
0:29this trend is one of three three neural
0:30scaling laws that have been broadly
0:32observed error rate scales in a very
0:34similar way with compute model size and
0:37data set size and remarkably doesn't
0:39depend much on model architecture or
0:41other algorithmic details as long as
0:44reasonably good choices are made the
0:46interesting question from here is have
0:48we discovered some fundamental law of
0:50nature like an ideal gas law for
0:52building intelligent systems or is this
0:55transist result of the specific neural
0:57network driven approach to AI that we're
0:59taking right now now how powerful can
1:01these models become if we continue
1:03increasing the amount of data model
1:05sizing compute can we drive errors to
1:08zero or will performance level off why
1:11are data model size and compute the
1:13fundamental limits of the systems we're
1:15building and why are they connected to
1:17model performance in such a simple
1:19way 2020 was a watershed year for open
1:22AI in January the team released this
1:25paper where they showed very clear
1:27performance Trends across a broad range
1:29of scales for language models the team
1:31fit a power law equation to each set of
1:34results giving a precise estimate for
1:36how performance scales with compute data
1:38set size and model size on logarithmic
1:40plots these power law equations show up
1:42as straight lines and the slope of each
1:45line is equal to the exponent of the fit
1:47equation larger exponents make for
1:49steeper lines and more rapid performance
1:51improvements the team observed no signs
1:54of deviation from these Trends on the
1:56upper end foreshadowing open AI strategy
1:58for the year the largest model the team
2:01tested at the time had 1.5 billion
2:03learnable parameters and required around
2:0510 petaflop days of compute to train a
2:08pedop flop day is the number of
2:10computations a system capable of one
2:11quadrillion floating Point operations a
2:13second can perform in a day the
2:15top-of-the-line gpus at the time the
2:17Nvidia V100 is capable of around 30 Tera
2:21flops so a system with 33 of these
2:23$10,000 gpus would deliver around a
2:26pedop flop of compute that summer the
2:28team's empirically predicted game would
2:30be realized with the release of GPT 3
2:33the open AI team had placed a massive
2:35beted on scale partnering with Microsoft
2:37on a huge supercomputer equipped with
2:39not 33 but 10,000 V100 gpus and training
2:44the absolutely massive 175 billion
2:46parameter gpt3 model using 3,640 pedop
2:50flop days of compute gpt3 performance
2:52followed the trend line predicted in
2:54January remarkably well but also didn't
2:56flatten out indicating that even larger
2:59models would further improve performance
3:02if the massive gpt3 hadn't reached the
3:04limits of neural scaling where were they
3:07is it possible to drive error rates to
3:08zero given sufficient compute data and
3:10model size in an October publication the
3:13open AI team took a deeper look at
3:15scaling the team found the same Clear
3:18scaling laws across a range of problems
3:20including image and video modeling they
3:23also found that on a number of these
3:24other problems the scaling Trends did
3:26eventually flatten out before reaching
3:28zero error this makes sense if we
3:30consider exactly what these error rates
3:32are measuring large language models like
3:35gpt3 are Auto regressive they are
3:37trained to predict the next word or word
3:39fragment in sequences of text as a
3:41function of the words that come before
3:44these predictions generally take the
3:45form of vectors of probabilities so for
3:48a given sequence of input words a
3:50language model will output a vector of
3:51values between 0o and one where each
3:54entry corresponds to the probability of
3:56a specific word in its
3:58vocabulary these vectors are typically
4:00normalized using a soft Max operation
4:03which ensures that all the probabilities
4:04add up to one gpt3 has vocabulary size
4:08at
4:0950257 so if we input a sequence of text
4:12like Einstein's first name is the model
4:15will return a vector of length
4:1750257 and we expect this Vector to be
4:19close to zero everywhere except at the
4:22index that corresponds to the word
4:23Albert this is index
4:2542590 in case you're wondering during
4:28training we know what the next word is
4:30in the text that we're training on so we
4:32can compute an error or loss value that
4:35measures how well our model is doing
4:36relative to what we know the word it
4:38should be this loss value is incredibly
4:40important because it guides optimization
4:43or learning of the model's parameters
4:45all those pedoph flops of training are
4:47performed to bring this loss number down
4:49there's a bunch of different ways we
4:51could measure the loss in our Ein sign
4:53example we know that the correct output
4:55Vector should have a one at the index of
4:5742590
4:59so we could Define our loss value as 1
5:02minus the probability returned by the
5:03model at this index if our model was
5:06100% confident the answer was Albert and
5:08returned a one our loss would be zero
5:11which makes sense if our model returned
5:13a value of 0.9 our loss would be 0.1 for
5:17this example if the model returned a
5:19value of 0.8 our loss would be 0.2 and
5:22so on this formulation is equivalent to
5:24what's called an L1 loss which works
5:26well in a number of machine learning
5:28problems however in practice we found
5:30that models often perform better when
5:32using a different loss function
5:33formulation called the cross entropy the
5:36theoretical motivation of cross entropy
5:38is a bit complicated but the
5:39implementation is simple all we have to
5:42do is take the negative natural
5:43logarithm of the probability output of
5:46the model at the index of the correct
5:48answer so to compute our loss in the
5:50Einstein example we just take the
5:52negative log of the probability output
5:54by the model at index
5:5742590 so if our model is 100% confident
6:00then our cross entropy loss equals the
6:02minus natural logarithm of one or zero
6:05which makes sense and matches our L1
6:07loss if our model is 90% confident of
6:10the correct answer our cross entropy
6:12loss equals the negative natural log of
6:140.9 or about 0.1 again close to our L1
6:18loss plotting our cross entropy loss as
6:20a function of the model's output
6:22probability we see that loss grows
6:24slowly and then shoots up as the model's
6:26probability of the correct word
6:27approaches zero this means that if the
6:29model's confidence in the correct answer
6:31is very low the cross entropy loss will
6:33be very high the model performance shown
6:35on the Y AIS and all the scaling figures
6:38we've looked at so far is this cross
6:40entropy loss averaged over the examples
6:42in the model's test set the more
6:44confident the model is about the correct
6:46next word in the test set the closer to
6:48zero the average cross entropy becomes
6:50now the reason it makes sense that the
6:52open AI team s some of their loss curves
6:54level off instead of reaching zero is
6:57because predicting the next element in
6:58sequences like this generally does not
7:00have a single correct answer the
7:03sequence Einstein's first name is has a
7:05very unambiguous next word but this is
7:08not the case for most text a large part
7:10of gpt3 is training data comes from text
7:12scraped from the internet if we search
7:14for a phrase like a neural network is a
7:17we'll find many different next words
7:19from various sources none of these words
7:21are wrong there's just many different
7:23ways to explain what a neural network is
7:26this fundamental uncertainty is called
7:27the entropy of natural language
7:30the best we can hope for our language
7:31models is that they give High
7:33probabilities to a realistic set of next
7:35word choices and remarkably this is what
7:38large language models do for example
7:40here's the top five choices for meta's
7:42llama
7:43model so we can never drive the cross
7:46entropy loss to zero but how close can
7:48we get can we compute or estimate the
7:51value of the entropy of natural language
7:54by fitting power law models to their
7:55loss curves that include a constant
7:57irreducible error term the the opening I
8:00team was able to estimate the natural
8:01entropy and low resolution images videos
8:04and other data sources for each problem
8:07they estimated the natural entropy of
8:08the data in two ways once by looking at
8:11where the model size scaling curve
8:12levels off and again by looking at where
8:14the compute curve levels off and they
8:17found that these separate estiment
8:18agreed very well know that the scaling
8:20power laws still work in these cases but
8:23by adding this constant term our trend
8:25line or Frontier on a log log plot is no
8:28longer a straight line interestingly the
8:30team was not able to detect any
8:32flattening out of performance on
8:33language data however noting that
8:36unfortunately even with data from the
8:38largest language models we cannot yet
8:40obtain a meaningful estimate for the
8:42entropy of natural language 18 months
8:45later the Google deepmind team published
8:46a set of massive neural scaling
8:48experiments where they did observe some
8:50curvature in the compute efficient
8:52Frontier on natural language data they
8:55used their results to fit a neural
8:56scaling law that broke the overall loss
8:59into into three terms one that scales
9:01with model size one with data set size
9:03and finally an irreducible term that
9:06represents the entropy of natural text
9:08these empirical results imply that even
9:10an infinitely large model with infinite
9:13data cannot have an average crossentropy
9:15loss on the massive Text data set of
9:17less than
9:181.69 a year later on Pi Day 2023 the
9:22open AI team released GPT
9:254 despite running for a 100 Pages the
9:28gp4 technical report contains almost no
9:31technical information about the model
9:32itself the open aai team did not share
9:35this information citing the competitive
9:37landscape and safety
9:39implications however the paper does
9:40include two scaling plots the cost of
9:43training GPT 4 is enormous reportedly
9:46well over $100
9:47million before making this massive
9:49investment the team predicted how
9:51performance would scale using the same
9:53simple power laws fitting this curve to
9:55the results of much smaller experiments
9:58note that this uses a linear and not
10:00logarithmic y-axis scale exaggerating
10:03the curvature of the scaling if we map
10:06this curve to a logarithmic scale we see
10:08some curvature but overall a close match
10:11to the other scaling plots we've seen
10:13what's incredible here is how accurately
10:15the open a team was able to predict the
10:17performance of GPT 4 even at this
10:19massive scale while gpt3 training
10:22required an already enormous 3,640 peda
10:25flop days some leaked information on GPT
10:284 training puts the training compute at
10:30over 200,000 peda flop days reportedly
10:34requiring 25,000 Nvidia a100 gpus
10:37running for over 3 months all of this
10:40means that neural scaling laws appear to
10:42hold across an incredible range of
10:44scales something like 13 orders of
10:46magnitude from 10 to the minus8 pedop
10:49Flop days reported in open ai's first
10:512020 publication to the leaked value of
10:53over 200,000 pedop flop days for
10:55training GPT 4 this brings us back to
10:58the question why does AI model
11:00performance follow such simple laws in
11:02the first place why are data model
11:05sizing compute the fundamental limits of
11:07the systems we building and why are they
11:09connected to model performance in such a
11:10simple way the Deep learning theory we
11:13need to answer questions like this is
11:15generally far behind deep learning
11:17practice but some recent work does make
11:19a compelling case for why model
11:21performance scales following a power law
11:24by arguing that deep learning models
11:25effectively use data to resolve a
11:27high-dimensional data manifold
11:30really getting your head around these
11:31theories can be tricky it's often best
11:33to build up intuition step by step to
11:36build up your intuition on llms and a
11:38huge range of other topics check out
11:40this video sponsor brilliant when trying
11:42to get my own head around theories like
11:44neural scaling I start with the papers
11:46but this only gets me so far I almost
11:49always code something up so I can
11:51experiment and see what's really going
11:52on brilliant does this for you in an
11:55amazing way allowing you to jump right
11:57to the powerful learning by doing part
12:00they have thousands of interactive
12:01lessons covering math programming data
12:03analysis and AI brilliant helps you
12:05build up your intuition through solving
12:07real problems this is such a critical
12:10piece of learning for me a few minutes
12:12from now you'll see an animation of a
12:13neural network learning a
12:14low-dimensional representation of the
12:16Imus data set solving small versions of
12:19big problems like this is an amazing
12:21intuition builder for me brilliant
12:23packages up this style of learning into
12:25a format you can make progress on in
12:26just minutes a day you'll be amazed at
12:28the progress you can stack up with
12:30consistent effort brilliant has an
12:32entire course on large language models
12:34including lessons that take you deeper
12:36into topics we covered earlier
12:38predicting the next word and calculating
12:39word probabilities to try the brilliant
12:42llm course and everything else they have
12:44to offer for free for 30 days visit
12:46brilliant.org Welch laabs or click the
12:49link in this video's description using
12:51this link you'll also get 20% off an
12:53annual premium subscription to brilliant
12:56big thank you to brilliant for
12:57sponsoring this video now back to neural
12:59scaling there's this idea in machine
13:01learning that the data sets our models
13:03learn from exist on manifolds in
13:06high-dimensional space we can think of
13:08natural data like images or text as
13:11points in this High dimensional space in
13:13the Imus data set of hand written images
13:15for example each image is composed of a
13:18grid of 28x 28 pixels and the intensity
13:21of each pixel is stored as a number
13:22between zero and one if we imagine that
13:25our images only have two pixels for a
13:27moment we can visualize these two pixel
13:29images as points in 2D space where the
13:32intensity value of the first pixel is
13:33the x coordinate and the intensity value
13:35of the second pixel is the y coordinate
13:38an image made of two white pixels would
13:40fall at 0 0 in our 2D space an image
13:43with a black pixel in the first position
13:45and a white pixel in the second position
13:47would fall at one Z and an image with a
13:49gray value of 0.4 for both pixels would
13:52fall at 0.4 comma 0.4 and so on if our
13:55images had three pixels instead of two
13:58the same approach still works just in
14:00three dimensions scaling up to our 28x
14:0328 mnist images our images become points
14:06in 784 dimensional space the vast
14:09majority of points in this High
14:10dimensional space are not handwritten
14:12digits we can see this by randomly
14:15choosing points in the space and
14:16displaying them as images these almost
14:19always just look like random noise you
14:21would have to get really really really
14:23lucky to randomly sample a handwritten
14:25digit this sparsity suggests that there
14:27may be some lower dimensional shape
14:29embedded in this 784 dimensional space
14:33where every point in or on this shape is
14:35a valid handwritten digit going back to
14:37our toy three pixel images for a moment
14:40if we learned that our third pixel
14:41intensity value let's call it X3 was
14:44always just equal to 1 plus the cosine
14:47of our second pixel value X2 all of our
14:49three pixel images would lie on the
14:51curved surface in our 3D space defined
14:53by X3 = 1 + the cosine of X2 this
14:57surface is two-dimensional we can
14:59capture the location of our images in 3D
15:01space just using X1 and X2 we no longer
15:04need X3 we can think of a neural network
15:06that learns to classify imist as working
15:08in a similar way in this network
15:11architecture for example our second to
15:13last layer has 16 neurons meaning that
15:15the network has mapped the 784
15:17dimensional input space to a much lower
15:2016-dimensional
15:21space very much like our 1 plus cosine
15:23function mapped our three-dimensional
15:25space to a lower two-dimensional space
15:28where the manifold hypothesis gets
15:29really interesting is that the manifold
15:31is not just a lower dimensional
15:33representation of the data the geometry
15:35of the manifold often encodes
15:37information about the data if we take
15:40the 16-dimensional representation of the
15:42Imus data set learned by our neural
15:44network we can get a sense for its
15:46geometry by projecting from 16
15:47Dimensions down to two using a technique
15:50like umap which attempts to preserve the
15:52structure of the higher dimensional
15:54space coloring each point using the
15:56number that the image corresponds to we
15:59can see that as the network trains
16:01effectively learning the shape of the
16:02manifold instances of the same digit are
16:04grouped together into little
16:05neighborhoods on the manifold this is a
16:08common phenomena across many machine
16:10learning problems images showing similar
16:13objects or text referring to similar
16:14Concepts end up close to each other on
16:16the Learned manifold one way to make
16:19sense of what deep learning models are
16:20doing is mapping high-dimensional input
16:23spaces to lower dimensional manifolds
16:25where the position of data on the
16:27manifold is Meaningful
16:29now what does the manifold hypothesis
16:31have to do with neural scaling laws
16:33let's consider the neural scaling law
16:35that links the size of the training data
16:37set with the performance of the model
16:39measured as the cross entropy loss on
16:41the test set if the manifold hypothesis
16:43is true then our trading data are points
16:46on some manifold in higher dimensional
16:48space and our model attempts to learn
16:50the shape of this manifold the density
16:52of our training points on our manifold
16:54depends on how much data we have but
16:56also on the dimension of the manifold in
16:59onedimensional space if we have D
17:01training data points and the overall
17:03length of our manifold is L we can
17:05compute the average distance between our
17:07training points s by dividing L by D
17:10note that instead of thinking about the
17:12distance between our training points
17:13directly it's easier when we get to
17:15higher Dimensions to think about a
17:16little neighborhood around each point of
17:18size as and since these little
17:19neighborhoods bump up against each other
17:22the distance between our data points is
17:23still just s moving to two Dimensions
17:25we're now effectively filling up an L by
17:27L square with small squares of side
17:30length s centered around each training
17:31point the total area of our large Square
17:34l^ s must equal our number of data
17:36points D * the area of each little
17:39square so D * s^ 2 rearranging and
17:42solving we can show that s is equal to l
17:45* D Theus 12 moving to three dimensions
17:49we're now packing an L by L by L cube
17:51with d cubes of side length s equating
17:54the volumes of our D small cubes and our
17:56large Cube we can show that s is equ Al
17:59to L * D Theus 1/3 so as we move to
18:02higher Dimensions the average distance
18:04between points scales as the amount of
18:06data we have to the power of minus1 over
18:09the dimension of the
18:11manifold now the reason we care about
18:13the density of the training points on
18:14our manifold is because when a testing
18:17Point comes along its error will be
18:19bounded by a function of its distance to
18:21the nearest Training point if we assume
18:24that our model is powerful enough to
18:25perfectly fit the training data then our
18:28learned man manold will match the true
18:30data manifold exactly at our training
18:32points a deep naral network using Ru
18:34activation functions is able to linearly
18:37interpolate between these training
18:38points to make predictions if we assume
18:41that our manifolds are smooth then we
18:43can use a tailor expansion to show that
18:45our error will scale as the distance
18:47between our nearest Training and testing
18:48points squared we establish that our
18:51average distance between training points
18:52scales as the size of our data set D to
18:55the power of minus1 over the dimension
18:56of our manifold so we can Square this
18:59term to get an estimate for how our
19:01error scales with data set size and
19:03compute D the^ of minus 2 over the
19:06manifold Dimension finally remember that
19:08our models are using a cross entropy
19:10loss function but thus far in our
19:12manifold analysis we've only considered
19:14the distance between the predicted and
19:16True Value this is equivalent to the L1
19:18loss value we considered earlier
19:20applying a similar tailor expansion to
19:22the Cross entropy function we can show
19:24that the cross entropy loss Will scale
19:26as the distance between the predicted
19:28and true value squared so for our final
19:31theoretical result we expect the cross
19:33entropy loss to scal as the data set
19:35size d to the power of Min -2 over the
19:37manifold Dimension squared so D ^ of-4
19:41over Little D this represents the worst
19:44case error making this an upper bound so
19:46we expect cross entropy loss to scale
19:48proportionally or better than this term
19:51the team that developed this Theory
19:52calls this resolution limited scaling
19:55because more data is allowing the model
19:57to better resolve the data manifold
20:00interestingly when considering the
20:01relationship between model size and lost
20:04the theory predicts the same fourth
20:05power relationship in this case the idea
20:08is that the additional model parameters
20:10are allowing the model to fit the data
20:12manifold at higher
20:14resolution so how does this theoretical
20:16result stack up against observation both
20:19the open aai and Google deepmind teams
20:21published their fit scaling values do
20:24these match what theory predicts in the
20:27January 2020 open AI paper the team
20:30observed the cross entropy loss scaling
20:32as the size of the data set to the power
20:34of minus
20:360.095 they refer to this value as Alpha
20:39subd if the theory is correct then Alpha
20:42subd should be greater than or equal to
20:434 over the intrinsic dimension of the
20:46data this final step is tricky since it
20:49requires estimating the dimension of the
20:51data manifold also known as the
20:53intrinsic dimension of natural language
20:56the team started with smaller problems
20:58where the intrinsic Dimension is known
21:00or can be estimated well they found
21:02quite good agreement between theoretical
21:04and experimental scaling parameters in
21:06cases where synthetic training data of
21:07known intrinsic Dimension is created by
21:09a teacher model and learned by a student
21:11model they were also able to show that
21:14the minus 4 overd prediction holds up
21:15well with smaller scale image data sets
21:18including
21:19imist finally turning to language if we
21:22plug in the observed scaling exponent of
21:24minus
21:250.095 we can compute that the intrinsic
21:27dimension of natural language should be
21:29something like 42 the team tested this
21:32result by estimating the intrinsic
21:34dimension of the manifolds learned by a
21:36language model and found the intrinsic
21:37Dimension to be significantly higher on
21:39the order of 100 note that the
21:42inequality from Theory still holds but
21:44we don't see nearly the same agreement
21:46that was observed in synthetic and
21:48smaller data sets what we're left with
21:50then is a compelling Theory with some
21:52real predictive power but definitely no
21:54unified theory of AI just yet we've seen
21:58some astounding AI progress in The Last
22:005 Years From open ai's first scaling
22:03paper in early 2020 to the release of
22:05GPT 4 in 2023 neural scaling laws showed
22:09us a path to better and better
22:11performance it's important to note here
22:13that while scaling laws have been
22:15incredibly predictive of next word
22:16prediction performance predicting the
22:19presence of specific model behaviors has
22:21remained more elusive abilities on tasks
22:23like word unscrambling arithmetic and
22:25multi-step reasoning seem to just pop
22:27into existence at various scales it's
22:30incredible to see how far our neural
22:32network powered approach has taken us
22:34and we of course don't know how far it
22:36can go many of the authors of the papers
22:39we've covered here have backgrounds in
22:40physics and you can feel in their
22:43approaches in language that they're on
22:44the hunt for unifying principles it's
22:47exciting to see this mindset applied to
22:48AI neural scaling laws are a powerful
22:51example of unification in AI delivering
22:54astoundingly accurate and useful
22:56empirical results and tantalizing Clues
22:59to a unified theory of scaling for
23:01intelligent systems it will be
23:03fascinating to see where scaling laws
23:05and other theories can take us in the
23:07next 5 years and to see if we can figure
23:10out if AI really can't cross this
23:15line if you enjoy Welch lab's videos I
23:18really think you'll like my book on
23:19imaginary numbers it's coming out later
23:22this year way back in 2016 I made a
23:24massive 13-part YouTube series on
23:26imaginary numbers it's such an
23:28incredible topic I released an early
23:30version of this book back then and I'm
23:32now in the process of revising
23:34correcting and significantly expanding
23:35it my goal is to create the best book
23:38out there on imaginary numbers
23:40highquality hardcover printed books will
23:42start shipping later this year you can
23:44pre-order a copy today at the link in
23:46the description below and your order
23:47includes a free PDF copy of the 2016
23:50version that you can download today I've
23:52also been working on some new poster
23:54designs I now have a dark mode version
23:56of my activation Atlas poster
23:59these are an incredible way to visualize
24:01the data manifolds learned by Vision
24:03models you'll find all of this and more
24:05at the Welch Labs store