Full transcript
Intro
0:00Okay, then let me make a
0:02controversial statement that again is
0:04going to get me a lot of friends in
0:05Silicon Valley.
0:06Um
0:06>> AI legend Yann LeCun has raised a
0:08billion dollars to pursue an alternative
0:11approach to AI.
0:13Unlike large language models, LeCun's
0:15approach is not rooted in language and
0:18is not generative.
0:20By design, it does not spit out text,
0:22images, or videos.
0:24Instead, LeCun has proposed Jepa.
0:26Jepa is not a single AI model, but
0:29instead an alternative framework for
0:31training AI models.
0:34Many successful approaches in AI and
0:36machine learning train models to predict
0:37some output Y given some input X.
0:40Large language models are given some
0:42input text X and trained to predict the
0:44text Y that comes next.
0:47Image classifier models are given an
0:48input image X [music] and trained to
0:50predict the corresponding label Y.
0:53Jepa does not work like this.
0:55Instead, our inputs X and outputs Y are
0:58each passed into models known as
0:59encoders.
1:01These encoders return a vector or matrix
1:03of numbers, often referred to as an
1:05embedding.
1:07From here, a third model known as a
1:09predictor is trained to predict the
1:11embedding of Y given the embedding of X.
1:15Why might this be a better way to build
1:17AI systems?
1:18>> Do you think that Jepa world model
1:20model-based approaches, do you think
1:21they'll replace LLMs one day or are they
1:23kind of solving different problems?
1:25>> Initially, they'll solve different
1:26problems.
1:28Eventually, they'll replace. Um I don't
1:30know.
1:31Okay. Because, you know, LLMs are really
1:32good at manipulating language, but
1:34basically nothing else.
1:36Um and they're really good in domains
1:38where the language itself
1:40>> [music]
1:40>> is the substrate of reasoning.
1:43>> Compared to the mainline generative
1:44language-driven approach to AI, Jepa
1:47lives on an alternative path of joint
1:49embedding architectures.
1:51Interestingly, LeCun played a
1:53significant role at the outset of both
1:55paths.
1:56In part one of this two-part series,
1:58we'll explore this alternative path to
2:00Jepa.
2:01We'll dig into why Yann moved away from
2:03generative architectures just as they
2:05were gaining traction in language,
2:08and explore Yann's epiphany for a new
2:10solution to the representation collapse
2:12problem that plagued joint embedding
2:14architectures for years.
2:16Finally, we'll dig into the Jepa
2:17architecture itself.
2:19In part two, we'll dive into Jepa
2:21implementations and see exactly how
2:23these models stack up against LLM-driven
2:26approaches.
The Problem with Deep Learning
2:31Yann LeCun saw the revolution coming.
2:34In the 1980s, while most of the AI field
2:36was busy building expert systems that
2:39were explicitly programmed instead of
2:40learned from data,
2:42Yann pioneered the convolutional neural
2:44network.
2:4625 years later, when deep learning began
2:48its rise to its now dominant position in
2:50AI, the breakthrough deep learning model
2:52AlexNet turned out to be uncannily
2:55similar to LeCun's convolutional nets
2:57from the 1990s.
3:00However, as deep learning continued to
3:01pick up steam through the 2010s, LeCun
3:04and other researchers became
3:05increasingly concerned by just how much
3:07this approach to AI depended on labeled
3:10training data.
3:12AlexNet was trained on the enormous and
3:13meticulously labeled ImageNet dataset
3:16using supervised learning, where AlexNet
3:18was trained to match the labels assigned
3:20to each image by human annotators.
3:23In contrast, children are able to learn
3:25very general representations for
3:27concepts like dog with very few
3:29explicitly labeled examples.
3:33As manually labeled data became a
3:35bottleneck for supervised learning,
3:37interest grew in alternative approaches.
3:40Reinforcement learning, where models
3:41learn from interacting with their
3:43environments instead of from labeled
3:44data, experienced a mini renaissance in
3:47the mid-2010s,
3:49highlighted by Google DeepMind's
3:50breakthrough performance on Atari games
3:52and the highly complex board game Go.
3:56Concurrently, LeCun and others explored
3:57unsupervised methods that learn from
4:00data without labels, including a variant
4:03called self-supervised learning, where
4:05the labels are taken from the data
4:06itself.
4:08>> Starting in 2015 or so, I started
4:10showing a a slide that has become a bit
4:12of a meme in the machine learning
4:13community, where I said like, you know,
4:15if
4:16uh it's the cake slide, right? So, if uh
Intelligence is a Cake
4:19intelligence is a takes the bulk of the
4:20cake, it is
4:22self-supervised learning, and the icing
4:24on the cake is supervised learning, and
4:25the cherry on the cake is reinforcement
4:26learning. And at the time, people were
4:27kind of crazy about reinforcement
4:29learning, so I was trying to tell them
4:30like, this is not never going to
4:33you know, take us to, you know, anywhere
4:34close to human-level intelligence uh cuz
4:37it's too inefficient. Um and uh
4:41uh turns out the success of
4:43self-supervised learning
4:44uh
4:46you know, happened in in text and and
4:48language way faster than it did in sort
4:51of more uh you know, natural uh
4:54modalities like like vision.
4:57>> Here, Yann is referring to the success
4:58of next-token prediction for training
5:00large language models.
5:02OpenAI was founded in 2015
5:05and initially focused their efforts on
5:06reinforcement learning, creating OpenAI
5:09Gym and Universe, and showing very
5:11impressive performance on complex video
5:13games.
The Rise of Generative AI
5:15While much of the company was focused on
5:17reinforcement learning, Ilya Sutskever,
5:19Alec Radford, and others became
5:21interested in a new neural network
5:22architecture from Google, the
5:24Transformer, initially designed for
5:26language translation.
5:29While experimenting, uh Radford tried an
5:31interesting modification. Instead of
5:33having the Transformer translate from a
5:35block of text in one language to a block
5:37of text in another language, he switched
5:39to a simpler self-supervised approach,
5:42where training text is broken into
5:43sequences, and the Transformer is given
5:46all but the last little piece of text,
5:48known as a token, in each sequence, and
5:51trained to predict what this final token
5:52will be.
5:55Radford and his OpenAI colleagues
5:56trained their transformer on a fairly
5:58large internal OpenAI data set of 7,000
6:01books. Note that we now call this phase
6:03pre-training.
6:05And then further trained their model
6:06using standard supervised learning from
6:08human-generated labels on specific
6:10language tasks.
6:12Their two-stage training approach worked
6:14well, setting new state-of-the-art
6:17results on nine language benchmarks,
6:19including tasks like high school-level
6:21reading comprehension questions,
6:23outperforming architectures and methods
6:25that were individually designed and
6:26trained for each individual task.
6:29Radford's model is now known as
6:31generative pre-trained transformer 1, or
6:34GPT-1.
6:36GPT-1 didn't receive much public
6:37attention at the time, but was a huge
6:40unlock, breaking models free from their
6:42dependence on human-labeled data, and
6:45opening up unprecedented levels of
6:46scale.
6:48Other researchers at OpenAI quickly
6:50grasped the significance of Radford's
6:52results, and the team went all in on
6:54this approach, aggressively scaling up
6:57to GPT-2 in 2019, GPT-3 in 2020, and
7:01ChatGPT in 2022.
7:04In 2012, AlexNet was trained on around a
7:07million examples. In 2020, GPT-3 was
7:10trained on hundreds of billions of
7:12examples.
7:13And interestingly, the new training
7:15paradigm that emerged exactly matched
7:17Yann LeCun's predictions from a few
7:19years earlier.
7:20An extensive self-supervised
7:22pre-training phase, followed by
7:24supervised learning, and finally
7:25reinforcement learning, to shape the raw
7:28next token predictor model into a
7:30helpful AI assistant.
7:32However, while these self-supervised
7:34generative approaches clearly broke
7:36through in language, the picture was
7:38much fuzzier for image and video data.
7:41>> But I I I kept working on vision and
7:43then initially uh
7:46my the the uh idea was to use um
7:50so to train a system to predict what
7:52happens in video
7:53but to use uh generative architectures.
7:56Um so basically train at a pixel level
7:59what's going to happen in a video.
Blurry Images
8:01>> Years before the success of GPT-1,
8:03researchers including LeCun had tried to
8:05apply the same self-supervised
8:07generative approach to video.
8:09In the most straightforward
8:10implementation, we configure our neural
8:12network to take in the RGB pixel values
8:15from a sequence of video frames and then
8:17predict the pixel values in the next
8:19frame just as the GPT models are trained
8:21to predict the next token in language.
8:24However, when we use these models to
8:26predict the next frame, the results are
8:28blurry.
8:29And this blurriness compounds
8:31dramatically in longer horizon
8:32predictions.
8:34Large language models are
8:35autoregressive.
8:37When ChatGPT answers a question, it
8:38generates one token at a time.
8:41At each step, feeding its latest
8:42generated token back into its input to
8:44create the next output.
8:46If we try this autoregressive approach
8:48with a next frame video prediction
8:50model, the results quickly devolve into
8:52blurry nothingness.
HRT is an awesome place to work
8:55Before we see exactly how Jeppa is able
8:57to get around this blurry prediction
8:58problem,
9:00let's look at another fascinating
9:01application of transformers beyond
9:03language models.
9:05This video is sponsored by Hudson River
9:07Trading and this is an order book.
9:10The left column shows all the bids to
9:12buy Nvidia stock ranked by bid price.
9:16And the right column shows all the
9:17current offers to sell Nvidia stock
9:19ranked by asking price.
9:22On a busy trading day, on the order of
9:241,000 new buy and sell orders like this
9:26come in every second.
9:28This deluge of orders is an incredibly
9:30rich information source.
9:33Is it possible to train a transformer,
9:35like the ones used in V-Jeppa, to find
9:37patterns in this data and use these
9:40patterns to predict future prices.
9:42Hudson River Trading has trillions of
9:44tokens of historical data.
9:46This is the same order of magnitude of
9:48training data used to train frontier
9:50LLMs.
9:51And their researchers are working to
9:53push the frontiers of machine learning
9:54on this data.
9:57The Vijeppa model we'll see later in the
9:58video maps patches of videos to
10:01individual embedding vectors. We could
10:03take a similar approach with order book
10:05data, tokenizing groups of orders using
10:07some financial intuition.
10:10However, this naive approach does not
10:11work well in practice. And the Hudson
10:14River Trading team has developed some
10:15really interesting [music] approaches to
10:17adapt cutting-edge transformer
10:18architectures to the complexities and
10:20constraints [music] of trading data.
10:22And all of this is happening in a
10:24setting where speed is everything.
10:26Models have to run under incredibly
10:28tight latency constraints.
10:31These fascinating and highly complex
10:33research and engineering challenges,
10:35combined with the resources to actually
10:37tackle [music] them, and an open highly
10:38collaborative environment, make Hudson
10:41River Trading an incredibly unique place
10:43to work.
10:44I hear a lot from potential sponsors
10:46these days and have been seriously
10:48impressed in my interactions with the
10:49Hudson River Trading team.
10:51The level of technical discussion and
10:53enthusiasm for these deep and
10:54interesting problems is unparalleled in
10:56my experience.
10:58If this sounds interesting, Hudson River
11:00Trading is currently hiring for AI
11:02researchers, algorithm developers, and
11:04software engineers.
11:06They're hiring globally and you don't
11:08need a finance background. You can learn
11:10more at
11:10hudsonrivertrading.com/welchlabs.
11:14Now, back to Jeppa.
But why so Blurry?
11:16Now, the blurry frames produced by our
11:18generative video prediction approach are
11:21not some huge mystery.
11:23Language is complex and unpredictable,
11:25but it's nothing compared to video.
11:28Language models used fixed-size
11:29vocabularies.
11:31GPT-2 has 50,257
11:34discrete outputs, one for each token
11:36that the model could say next.
11:38This complete enumeration approach is
11:40hopeless in video.
11:42For full HD video in the most general
11:44case, each pixel can take on 256
11:47discrete values, and we have 1920 * 1080
11:51* 3 color pixels, meaning there are
11:54something like 10 to the power of 15
11:56million possible next video frames,
12:00dwarfing the number of atoms in the
12:01observable universe.
12:03So, there's no way our video prediction
12:05model can have a discrete output for
12:07each possible next video frame, as our
12:09language model has a discrete output for
12:11each next possible token.
12:14Instead, many generative video
12:16approaches of this era have the network
12:18directly output pixel intensity values.
12:22The big challenge with this approach is
12:23how the model learns to handle
12:25uncertainty.
12:27If we compare an LLM learning to
12:28complete the sentence the ball bounced
12:30to the and a neural network predicting
12:33the next frame of a video of a ball
12:35actually bouncing, we can see exactly
12:37what goes wrong.
12:39In the LLM training case, the model will
12:41see various examples in its training set
12:44of the ball bouncing left and right.
12:47And since the model has separate outputs
12:48for each of these tokens, it can
12:50essentially independently update these
12:52probabilities.
12:54Our video model doesn't have it so easy.
12:57If our data set includes videos of the
12:59ball starting down the same path and
13:01then bouncing in various directions,
13:03since our model is forced to directly
13:05predict a single output frame for a
13:07given input,
13:08the best it can do in the face of this
13:10ambiguity is to predict the average of
13:12these outcomes.
13:14When we average the pixel values of our
13:15videos, we end up with a blurry,
13:18washed-out mess.
13:20Now, this is only the most naive
13:21approach, and there have been many, many
13:24interesting video and image prediction
13:25strategies tried with various degrees of
13:27success over the last couple decades.
Do our models need to be generative?
13:30However, the challenges that naturally
13:32arise led LeCun and other researchers to
13:35ask an interesting question.
13:37Do our models really need to be
13:39generative?
13:41In our GPT example, during the crucial
13:43pre-training phase, it really doesn't
13:45matter that our model is generative.
13:48After pre-training on next token
13:50prediction, we're left with a model
13:52that's essentially a really good auto
13:54complete.
13:55But this is not the point.
13:57What actually matters are the internal
13:59representations and features that the
14:01model learns to solve the next token
14:03prediction task.
14:05These learned internal representations
14:07are what allows pre-trained models to be
14:10quickly adapted into powerful AI
14:12assistants.
14:14Next token prediction on language is a
14:16proxy for intelligence that has turned
14:18out to work shockingly well.
14:22But are there other signals and methods
14:23that we can use to learn these powerful
14:25internal representations that we need to
14:28build intelligent systems?
14:30>> Simultaneously, we started realizing in
14:33the
14:34you know, round
14:362017-18
14:37that uh the the best system to learn
14:41representations of images are systems
14:43that
14:44do not are not generative. They don't
14:46reconstruct. They they you know, you you
14:49you get an image and
14:50you run it to an encoder
14:52and then you try to kind of coerce this
14:54encoder to extract as much information
14:57as possible with certain properties. So,
14:59for example,
15:00you take two images of the same scene
15:02or you take an image and you corrupt it
15:04or transform it in some ways. You run
15:06them both through encoders and you tell
15:08the system the representation, whatever
15:10you extract, should really be the same
15:11for those two images because they
15:13semantically represent the same thing.
15:15Um and I've been working on things like
Siamese Networks
15:17this since the '90s. So, this is not a
15:19new idea. This this idea of joint
15:21embedding. Uh we used to call this
15:23Siamese neural nets.
15:25>> The method Yan is referring to here,
15:26Siamese networks, was created by Yan and
15:29his collaborators at Bell Labs in the
15:31early 1990s
15:33when developing systems to detect
15:34fraudulent signatures.
15:37The system worked by passing a pair of
15:38signature images into two copies of the
15:41same neural network.
15:43The network copies were not trained to
15:44generate any kind of data. Instead, they
15:47output vectors of numbers, often
15:49referred to as embedding vectors.
15:51These network copies were trained on two
15:53types of examples.
15:55Positive examples that contain a
15:56reference signature and a non-fraudulent
15:59signature, so these are by the same
16:00person, and negative examples that
16:03contain a reference signature and a
16:04fraudulent signature.
16:07For fraudulent examples, the network
16:09copies are trained to produce embedding
16:10vectors that are maximally different.
16:13And for positive examples, produce
16:15embedding vectors that are maximally
16:16similar.
16:18When a new signature comes along, we can
16:20pass it into our network to compute an
16:22embedding vector, and compare it to the
16:24embedding vector produced from our
16:25reference signature.
16:27If the resulting embedding vectors are
16:29not similar enough,
16:31the signature is detected as fraudulent.
16:34By jointly embedding our signatures, our
16:36Siamese network learns a very useful
16:38internal representation of the images of
16:40our signatures,
16:41notably without learning to predict or
16:43generate any actual signature images as
16:46a GPT-based approach would.
16:49Joint embeddings offer potentially
16:50viable solution to our blurry video
16:52problem.
16:54As Yan explains,
16:55>> You you get an image and
16:57you run it through an encoder,
16:59and then you try to kind of coerce this
17:01encoder to extract as much information
17:04as possible with certain properties. So,
17:05for example, you take two images of the
17:08same scene,
17:09or you take an image and you corrupt it
17:10or transform it in some ways. You run
17:13them both through encoders, and you tell
17:14the system the representation, whatever
17:17you extract, to really be the same for
17:18those two images because they
17:20semantically represent the same thing.
17:22>> So, the idea here is that we sidestep
17:23the blurry video problem we saw with
17:25generative models
17:27by using a joint embedding architecture
17:30to map copies of images or videos with
17:32one or both corrupted or transformed to
17:35similar embedding vectors.
17:37This trained model will ideally learn a
17:39useful internal representation of images
17:41or video
17:43that we can repurpose for other tasks.
17:45Just as GPT models learn internal
17:47representations during pre-training that
17:49can be adapted into AI assistant
17:51behaviors.
Representation Collapse
17:53However, this joint embedding strategy
17:55has a huge problem.
17:57Since we're training our network to make
17:59the embeddings of our original and
18:01corrupted images or videos as similar as
18:03possible,
18:04the network can find a trivial solution
18:07where it simply returns the same
18:08embedding vector for any input that we
18:10pass in.
18:12If our network learns to output, for
18:14example, a vector of all ones for any
18:16input, then the network will return all
18:18ones for a corrupted and non-corrupted
18:20view of the same image,
18:22maximizing the resulting similarity, but
18:24without actually learning anything
18:26useful.
18:28This problem is known as representation
18:30collapse.
18:32In LeCun's original Siamese network
18:33approach, the team used what's now known
18:36as contrastive learning to avoid
18:38representation collapse
18:40giving the network both positive and
18:42negative examples.
18:44It turns out that we can apply the same
18:46contrastive approach to images and
18:47video.
18:49Training our network to output similar
18:50embeddings for views of the same
18:52underlying images or videos and
18:54dissimilar embeddings for different
18:55images or video.
18:57These contrastive methods have been
18:59successfully implemented on images and
19:01videos, but can run into issues when
19:03they're scaled up, requiring large
19:05amounts of computation and many negative
19:07examples to learn meaningful
19:09representations.
19:10And LeCun has argued that in the worst
19:12case, the number of contrastive samples
19:15may grow exponentially with the
19:16dimension of the representation.
19:19By the end of the 2010s, it was clear to
19:21LeCun and others that using generative
19:23models to fully reconstruct images and
19:25video was not a good strategy for
19:28self-supervised learning.
19:30But there wasn't a straightforward
19:31solution to the representation collapse
19:33problem that would allow joint embedding
19:35architectures to learn the same level of
19:37powerful and general internal
19:39representations that large language
19:41models were enjoying.
19:43>> And so, it was pretty clear that
19:44ego-structure was a bad idea for
19:47signals like like images and a fortiori
19:50for video.
19:52>> And
Yann’s Epiphany & Barlow Twins
19:54I had a bit of an epiphany
19:56because uh
19:57the uh
19:59the the methods that we were using to
20:01train those joint embedding
20:02architectures were kind of hacks a
20:04little bit
20:05until
20:06um I did some work with uh
20:09uh a couple of postdocs at um at at
20:11Meta, particularly back with Stefan
20:13Deny,
20:14who uh came up with um technical Barlow
20:17twins. So, it it's based on another idea
20:20in uh
20:21in computational neuroscience and
20:23machine learning that you know, Jeff
20:24Hinton also played on with similar
20:25ideas, which is that you see you should
20:27have time to have some measure of
20:29information content and try to maximize
20:31that.
20:32And there's some really old work by uh
20:35by Barlow uh but he's a famous
20:38computational neuroscientist
20:40>> And right.
20:40>> theoretical neuroscientist.
20:42>> Here, Yann is referencing the work of
20:43Horace Barlow,
20:45who hypothesized in 1961 that the
20:48neurons in animal and human vision
20:50systems operate by reducing redundant
20:52information between neurons.
20:55Stefan Deny, a postdoc LeCun was working
20:58with in 2020, was familiar with Barlow's
21:00work
21:01and proposed that one way to avoid
21:03representation collapse could be to
21:05apply Barlow's idea to the outputs of
21:08their networks.
21:09In the joint embedding architectures
21:11we've been considering, our embedding
21:13vectors are produced by a final layer of
21:15artificial neurons in our embedding
21:17networks.
21:18So, if our embedding vectors are of
21:20length 128,
21:21then the output layer of each of our
21:23networks contains 128 neurons.
21:27If we pass in a batch of various images
21:29into each of our networks and plot the
21:31output activation of the first neuron as
21:33we step through our images,
21:35we can see that this fires strongly on
21:37this first picture of a dog, not so much
21:40on this cat picture, and so on.
21:43Following our joint embedding approach,
21:45our network takes in a distorted view of
21:47the same batch of images.
21:50The whole point of our joint embedding
21:51architecture is to make the resulting
21:53embeddings of the same underlying images
21:55or videos similar.
21:57So, we want the output of our first
21:59neuron in our second network to be
22:01similar to the output of our first
22:02neuron in our first network.
22:05In a standard joint embedding
22:06architecture, we would simply measure
22:08and maximize the similarity between
22:10these two vectors.
22:12However, as we've seen, this approach is
22:14susceptible to representation collapse,
22:17with the network simply learning to
22:18output the same values for any input
22:20image.
22:22But now, applying Barlow's hypothesis as
22:24proposed by Stéphane Deny, we should
22:26reduce the redundancy between the
22:28outputs of different neurons.
22:31We have a bit of a choice to make here.
22:33We could compare the output of the first
22:35neuron in our first network to the
22:36output of our second neuron in our first
22:38network, or to the output of the second
22:40neuron in our second network.
22:42The team chose to compare to the output
22:44of the second network.
22:46As we'll see, this results in a simpler
22:47implementation, and the team further
22:49notes in the appendix of their paper
22:51that in practice they didn't see much
22:52difference between these alternatives.
22:55Here's the output of the second neuron
22:57in our second model.
22:59To measure reduce
23:00between neuron outputs, the team
23:02computed the cross-correlation between
23:05these output vectors.
23:07This computation consists of scaling
23:08each vector and taking the dot product,
23:11resulting in a single number, the
23:13correlation,
23:15or more precisely, the Pearson
23:16correlation coefficient between our
23:18vectors.
23:20To reduce the redundancy between our
23:21neurons as proposed by Barlow, we want
23:24this correlation to be close to zero.
23:27If we arrange the neuron outputs of our
23:29first encoder vertically and the outputs
23:31of our second encoder horizontally, we
23:33can compute and place the correlations
23:35between all pairs of neurons into a
23:37single matrix.
23:39This cross-correlation matrix has one
23:41row for each output neuron in our first
23:43encoder and one column for each output
23:46neuron in our second encoder.
23:48The elements along the diagonal capture
23:50the correlations between corresponding
23:52neurons.
23:54Since the whole idea here of this joint
23:55embedding architecture is to produce
23:57similar outputs for distorted versions
23:59of the same image, we want the
24:01corresponding neurons in our two
24:02encoders to have high correlations.
24:06Alternatively, all of the off-diagonal
24:08entries in our cross-correlation matrix
24:10correspond to different neurons in our
24:12two encoders.
24:14And following Barlow's hypothesis, we
24:16want to reduce the redundancy between
24:18these neurons.
24:20So, we want these correlations to be
24:21zero.
24:23So, ideally, our cross-correlation
24:24matrix looks like the identity matrix.
24:28The Nee and LeCun and their
24:29collaborators designed a new loss
24:31function for their joint embedding
24:32architecture
24:34that measured the deviation of their
24:35cross-correlation matrix from the
24:37identity matrix.
24:39Their new method, which they called
24:41Barlow Twins, worked surprisingly well,
24:44avoiding representation collapse while
24:46learning a powerful internal
24:47representation of the images that it was
24:49trained on.
24:52The team used a few different methods to
24:54measure the quality of these internal
24:55representations.
24:58Earlier, we saw how by using
25:00self-supervised pre-training, GPT-1 was
25:02able to outperform purely supervised
25:04models that had been adapted to specific
25:07language tasks.
25:09For vision tasks, one of the most
25:11important benchmarks at the time was
25:13accuracy on the ImageNet data set.
25:16This is the same image classification
25:18data set that the AlexNet model had
25:19shown breakthrough performance on back
25:21in 2012.
25:23The original AlexNet paper achieved an
25:25accuracy of 59.3% on the ImageNet
25:28validation set. To compare the
25:30self-supervised Barlow Twins approach to
25:33fully supervised models like AlexNet,
25:36the team used a common approach known as
25:37a linear probe,
25:40where a single layer of neurons are
25:41tacked onto the output of the Barlow
25:43Twins trained encoder model,
25:46and trained using supervised learning to
25:48classify the ImageNet data set.
25:51Importantly, the main encoder model is
25:52frozen during this training process.
25:56So, the simple linear probe is
25:57effectively adapting the Barlow Twins
25:59encoder's learned representation to
26:02solve the ImageNet classification task.
26:05Impressively, the frozen Barlow Twins
26:07encoder with a linear probe achieved an
26:10ImageNet accuracy of 73.2%
26:13outperforming the original fully
26:15supervised AlexNet model by over 10
26:17percentage points.
26:19However, in the 9 years from the AlexNet
26:21paper in 2012 to the Barlow Twins paper
26:24in 2021,
26:26fully supervised approaches had made
26:28significant improvements over AlexNet.
26:31In 2020, a team at Google applied the
26:33transformer architecture to image
26:34classification,
26:36achieving a new state-of-the-art
26:38ImageNet accuracy of 88.6%.
26:42So, by 2021, thanks to the Barlow Twins
26:45epiphany and other joint embedding
26:46approaches,
26:48self-supervised learning was advancing
26:50rapidly for vision tasks,
26:52but it was still inferior to fully
26:54supervised methods.
26:56The general and clearly superior
26:58self-supervised generative pre-training
27:00methods in language that were fueling
27:02the rapid advancement of LLMs were still
27:05out of reach for image and video
27:07applications.
27:09>> And so, it became clear that this video
27:10was
27:12the the right way to go. So, we kind of
27:14uh
27:14after that, publish another version, a
27:16simplified version basically of bottle
27:18twins called VICReg.
27:20Which turned out to be quite good. Uh
DINO
27:22and then simultaneously, another group,
27:23some of our colleagues at Fair Paris,
27:25were working on uh
27:27uh similar methods, which eventually
27:30came to be known as DINO.
27:33Uh DINO V1, V2, V3. Uh now they have a
27:36new version, um which is not called DINO
27:38anymore. Uh
27:39and and this is also a joint embedding
27:41uh technique. So,
27:43so, it was really clear joint embedding
27:46was better for represent learning, you
27:49know,
27:50>> Right.
27:50>> self-supervised learning to to represent
27:52images.
27:53>> The DINO V3 paper, released in August
27:562025,
27:57marked an important turning point,
28:00achieving a very near state-of-the-art
28:01image net accuracy of 88.4%
28:05using a joint embedding architecture.
28:08As the authors say in their paper,
28:11"All in all, this is the first time that
28:12a self-supervised model has reached
28:14comparable results to weakly and
28:16supervised models on image
28:18classification."
28:20The quality of representations that DINO
28:22V3 is able to learn without access
28:24[music] to any human-generated labels is
28:27astounding. DINO outputs an embedding
28:29vector for each patch of image that it
28:31analyzes.
28:33If I take this image of myself and take
28:35DINO's embedding vector from this image
28:37patch on my hand,
28:39and compare this embedding vector to the
28:40rest of the patches in the image,
28:42visualizing how similar each patch is to
28:44the hand patch using a color map,
28:47>> [music]
28:47>> Dino does a remarkably good job
28:49segmenting my hand from the background.
28:51[clears throat]
28:52Here's the same approach applied to a
28:53ball,
28:55a cat,
28:56and a book.
JEPA & World Models
28:58Following the success of Barlow Twins,
29:00VicRag, and Dino V1, in 2022, LeCun
29:04brought these and many other threads
29:06together into a 60-page position paper
29:08called a path towards autonomous machine
29:10intelligence.
29:12Unlike the great majority of LeCun's
29:14papers, [music] where he works on
29:15specific and technical pieces of machine
29:17learning theory or practice,
29:20a path towards autonomous machine
29:21intelligence takes a holistic first
29:24principles approach to how we should
29:25build intelligent machines.
29:28LeCun begins by arguing that our current
29:30approaches to AI are nowhere near the
29:32capabilities of human learning,
29:34giving the example of a teenager that
29:36can learn to drive a car in around 20
29:38hours of practice.
29:40>> How is it that we have those millions of
29:41hours of training data
29:43where we have we can train kind of level
29:45two system with it, which is what
29:47Tesla's doing basically.
29:48>> Yeah.
29:49>> Um
29:50but
29:51nowhere near level three, four, five.
29:53Okay?
29:54Uh
29:54yet a 17-year-old can learn to drive in
29:56a few hours of practice. Like how does
29:58that happen, right? Shouldn't we figure
30:00out what's the what's the secret there?
30:02>> Yeah.
30:03>> And my guess about it is the secret is
30:05world models.
30:06>> LeCun's billion-dollar bet is that the
30:08missing piece of modern AI is world
30:11models.
30:12Models that make predictions about the
30:13physical world.
30:16As he says in his 2022 position paper,
30:18common sense can be seen as a collection
30:20of models of the world that can tell an
30:22agent what is likely, what is plausible,
30:25and what is impossible.
30:27Using such world models, animals can
30:30learn new skills with very few trials.
30:33They can predict the consequences of
30:34their actions.
30:36They can reason, plan, explore, and
30:38imagine new solutions to problems.
30:41LeCun goes on to argue that joint
30:42embedding architectures offer the right
30:44foundation to build world models on top
30:47of.
30:48>> So, JEPA means joint embedding
30:49predictive architecture, and it's you
30:52you take an observation in the world,
30:55and then the next observation in the
30:56world, uh you run them through encoders,
30:58so it's like a joint embedding type
31:00architecture, and then you have a
31:01predictor that tries to predict that the
31:03state at time t + 1 from the state at
31:05time t, and you might condition this on
31:07an action, and then you have a world
31:08model.
31:09>> As a concrete example, instead of using
31:11a generative architecture to predict the
31:14pixel values in the next frame of video,
31:16we can map the video and next frame to
31:18embeddings, and then train a predictor
31:21model to predict the embedding of the
31:22next frame
31:24given the embedding of the video.
31:26In this implementation, the JEPA
31:28architecture frees the model of the
31:30intractable task of predicting every
31:33pixel in the next frame of video,
31:35and theoretically allows the predictor
31:37to focus on predicting only the salient
31:39features of the scene that make it
31:41through the encoder.
31:42Yann gives a nice example here.
31:45>> If you're trying to generate a model,
31:46you know, to predict what's going to
31:47happen in a dashcam video,
31:50uh it will spend most of its resources
31:52predicting the random motion of the
31:53leaves on the trees that border are
31:55bordering the road.
31:56And
31:57and those are things that are
31:58essentially not predictable, but they
31:59have a lot of pixels,
32:01you know, that move around.
32:03>> As Yann mentioned earlier, we can take
32:04JEPA one step further by conditioning on
32:07actions.
32:08In the VJEPA 2 paper, which we'll dig
32:10into in part two,
32:12the team conditions a JEPA model on the
32:14action signals sent to a robot arm.
32:17So, the JEPA model sees a sequence of
32:19images of the robot's arm and
32:20environment, and then is trained to
32:23predict the embedding of the next video
32:24frame,
32:26but is also given the control signals
32:27that are sent to the robot arm.
32:30This allows the predictor to learn to
32:31predict how various control signals will
32:33change the robot arm's position in the
32:36embedded image.
32:38This learned world model can then be
32:40used for robot planning and control.
32:43Given an image of some goal state, for
32:45example, moving a cup off of a platform,
32:48this image is passed into the next frame
32:50encoder, resulting in an embedding of
32:52the goal state of the robot.
32:55From here, a controls algorithm can be
32:56used to explore the world model's
32:58predictions given various hypothetical
33:01actions,
33:02and find a set of actions that will lead
33:04the model's predicted future state to
33:05match its goal state.
33:08As Yann says, this is really a new twist
33:10on an old idea.
33:11>> You build a model that gives you the
33:13state of the world at time T plus one as
33:14a function of state of the world at time
33:16T, and an action you imagine taking or
33:18intervention or control, right? And then
33:21if you have this, you can uh predict the
33:24outcome of a sequence of actions, and
33:25you can by optimization, you can figure
33:27out an optimal sequence of actions to
33:30arrive at a particular um outcome,
33:33right? This is classical optimal
33:34control. This is, you know, this is
33:36going back to the late '50s
33:38in the Soviet Union, early '60s in the
33:41in the West.
33:42>> Mhm.
33:42>> Uh very classical stuff.
33:44>> Yeah.
33:45>> What is not classical is you learn the
33:47model.
33:48>> Sure, yeah.
33:48>> a machine learning to learn the model.
33:50>> Right, yeah.
33:50>> What is even less classical is you learn
33:53a representation of the input
33:55that computes a state, an abstract state
33:59representation, and you learn the
34:02you know, the the model in that uh in
34:04that state.
34:06And that's Jepa.
But is JEPA good?
34:09>> But will Jepa or other world model-based
34:11approaches really overtake large
34:13language models?
34:15Since LeCun first proposed Jepa in 2022,
34:18the architecture has been applied by
34:20various teams to a wide range of
34:22problems.
34:23How exactly do these models stack up?
34:27In part two, we'll dive deeper into
34:28V-Jepa-2 to get a sense for what's
34:30really happening inside the model's
34:32embedding space.
34:34And see how V-JEPA 2 fares as a robotics
34:36control algorithm against the rapidly
34:38advancing VLA approaches.
34:41We'll also explore VL-JEPA, which solves
34:44many of the same vision language
34:46problems we solve today with multimodal
34:48LLMs, but in a very different way and
34:51with impressive results.
34:53Finally, we'll spend some time on an
34:54implementation of JEPA called Le World
34:56Model.
34:57Le World Model gives perhaps the most
34:59complete, albeit early, picture of what
35:01JEPA-based systems can do.
35:03Until next time, I'll leave you with
35:05Yann's take.
35:07>> Okay, then I will make a
35:08controversial statement that again is
35:10going to give me a lot of friends in
35:12Silicon Valley. Um
35:14I do not understand how you can even
35:16think of building an agentic system
35:20without a agentic system
35:23having the ability of predicting the
35:24consequences of its actions.
35:26>> Mhm.
35:26>> Okay?
35:27And VLA doesn't doesn't do that.
35:30>> Sure. Sure.
35:31>> And LLMs do not have world models. They
35:32cannot predict the consequences of their
35:34actions beforehand. They just take the
35:36action and then
35:38apres moi le deluge as
35:41you know, as some famous
35:43French kings said. So,
35:46uh
35:46if you really want to build reliable
35:49agentic systems, they absolutely have to
35:51be able to predict the consequences of
35:53their actions.
35:54So that you can plan a sequence of
35:55actions to do something, first of all to
35:58uh
35:59fulfill the task that they are being
36:01asked to fulfill, but also
36:03uh
36:04perhaps to, you know, guarantee some
36:06safety guardrails, right?
36:08>> And
36:08>> And the inference process now becomes a
36:10search as opposed to just a
36:12auto-regressive prediction.
36:13>> Right.
36:14>> Uh
36:15so, that's a world model. That that's
36:16the whole idea of a world model.
Welch Labs Book
36:19>> If you enjoyed this video, check out the
36:21Welch Labs Illustrated Guide to AI.
36:24Its cover produces highly consistent
36:26deno representations, so you know it has
36:29to be good.
36:31The book is beautifully illustrated and
36:33is a great way to dig deeper into many
36:34of the topics we touched on in this
36:36video.
36:37Chapter five on AlexNet is a great way
36:39to learn more about embedding vectors
36:41and the rise of deep learning.
36:44Chapter six on neural scaling laws takes
36:46a deeper look at the fascinating
36:47build-up from GPT-1 to GPT-3 at OpenAI.
36:52Chapter nine covers diffusion models,
36:55which are able to reconstruct highly
36:56accurate pixel-level representations of
36:58images and video,
37:00but with some notable trade-offs.
37:03Chapters 1 through 4 give some great
37:05background on all of these topics,
37:07covering the fundamentals of neural
37:08networks, backpropagation, and deep
37:10learning.
37:12Each chapter includes thought-provoking
37:13exercises and supporting code. The book
37:16is now shipping to 24 countries. You can
37:19pick up a copy today at welchlabs.com.