LeCun Team Paper: 109M-Parameter Text JEPA Outperforms 72B-Parameter Qwen2.5 in Cross-Modal Alignment

A 660-fold parameter gap, and the small model won?!

Wow, LeCun has really put the pressure on Scaling Law this time.

A small model with only 109 million parameters actually beat the 72-billion-parameter Qwen2.5 on a cross-modal representation alignment task.

What's even more striking is that the paper also found:

Give the model a different training method, and the number of training samples needed to achieve the same result plummets to 1/42 of the original.

Now the big model vendors who obsess over parameters and data every day are probably feeling a bit awkward...

In its paper analysis, alphaXiv directly pointed out the most stinging conclusion:

How a model is trained may be more important than simply making the model bigger.

Now, the view LeCun has been preaching for years finally has new theoretical and experimental backing:

LLMs are of course powerful, but they are mainly accumulating knowledge; if a model truly wants to understand the world, simply stacking up parameters may be the wrong path.

Even Shruti Mishra, a former Google DeepMind researcher, lamented after reading it:

Yann LeCun is right, it's just that the industry has been unwilling to admit it.

Looking closely at the paper, this contest is actually comparing a very special capability—cross-modal representation alignment.

For example, suppose there is an orange cat in front of you; the vision model sees a photo of the cat, and the language model reads "an orange cat lying on the sofa."

Although the inputs are completely different (one processes pixels, the other processes text), they are clearly talking about the same cat.

So the question is:

If both models truly understand this cat, shouldn't the internal representations formed in their minds also have something in common?

This is the core discussed in the paper—the Platonic Representation Hypothesis (PRH).

In 2024, researchers from MIT and other institutions proposed this hypothesis:

Previous research has indeed observed this trend.

But the question is, must models be stacked large enough to achieve this?

This time, LeCun and his collaborator Randall Balestriero discovered:

What LeCun's team used this time is exactly JEPA (Joint Embedding Predictive Architecture), which he has been strongly advocating for years.

Don't be intimidated by the name; to understand this paper, you only need to grasp the difference between two training approaches:

Good Views and Bad Views.

The names sound casual, but the results are vastly different.

Simply put, Good Views let the model capture the common information behind different observations, while Bad Views may make the model spend too much effort on details such as lighting and wording.

The difference between the two training approaches also became the breakthrough point for the mathematical proof in this paper.

The research team proved that under certain idealized conditions, if JEPAs of different modalities are all trained with Good Views, then even if they learn independently, the resulting representations can be matched through a rotation transformation.

Here, "rotation" refers to a coordinate transformation in high-dimensional space.

To put it more plainly:

The vision model and the language model may simply be using two different coordinate systems to describe the same world; as long as the relationship between the coordinate systems is found, the representations on both sides can be aligned.

At this point, the "Platonic Representation Hypothesis," which previously relied mainly on experimental observation, has a mathematical proof under a specific JEPA framework (see Section 2.1 of the paper).

But LeCun's team wasn't done; they even classified the most mainstream approach in current large language models, "Next-Token Prediction," as Bad-View JEPA.

Okay, okay—before it was just talk, but this time it's for real~

Another author of the paper, Randall Balestriero (currently an assistant professor of computer science at Brown University and a researcher at AMI Labs), also said after the paper was released:

We proved the Platonic Representation Hypothesis for JEPA: as long as Good Views are used, even if encoders of different modalities are trained independently, the representations they learn can be aligned through a single rotation. Next-token prediction uses Bad Views. Our 109-million-parameter text JEPA outperforms the 72-billion-parameter Qwen2.5 in cross-modal alignment, and the training samples needed to achieve the same effect drop to 1/42 of the original.

Their reasoning is that "predicting the next token" essentially asks the model to guess what will appear next based on the preceding text.

Of course, this kind of training can learn a lot of knowledge, but it inevitably spends considerable effort on wording, grammar, context, and other regularities of language itself—yet these things may not help the model better understand the world behind the text.

Good Views is more direct: it lets the model learn from different observations of the same scene and capture the common information.

In comparison, one has to slowly grope through massive amounts of text, while the other goes directly for common scene information.

If what is being compared is cross-modal alignment, the gap may open up just like that.

Maybe the model doesn't learn well enough because we didn't teach it right from the start.

As for exactly how big the gap can be, they first validated the idea on the Flickr30k dataset, then expanded the experiment to text data corresponding to about 940 million web images.

Each image has different textual descriptions, including web page Alt-text and descriptions generated by vision-language models.

In this way, the same scene has multiple textual expressions, which is exactly suitable for constructing Good Views.

The results were quite striking.

Under the same 109M-parameter encoder and training corpus conditions:

Good-View JEPA needed only 21.6 million training samples to reach the cross-modal alignment level that the "next-token prediction" model achieved with 900 million samples.

That's a full 42-fold difference.

Moreover, even when counted by number of tokens, the tokens required for Good Views training dropped to 1/14 of the original.

Next, the research team took this 109M text JEPA and compared it head-on with a range of large language models.

On the WIT-1024 benchmark, the representation alignment scores with the vision model DINOv2 are as follows:

Wow, with a parameter gap of about 660 times, the small model actually got a higher score.

Moreover, the research team also tried connecting independently trained vision models and text models through a simple linear mapping.

As a result, on the COCO image captioning task, it achieved a CIDEr score of 0.838.

When further attempting to fit the mapping without using image-text paired data, the model also achieved 64% Top-5 accuracy in zero-shot classification experiments related to ImageNet-100.

This means that even if models of different modalities are trained separately, as long as the representations they learn are close enough, they can later be connected through relatively simple methods.

Of course, don't rush to declare a comprehensive victory for small models.

The 109M surpassing 72B compares a specific cross-modal representation alignment metric; it does not mean it is also stronger on general tasks such as reasoning and coding.

In addition, the paper's mathematical proof relies on idealized conditions such as a finite world and invertible observations, and real experiments have not yet fully satisfied all the assumptions.

But at least LeCun has offered a new line of thinking worth taking seriously.

If the training method is chosen correctly, a small model may also learn more efficient cross-modal representations than a large model.

In the past, everyone was busy studying how big a model must be to better understand the world.

Now LeCun asks the reverse question:

Is it possible that the model doesn't learn well enough because we didn't teach it right from the start?

Paper address:

Reference link:

This article comes from the WeChat public account "Quantum Bit", author: Yishui, published by 36Kr with authorization.