Scaling and Distilling Text Embeddings for Better Diffusibility
1University of Michigan
TL;DR. Scaling the latent embeddings of continuous diffusion language models (DLMs) improves their performance but is not enough. Distilling them with soft labels further makes them more diffusible.
Which embedding should a continuous diffusion language model use?
A typical family of continuous diffusion language models (DLMs) applies latent diffusion to text, where we first use a frozen embedding model to , then train a denoiser to generate these embeddings, and finally decode them back to tokens.
A token sequence \(\bm{s}\) becomes \(\bm{x}=\mathrm{Emb}(\bm{s})\in\mathbb{R}^{L\times d}\), and we denoise \(\bm{x}\) as continuous vectors. Early work tried co-trained or simple embeddings (from a lookup table), but later people found that pretrained embeddings improve over them, and we seek their full potential.
The image diffusion guys have asked the same question for years: which latent is easy to diffuse? For text it is mostly open. So we keep the diffusion fixed and change only the embedding. We follow , a recent continuous DLM, but our argument is not restricted to a specific diffusion recipe.
ELF runs standard Gaussian diffusion on the embeddings:
$$ \mathcal{L} = \mathbb{E}_{t,\bm{x},\bm{\epsilon}}\Big[\tfrac{1}{(1-t)^2}\,\big\|\bm{x}_\theta(\bm{z}_t,t)-\bm{x}\big\|^2\Big], \qquad \bm{z}_t = t\bm{x}+(1-t)\bm{\epsilon}. $$We change only \(\mathrm{Emb}\) and keep everything else as is.

Overview. Left: the same DLM on different embeddings. Right: on T5Gemma-2 a generated embedding can deviate from the plausible words; on the distilled student it lands among them.
Scaling the latent embeddings improves continuous DLMs.
We find a behavior within the T5 family of encoder-decoder models (T5-small/base → T5Gemma-1 → T5Gemma-2). The generative performance improves along with the representation power of the embeddings. Using T5Gemma-2 cuts the generative perplexity by about 40% at the same entropy as T5-small.
By scaling we mean updating to a stronger embedding model of the same family, pretrained with more data and a better recipe, and not only adding parameters.

Same DLM, different embedding models. Lower perplexity at higher entropy is better; the star is real text.
But scaled embeddings are hard to generate perfectly.
When we look at the generated embeddings, we find that some are far from every possible word for their position. We call such an embedding . It can still be decoded, but with low confidence or to a wrong word, and it tends to cause grammar mistakes and hurt generation.
For a position, take the top-\(k\) candidate words of the decoder and their real embeddings \(\bm{x}_1,\dots,\bm{x}_k\). A generated embedding \(\hat{\bm{x}}\) is invalid if
$$ \min_i \|\hat{\bm{x}}-\bm{x}_i\| \;>\; 0.91\,\mathrm{nn}, $$where nn is the median nearest-neighbor distance between real embeddings.

Embedding errors in a generated sample. Purple tokens are decoded with low confidence. Below, each disk is the range of a candidate word; an invalid embedding lands outside every disk.
This may be an inherent mismatch. Language is discrete and multimodal, where several separate words can fit the same position. MSE (used in continuous diffusion), however, is mode-averaging. The MSE-optimal denoiser predicts the , a weighted combination of the candidate words.
The minimizer of \(\mathbb{E}\,\|\bm{x}_\theta(\bm{z}_t,t)-\bm{x}\|^2\) is \(\bm{x}_\theta(\bm{z}_t,t)=\mathbb{E}[\bm{x}\mid\bm{z}_t]\). If the position could hold the words \(\bm{x}_1,\dots,\bm{x}_k\) with probabilities \(p_1,\dots,p_k\), this is \(\sum_i p_i\bm{x}_i\): a point inside their convex hull.
A scaled encoder keeps these words far apart, so the combination falls into the empty space between them, and the sampling is guided there.
Distilled embeddings are easier to generate.
To mitigate this, we distill the T5Gemma-2 encoder into a student encoder that learns the teacher decoder's probabilities as soft labels. These carry information about the plausible words of a position, so the student places these words closer and becomes more robust.

Distillation. The student matches the teacher's decoded probabilities, and the embeddings of plausible words move closer.
As a result, the same DLM trained on the student's embeddings generates fewer invalid embeddings.

Teacher against student. The student's generated embeddings lie in the region formed by the candidate words.
Results.
On OpenWebText with 1024-token sequences, the student wins at every sampling budget:

Over sampling settings. The student outperforms T5Gemma-2 from 64 to 512 sampling steps, while the teacher collapses at few steps.
The distillation trades discrimination for diffusibility (SST-2 probing drops from 89.4 to 78.1).
What's next?
- Few-step generation: ELF trained on raw T5Gemma-2 embeddings collapses at small NFEs. The student embeddings collapse much less, but can still be improved.
- Scaling to larger models for practical tasks such as QA, instruction following, and even reasoning.
- Using autoregressive models directly as embedding models, dropping the Gemma → T5Gemma → ELF detour. If one can directly make an LLM's activations diffusible, continuous DLMs can ride on every LLM release.
BibTeX
@article{zhang2026scaling,
title={Scaling and Distilling Text Embeddings for Better Diffusibility},
author={Zhang, Zekai and Tian, Yunjie and He, Yanjin and Zhang, Xiaoyan and Zhao, Dongdi and Qu, Qing and Fu, Di},
journal={arXiv preprint arXiv:2610.01016},
year={2026}
}