Back to Zekai's blogs

Code arXiv alphaXiv Checkpoints

Zekai Zhang1, Yunjie Tian, Yanjin He1, Xiaoyan Zhang1, Dongdi Zhao, Qing Qu1, and Di Fu

1University of Michigan

TL;DR. Scaling the latent embeddings of continuous diffusion language models (DLMs) improves their performance but is not enough. Distilling them with soft labels further makes them more diffusible.


Which embedding should a continuous diffusion language model use?

A typical family of continuous diffusion language models (DLMs) applies latent diffusion to text, where we first use a frozen embedding model to , then train a denoiser to generate these embeddings, and finally decode them back to tokens.

The image diffusion guys have asked the same question for years: which latent is easy to diffuse? For text it is mostly open. So we keep the diffusion fixed and change only the embedding. We follow , a recent continuous DLM, but our argument is not restricted to a specific diffusion recipe.

Left: generative perplexity against entropy for the same diffusion model trained on different embeddings. Right: generated embeddings among candidate words, for T5Gemma-2 and for the distilled student.

Overview. Left: the same DLM on different embeddings. Right: on T5Gemma-2 a generated embedding can deviate from the plausible words; on the distilled student it lands among them.


Scaling the latent embeddings improves continuous DLMs.

We find a behavior within the T5 family of encoder-decoder models (T5-small/base → T5Gemma-1 → T5Gemma-2). The generative performance improves along with the representation power of the embeddings. Using T5Gemma-2 cuts the generative perplexity by about 40% at the same entropy as T5-small.

Generative perplexity against entropy for the same diffusion model trained with different encoders.

Same DLM, different embedding models. Lower perplexity at higher entropy is better; the star is real text.


But scaled embeddings are hard to generate perfectly.

When we look at the generated embeddings, we find that some are far from every possible word for their position. We call such an embedding . It can still be decoded, but with low confidence or to a wrong word, and it tends to cause grammar mistakes and hurt generation.

A generated sample with uncertain tokens marked, and four positions drawn among their candidate words.

Embedding errors in a generated sample. Purple tokens are decoded with low confidence. Below, each disk is the range of a candidate word; an invalid embedding lands outside every disk.

This may be an inherent mismatch. Language is discrete and multimodal, where several separate words can fit the same position. MSE (used in continuous diffusion), however, is mode-averaging. The MSE-optimal denoiser predicts the , a weighted combination of the candidate words.

A scaled encoder keeps these words far apart, so the combination falls into the empty space between them, and the sampling is guided there.


Distilled embeddings are easier to generate.

To mitigate this, we distill the T5Gemma-2 encoder into a student encoder that learns the teacher decoder's probabilities as soft labels. These carry information about the plausible words of a position, so the student places these words closer and becomes more robust.

The distillation pipeline and its effect on the embeddings of candidate words.

Distillation. The student matches the teacher's decoded probabilities, and the embeddings of plausible words move closer.

As a result, the same DLM trained on the student's embeddings generates fewer invalid embeddings.

Generated samples and embedding panels for T5Gemma-2 and for the distilled student.

Teacher against student. The student's generated embeddings lie in the region formed by the candidate words.


Results.

On OpenWebText with 1024-token sequences, the student wins at every sampling budget:

Generative perplexity against entropy over sampling settings, for T5Gemma-2 and the student.

Over sampling settings. The student outperforms T5Gemma-2 from 64 to 512 sampling steps, while the teacher collapses at few steps.

The distillation trades discrimination for diffusibility (SST-2 probing drops from 89.4 to 78.1).


What's next?

  • Few-step generation: ELF trained on raw T5Gemma-2 embeddings collapses at small NFEs. The student embeddings collapse much less, but can still be improved.
  • Scaling to larger models for practical tasks such as QA, instruction following, and even reasoning.
  • Using autoregressive models directly as embedding models, dropping the Gemma → T5Gemma → ELF detour. If one can directly make an LLM's activations diffusible, continuous DLMs can ride on every LLM release.

BibTeX

@article{zhang2026scaling,
  title={Scaling and Distilling Text Embeddings for Better Diffusibility},
  author={Zhang, Zekai and Tian, Yunjie and He, Yanjin and Zhang, Xiaoyan and Zhao, Dongdi and Qu, Qing and Fu, Di},
  journal={arXiv preprint arXiv:2610.01016},
  year={2026}
}