Doc-to-LoRA (D2L, arXiv 2602.15902, ICML 2026) is context distillation amortized into a hypernetwork: a Perceiver of about 309M parameters reads a frozen gemma-2-2b-it's per-layer activations over a document and writes a rank-8 A and B for every one of the 26 MLP down projections in a single forward pass. Internalizing a SQuAD passage took 0.086 s against 8.763 s for oracle context distillation, and on 2WikiMultihopQA D2L scored 0.857 against 0.901 for oracle distillation at 0.209 s instead of 40.2 s. The base weights never change, the adapter starts at exactly zero because its B scaler is initialized to zero, and long documents are chunked and concatenated along the rank axis with a learned bias block, giving rank 8(K+1). The cost moved to meta-training: around 5 days on 8 H200 GPUs for gemma-2-2b-it, repeated for every new target model, with released hypernetworks for only three base models. It breaks in known places: QASPER scores 0.543 against 0.945 for oracle distillation, unrelated queries fall below the base model (0.096 vs 0.201 on SQuAD), and the released internalize() call does not chunk long inputs. Start by picking one document set that is queried repeatedly and checking it against those three failure modes on paper before touching a GPU.
Doc to LoRA, written Doc-to-LoRA or D2L, is a hypernetwork from researchers at Sakana AI that turns a document into a LoRA adapter for a frozen language model in a single forward pass. Once it is trained, internalizing a SQuAD passage takes 0.086 seconds against 8.763 seconds of gradient updates for oracle context distillation (Table 4 of arXiv 2602.15902), and the base model then answers questions about that passage with the passage gone from its prompt. The base weights never change. What changes is a removable rank-8 delta on every MLP down projection, written by a network of about 309M parameters that has read the document.
The cost did not vanish, it moved. Training the hypernetwork for one target model, gemma-2-2b-it, took around 5 days on 8 H200 GPUs, and a different base model means a different training run. Quality sits below both the model reading the document and oracle per-document distillation, sometimes by four points and sometimes by forty.
Every D2L figure below comes from the paper (published at ICML 2026), the official repository pinned at commit baa85db (the last commit on main before this post; the repo has no releases or tags), the released config files and Hugging Face file listing, and the maintainers' answers on the issue tracker; sibling methods are cited by their own arXiv papers. Nothing here was run by us.
What Doc-to-LoRA learns: context distillation, amortized
Context distillation (CD) is the starting point. Given a context c, you train context-specific parameters so the model without c reproduces the model with c: minimize KL(p_θ(y | x, c) ‖ p_θc(y | x)) over queries x. Done per document, that means generating queries, running the teacher, and taking gradient steps on a LoRA. It works, and in the paper's measurements it costs from several seconds to over ten minutes per document.
D2L learns the map instead of solving each instance. The meta-objective (Eq. 4) is: minimize over the hypernetwork parameters φ the expectation over documents and query-response pairs of KL(p_θ(y | x, c) ‖ p_(θ + H_φ(c))(y | x)). The teacher is the frozen base model with the document in context. The student is the same frozen model with no document and the generated adapter H_φ(c) added. Only φ receives gradients.
The training targets are built offline. For each context-query pair, the paper samples one response from gemma-2-2b-it itself and stores the top-16 token logits for every generated token. Queries come from gemma-3-12b-it, 10 per FineWeb-Edu sample, alongside PwC, SQuAD, ROPES and DROP passages.
The loss in the code (trainer.py at baa85db) computes the student's log-softmax over the full vocabulary, gathers it at the teacher's stored top-16 indices, and takes the cross-entropy against the teacher's probabilities. That equals the KL up to the teacher's entropy, restricted to the top-K tokens. On top sits an L1 penalty on the generated A and B matrices, with --gen_lora_l1_reg_coef=0.1 in the stage-one training script.
Distilling a distribution beats training on answers. In the paper's ablation on an early checkpoint, the KL objective scored 0.819 normalized F1 on SQuAD against 0.763 for next-token prediction (Table 6).
Compare that with fine-tuning on a document. Fine-tuning needs per-document training data, an optimizer, backward passes and a stopping decision, every time the document arrives. D2L at inference time runs no backward pass, holds no optimizer state, and generates no synthetic questions. All of that was paid once, during meta-training.
Doc-to-LoRA architecture: from activations to A and B
The update is ordinary LoRA (Eq. 5): W'_l = W_l + α B_l A_l, with A_l of shape r × d_in and B_l of shape d_out × r. What is unusual is which modules get it and where A and B come from.
Only the MLP down projection is adapted. The paper says each generated adapter is applied to the down projection of each MLP block, and both the repo config and the released args.yaml set target_modules: [down_proj] and lora_r: 8. For gemma-2-2b-it (hidden size 2,304, intermediate size 9,216, 26 layers), that is a 9,216 to 2,304 projection in each of 26 layers.
# Doc-to-LoRA, released Gemma-2-2b-it checkpoint (commit baa85db)
# K = number of document chunks, T = tokens in a chunk
document chunk ids [K, T]
-> ctx encoder: 4-bit copy of gemma-2-2b-it, last layer dropped,
output_hidden_states stacked [K, 26, T, 2304] # Z_0 .. Z_25, one per target layer
-> fold layers into batch [26*K, T, 2304] # one Perceiver shared by all layers
-> modality_projection MLP [26*K, T, 512]
-> Perceiver encoder (8 latents) [26*K, 8, 512] # cross-attention stack
-> Perceiver decoder (8 queries) [K, 26, 1, 8, 512] # one latent per rank slot
-> per-layer ResMLP + L2 normalize [K, 26, 1, 8, 512]
-> per-layer head 512 -> 11520 [K, 26, 1, 8, 11520]
-> split 11520 = 9216 + 2304
A [K, 26, 8, 9216] # x scaler_A (init 1)
B (stored as [r, d_out]) [K, 26, 8, 2304] # x scaler_B (init 0)
-> combine_lora: concat chunks on rank, append learned bias block
A [1, 26, 8*(K+1), 9216]
B [1, 26, 8*(K+1), 2304]
-> every mlp.down_proj (9216 -> 2304):
y = W x + 45.25 * B^T (A x) # lora_alpha = 2 * r**1.5 at r = 8The context encoder is a second copy of the base model
The document is not embedded by a separate text encoder. The class PerLayerActivations loads another copy of the base model, removes the LM head, drops the last decoder layer, and runs it with output_hidden_states=True. Stacking the hidden states gives one feature map per target layer: the embeddings plus the outputs of the remaining 25 layers, so layer l's adapter is generated from Z_(l-1). With quantize_ctx_encoder: true in the released args, this copy loads in 4-bit NF4 with double quantization and bf16 compute.
One Perceiver, shared by all 26 layers
The layer axis is folded into the batch axis, so a single set of Perceiver weights serves every layer. Per layer, a projection MLP takes 2,304 dimensions down to a 512-dimensional latent, an encoder with 8 learned latent queries cross-attends over the document tokens through a stack of cross-attention blocks without self-attention (eight per the paper; the released config sets num_blocks: 9, and the released gemma checkpoint holds nine), and a one-block decoder emits 8 output queries: one latent per rank slot.
The head writes one rank-1 slice per latent
Each latent passes through a per-layer residual MLP, is L2-normalized, and goes through a per-layer linear head with weight shape 26 × 512 × 11,520 and no bias. The 11,520 outputs split into 9,216 values for a row of A and 2,304 for a row of B. With per_rank_gen: true, each of the 8 latents writes one rank-1 slice, and 8 slices make the rank-8 adapter. The scaler pair matters. A and B are multiplied elementwise by learned scalers of shape [1, 26, 8, 1]: scaler_A initialized to ones, scaler_B to zeros. At step zero the generated delta is exactly zero, so the student starts as the base model without context, the same reasoning as standard LoRA's zero-initialized B. Here are the shapes for the released Gemma-2-2b checkpoint, read from the code, not run: Two derived counts follow from the shapes. The head weight alone is 26 × 512 × 11,520 = 153,354,240 parameters, about half of the paper's 309M total. One chunk's generated adapter is 26 × 8 × 11,520 = 2,396,160 parameters. The released gemma_2b_d2l/checkpoint-20000/pytorch_model.bin is 1,261,017,534 bytes, about 315M fp32 parameters, which matches the nine-block released config rather than the paper's eight-block 309M; the base model is not in it and is loaded by name.
How the delta is applied
D2L does not go through PEFT's LoRA layers. It disables them and monkey-patches mlp.down_proj.forward on each layer to compute W x plus scaling times the low-rank term via einsum. The scaling is lora_alpha passed directly, and the loader sets lora_alpha = r ** (3/2) * 2, about 45.25 at rank 8. PEFT-style loaders divide alpha by rank, so any export to another serving stack has to fold that factor in deliberately. Generation runs in batched mode (all 26 layers in one pass, faster) or iterative mode (one layer at a time, less memory). The paper states the two are mathematically equivalent.
Doc-to-LoRA on long documents: chunking and rank
A document longer than one chunk produces one adapter per chunk, and Eq. 9 concatenates them along the rank axis. The code adds one thing the equation omits: combine_lora appends a learned, context-independent bias adapter (bias_A of [26, 8, 9216], bias_B of [26, 8, 2304]) as one more rank-8 block, and compute_rank returns (n_lora + 1) × rank. K chunks therefore give rank 8(K+1) on every down projection, not 8K. Four chunks mean rank 40.
The model learned to compose chunks in a second stage. Stage one ran 80,000 steps with one chunk per context (--max_steps=80000, 8 processes, --max_packed_ctx_len=4096). Stage two ran 20,000 steps with random chunking: --max_ctx_chunk_len=512, --min_ctx_chunk_len=25, --learning_rate=2e-5, with probability 0.5 of one chunk, 0.125 of two, and 0.0625 for each count from three to eight. The paper's reason for the split is that training was unstable when the hypernetwork had to emit several adapters from the start.
The headline long-context result comes from a separate hypernetwork trained only for needle-in-a-haystack retrieval: 640K samples of 32 to 256 tokens, about 3 hours on one H200, trained on ground-truth tokens rather than self-distillation. Evaluated with chunks of up to 1,024 tokens on an 8K-window gemma-2-2b-it, retrieval stayed close to perfect up to 40 chunks (40K tokens), five times the chunk count seen in training. The abstract states it as near-perfect zero-shot accuracy beyond the native window by more than 4x.
The memory comparison is the number that matters for serving. To answer from a 128K-token haystack, the base model used more than 12 GB of additional memory; with the needle internalized, under 50 MB regardless of haystack length. The paper calls this additional memory, not a KV-cache measurement. If your long-context pressure is memory, the other lever is KV cache quantization and its accuracy cost.
import torch
from ctx_to_lora.model_loading import get_tokenizer
from ctx_to_lora.modeling.hypernet import ModulatedPretrainedModel
ckpt = "trained_d2l/gemma_demo/checkpoint-80000/pytorch_model.bin"
state_dict = torch.load(ckpt, weights_only=False) # full unpickle: vet the file first
model = ModulatedPretrainedModel.from_state_dict(
state_dict, train=False, use_sequence_packing=False
)
model.reset()
tokenizer = get_tokenizer(model.base_model.name_or_path)
chat_ids = tokenizer.apply_chat_template(
[{"role": "user", "content": "What does clause 14.2 require?"}],
add_special_tokens=False,
return_attention_mask=False,
add_generation_prompt=True,
return_tensors="pt",
).to(model.device)
model.internalize(open("policy.txt").read()) # one forward pass, no gradient step
out = model.generate(input_ids=chat_ids, max_new_tokens=512)
print(tokenizer.decode(out[0]))
model.reset() # drop the generated adapterThe Python API, and what it does not do
The README's interface is three calls. The checkpoint load uses torch.load(..., weights_only=False), a full unpickle that can execute arbitrary code from the file, so treat the download the way you would any pickled model and vet pickled checkpoints before loading them. Three caveats sit behind that snippet. The README says the interface supports non-batched inputs only. At baa85db, internalize() tokenizes the whole string as one context with no chunking, so long documents are the caller's job: one user reported a roughly 20K-token document running out of memory on an 80 GB GPU, and the maintainers answered that memory and compute scale linearly with document length, above 64 GB for a 128K-token document, and advised chunking. And the README's demo checkpoint path points at the 80,000-step folder, whose step count matches the single-chunk first stage; the chunk-trained releases are the checkpoint-20000 folders. Stacking is the documented weak point. In another issue a user reported chunked multi-adapter inference on the released Qwen checkpoint collapsing into repeated tokens on longer documents, and the maintainers replied that this is consistent with what they observe internally, adding that stacking many adapters can cause knowledge interference and instability. Treat chunk count as something to test on your own documents, not a solved parameter.
Doc-to-LoRA vs context distillation vs fine-tuning: cost
Table 1 of the paper carries the cost comparison: 2WikiMultihopQA, documents up to 32K tokens, while the longest training sample was 2,344 tokens.
Read the memory column carefully. Batched D2L used more update memory than oracle distillation, 11.5 GB against 7.8 GB, and only iterative mode came in lower. On QASPER the gap is wider: batched D2L peaked at 31.153 GB against 7.665 GB for oracle CD, iterative at 4.781 GB (Table 10). If update memory is your constraint, run iterative and accept roughly 0.3 to 0.35 s more latency per document.
The oracle row is also not a deployable baseline. It distills with the real test query, which you do not have in production. The realistic per-document alternative is CD with generated queries, and against that D2L wins on every column: higher quality, a fraction of the memory, and hundreds to thousands of times less latency. On the three reading-comprehension benchmarks, the paper reports D2L and oracle CD both under 2 GB of VRAM during the update, CD with generated queries above 40 GB, and vanilla CD above 100 seconds including query generation.
| Method (2WikiMultihopQA) | Trained per document | Normalized performance | Update memory (GB) | Update latency (s) |
|---|---|---|---|---|
| CD, oracle query | LoRA, gradient steps | 0.901 | 7.820 | 40.171 |
| D2L, batched | Nothing, forward pass | 0.857 | 11.522 | 0.209 |
| D2L, iterative | Nothing, forward pass | 0.844 | 3.791 | 0.551 |
| CD, 25 generated queries | LoRA, gradient steps | 0.745 | 59.925 | 465.454 |
| CD, 5 generated queries | LoRA, gradient steps | 0.704 | 79.371 | 72.537 |
| Cost | D2L | Per-document CD | Fine-tune on the document |
|---|---|---|---|
| One-time training | About 5 days on 8 H200s for gemma-2-2b-it, repeated per base model | None | None |
| Per-document compute | One forward pass, under 1 s | Query generation plus gradient steps | Data prep plus gradient steps |
| Artifact per document | About 2.4M parameters per chunk | One LoRA | One LoRA or full checkpoint |
| Retrain on base model change | Hypernetwork | Every adapter | Every adapter |
Where Doc-to-LoRA breaks
The gap to the teacher depends on the text
On SQuAD, the paper reports D2L at 82.5% of the in-context upper bound. On 2WikiMultihopQA it trails oracle CD by 0.044. On QASPER, question answering over scientific papers, it scores 0.543 against 0.945 for oracle CD (Table 10, relative to truncated in-context performance). The architecture is the same in all three; the documents differ. Of these three, scientific papers opened the widest gap, and MultiFieldQA was wider still: 0.481 batched against 1.041 for oracle CD (Table 9).
Unrelated queries make the model worse
D2L is trained to assume the next query is about the internalized document. When it is not, the model drops below the base model. With an unrelated "assistant" context internalized, SQuAD fell to 0.096 against 0.201 for the base model and 0.211 for CD; with a roughly 4K-token distracting book chapter, 0.126 against 0.175 and 0.203 (Table 8). The paper says plainly that D2L is not trained to retain existing knowledge. A shared endpoint where some queries concern the document and others do not needs routing in front of it.
Capacity is a ceiling you can see
Doubling the rank to 16 lifted SQuAD from 0.814 to 0.896 and DROP from 0.655 to 0.711, while ROPES moved from 0.906 to 0.895 (Table 7). Rank helps two of three benchmarks, which says rank 8 is binding. Independent evidence points the same way at small scale: a single-author preprint from August 2026 (arXiv 2608.21386) found one-pass emitted adapters recovering 14.0 ± 0.9% (5M-parameter model) and 11.2 ± 0.5% (15M) of the in-context gain, with a rank sweep moving capture from 5% to 21% before plateauing. It does not measure D2L, but it is the same mechanism hitting the same wall.
Everything else
The hypernetwork is tied to its base model. Released checkpoints cover three: gemma-2-2b-it, Mistral-7B-Instruct-v0.2 and Qwen3-4B-Instruct-2507. On Mistral, SQuAD ROUGE-L recall was 0.835 for D2L against 0.919 with the context in the prompt, and the paper notes Mistral struggles to internalize long contexts. After reset(), the README warns the model will hallucinate about the document, and with the adapter applied, nothing marks which part of an answer came from the document. Weights carry no provenance.
Doc-to-LoRA vs Text-to-LoRA and other LoRA hypernetworks
D2L is one point in a line of methods where one network writes or edits another network's weights. The LLM models pillar covers the fine-tuning side these methods sit beside.
Text-to-LoRA is the sibling with the near-identical name. Its hypernetwork embeds a task description, adds learned layer and module embeddings, and outputs an adapter for the attention projections; its three head variants have 55M, 34M and 5M trainable parameters. It teaches a behaviour. D2L stores content, which is why it reads activations of the document through the target model rather than an embedding of a sentence.
Internalizer is the scale test. It applies the same document-to-LoRA idea to a frozen 284B DeepSeek v4 Flash, about 20 times the largest target in prior work, using a model-agnostic trunk with thin per-model entry and exit layers, a Perceiver with 256 encoder queries, and rank 16(k+1). Its 84.9% top-1 and 97.8% top-5 are teacher-forced token accuracy at reproducing documents up to 4,096 tokens, not question answering. At 16,384 tokens, beyond its training length, adapter top-1 falls to 22 to 25% against 59 to 68% for the base model.
SEAL and MEND/MEMIT are on the table for contrast. SEAL has the model write its own training data, then runs real gradient updates. MEND and MEMIT edit the base weights in place at the level of individual facts. D2L's property that matters operationally is that its output is additive and removable: the approved base model stays the approved base model.
The D2L paper's appendix also shows the same architecture emitting a KV-cache prefix (the equivalent of 20 prefix tokens) instead of a LoRA for Qwen3-4B, near-perfect on needle retrieval to 8K tokens before degrading. The hypernetwork pattern is not tied to LoRA as the output format.
| Method | Input | Output | Base weights changed | Per-update compute | Published evidence |
|---|---|---|---|---|---|
| Text-to-LoRA (arXiv 2506.06105) | Task description | Rank-8 attention LoRA, Mistral-7B | No | Forward pass | 10-task avg 67.7 vs base 55.8, multi-task LoRA 66.3 |
| Drag-and-Drop LLMs (arXiv 2506.16406) | A few unlabeled task prompts | LoRA weights | No | Forward pass | Authors report up to 12,000x lower overhead than full fine-tuning, up to 30% gains |
| Doc-to-LoRA (arXiv 2602.15902) | Document, via frozen model activations | Rank 8 per chunk on down_proj | No | Forward pass | 0.857 vs oracle CD 0.901 on 2WikiMultihopQA |
| SHINE (arXiv 2602.06358) | Context | LoRA, hypernetwork reuses frozen LLM parameters | No | Forward pass | No numbers in the abstract |
| Internalizer (arXiv 2610.11715) | Document | Rank 16 per chunk on shared-expert MLP, 284B model | No | Forward pass | 84.9% teacher-forced top-1 vs 63.4% base |
| SEAL (arXiv 2506.10943) | New context or few-shot examples | Fine-tuning data and directives | Yes, via SFT | Gradient steps | RL on downstream reward |
| MEND / MEMIT (arXiv 2110.11309, 2210.07229) | One fact or thousands of associations | Direct edit of base weights | Yes, in place | Gradient transform or closed-form edit | MEMIT: thousands of edits on GPT-J 6B and GPT-NeoX 20B |
Doc-to-LoRA vs RAG vs long context on-prem
The D2L paper does not evaluate against retrieval. What follows is reasoning from the published numbers, labelled as such.
For changing corpora, compare the adapter's regenerate-on-change model with incremental RAG index updates; both move work from query time to ingest time, as does compiling documents into an LLM-maintained wiki.
Inside a regulated perimeter, the answer shifts in four places.
The adapter is a derivative of the document. A delta generated from a confidential credit file or clinical protocol carries that document's content in a form no DLP scanner reads. It inherits the source's classification, access control and retention. Deleting the source means deleting every adapter generated from it. The repo's reset() clears memory; anything you persist is your records problem.
There is no export path at baa85db. Generated adapters live in generated_loras and are applied by patching forward. The only code that writes an adapter to disk belongs to the baselines (context distillation and the bundled Text-to-LoRA), not to D2L. Serving a D2L adapter from another stack means writing your own export from the A and B tensors, handling the 45.25 scaling and a rank that grows as 8(K+1). Whether a given multi-LoRA server accepts that shape is something to test; the flags and memory math live in serving many LoRA adapters on vLLM.
Supply chain. The README, demo, example and evaluation loaders pass weights_only=False. The base model is fetched by name from a gated Hugging Face repository, so an air-gapped host needs it pre-staged after accepting the Gemma terms, and the package pins transformers==4.51.3.
Where the compute sits. Generating an adapter is a forward pass. If you train your own hypernetwork, that run is the compute that could matter for the EU AI Act's modification threshold.
| Situation | Use | Why |
|---|---|---|
| Same document queried many times by one team | Generated adapter | Removes the document from every prompt; under 50 MB vs over 12 GB extra at 128K tokens |
| Answers must cite a clause or page | Retrieval | Weights carry no provenance |
| Documents change daily | Retrieval, or regenerate per version | Maintainers' answer to an update is delete the adapter and generate a new one |
| Endpoint also takes unrelated queries | Retrieval, or route before the adapter | Unrelated queries fell to 0.096 vs 0.201 for the base model |
| Dense technical text, exact figures matter | Long context or retrieval | QASPER 0.543 vs 0.945 for oracle CD |
| Document fits the window, few queries | Long context | No hypernetwork to train; watch long-context degradation |
How to evaluate Doc-to-LoRA this week
Pick one internal document set that is queried repeatedly by the same group and chunks cleanly into short passages. On paper, before any GPU time, write down three numbers: the context tokens per query you would stop sending, the oracle-CD gap for the closest benchmark in the paper (0.044 on multi-hop QA, 0.402 on scientific papers), and the share of that endpoint's traffic that is about something else, because that traffic lands in the 0.096 interference case. If the second number is large or the third is not near zero, keep retrieval and stop there.
FAQ
Quick answers to the questions this post tends to raise.
