Train Overcomplete, Deploy Compact: Scaling Recovery Capacity for Structured LLM Pruning

Seungmin Oh; Donggeon Lee; Jongbin Ryu

EMNLP 2026

Train Overcomplete, Deploy Compact: Scaling Recovery Capacity for Structured LLM Pruning

Scale recovery capacity during training, then merge it away: better pruned LLMs at the same deployment cost.

Seungmin Oh¹
Donggeon Lee¹
Jongbin Ryu^1†

¹Ajou University
^†Corresponding author

retained performance on six LLMs at 25% and 50% pruning: SOTA
points of retained reasoning over the best recovery baseline (best case, 50% pruning): +8.4
extra parameters, memory or latency after merging: 0

See it prune, grow and merge

A dense LLM: eight blocks, each built from linear projections like the weight matrix P.

Blocks at inference: 8
Trainable recovery parameters: -
Extra inference cost: -

A schematic eight-block model; drag the timeline, pick a step or press Play. The matrices are a toy 6×8 example (the merge is computed exactly), the α schedule is the paper's, and parameter counts are its LLaMA3-3B setting at 25% pruning.

Train overcomplete, deploy compact

LoRA: low-rank adapters on every remaining layer, with a long backpropagation path. — (a) LoRA

OverRep: overcomplete modules on the two blocks after the pruned gap, merged into one weight per projection at deployment. — (a) LoRA

Overcomplete recovery module

After the pruned blocks are removed, the two blocks right after the gap become recovery blocks. Each linear projection P_i in them is temporarily expanded to D_i(P_i + W_i): an additive parallel branch W_i (initialized to zero) broadens the optimization space, and a cascaded multiplicative transformation D_i in the output space (initialized to the identity) reshapes it, so training starts exactly at the pretrained weights. The first recovery block trains its own weights and the added ones; the second keeps its weights frozen and trains only the added ones. In the reported setup, attention D_i and MLP W_i sit only on selected projections of that frozen second block (the hybrid placement).

Annealed activation

The annealed activation 𝒜_s(x) = α_sσ(x) + (1 − α_s)x sits between P_i + W_i and D_i, with σ the model’s own activation. α_s rises from 0 to 1 over the first 1% of steps, follows a cosine decay and stays at 0 for the last 20%, so the module trains with nonlinear dynamics but starts and ends as a purely linear map.

Exact re-parameterization

Once linear, each triplet merges into one standard weight, P_i = D_i(P_i + W_i). The deployed checkpoint is a plain pruned model: same architecture, parameters, memory and latency.

Better recovery, nearly everywhere

On average over six LLMs, OverRep retains the most performance in reasoning and generation at both pruning ratios, and it leads in 23 of the 24 backbone-by-setting comparisons.

AdaLoRAICLR’23
RankAdaptorNAACL’25 Findings
RestoreLCCNeurIPS’25
LaCoEMNLP’24 Findings
ShortGPTACL’25 Findings
StreamlineICLR’25
OverRep (ours)

Figure 2.Retained performance (RP, pruned accuracy over dense accuracy × 100), averaged over six LLMs, as a gain over LoRA recovery. Hover for exact values. AdaLoRA, RankAdaptor and RestoreLCC recover the same pruned model; LaCo, ShortGPT and Streamline are complete pruning pipelines. Paper, Tables 1-3.

	Reasoning		Generation
	25%	50%	25%	50%
LLaMA2-7B	+3.2	+3.9	+1.6	+4.2
LLaMA2-13B	+0.7	+3.4	+2.9	+5.2
LLaMA3-3B	+4.1	+3.9	+1.4	+1.3
LLaMA3-8B	+2.4	+4.5	+2.2	−1.7
Qwen3-4B	+1.1	+2.4	+4.1	+3.3
Qwen3-8B	+0.4	+1.4	+0.4	+1.3

behind the best other method ahead of the best other method 23 of 24 cells ahead

Figure 3.OverRep's retained performance minus the best of the seven other methods, for each backbone and setting. Hover a cell for both values. Paper, Tables 1-3.

Not just more parameters

More trainable parameters alone do not explain the gain: at 25% pruning, LoRA scaled to OverRep's 650M parameters gets worse, and full fine-tuning and logit distillation cost more yet still trail OverRep in reasoning. OverRep and OverRep-KD sit at the top, and OverRep trains in the least time (4.6 GPU-hours with cached features, tied with Streamline).

Retained reasoning vs. trainable parameters

Figure 4.LLaMA3-3B at 25% pruning. Bubble area is training time in GPU-hours; OverRep and Streamline reuse cached features (a one-time precomputation under an hour), and Streamline uses its own pruning mask. Paper, Tables 1, 2, 5, 6 and the runtime table; Full FT and Logit-KD parameter counts from the poster.

Cheaper to train as models grow

Recovery is local to two blocks with cached inputs, so its cost barely grows with model size, while LoRA backpropagates through the whole model.

Figure 5.End-to-end recovery speedup over LoRA at 25% pruning, counting OverRep's one-time feature cache. LoRA's cost grows with model size because it backpropagates through the whole model; OverRep trains two blocks. Hover for GPU-hours and accuracy. Paper, Appendix (end-to-end cost).

Every piece adds recovery

Components on the MLP side help most, W and D are complementary, and the annealed activation adds the last step on top of the hybrid placement.

Reasoning RP, gain over Plain

Generation RP, gain over Plain

Figure 6.LLaMA3-3B at 25% pruning. Plain fine-tunes the same recovery blocks without W, D or the activation. Attn/MLP: the component on attention or MLP projections only; Uniform: W and D on every projection of both recovery blocks; Hybrid: attention W and MLP D in both recovery blocks, attention D and MLP W only on selected projections of the frozen second block; 𝒜s: the annealed activation. Paper, Table 4.

Citation

@inproceedings{oh2026overrep,
  title     = {Train Overcomplete, Deploy Compact: Scaling Recovery Capacity for Structured {LLM} Pruning},
  author    = {Oh, Seungmin and Lee, Donggeon and Ryu, Jongbin},
  booktitle = {Proceedings of the Conference on Empirical Methods in Natural Language Processing},
  year      = {2026},
  publisher = {Association for Computational Linguistics},
  url       = {https://arxiv.org/abs/2609.06974}
}