CoSpaDi: Compressing LLMs via Calibration-Guided Sparse Dictionary Learning

AI Digest

CoSpaDi通过校准引导的稀疏字典学习，实现LLM压缩，提升压缩比与模型性能的平衡。

CoSpaDi uses calibration-guided sparse dictionary learning to compress LLMs, improving accuracy-compression trade-offs at 20-40% compression ratios.

Key points

• 替代传统低秩近似，采用稀疏字典学习建模多子空间 Replaces low-rank approximation with sparse dictionary learning for union-of-subspaces modeling
• 通过校准激活将功能重建转化为标准字典学习问题 Transforms functional reconstruction into standard dictionary learning via calibration activations
• 在LLaMA和Qwen模型上超越SVD和结构化剪枝基线 Outperforms SVD and structured pruning baselines on LLaMA/Qwen at 20-40% compression
• 支持稀疏-密集计算与后训练量化 Supports sparse-dense computation and post-training quantization

Takeaway: 无需训练的稀疏字典学习为LLM压缩提供新范式。 / Calibration-guided sparse dictionary learning offers a novel training-free compression paradigm for LLMs.

Why it matters 提供无需训练的高效压缩方案，适合资源受限场景部署。