Iterative Combinatorial Brain Surgeon (iCBS): Scalable Pruning of Large Language and Vision Models (LLVMs)

The Challenge

State-of-the-art large language and vision models (LLVMs) have seen tremendous success, but their massive scale comes with a hefty price in terms of computational resources. The need to balance performance and efficiency has led to a growing interest in model compression techniques. By using methods like pruning, quantization, or distillation, researchers aim to streamline these models without sacrificing their impressive accuracy.

The Impact

With the integration of advanced methods — such as the one proposed below — and specialized hardware support for sparse models, we can significantly decrease the computational power and energy required to run AI models, all while maintaining their original performance. This can enable the deployment of smaller, more efficient models directly on devices, rather than relying on server-side processing — ultimately helping to enhance data privacy.

The Outcomes

We proposed iterative Combinatorial Brain Surgeon (iCBS), a scalable iterative pruning algorithm that optimizes over small blocks of weights in neural networks using block gradient descent. This blockwise approach can allow iCBS to scale to very large models, including LLVMs with billions of parameters, while helping to achieve higher performance compared to existing one-shot pruning techniques.

For further details on this project, read the full paper.