A new Transformer architecture for powerful LLMs without GPUs

Matrix multiplications (MatMul) are probably the most computationally costly operations in giant language fashions (LLMs) that use the Transformer structure. As LLMs develop to giant sizes, the price of MatMul will increase considerably, growing reminiscence utilization and latency throughout coaching and output.

Now, researchers from UC Santa Cruz, Suchow College, and UC Davis have developed a brand new structure that fully eliminates matrix multiplication from language fashions whereas sustaining excessive efficiency at scale.

Of their paper, the researchers current MatMul-free language fashions that obtain efficiency on par with state-of-the-art Transformers, whereas requiring considerably much less reminiscence throughout inference.

MatMul

Matrix multiplication is a elementary operation in deep studying, the place it’s used to mix knowledge and weights in neural networks. MatMul is important for duties equivalent to remodeling enter knowledge by way of neural community layers to make predictions throughout coaching and inference.

GPUs are designed to carry out many MatMul operations concurrently as a result of their extremely parallel structure. This parallelism permits GPUs to deal with the large-scale computations required for deep studying a lot quicker than conventional CPUs, making them important for coaching and effectively operating complicated neural community fashions.

Nevertheless, with LLMs scaling to tons of of billions of parameters, MatMul operations have develop into a bottleneck, requiring very giant GPU clusters in each the coaching and inference levels. Changing MatMul with an easier operation may end up in vital financial savings in reminiscence and computation. However earlier efforts to interchange MatMul’s operations have had blended outcomes, lowering reminiscence consumption however making operations slower as a result of they do not carry out properly on GPUs.

Changing MatMul with ternary operations

In a brand new paper, the researchers suggest changing the normal 16-bit floating-point weights utilized in Transformers with 3-bit ternary weights that may take one in all three states: -1, 0, and +1. Additionally they change MatMul with additive operations that present equally good outcomes at a lot decrease computational price. The fashions include “BitLinear layers” that use ternary weights.

“By limiting the weights to the set {−1, 0, +1} and making use of further quantization strategies, MatMul operations are changed by addition and negation operations,” the researchers write.

Additionally they make deeper adjustments to the structure of the language mannequin. Transformer models include two essential elements: a token mixer and a channel mixer. The token mixer is chargeable for integrating info into the assorted tokens in sequence. In conventional Transformer fashions, that is often achieved with self-management mechanisms that use MatMul operations to compute relationships between all token pairs to seize dependencies and context info.

Nevertheless, within the non-MatMul structure described within the paper, the token mixer is applied utilizing a non-MatMul linear recurrent unit (MLGRU). GRU is deep studying for sequence modeling that was well-liked earlier than Transformers. The MLGRU processes the sequence of tokens by updating the hidden states by way of easy ternary operations with out the necessity for costly matrix multiplications.

A channel mixer is chargeable for integrating info throughout a number of characteristic channels right into a single token illustration. The researchers applied their channel mixer utilizing a Gated Linear Unit (GLU), which can also be used within the Llama-2 and Mistral. Nevertheless, they modified GLU to additionally work with triple weights as an alternative of MatMul operations. This allowed them to scale back computational complexity and reminiscence utilization whereas sustaining the effectivity of characteristic integration

“By combining the MLGRU token mixer and the GLU channel mixer with triple weights, our proposed structure depends completely on addition and element-wise merchandise,” the researchers write.

Estimating language fashions with out MatMul

The researchers in contrast two variations of their MatMul-free LM with the superior Transformer++ structure utilized in Llama-2 on totally different mannequin sizes.

Apparently, their scaling predictions present that the LM with out MatMul is extra environment friendly in utilizing further computing sources to enhance efficiency in comparison with the Transformer++ structure.

The researchers additionally evaluated the standard of the fashions on a number of language duties. The two.7B MatMul-free LM outperformed its Transformer++ counterpart in two superior benchmarks, ARC-Problem and OpenbookQA, whereas sustaining comparable efficiency in different duties.

“These outcomes spotlight that non-MatMul architectures are in a position to obtain robust null efficiency in a wide range of language duties starting from query answering and reasoning to bodily understanding,” the researchers wrote.

LM with out MatMul is anticipated to have decrease reminiscence utilization and latency in comparison with Transformer++, and its reminiscence and latency benefits develop into extra obvious because the mannequin measurement will increase. For the 13B mannequin, LM with out MatMul used solely 4.19 GB of GPU reminiscence with a latency of 695.48 ms, whereas Transformer++ required 48.50 GB of reminiscence with a latency of 3183.10 ms.

Optimized implementations

The researchers created an optimized GPU implementation and a customized FPGA configuration for language fashions with out MatMul. By implementing triple-dense GPU layers, they have been in a position to velocity up coaching by 25.6% and scale back reminiscence consumption by as much as 61.0% in comparison with the unoptimized baseline implementation.

“This work goes past simply software program implementations of light-weight fashions and reveals how scalable but light-weight language fashions can each scale back computational necessities and energy consumption in the actual world,” the researchers write.

The researchers imagine their work might pave the best way for the event of extra environment friendly and hardware-friendly deep studying architectures.

Attributable to computational limitations, they have been unable to check the structure with out MatMul on very giant fashions with greater than 100 billion parameters. Nevertheless, they hope their work will function a name to motion for establishments and organizations with the sources to construct the biggest language fashions to spend money on accelerating light-weight fashions.

Ideally, this structure would make language fashions a lot much less depending on high-end GPUs like Nvidia and permit researchers to run highly effective fashions on different, inexpensive and fewer supply-constrained varieties of processors. The researchers launched code for the algorithm and fashions that the analysis neighborhood can construct on.

“By prioritizing the event and deployment of MatMul-free architectures equivalent to this one, the way forward for LLM will solely develop into extra accessible, environment friendly, and sustainable,” the researchers write.