Meet Medusa: An Efficient Machine Learning Framework for Accelerating Large Language Models (LLMs) Inference with Multiple Decoding Heads

The newest development within the subject of Synthetic Intelligence (AI), i.e., Massive Language Fashions (LLMs), has demonstrated some nice enchancment in language manufacturing. With mannequin sizes reaching billions of parameters, these fashions are moving into each area, starting from healthcare and finance to schooling.

Although these fashions have proven superb capabilities, the event of the mannequin’s measurement has led to an elevated inference latency, which poses an issue for real-world functions. Reminiscence-bound operations symbolize the principle bottleneck in LLM inference, as it’s inefficient to move all mannequin parameters from Excessive Bandwidth Reminiscence (HBM) to the accelerator’s cache throughout auto-regressive decoding.

Researchers have been placing in efforts to discover a answer to those limitations, one among which is to lower the variety of decoding steps and enhance the arithmetic depth of the decoding course of. Utilizing a smaller draft mannequin for speculative decoding, which produces a sequence of tokens which are then improved upon by the larger unique mannequin, has been prompt. Nonetheless, there are difficulties with incorporating a draft mannequin right into a distributed system.

To beat these challenges, a workforce of researchers in a latest examine has introduced MEDUSA, an environment friendly strategy that enhances LLM inference by incorporating extra decoding heads to foretell a number of subsequent tokens in parallel. It makes use of the spine mannequin’s quite a few decoding heads to hurry up inference. These heads overcome the difficulties of speculative decoding by concurrently predicting quite a few tokens. 

MEDUSA doesn’t require a separate draft mannequin like speculative decoding requires, which makes it able to getting simply built-in into present LLM programs, even in dispersed conditions. The workforce has shared that MEDUSA builds a number of candidate continuations in every decoding part and verifies them concurrently utilizing a tree-based consideration mechanism. By using parallel processing, MEDUSA lowers the variety of vital decoding steps whereas introducing little or no overhead when it comes to single-step latency.

Two new insights have been added to MEDUSA. First, quite a few candidate continuations have been generated utilizing MEDUSA heads, and so they have been verified concurrently. Secondly, an acceptance process has been used to decide on appropriate candidates. The workforce has shared the rejection sampling technique utilized in speculative decoding, which a temperature-based threshold can successfully substitute to deal with deviations.

The examine has prompt two strategies for fine-tuning LLMs’ predictive MEDUSA heads, that are as follows.

MEDUSA-1: This enables lossless inference acceleration by instantly fine-tuning MEDUSA on prime of a frozen spine LLM. MEDUSA-1 has been prompt for use when incorporating MEDUSA into an current mannequin or in settings with restricted computational sources. It makes use of much less reminiscence and may be made much more environment friendly by making use of quantization strategies.

MEDUSA-2: This methodology adjusts MEDUSA and the principle LLM concurrently. Whereas it presents a better speedup and improved prediction accuracy for MEDUSA heads, it necessitates a novel coaching recipe to keep up the spine mannequin’s performance. MEDUSA-2 is suitable when sources are plentiful and permits simultaneous coaching of MEDUSA heads and the spine mannequin with out sacrificing output high quality or next-token prediction means.

The analysis has additionally prompt a number of additions to reinforce or broaden the usage of MEDUSA. These embrace a standard acceptance scheme to extend the acceptance charge with out sacrificing technology high quality and a self-distillation methodology within the absence of coaching knowledge. The workforce has shared that the analysis technique of MEDUSA included testing on fashions of various sizes and coaching protocols. The outcomes have demonstrated that MEDUSA-1 can speed up knowledge by greater than 2.2 instances with out sacrificing technology high quality. Furthermore, the acceleration is improved to 2.3-3.6× utilizing MEDUSA-2. 

Try the Paper and Github. All credit score for this analysis goes to the researchers of this mission. Additionally, don’t neglect to observe us on Twitter. Be a part of our 36k+ ML SubReddit, 41k+ Fb Group, Discord Channel, and LinkedIn Group.

In the event you like our work, you’ll love our publication..

Don’t Neglect to affix our Telegram Channel