A totally offline use of Whisper ASR and LLaMA-2 GPT Mannequin
These days, no person might be stunned by operating a deep studying mannequin within the cloud. However the state of affairs will be far more difficult within the edge or client gadget world. There are a number of causes for that. First, the usage of cloud APIs requires gadgets to all the time be on-line. This isn’t an issue for an online service however could be a dealbreaker for the gadget that must be purposeful with out Web entry. Second, cloud APIs price cash, and clients doubtless won’t be completely satisfied to pay one more subscription price. Final however not least, after a number of years, the undertaking could also be completed, API endpoints might be shut down, and the costly {hardware} will flip right into a brick. Which is of course not pleasant for purchasers, the ecosystem, and the atmosphere. That’s why I’m satisfied that the end-user {hardware} ought to be absolutely purposeful offline, with out additional prices or utilizing the net APIs (effectively, it may be elective however not necessary).
On this article, I’ll present how you can run a LLaMA GPT mannequin and computerized speech recognition (ASR) on a Raspberry Pi. That may enable us to ask Raspberry Pi questions and get solutions. And as promised, all this can work absolutely offline.
Let’s get into it!
The code introduced on this article is meant to work on the Raspberry Pi. However many of the strategies (besides the “show” half) may even work on a Home windows, OSX, or Linux laptop computer. So, these readers who don’t have a Raspberry Pi can simply check the code with none issues.
{Hardware}
For this undertaking, I might be utilizing a Raspberry Pi 4. It’s a single-board pc operating Linux; it’s small and requires solely 5V DC energy with out followers and lively cooling:
A more recent 2023 mannequin, the Raspberry Pi 5, ought to be even higher; based on benchmarks, it’s virtually 2x quicker. However it’s also virtually 50% costlier, and for our check, the mannequin 4 is nice sufficient.
