How to Run Large Language Models with Small GPUs

Quick Navigation

Difficulty: Intermediate
Estimated Time: 10-20 minutes
Prerequisites: basic-linux-command-line, nvidia-gpu-with-cuda-drivers, familiarity-with-llm-concepts, python-and-pip-installed

The problem

Large language models (LLMs) have become increasingly powerful, but their size often poses challenges for deployment, especially on hardware with limited resources. LLMs with over 8 billion parameters usually require GPUs with more than 8GB of VRAM. However, it's possible to perform inference with LLMs on smaller GPUs by employing specific strategies and optimizations.

Challenge

High Memory Requirements: Large models demand significant VRAM for both model weights and data processing.

Solutions

1. Scale Resources with Multiple Graphics Cards

Single Machine Multi-GPU Setup

Sharding Method:

Shard the model by splitting it into smaller parts and distributing these across multiple GPUs. Each GPU handles only a portion of the model, which reduces the memory requirement per GPU. This approach involves careful data transfer management between GPUs to minimize communication overhead.

Implementation Example with Ollama:

Ollama provides support for utilizing multiple GPUs.

Check that you have multi GPUs installed

$ nvidia-smi
Tue Apr 4 11:17:31 2023
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 470.161.03 Driver Version: 470.161.03 CUDA Version: 11.4 |
|-------------------------------+----------------------+----------------------+
...

1-Install ollama:

curl -fsSL https://ollama.com/install.sh | sh

2-Activate env vars

You can activate environment variables to manage multi-client requests:

OLLAMA_NUM_PARALLEL=4 # Number of requests per model
OLLAMA_MAX_LOADED_MODELS=4 # Number of model instances

Run your model

ollama run llama3.1:70b

For more information, refer to the Ollama GitHub repository.

curl http://5.104.100.192:11434/api/chat -d '{
"model": "llama3.1",
"messages": [
{ "role": "user", "content": "why is the sky blue?" }
]
}'

Distributed Multi-GPU Setup Across Multiple Machines

Networked GPU Clusters:

Implement a networked setup where multiple machines, each with their own GPUs, collaborate to process model shards. This can significantly increase computational capacity and enable larger model handling.

GitHub - run-ai/genv: GPU environment and cluster management with LLM support

GPU environment and cluster management with LLM support - run-ai/genv

github.com

sudo apt install python3.10-venv
python3 -m venv genv
source genv/bin/activate

pip install genv[admin]
eval "$(genv shell --init)"

genv llm serve llama2 --gpus 1
genv llm attach llama2

GitHub - Juice-Labs/Juice-Labs: Juice Community Version Public Release

Juice Community Version Public Release. Contribute to Juice-Labs/Juice-Labs development by creating an account on…

github.com

you can use llama cpp project with rpc

https://github.com/ggerganov/llama.cpp/tree/master/examples/rpc