Running models with llama cpp mono / cluster mode
How to run GGUF models with llama.cpp in single-node (mono) mode and across multiple nodes using RPC cluster mode.
2 minutes(207 words)complex
Quick Navigation
Difficulty: Intermediate
Estimated Time: 10-20 minutes
Prerequisites: Linux with a CUDA-capable GPU, CMake and a C++ build toolchain, Python with pip, Basic command-line experience
Mono node
Download llama cpp from : https://github.com/ggerganov/llama.cpp/releases
pip install huggingface-hub
huggingface-cli download \
bartowski/Meta-Llama-3.1-8B-Instruct-GGUF \
--local-dir downloads \
--local-dir-use-symlinks False
./build/bin/llama-cli -m downloads/Meta-Llama-3.1–8B-Instruct-Q8_0.gguf -p "You are a helpful assistant" -cnv
./build/bin/llama-server -m downloads/Meta-Llama-3.1–8B-Instruct-Q8_0.gguf — port 8080 — host 0.0.0.0
Multi nodes
on each device
Download llama cpp from : https://github.com/ggerganov/llama.cpp/releases
git clone https://github.com/ggerganov/llama.cpp.git
mkdir build-rpc-cuda
cd build-rpc-cuda
cmake .. -DGGML_CUDA=ON -DGGML_RPC=ON
cmake --build . --config Release
sudo cp/llama.cpp/build-rpc-cuda/ggml/src/libggml.so /lib
cd llama.cpp/examples/rpc/
CUDA_VISIBLE_DEVICES=0 ./rpc-server -p 50052 -H 0.0.0.0
./build/bin/llama-cli -m downloads/Meta-Llama-3.1–8B-Instruct-Q8_0.gguf -p “You are a helpful assistant” — repeat-penalty 1.0 -n 64 — rpc 127.0.0.1:50052 -ngl 99
interactive mode
./build/bin/llama-cli -m downloads/Meta-Llama-3.1–8B-Instruct-Q8_0.gguf -p “You are a helpful assistant” — repeat-penalty 1.0 -n 64 — rpc 127.0.0.1:50052 -ngl 99 -cnv
./build/bin/llama-server -m downloads/Meta-Llama-3.1–8B-Instruct-Q8_0.gguf — port 8080 — host 0.0.0.0 — rpc 127.0.0.1:50052
Test with cli
curl --request POST \
--url http://5.104.100.192:8080/completion \
--header "Content-Type: application/json" \
--data '{"prompt": "Building a website can be done in 10 simple steps:","n_predict": 128}
Access UI