llama.cpp on Linux - maximum performance without overhead

1. Dependencies

sudo apt install -y build-essential cmake git
# Optional CUDA
sudo apt install -y nvidia-cuda-toolkit

2. Build

git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
cmake -B build -DGGML_CUDA=ON
cmake --build build -j

3. Download a GGUF model

Use `huggingface-cli download` to fetch variants such as `Qwen2.5-7B-Instruct-GGUF` in Q4_K_M format.

4. Serve it as OpenAI-compatible

./build/bin/llama-server \
  -m ./models/qwen2.5-7b-instruct-q4_k_m.gguf \
  --port 8080 --host 0.0.0.0 -c 8192 -ngl 99

Performance tips

• •`-ngl 99` pushes all layers to the GPU
• •`-c` controls context size; watch VRAM carefully
• •`--flash-attn` speeds up long prompts