In this post, you’ll go from cloning the repo to asking your locally hosted model for a prompt completion, all running CPU-only on your Apple Silicon Mac (M1).
🧰 What You'll Need
Before we start, here’s what you need:
✅ Docker Desktop (with BuildKit and buildx support)
✅ macOS with Apple Silicon
✅ At least 16 GB RAM (some models need memory!)
✅ Terminal and curl installed
🔧 Step 1: Clone the vLLM Repository
Let’s grab the latest vLLM code from GitHub:
git clone git@github.com:vllm-project/vllm.git
cd vllm
This repo has everything you need, including Dockerfiles for CPU and GPU builds.
🏗️ Step 2: Build the vLLM CPU Image
DOCKER_BUILDKIT=1 docker buildx build \ 130 ↵
--platform=linux/arm64 \
-f docker/Dockerfile.cpu \
--target=vllm-openai \
-t vllm-cpu-arm64 \
--load \
.
🧠 Step 3: Start the Inference Server
Let’s fire up the server with a small model (facebook/opt-125m) to keep RAM usage light:
docker run -it --rm \
-p 8000:8000 \
-e VLLM_CPU_OMP_THREADS_BIND=nobind \
vllm-cpu-arm64 facebook/opt-125m
💡 Why VLLM_CPU_DISABLE_AFFINITY=1?
On macOS, Docker doesn’t expose NUMA or low-level CPU affinity. This env var disables thread pinning to prevent runtime errors like AssertionError: Not enough allowed NUMA nodes to bind threads of 1 CPUWorkers. Allowed NUMA nodes are []. Please try to bind threads manually.]
🌐 Step 4: Send a Completion Request via curl
Let’s ask the model to complete a prompt:
curl http://localhost:8000/v1/completions \
-H "Content-Type: application/json" \
-d '{
"model": "facebook/opt-125m",
"prompt": "What are three healthy snacks I can keep at my desk? ",
"max_tokens": 1000,
"temperature": 0.9,
"top_p": 0.95
}'
The model replies with its best attempt.
{"id":"cmpl-b180a957fc581c3e","object":"text_completion","created":1770101699,"model":"facebook/opt-125m","choices":[{"index":0,"text":" The items below are snacks, so I’m assuming snack #1. I need some fruits, vegetables, and nuts. The ones I’ve had for a while.\nTin foil, peanut butter, and peanut butter and orange juice (I found them at some local shop you go to) Also some dried fruit\nNothing too “healthy” at the moment but I’m thinking of making some almond flour from this lol\nI'd rather use the butter, but I do have avocado so I'm probably gonna make some peanut butter too\nIt's definitely worth it! It's hard to find almond flour, but I’m sure you can find it there","logprobs":null,"finish_reason":"stop","stop_reason":null,"token_ids":null,"prompt_logprobs":null,"prompt_token_ids":null}],"service_tier":null,"system_fingerprint":null,"usage":{"prompt_tokens":14,"total_tokens":160,"completion_tokens":146,"prompt_tokens_details":null},"kv_transfer_params":null}
and server side log.
🧪 Tips, Tricks, and Troubleshooting
🐢 It's slow!
Yes, CPUs aren't great at real-time inference for large models. Try lighter models like:
facebook/opt-125m
EleutherAI/pythia-70m
🧠 Want smarter answers?
Try a bigger model like facebook/opt-1.3b, but be warned, you’ll need more RAM.
🎉 Wrapping Up
In just a few steps, you:
Built a CPU-only vLLM image tailored for Apple Silicon
Ran a local OpenAI-compatible inference server
Sent a live request and got a generated response
No GPU. No cloud. All local. 🧠💻
🚀
May the Forge be with you!
