Self-hosting a small language model on a VPS in 2026
Which VPS tier can actually run a local LLM, what fits in 8 GB of RAM, why CPU inference still makes sense, and the privacy argument for hosting your own model.
Self-hosting a language model is the most common reason people upgrade from a $6 box to a dedicated server. The hardware math is less intimidating than it looks — quantised models run happily on modest CPUs — and the privacy argument is stronger than most people expect, because an inference server you control never logs your prompts to a third party.
The hardware reality
Inference is memory-bound. The rule of thumb: a quantised model needs roughly the size of its quantisation in RAM, plus overhead. A 3B model at 4-bit runs comfortably in about 3–4 GB; a 7B at 4-bit wants 6–8 GB; larger models start needing dedicated RAM. You do not need a GPU to get started — CPU inference on a modern Xeon, Ryzen or EPYC core is entirely usable for chat, summarisation and coding help.
Matching a model to a plan
Note · The jump from VPS to dedicated matters here: 64 GB of ECC RAM is what turns "a small model" into "a real model you can actually use".
| Plan | RAM | Model that fits (4-bit) | Good for |
|---|---|---|---|
| Wisp | 1 GB | Nothing useful — too small | Not suitable for inference |
| Pulse | 2 GB | Tiny quantised models, ~1B | Light classification, not real chat |
| Umbra | 4 GB | 3B-class models | Private chat, summarisation, embeddings |
| Nocturne | 8 GB | 7–8B-class models | Serious local assistant, code help |
| R1 dedicated | 64 GB ECC | 30B+ quantised, multiple models | Full open-weight model, high concurrency |
The software stack
A practical first step: pick Umbra or Nocturne, install Ollama, pull a 3B or 7B quantised model, and bind it to localhost behind your own reverse proxy. You now have inference that no third party logs.
- ▹Ollama or llama.cpp — the two most common ways to run quantised open-weight models locally.
- ▹Open WebUI — a browser interface over a local model, if you want a chat UI rather than a CLI.
- ▹Quantisation via GGUF or GPTQ — this is what makes a big model fit in small RAM at the cost of some precision.
Why host it yourself
Every prompt you send to a hosted AI service is data that leaves your machine and lands on someone else's infrastructure, subject to their retention and training policies. A self-hosted model on a VPS you control changes the default: the prompts never leave your server, and there is no API key or vendor to leak. For sensitive documents, client code, or just general dislike of sending context to a third party, that is a real and immediate benefit.
Where a VPS falls short
Be honest about the ceiling. There is no GPU on these plans, so inference is CPU-bound and slower than a local-GPU or hosted-API experience. For interactive chat at human conversation speed on a large model, a VPS is not competitive with a GPU box. Where it does shine: always-on availability, batch and scripted workloads, summarisation and extraction pipelines, and any time the privacy benefit outweighs raw tokens-per-second.
Recommendation
- ▹Curious, small models, chat and summaries → Umbra with Ollama and a 3B model.
- ▹A daily-driver local assistant or code help → Nocturne with a 7–8B model.
- ▹Full open-weight models, batch work, or many users → a dedicated R1/R2 with 64–128 GB of ECC RAM.
FAQ
QDo I need a GPU to run a local LLM?
ANo. Quantised models run acceptably on CPU; a GPU mostly buys you speed, not the ability to run at all.
QHow much RAM do I need for a 7B model?
AAt 4-bit quantisation, roughly 6–8 GB — which is why Nocturne at 8 GB is the natural tier for it.
QIs a self-hosted model private?
AYes, if you control the server. Prompts never leave your VPS; there is no third-party API key, log or training policy involved.