Self-Hosting Local LLMs for Private Coding: Ollama, vLLM, and llama.cpp

The Monday Morning Panic

A good friend of mine working at a fintech startup in Gurgaon called me in a panic last year. An eager junior engineer on his team had pasted a 400-line database migration script containing production schemas, payment gateway keys, and bank account table structures directly into public ChatGPT to fix a syntax error.

Their automated security proxy caught the unencrypted outbound payload immediately. The entire engineering team had to rotate every API token, spend three days auditing audit logs, and submit a security incident report to their compliance board.

If you write code for a living, you already know that AI assistants save you hours of mundane boilerplate typing every week. But pasting proprietary business logic into public cloud endpoints is an unnecessary risk. You do not need to send your code to remote servers. You can run private open-source models right on your local machine or your private office server for zero subscription fees.

Choosing Your Runtime: Ollama vs. llama.cpp vs. vLLM

Setting up local AI feels intimidating if you only read academic research papers. In practice, you only need to choose between three production engines based on what hardware you have:

The Hardware Math: Calculating Your VRAM Budget

Before you run a docker pull command, check your memory limits. An LLM requires memory proportional to its total parameter count and quantization level, plus extra room for the Key-Value (KV) context buffer:

Base VRAM = (Parameter Count in Billions * Bits Per Parameter) / 8
Total Required VRAM = Base VRAM * 1.20 (adds 20% for 16k context window)

Here is what that math means for real laptops and GPUs:

• Qwen 2.5 Coder 7B (Q4_K_M quantization): Needs ~5.5 GB VRAM. Runs smoothly on an 8GB or 16GB M1/M2/M3 MacBook or an entry-level RTX 3060 graphics card.
• Qwen 2.5 Coder 32B (Q4_K_M quantization): Needs ~22 GB VRAM. Fits on a 24GB RTX 3090, RTX 4090, or a Mac with 32GB unified memory. This model matches GPT-4 level coding accuracy for daily web tasks.
• DeepSeek-Coder-V2 (236B MoE with 21B active): Requires ~80 GB VRAM for 4-bit precision. You need a dedicated workstation with multiple GPUs to run this one.

Option 1: Solo Developer Setup with Ollama in Two Minutes

If you want a private coding copilot running on your personal laptop without tinkering with Python virtual environments, Ollama is your best choice:

# 1. Install Ollama on Linux, macOS, or Windows WSL2
curl -fsSL https://ollama.com/install.sh | sh

# 2. Download and run Qwen 2.5 Coder (7B for 8GB RAM, 32B for 32GB RAM)
ollama run qwen2.5-coder:7b

# 3. Test the OpenAI-compatible HTTP endpoint locally
curl http://localhost:11434/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen2.5-coder:7b",
    "messages": [{"role": "user", "content": "Write a TypeScript function to format Indian currency (INR)"}]
  }'

Option 2: Running a High-Concurrency Team Server with vLLM

If you have an old office PC with a 16GB or 24GB NVIDIA GPU and want five junior engineers to share the same model, Ollama will choke because it queues requests sequentially. That is where vLLM shines. It uses PagedAttention to batch multiple developer completions in parallel:

# docker-compose.yml
services:
  vllm-coder:
    image: vllm/vllm-openai:latest
    container_name: private-ai-coder
    runtime: nvidia
    ports:
      - "8000:8000"
    environment:
      - HUGGING_FACE_HUB_TOKEN=${HF_TOKEN}
    volumes:
      - ~/.cache/huggingface:/root/.cache/huggingface
    ipc: host
    command: >
      --model Qwen/Qwen2.5-Coder-32B-Instruct-AWQ
      --quantization awq
      --dtype half
      --max-model-len 16384
      --gpu-memory-utilization 0.95
      --enforce-eager
      --port 8000
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]

Start the server on your local office network:

docker compose up -d

# Inspect the container logs to verify weights are loaded
docker compose logs -f

Connecting VS Code with the Continue Extension

You do not need to leave your code editor to interact with your local server. Install the free open-source Continue extension in VS Code. Open ~/.continue/config.json and configure both tab autocomplete and the chat sidebar:

{
  "models": [
    {
      "title": "Local Qwen Coder",
      "provider": "openai",
      "model": "qwen2.5-coder:7b",
      "apiBase": "http://localhost:11434/v1",
      "apiKey": "empty-not-needed-locally"
    }
  ],
  "tabAutocompleteModel": {
    "title": "Local Autocomplete",
    "provider": "openai",
    "model": "qwen2.5-coder:7b",
    "apiBase": "http://localhost:11434/v1",
    "apiKey": "empty-not-needed-locally"
  }
}

You can check our related article on how AI code generators work under the hood to understand how editor plugins pass AST context to local models.

Verifying Zero Network Egress

If your manager or senior security lead asks for proof that source code never leaves the office perimeter, show them this simple network packet verification:

# 1. Check which local port the model process is listening on
lsof -i :11434

# 2. Capture packets originating from your workstation to verify zero external calls
sudo tcpdump -i any host not 127.0.0.1 and port not 22 -c 50

As you trigger completions in VS Code, you will see all network traffic stays pinned to 127.0.0.1. Zero outbound packets leave your network. No external company logs your code, and your corporate compliance stays spotless.

Real Performance on Normal Hardware

A lot of beginners assume you need a $10,000 server to run AI. Here is what real-world performance looks like on machines you probably already own:

• Base M1 MacBook Air (8GB RAM): Qwen 2.5 Coder 7B runs at 18 to 22 tokens per second. That is faster than you can read code.
• Desktop with NVIDIA RTX 3060 (12GB VRAM): Generates 35 to 42 tokens per second with zero CPU fan noise. Tab autocomplete feels instantaneous.
• Cloud Cost: Exactly ₹0 per month. No token quotas, no sudden rate limit errors, and zero surprise credit card deductions.

Stop paying monthly subscriptions for tools that treat your source code as training data. Download Ollama, pull a 7B model, and experience private, offline coding on your own terms.