ds4 + pi — Local LLM Harness

Prerequisites

Hardware requirements

ds4 (DwarfStar) supports multiple backends: Metal on Apple Silicon, CUDA on NVIDIA (DGX Spark), and ROCm on AMD (Strix Halo). The model requires significant memory to run efficiently.

Apple Silicon (Metal)

Primary target. Uses Apple's Metal Shading Language for GPU acceleration. Mac-native, no Docker, no CUDA.

RAM: 128 GB for full q2 (~81 GB model); 96 GB minimum with SSD streaming
Storage: 100 GB+ free SSD for weights, KV cache, and streaming expert cache
Chip: M3 Max, M3 Ultra, M4 Max, or M4 Ultra recommended
Supported: M1/M2/M3/M4 series (performance varies)

64 GB machines can run q3 quantization (~63 GB) at reduced reasoning quality, or use SSD streaming for q2.

NVIDIA CUDA / DGX Spark

CUDA backend with special care for the DGX Spark (GB10). Runs the full q2 model at 13+ t/s.

Device: DGX Spark / GB10, or any local NVIDIA GPU
Build: make cuda-spark (DGX Spark) or make cuda-generic
RAM: 128 GB unified on DGX Spark
Support: SSD streaming, distributed inference across multiple machines

AMD ROCm / Strix Halo

ROCm backend for AMD GPUs — supports the Framework Desktop and other Strix Halo systems with unified RAM design.

Device: Strix Halo (Framework Desktop, similar AMD unified RAM systems)
RAM: 128 GB+ unified, runs q2 quantization
Build: ROCm branch — git clone https://github.com/antirez/ds4 -b rocm
Support: SSD streaming, MTP speculative decoding, fused GPU ops

Step-by-Step

Setting up your local harness

From zero to a running agent in about 10 minutes.

Get the model weights

Clone the DwarfStar repository and download your chosen quantization. Multiple variants are available — imatrix-tuned quants are recommended for best quality.

terminalbash

# Clone DwarfStar and download your quantization
git clone https://github.com/antirez/ds4.c
cd ds4.c

# q2-imatrix (recommended) — 96/128 GB RAM machines, imatrix-tuned
./download_model.sh q2-imatrix

# q2-q4-imatrix — q2 with last 6 layers boosted to q4 for better quality
./download_model.sh q2-q4-imatrix

# q4-imatrix — for 256 GB+ machines, highest quality
./download_model.sh q4-imatrix

# PRO q2 — for 512 GB machines (DeepSeek V4 PRO)
./download_model.sh pro-q2-imatrix

Rationale: The q2 quantization is asymmetrical — only MoE expert tensors are aggressively quantized (up/gate at IQ2_XXS, down at Q2_K). Shared experts, projections, and routing stay at full precision. The q2-q4-imatrix variant boosts the last 6 layers to Q4_K for better output quality. The download script uses curl -C - for resume support and stores files under ./gguf/. The --tensor-type flag lets you create custom per-layer boosted quants.

Build the engine

Compile the inference engine for your platform. A single C codebase maps the model via mmap and executes on GPU through Metal Shading Language, CUDA, or ROCm.

terminalbash

# macOS (Metal) — default build, produces ds4 + ds4-server
make

# Linux CUDA — DGX Spark / GB10
make cuda-spark

# Linux CUDA — generic GPU
make cuda-generic

# CPU-only diagnostics build
make cpu

Rationale: DwarfStar is not a generic GGUF runner. It's a purpose-built engine for DeepSeek V4 Flash's specific tensor layout, KV compression scheme, and MoE routing. By being narrow, it avoids the overhead of general-purpose frameworks and can optimize the GPU graph end-to-end. The build produces two binaries: ds4 for interactive chat and ds4-server for agent integration.

Start the server

Launch the OpenAI-compatible HTTP server with disk KV persistence and optional SSD streaming. This is the bridge between the model and your agent.

terminalbash

# Standard mode — model fully resident in RAM (128 GB+ machines)
./ds4-server --ctx 100000 --kv-disk-dir /tmp/ds4-kv --kv-disk-space-mb 8192

# SSD streaming mode — for 96 GB machines, experts served from SSD
./ds4-server --ctx 100000 --kv-disk-dir /tmp/ds4-kv --kv-disk-space-mb 8192 --ssd-streaming

Rationale: --ctx 100000 gives 100k token context. --kv-disk-dir enables disk-backed KV cache: checkpoints written to SSD as <sha1>.kv files, keyed by token hash. Agent sessions survive server restarts. --ssd-streaming keeps non-routed weights resident while serving routed MoE experts from the GGUF file on cache miss — this lets the q2 model run on 96 GB machines where it wouldn't otherwise fit.

Configure Pi to use ds4

Add ds4 as a provider in Pi's model registry. This tells Pi where to find your local model and how to talk to it.

~/.pi/agent/models.jsonjson

{
  "providers": {
    "ds4": {
      "name": "ds4 local",
      "baseUrl": "http://127.0.0.1:8000/v1",
      "api": "openai-completions",
      "apiKey": "dsv4-local",
      "compat": {
        "supportsStore": false,
        "supportsDeveloperRole": false,
        "supportsReasoningEffort": true,
        "supportsUsageInStreaming": true,
        "maxTokensField": "max_tokens",
        "supportsStrictMode": false,
        "thinkingFormat": "deepseek",
        "requiresReasoningContentOnAssistantMessages": true
      },
      "models": [
        {
          "id": "deepseek-v4-flash",
          "name": "DeepSeek V4 Flash (ds4 local)",
          "reasoning": true,
          "thinkingLevelMap": {
            "off": null,
            "minimal": "low",
            "low": "low",
            "medium": "medium",
            "high": "high",
            "xhigh": "xhigh"
          },
          "input": ["text"],
          "contextWindow": 100000,
          "maxTokens": 384000,
          "cost": {
            "input": 0,
            "output": 0,
            "cacheRead": 0,
            "cacheWrite": 0
          }
        }
      ]
    }
  }
}

Rationale: The compat block is critical — it tells Pi that this provider speaks OpenAI-compatible chat completions, supports reasoning effort (thinking modes), and uses DeepSeek's native thinking format. The thinkingLevelMap maps Pi's abstract levels (off/minimal/low/medium/high/xhigh) to ds4-server's reasoning_effort values. Setting cost to zero means Pi won't warn about token budgets — because local inference is free.

Make it the default

Set ds4 as Pi's default provider and model so every session uses your local harness automatically.

~/.pi/agent/settings.jsonjson

{
  "defaultProvider": "ds4",
  "defaultModel": "deepseek-v4-flash"
}

Rationale: Once set, every pi command — pi "refactor this module", pi "explain this bug", pi "write tests for this" — routes through your local ds4-server. No cloud dependency. No API key management. No rate limits.

Verify the connection

Run a quick smoke test to confirm Pi can reach your local model and generate a response.

terminalbash

# Smoke test — Pi will print the model name and respond
pi "Hello from the local harness. What model are you running?"

Rationale: Pi prints the model name and provider in its output. If you see ds4 local and a response, the harness is working. The first request will include a cold prefill (the server renders and processes your prompt), but subsequent requests with overlapping prefixes reuse the cached KV state — no re-prefill.

Architecture Deep Dive

How the harness works

Understanding the moving parts helps you tune and debug your setup. Here's what happens under the hood.

Data flows from Pi (your agent) through ds4-server's HTTP API to the Metal GPU. KV checkpoints are persisted to SSD.

Single Live Session

ds4-server keeps exactly one mutable GPU graph + KV checkpoint in memory. Concurrent requests queue behind a single worker. This isn't a limitation — it's the design: agent tools call the model sequentially anyway, and the single session means the KV cache is always hot for the active conversation.

Disk KV Cache

When a new session replaces the live one, the old checkpoint is written to SSD as a .kv file. The key is a SHA1 of the exact token IDs. On the next request with the same prefix, the server loads the cached checkpoint and resumes from the exact token — logits, attention state, and all.

SSD Streaming

Routed MoE experts are served from the GGUF file on SSD instead of staying resident in RAM. Non-routed weights, KV cache, and a hot expert cache stay in memory. Cache misses load from SSD at ~6 GB/s — fast enough for tolerable decode. This turns RAM from a hard cutoff into a continuous speed spectrum: 96 GB machines can run the full q2 model.

Tool Calling

ds4-server renders OpenAI tool schemas into DeepSeek's DSML format, and maps generated DSML tool calls back to OpenAI tool calls. The exact-DSML replay map stores verbatim sampled text for each tool call ID, so client JSON history can be re-rendered byte-for-byte. The server now recovers tool calls started inside unclosed thinking blocks — the model sometimes opens a DSML stanza without closing its reasoning first; the server force-feeds a response marker and lets the model continue cleanly.

Mixed-Precision Experts

Supports per-layer boosted quants (e.g., IQ2_XXS/Q2_K everywhere with a few layers upcast to Q4_K via --tensor-type). The q2-q4-imatrix variant uses this: last 6 layers at Q4_K for better output quality. SSD streaming handles non-uniform quant layers automatically — boosted layers bypass the expert cache and use the mapped-view path.

Official Vector Validation

Every engine change is validated against official DeepSeek API logprobs at multiple context sizes. The test vectors capture greedy continuations with top_logprobs from the hosted API. Local --dump-logprobs output must match token-by-token. Includes long-context story recall regression and MTP speculative verify correctness tests.

Dimension	Local (ds4 + pi)	Cloud API
Privacy	✔ Your code never leaves your machine	✖ Prompts & code uploaded to remote servers
Cost	✔ One-time hardware, $0/token	✖ Recurring per-token billing
Speed	✔ 26–36 t/s on M3 Max/Ultra	✔ Fast, but adds latency
Context Window	✔ Up to 1M tokens (configurable)	✔ Usually 128k–1M
Model Quality	✔ Same 284B frontier model	✔ Same model (may be newer)
Availability	✔ Always on, no outages	✖ Rate limits, outages, deprecations
Thinking	✔ Native chain-of-thought, scales with complexity	✔ Supported (but costs extra)
Tool Calling	✔ Full OpenAI/Anthropic tool format	✔ Native tool support
Setup Effort	✖ One-time download + config (~10 min)	✔ Just an API key
Hardware Required	✖ Mac with 128 GB+ RAM	✔ Nothing — runs on their servers

Live Example

A complete agent session

Here's what a real harness session looks like — from starting the server to having Pi refactor code using your local model.

terminal — full harness sessionbash

# ── 1. Start the server (terminal 1) ──
./ds4-server --ctx 100000 --kv-disk-dir /tmp/ds4-kv --kv-disk-space-mb 8192

# ── 2. Configure Pi (done once) ──
# Edit ~/.pi/agent/models.json and settings.json as shown above

# ── 3. Use Pi with your local model (terminal 2) ──
pi "Read the current project structure and suggest a refactor."

# Pi will:
#   1. Read files using its built-in tools
#   2. Send the conversation to ds4-server at http://127.0.0.1:8000
#   3. ds4-server renders the prompt, runs inference on Metal
#   4. Streams the response back with tool call suggestions
#   5. Pi executes the tools (edit files, run commands)
#   6. Sends the results back to the model for the next turn
# All locally, all private, all free.

# ── 4. Check the KV cache (debugging) ──
ls /tmp/ds4-kv/          # See .kv checkpoint files
hexdump -C /tmp/ds4-kv/*.kv | head 20  # Inspect cached prompt text

This is the key insight: the agent (Pi) and the model (ds4) are separate processes communicating over HTTP. Pi doesn't know or care that the model is running locally — it just sees an OpenAI-compatible API. ds4-server doesn't know about Pi's tool system — it just generates tokens and maps tool calls. The harness is the glue between them.

— Loose coupling, local execution

Performance Tuning

Getting the most out of it

Tips from running DwarfStar + Pi in production for real agent work.

Match context to RAM

With 128 GB RAM and q2 quant (~81 GB), you have ~47 GB left for KV cache. At 1M tokens the compressed cache uses ~26 GB. Use --ctx 100000 to --ctx 300000 for a comfortable margin. On 96 GB machines with SSD streaming, limit to --ctx 32768 to leave room for the expert cache.

Enable disk KV always

Always use --kv-disk-dir. Agent clients resend the full conversation every request — without disk cache, every new session re-prefills from scratch. The cache directory is disposable: stop the server, rm -rf /tmp/ds4-kv, restart.

SSD streaming tuning

With --ssd-streaming, the server keeps non-routed weights and a hot expert cache in memory. Cache misses load from the GGUF file on SSD. For best performance, place the GGUF on your fastest internal SSD. The expert cache automatically evicts cold entries — no manual tuning needed, but larger --kv-disk-space-mb helps if KV cache pressure forces evictions.

Use trace for debugging

Start the server with --trace /tmp/ds4-trace.txt. The trace log shows rendered prompts, cache decisions, generated text, and tool-parser events. It's the single best debugging tool for understanding what's happening under the hood.

Mixed-precision tuning

Use --tensor-type to create custom per-layer boosted quants. The q2-q4-imatrix variant boosts the last 6 layers to Q4_K for better output quality. You can target any layer range: ./download_model.sh q2-q4-imatrix for the ready-made variant, or pass --tensor-type to quantize_model.sh for custom layer boosts.

Thinking mode tradeoffs

Thinking mode produces better results for complex tasks but uses more tokens. For simple edits, use /nothink or set reasoning_effort: "off". For architecture or debugging, use reasoning_effort: "high". The model's thinking length adapts to the problem.

Help

FAQ & Troubleshooting

Common issues and solutions.

Model fails to load with OOM (out of memory) error

You need 128 GB of RAM free when the model loads. Check Activity Monitor — close other apps, especially browser tabs and Docker. If you have 96 GB, use --ssd-streaming to run the q2 model — experts are served from SSD instead of RAM. If you have 64 GB, use q3 quantization: ./download_model.sh q3. You can also reduce --ctx to free up KV cache memory: ./ds4-server --ctx 32768 ...

First response is slow (>30s) but subsequent ones are fast

This is expected cold-start behavior. The first request does a full prefill — the server renders your full prompt and processes it token-by-token. Use --trace /tmp/ds4-trace.txt to see the prefill phase. After the cold start, ds4-server saves a checkpoint and subsequent requests reuse it. For interactive work, keep the server running rather than restarting between sessions.

Pi reports a different model name or goes to cloud API

Verify your config files are correct: ~/.pi/agent/models.json should have ds4 as a provider, and ~/.pi/agent/settings.json should set defaultProvider to "ds4". Run pi --debug "hello" to see which API endpoint Pi is hitting. Check that ds4-server is running (curl http://127.0.0.1:8000/v1/models).

KV cache isn't being reused — every restart re-prefills

The cache is keyed by the exact token IDs of your prompt. If the system prompt or conversation format changed slightly (different whitespace, newlines, or formatting), the token stream hash changes and no cache hit occurs. Run with --trace /tmp/ds4-trace.txt and look for cache HIT or cache MISS entries. Ensure Pi is sending the same prompt format each time.

Throughput is low (<10 t/s) on M3 Max/Ultra

Metal throttling can occur in certain conditions. Try: (1) closing other GPU-heavy apps, (2) ensuring the model file is on a fast SSD (not network storage), (3) running sudo mdutil -a -i off to disable Spotlight indexing during inference. For sustained workloads, consider raising --kv-disk-space-mb to keep more cache on SSD.

Tool calls aren't working — model says "I'll use a tool" but nothing happens

ds4-server uses DSML format internally and maps to OpenAI tool calls. Make sure your ~/.pi/agent/models.json has "thinkingFormat": "deepseek" and

"requiresReasoningContentOnAssistantMessages":
                        true

in the compat block. Also run the server with --trace to see the tool call mapping in action. Recent versions recover tool calls started inside unclosed thinking blocks — if the model opens a DSML stanza mid-reasoning, the server force-feeds a response marker to let it continue cleanly.

How do I upgrade to a newer model or quantization?

Download the new quantization: ./download_model.sh q2-imatrix (recommended), q2-q4-imatrix (boosted), q3, q4, or q5 for higher quality if you have more RAM. Stop the server, swap the file, restart. Your Pi config stays the same — just restart ds4-server with the new model file.

SSD streaming — how does it work on different platforms?

SSD streaming keeps non-routed weights resident while serving routed MoE experts from the GGUF file on cache miss. Works on all backends: Metal (macOS), CUDA (DGX Spark), and ROCm (Strix Halo). The expert cache automatically evicts cold entries. On ROCm, mixed-precision streaming handles non-uniform quant layers by bypassing the cache and using the mapped-view path. For best performance, place the GGUF on your fastest internal SSD.

How do I run on AMD ROCm (Strix Halo) or NVIDIA CUDA?

For CUDA (DGX Spark/GB10): make cuda-spark or make cuda-generic. For ROCm: clone the rocm branch — git clone https://github.com/antirez/ds4 -b rocm. Both backends support SSD streaming, distributed inference, and MTP speculative decoding. The ROCm backend uses fused GPU ops for better throughput. See the Hardware section for detailed requirements.

How was this page generated?

This page was built by Pi, an AI coding agent running locally with the ds4 harness. Pi read the ds4.c repository's git history and documentation, analyzed the key features and changes, then wrote the entire HTML page — structure, CSS, SVG diagrams, and FAQ content — using its file editing tools. The process was iterative: Pi read source files, planned the content, wrote the markup, and refined the design through multiple edit cycles. No cloud AI was involved — the model (DeepSeek V4 Flash) ran entirely on local hardware through ds4-server.

How do I use ds4 directly (without Pi)?

./ds4 opens an interactive REPL with the model. Useful for quick experiments and debugging. Run ./ds4 --help to see all flags. For programmatic access, ./ds4-server exposes an HTTP API —

curl -X POST http://127.0.0.1:8000/v1/chat/completions
                        -d
                        '{"model":"deepseek-v4-flash","messages":[{"role":"user","content":"Hello"}],"max_tokens":256}'

Your Local
LLM Harness

Why build a local LLM harness?

Total Privacy

Zero Recurring Cost

Real Speed

Frontier Reasoning

Agent-Native

Disk KV Persistence

Hardware requirements

Apple Silicon (Metal)

NVIDIA CUDA / DGX Spark

AMD ROCm / Strix Halo

Setting up your local harness

Get the model weights

Build the engine

Start the server

Configure Pi to use ds4

Make it the default

Verify the connection

How the harness works

Single Live Session

Disk KV Cache

SSD Streaming

Tool Calling

Mixed-Precision Experts

Official Vector Validation

Local vs. Cloud LLMs

A complete agent session

Getting the most out of it

Match context to RAM

Enable disk KV always

SSD streaming tuning

Use trace for debugging

Mixed-precision tuning

Thinking mode tradeoffs

FAQ & Troubleshooting

Your Local LLM Harness

Why build a local LLM harness?

Total Privacy

Zero Recurring Cost

Real Speed

Frontier Reasoning

Agent-Native

Disk KV Persistence

Hardware requirements

Apple Silicon (Metal)

NVIDIA CUDA / DGX Spark

AMD ROCm / Strix Halo

Setting up your local harness

Get the model weights

Build the engine

Start the server

Configure Pi to use ds4

Make it the default

Verify the connection

How the harness works

Single Live Session

Disk KV Cache

SSD Streaming

Tool Calling

Mixed-Precision Experts

Official Vector Validation

Local vs. Cloud LLMs

A complete agent session

Getting the most out of it

Match context to RAM

Enable disk KV always

SSD streaming tuning

Use trace for debugging

Mixed-precision tuning

Thinking mode tradeoffs

FAQ & Troubleshooting

Your Local
LLM Harness