Local-first inference
A 261,888-token context on a single RTX 3090. Quantised weights, KV-cache tuning, and lazy expert streaming buy frontier-scale context on desk hardware.
Bench online NODE 01 · SELF-HOSTED
Local intelligence,
bare metal
An independent engineering lab. Language models, semantic memory, and autonomous agents — compiled for one machine, measured on the bench, and kept entirely in-house.
- Context window
- 261,888tok
- Accelerator
- 24GiB
- Nodes
- 01host
- Cloud inference
- 00calls
SEC 01 — Systems
Three subsystems, one machine
Inference, memory, and autonomy are separate instruments on the same bench — each with its own resource budget and its own failure mode.
Semantic memory graph
Four tiers of persistence: atomic fact extraction, scene synthesis, persona modelling, and recall — run on dedicated CPU models so the accelerator stays free.
Bill of materials
Autonomous engineering
Repo-scale indexing, verifiable execution loops, and skill distillation. Agents propose, run, and prove their own changes against real builds before anything ships.
Bill of materials
SEC 02 — Signal path
Follow a request through the lab.
Four stages from the web-facing agent to local inference. Select a component to inspect what enters, what happens, and what leaves.
IN / Client requestOUT / Streamed response
The reply travels back through the same chain. Tool calls can start another round.
INSPECT / 01 · ACCESS BOUNDARY
Terminate TLS. Enforce access.
freenginx accepts the connection and routes it to the intended application. Hostname rules, authentication, and network allow-lists keep internal services behind their own access boundaries.
This page is a static origin. Visiting it does not send a request to the model.
- Input
- HTTPS request from the client
- Work
- TLS termination and access checks
- Output
- Request routed to the agent application
INSPECT / 02 · EXECUTION BOUNDARY
Plan the work. Run the tools.
The agent assembles the conversation, requests a completion, and executes any tool calls. Tool results return to the conversation for the next model round; the loop ends with a final answer.
Local CLI agents enter here directly, without the web ingress hop.
- Input
- User task and conversation history
- Work
- Tool execution and agent control loop
- Output
- Model request sent through the memory proxy
INSPECT / 03 · PERSISTENCE BOUNDARY
Recall context. Capture the answer.
The memory proxy enriches the request with relevant context before forwarding it. On the return path, round-level capture stores the completed exchange rather than duplicating every intermediate tool round.
Extraction, synthesis, and embedding run on dedicated CPU services, separate from GPU inference.
- Input
- Conversation and agent identity
- Work
- Memory recall and final-answer capture
- Output
- Context-enriched request to the local model
INSPECT / 04 · COMPUTE BOUNDARY
Generate locally. Stream the reply.
llama.cpp evaluates the prompt and generates tokens on the RTX 3090, using host memory and expert streaming to fit the workload. The response returns through the proxy to the agent.
A tool-call response resumes the agent loop; a final answer completes the turn.
- Input
- Prompt, context, and tool definitions
- Work
- Prompt evaluation and token generation
- Output
- Streamed answer or structured tool calls
CPU / MEMORY WORKERS
L1 extraction · L2/L3 synthesis · ONNX embeddings
OPS / SERVICE SUPERVISION
systemd isolation · health gates · versioned rollback
SEC 03 — Bench
The whole lab is one node
Nominal specifications of the machine everything runs on. Figures are the build sheet, not a live utilisation dashboard.
| Ref | Component | Specification | Notes |
|---|---|---|---|
| U1 | NVIDIA RTX 3090 | 24 GiB GDDR6X · SM86 | KV cache, expert LRU and the CLIP projector are resident at once — the card runs deliberately near saturation. |
| U2 | Ryzen 5 5600X | 6 cores · 12 threads · Zen 3 | Six threads to inference, the remainder to memory extraction and the edge. |
| U3 | Host memory | 125 GiB DDR4 | Streams an 85 GB weight file without thrashing the page cache. |
| U4 | freenginx 1.31.4 | Source build · OpenSSL 3.5 · LTO | PCRE JIT and -march=native; one edge for every hostname. |
HTTP/2HTTP/3 readyTLS 1.3
ModSecuritygzip staticSSL preread
- Public surface1 origin
- Analyticsnone
- Third-party assetsnone
- Rollback pathalways
mail contact@dgonzo.de
Got a beautifully difficult problem?
Technical collaborations, infrastructure experiments, and long arguments about scheduling are all welcome. Replies come from a human on this machine.