Wiring a Self-Hosted Agentic Coding Environment to Your Own LLM Endpoints
Pointing an agent at your own model URL secures only one of five data paths.
Wiring a self-hosted coding agent to your own LLM endpoint is mostly configuration. The OpenAI-compatible /v1 API makes that possible. But changing the model URL controls only one of five data paths. Keeping code inside your network also means controlling where the agent runs and where its coordination, tool output, and artifacts go.
Why coding teams are redirecting agents away from cloud APIs in 2026
Three forces have converged in 2026. Open-weight coding models have narrowed the quality gap with closed models, inference servers have adopted a common API format, and regulated teams face restrictions on sending source code or PHI to outside inference services. Teams can now separate where an agent runs from who processes its prompts.
Cost adds pressure, but it is not the whole argument. Some teams need internal inference regardless of whether GPUs beat API prices. Others can keep everyday work local and use outside models for harder tasks. The architectural hinge is the shared API format, which lets teams redirect model requests without rebuilding the agent.
The five data-flow paths a "self-hosted" agent actually moves data through
AQ’s self-hosting guide lays out five paths that agent traffic takes. The first is host execution, including the checkout, shell, build tools, credentials, and network access. This is what “self-hosted” usually describes in vendor documentation. It is also the highest-stakes path. Whatever the host can reach, the agent can reach.
The second path is the model request. It includes the system prompt, conversation history, files the agent read, command output, tool definitions, and the reply. A local agent can still send all of this outside. The third path is hosted coordination, including session state, transcripts, orchestration metadata, telemetry, and update checks. A plain CLI on a VM has less of this than a platform with a hosted dashboard.
The fourth path is tool and MCP output. File contents, command results, and MCP responses feed back into the model request, so this path can expose secrets or internal hostnames. The fifth is artifacts, including screenshots, recordings, logs, and PR text uploaded for dashboards or pull requests. These paths need separate controls. The configuration that follows starts with path two, because the endpoint URL controls only model traffic.
How the OpenAI-compatible API became the universal interchange format
“OpenAI-compatible” means a server accepts the API request format an agent already uses, allowing the destination to change without rewriting the client. Ollama exposes that interface at localhost:11434/v1. vLLM, LM Studio, and llama.cpp also offer compatible endpoints. The URL becomes the point of interchange. The server still needs to support the specific API the agent expects.
Claude Code illustrates that distinction. Ollama added its native Anthropic-compatible /v1/messages endpoint in v0.14.0 in January 2026, and vLLM documents a Claude Code setup. Claude Code uses ANTHROPIC_BASE_URL, while Codex CLI requires wire_api = "responses" for custom providers.
Bifrost provides a gateway between agents and supported providers, including vLLM and Ollama. It ships as a single binary for in-VPC deployment on AWS, GCP, and Azure. In its documented air-gapped mode, it loads pricing data from local files. Teams get one internal API surface, and their sensitive requests stay on internal runtimes.
Choosing the agent harness: what each platform keeps local and what it sends out
A plain Linux VM with tmux and SSH is the simplest baseline to audit. Claude Code and Codex can run headless, while tmux keeps the session alive after a connection drops. There is little coordination machinery to inspect. For a single developer, Goose offers another simple arrangement by embedding llama.cpp directly, without a separate inference server.
For teams needing all five paths internal, Coder Agents runs its control plane, orchestration, and execution in the customer’s VPC, on-premises, or fully air-gapped. It became generally available September 9, 2026. Tabby is built primarily for code completion, with agentic capabilities layered on, and fits air-gapped completion workloads. AQ addresses shared work through dedicated-machine workspaces with isolated worktrees and shared terminal, editor, and preview access.
Where endpoint flexibility is the main need, OpenCode supports local models and OpenAI-compatible servers. Cline, Continue, Zed, and Crush also accept a compatible base URL, though their telemetry and update behavior still need auditing. Codex has built-in provider IDs for Ollama and LM Studio. The right harness is the one whose non-model paths meet the team’s residency requirements.
Picking the open-weight model to serve at your endpoint
Quantization comes before benchmark scores because tool calling can break when precision drops too far. Memory capacity is the other hard limit. The supplied material gives no numeric quantization floor, so there is no defensible universal bit-width recommendation here. The model needs to pass a test of how reliably it calls tools in the agent loop it will use. A model that fits but fails that test is not usable.
Qwen 3 Coder has an Apache 2.0 license and comes in a wide range of parameter sizes to fit different hardware budgets. Kimi K2.6, released in April 2026, has open weights and a very long context window. DeepSeek V4-Flash and Xiaomi’s MiMo-V2.6-Pro are further open-weight candidates named in the brief. Kimi and DeepSeek also offer API access when local capacity is insufficient.
The shortlist is only a starting point. The harness can change a model’s benchmark results, so teams need to test it with the tools and stack they plan to use. The routing pattern in the notes uses self-hosted Qwen or Kimi for routine traffic, with Claude Opus 4.7 or Gemini 3.1 Pro handling harder tasks through APIs. That escalation is available only where the data may leave the network.
Setting up vLLM as the inference server your agent points at
vLLM combines PagedAttention with an OpenAI-compatible API, making it suitable for a shared agent endpoint. The way it serves requests gives it higher throughput than naive serving. To configure it for agent workloads, enable prefix caching and chunked prefill. Agents reuse large prompt prefixes across turns, so avoiding repeated processing is a substantial latency improvement.
vLLM’s documentation says prefix caching lets “the new query skip the computation of the shared part”. It also states that caching “only reduces the time of processing the queries (the prefilling phase) and does not reduce the time of generating new tokens (the decoding phase)”. That distinction governs the hardware choice. Caching does not make token generation free.
For production Kubernetes deployments, the setup includes startup, readiness, and liveness probes, topology spread, and GPU resource limits. vLLM uses Ray for distributed orchestration and NCCL for inter-GPU communication. Large multi-node deployments need high-bandwidth networking, with InfiniBand or RoCE recommended in the notes. Autoscaling replicas down during quiet periods reduces idle cost.
Once the server is available, the agent’s base URL points to its address. An unauthenticated server can use any non-empty OPENAI_API_KEY value where the client requires one. Teams that do not want to operate the serving cluster can consider Inferact, the managed inference layer built by vLLM’s creators, deploying on customer GPUs or its own.
Hardware sizing for the inference server
For agents, how much memory is available comes first, followed by how fast the model can generate tokens. Prompt caching removes much of the repeated prefill work, leaving decode bandwidth as the constraint on each turn. Reasoning models sharpen that trade-off because their thinking tokens also require decoding.
The September 2026 hardware snapshot includes the Radeon AI PRO R9700 with 32GB and 640 GB/s bandwidth, with a recommendation to test ROCm against Vulkan for the chosen model. The RTX 5090 offers 32GB and a straightforward CUDA setup. Mac Studio M5 Ultra configurations offer 96GB or 256GB with 1.2 TB/s bandwidth, while the M5 Max 128GB offers 614 GB/s. These workload-sizing options call for hardware-specific vLLM deployment instructions.
Cloud rental provides another route through H100 instances with 80GB on specialist providers including Lambda, CoreWeave, RunPod, and Crusoe. Utilization decides whether that rental pays off. An underloaded H100 can cost more per token than a premium API, and fuller usage often improves economics more than higher peak throughput.
The notes place a rough break-even point around 11 billion tokens per month, with high utilization required for self-hosting to win. Hardware costs in the September 2026 snapshot are point-in-time because memory prices are volatile. Contracts that commit to usage and GPU types that fit the work can change the costs.
The environment variable wiring for each major agent
OpenCode uses OPENAI_BASE_URL for the inference server address. Cline, Continue, Zed, and Crush accept an OpenAI-compatible base URL in their configuration, plus a placeholder API key where the server does not require authentication. The destination is the essential setting. If a model name is required, it must match the model the team plans to use.
Codex CLI uses the built-in provider IDs ollama or lmstudio for those runtimes. Custom providers need a base URL and wire_api = "responses". That extra setting is easy to miss.
Claude Code uses ANTHROPIC_BASE_URL with an Anthropic-format endpoint, including Ollama v0.14.0 or later and LM Studio. vLLM also documents this setup. Anthropic does not support routing Claude Code through a gateway to non-Claude models, even though local pairings work in practice.
OpenHands selects a local LLM through its provider configuration and can target a cluster-internal vLLM service on Kubernetes. Goose embeds llama.cpp, so that arrangement needs no separate server URL. Coder Agents sets the model endpoint at platform level. Tabby is configured as the inference backend in VS Code or its client.
With Bifrost, agents point to the gateway inside the VPC, which routes requests to internal runtimes or permitted external APIs. Before starting the agent, send a simple completion request directly with curl. This checks reachability and model loading before agent-layer debugging begins.
Keeping all five paths inside the network: compliance and security controls
For host execution, scope filesystem permissions to the worktree and use credentials with the minimum access required. A repository-scoped push token is narrower than an organization-wide admin token. A git worktree isolates the checkout, but permissions still need attention.
For model traffic, place inference inside the VPC and block public inference endpoints through network egress policy. For coordination, audit telemetry and update checks or use an internal control plane. For artifacts, check whether screenshots, logs, and PR text go to external storage. Each path needs its own evidence. An air-gapped configuration must account for all of them.
Tool output needs particular care because it becomes prompt content. Treat repository files, issue text, websites, dependencies, and MCP responses as untrusted input that may contain prompt injection. Keep credentials out of environment variables the agent can read and log. CVE-2026-41182 exposed LangSmith redaction failures for streaming token events, while CVE-2026-15737 exposed raw prompt and response logging in Amazon Bedrock AgentCore SDK telemetry.
Regulated deployments need controls matched to their FedRAMP High, HIPAA, PCI-DSS v4.0, or EU AI Act obligations. A vendor’s SOC 2 certification does not authorize prompt disclosure. A vendor processing covered prompt content can still require a GDPR Article 28 DPA or a HIPAA BAA. For PHI, the business-associate obligations include a 60-day breach notification duty back to the covered entity.
When self-hosting loses: the utilization and overhead argument
Self-hosting loses when the GPUs sit idle or the team cannot maintain the serving stack. The bill covers operations work, model updates, and infrastructure overhead as well as the cost to rent or buy hardware. Small teams can pay more for less convenience. A managed inference layer can reduce the operating burden, but it does not erase the utilization problem.
To improve utilization, the notes recommend consolidating internal AI workloads on a shared cluster and running background batches beside interactive sessions. Autoscaling down during quiet periods limits waste. Even with these steps, teams need enough work for the cluster to make its cost worthwhile.
For many teams, the practical resolution is hybrid routing. A right-sized internal endpoint handles routine and compliance-sensitive work, while permitted difficult requests go to a frontier API. Data that must stay local stays there even when the local model struggles. The common API format makes that split a configuration choice rather than a commitment to own every part of inference.
