Deploying Hugging Face Models to Production Without MLOps Teams

Hugging Face Inference Endpoints are appealing for an obvious reason: you point the platform at a model card and get a production API in minutes. No Kubernetes, no CUDA version management. Autoscaling and basic observability come included. The supported runtimes span vLLM, SGLang, llama.cpp, TGI, TEI, and custom containers. For a small team, that breadth of choice within a single managed surface saves real time.

The scale-to-zero behavior is where the platform starts asking something of you. After 15 minutes of inactivity, an endpoint drops to zero replicas. The next request returns a 502 while a replica boots, and there is no built-in queue. Retry logic lives entirely on the client. For internal async workloads or batch jobs, that is fine. For a synchronous user-facing product, that cold path is a liability you will feel in user behavior before any dashboard surfaces it.

Pricing communicates the tradeoff clearly. As of mid-2026, an A100 80GB runs at $2.50 per hour and an H100 at $4.50 per hour on the AWS tier. Serverless Inference Providers route through partners like Together, Fireworks, and Replicate with a convenience markup over direct provider pricing. Neither model is wrong; they serve different stages of a product's life. The economics invert somewhere around 50 million output tokens per month, and teams consistently miss that inflection point until the bill arrives.

Three things reliably push teams off Inference Endpoints: volume economics that favor dedicated infrastructure, cold-start latency that synchronous products cannot tolerate, and data residency requirements that prohibit shared managed infrastructure. On that last point: HF Endpoints carry SOC 2 Type 2, BAA, GDPR DPA, and VPC-accessible offline configurations at the Enterprise tier. A surprising number of teams rule the platform out on security grounds without ever checking whether their actual requirements are satisfied there.

Choosing a Serving Runtime: TGI Is in Maintenance Mode, vLLM Is the Default, SGLang and TensorRT-LLM Have Specific Niches

On December 11, 2025, Hugging Face announced that TGI had entered maintenance mode. Bug fixes and documentation updates only; no new features. Teams already running TGI are not broken, but new deployments should avoid building on it. The migration path is low-friction because TGI exposes an OpenAI-compatible endpoint at /v1/chat/completions. Swapping the base URL is the full migration.

vLLM is the practical default for new deployments. Its PagedAttention architecture delivers higher throughput and better GPU utilization under high-concurrency workloads than TGI, per a November 2025 arXiv analysis. The community is the largest of any open serving runtime, which means more production references and faster resolution when something goes sideways. Multi-LoRA serving via S-LoRA lets you run dozens of fine-tuned variants on a single GPU pool, which matters considerably for teams shipping per-customer adapters. The ecosystem signal here is not subtle: Inferact, formed by core vLLM maintainers, raised a $150 million seed round at an $800 million valuation backed by a16z and Lightspeed. Institutional conviction at that scale does not guarantee a roadmap, but it does suggest sustained development for the foreseeable future.

SGLang earns serious consideration when time to first token is the primary constraint. PremAI benchmarks from 2026 on H100 hardware show SGLang delivering roughly 16,200 tokens per second versus vLLM's 12,500 on smaller models. That delta narrows substantially at 70B scale, but SGLang's p95 TTFT consistently runs lower than vLLM across concurrency levels tested. If you are building a real-time chat interface where sub-50 millisecond first-token latency matters at scale, that difference is not academic.

TensorRT-LLM is the fastest option at every concurrency level once compiled, running meaningfully faster than vLLM at 50 concurrent requests. The cost is a lengthy compile run per model version; subsequent starts reuse the saved engine considerably faster. That tradeoff makes sense when throughput-per-dollar is the primary constraint, the model version is stable, and you have already exhausted vLLM tuning. Most teams reach for TensorRT-LLM before they have hit the ceiling that would justify it.

How to Think About GPU Hardware Selection Before You Pick a Platform

GPU compute typically represents 40 to 60 percent of an AI startup's technical budget in the first two years, and serving infrastructure now accounts for roughly 80 percent of AI infrastructure spend versus about 20 percent for training, per GPUnex's 2026 data. That ratio has a direct implication most teams have not fully internalized: optimizing inference hardware has far more budget impact than optimizing training.

Utilization compounds the problem. Cast AI's 2026 State of Kubernetes Optimization Report found average GPU utilization across tens of thousands of Kubernetes clusters sitting around 5 percent. Teams are either provisioning for peak and never scaling down, or scaling to zero and absorbing cold-starts on every re-activation. The right platform choice eliminates the waste structurally.

Hardware selection should follow model size, not instinct or availability. For 7B to 13B parameter models, an L4 or A10G is often sufficient; defaulting to H100 because it is available is exactly how GPU budgets evaporate before anyone notices. For the 30B to 70B range, an A100 80GB is the practical minimum for reasonable throughput. Models above 70B, or long-context deployments, require H100 or multi-GPU configurations with tensor parallelism. GCP on-demand H100 rates have hovered around $3.00 per GPU per hour as of April 2026, which is a useful baseline when evaluating the markup embedded in managed platform pricing.

Inference cost per token has fallen considerably over the past three years as hardware efficiency and software optimization have compounded. Total spend keeps rising anyway, because usage scales faster than unit cost drops. Model token volume growth trajectories, not just current monthly cost, are what should drive platform commitments.

The Platform Decision at Each Stage of Traffic and Compliance Maturity

Early stage: low traffic, no compliance requirements, fast iteration as the priority. HF Inference Endpoints or Modal both serve this stage well. Modal's Python-first SDK with pay-per-second billing and idle shutoff keeps overhead close to zero. L4 GPU access on Modal runs roughly 60 percent cheaper than HF Dedicated Endpoints at equivalent throughput. If the model is likely to change weekly and operational overhead would genuinely slow the iteration cycle, one of these two is the right call.

The managed API crossover point arrives when traffic grows, latency expectations solidify, and cost becomes a line item someone is actively watching. That threshold sits around 50 million output tokens per month: below it, managed per-token APIs win on simplicity; above it, dedicated GPU infrastructure is almost always cheaper. Fireworks AI benchmarks suggest meaningful savings versus HF Dedicated Endpoints at equivalent volume. Together AI offers batch inference at half the cost alongside dedicated H100 and H200 endpoints. Baseten is the option for teams that need deployment tooling, including rollouts, versioning, and custom packaging, without managing their own cluster.

Compliance requirements create a hard forcing function that overrides everything else. HIPAA, SOC 2 with audit expectations, data residency clauses, or enterprise contracts requiring VPC isolation push teams off shared-tenant managed platforms. Together AI supports HIPAA with encryption in transit and at rest, audit logging, and strict BAAs. But for most teams hitting genuine enterprise compliance requirements, the real constraint is simpler: data cannot leave their own cloud account. A PaaS layer that deploys into your own AWS, GCP, or Azure account resolves this without requiring a platform engineering team to build the whole stack from scratch.

RunPod is the self-serve cost-optimization play for teams whose HF bill is growing and who have genuine capacity to operate their own serving stack. It is a graduation point, not a starting point.

Autoscaling and Cold-Start Behavior as a Product Decision, Not Just an Ops Detail

A 502 on the first request after 15 minutes of idle is not an ops metric. It is a user experience failure. Teams that treat cold-start as an infrastructure problem to be dealt with later are the ones who ship it to users and then spend two sprints diagnosing something that was preventable at configuration time.

Scale-to-zero is correct for async workloads, batch jobs, and internal tools with infrequent usage. It is the wrong choice for any synchronous user-facing API. A minimum of one replica eliminates cold-start at the cost of roughly one GPU-hour per hour of availability; once traffic is consistent, this is the right default for production user-facing endpoints. Predictive or scheduled scaling makes sense when traffic patterns are knowable: business-hours spikes, daily batch windows, predictable product usage curves. It reduces waste without sacrificing availability, but only for teams that actually have the traffic data to act on.

Four configuration decisions should be made deliberately, regardless of platform. Set minimum replicas based on your latency SLA, not cost intuition; these are different inputs and conflating them is expensive in ways that are hard to trace. Build a health check endpoint that reflects model readiness, not just container startup, because those two events do not coincide. Implement client-side retry with exponential backoff on any platform that can cold-start. And address request queuing explicitly: most managed platforms omit it, and the decision about whether to add it at the application layer needs to happen before deployment, not during an incident postmortem at 2am.

Packaging a Hugging Face Model for a Repeatable Production Deployment

The core packaging requirement is a container image that includes the model weights, the serving runtime, and a deterministic startup sequence. Not a script that downloads weights at container boot. Not a manual step someone runs on deployment day. A reproducible artifact that moves through the build pipeline the same way every time, on every deployment, without anyone having to remember the incantation.

Two paths get you there. BentoML is framework-agnostic across PyTorch, TensorFlow, and Transformers, and it packages the model, dependencies, and API layer into a single deployable artifact. It suits teams that want a structured packaging workflow without writing Dockerfiles from scratch. A custom Dockerfile with vLLM is the alternative: more control, more portable, and the right choice for teams that already have a container build pipeline and just need the serving layer standardized.

Weight storage strategy depends on model size. For smaller models, baking weights into the image is viable. For larger models, storing weights in object storage and loading at startup while caching on the node keeps images portable and build times reasonable. Either way, pin the model revision hash from the HF Hub rather than pulling the latest version. "Latest" is not a reproducible reference, and production deployments require deterministic artifacts.

Environment reproducibility is where teams accumulate silent debt. CUDA version, torch version, and transformers version all interact and can drift across environments without any obvious warning. Pin all three explicitly. Test the packaged container locally before pushing to any platform. The majority of deployment failures are packaging failures, not platform failures, and a local test catches them at the cheapest possible moment.

The CI/CD hook is where the discipline either exists or it doesn't. The container build should be the artifact that moves through staging to production, not a re-run of a deployment script, not a manual step initiated on release day. If someone has to do something by hand to ship a model, that step is a future incident waiting for a bad week.

Observability Minimums That a Small Team Can Actually Maintain

The anti-pattern is instrumenting everything and watching nothing, which is extremely common when teams adopt enterprise observability setups without the staff to triage the alerts those setups generate. The result is noise that desensitizes the team to real signal. Minimal instrumentation done well beats comprehensive instrumentation that nobody trusts.

Request latency at p50 and p95 tells you two different things. The p95 figure is the user-experience metric; it shows what the worst-case experience looks like for a meaningful portion of your traffic. The p50 tells you whether the median is genuinely healthy or whether you are averaging a good half against a broken half and the aggregate looks acceptable. You need both to make any responsible decision about scaling or capacity.

Time to first token matters especially for streaming endpoints. Degradation in TTFT is invisible to throughput metrics because the request will eventually complete successfully. If TTFT is climbing and you are not measuring it, users will know before you do.

Error rate by status code, not aggregate error rate. 502s indicate cold-start failures or unhealthy replicas. 503s indicate capacity exhaustion. 429s indicate rate limiting. Each tells you something different about where the system is failing, and conflating them into a single error rate delays diagnosis in ways that feel genuinely maddening at 2am.

GPU memory utilization as a leading indicator. Memory pressure almost always precedes an out-of-memory crash. If utilization is trending toward capacity, you have time to act. If you are not watching it, the first signal is a crashed replica and a very unpleasant conversation with whoever is on call.

These four, collected and surfaced to whoever is responsible for the system, cover the vast majority of inference production failures a small team will encounter. Add more instrumentation when the team has the capacity to act on it.