The Infra Engineer's Decision Matrix for AI-Assisted Internal App Platforms That Never Leave Your VPC

The VPC boundary is now a procurement gate for AI-assisted internal app platforms. For regulated or security-sensitive teams, the decision rests on four linked criteria, which are data residency, agent operability, compliance surface, and infrastructure ownership. A platform is fit for that boundary only when the organization can govern what runs inside it and independently prove what stays there.

What "data residency" actually requires from a platform, not just from your cloud account

Prompts, logs, tool-call payloads, and model outputs have their own paths through the stack, and those paths can cross a vendor’s control plane. The region setting is not the whole answer. A platform that stores or queues payloads, even briefly, brings that processing into audit scope. Stateless proxying leaves no payload behind on vendor infrastructure, but it still needs scrutiny when the requirement prohibits any payload from crossing the perimeter.

“Self-hosted” obscures several different arrangements. In Tier A, workloads run in a customer-owned VPC while the vendor manages the control plane, leaving dependencies on vendor infrastructure for updates and licensing. In Tier B, the whole platform runs on infrastructure that the customer operates, and the customer owns the runtime, keys, and audit trail. Tier C has no network connectivity to vendor infrastructure. Model weights and dependencies then arrive through physical media or isolated transfer, which makes air-gapped operation a distinct operational commitment.

These distinctions affect which guarantees a platform can support. GDPR’s territorial scope depends on the controller’s or processor’s EU establishment and/or the location of data subjects. HIPAA requires signed BAAs with every party handling PHI, regardless of geography. Location alone cannot settle both questions. SOC 2 network security controls and DORA’s Article 28 contractual-guarantee requirements also call for documented evidence, including audit logs that the customer controls.

Does the orchestration layer see prompts, tool-call payloads, and outputs, or only metadata such as which agent, which skill, and which model class? A private runtime does not answer that question. The payload-versus-metadata distinction exposes whether the platform’s own architecture supports the residency claim made for the cloud account.

Why AI agents inside a VPC require a different operability model than human-facing tooling

Residency defines where processing may happen, but agents change what can happen there. An agent can access data, create artifacts, escalate privileges, and trigger downstream actions without a person guiding it at every step. That expands the blast radius of a perimeter failure. It also means a platform designed around human interaction needs more than an agent interface to govern autonomous execution.

The access model must account for agents querying systems a person would not think to query in a single session. Generated code, configuration, and documents also need an inspection point before entering the pipeline on their path to deployment. When agents create infrastructure or change IAM policies, the platform must set clear limits on what they can do through least-privilege scoping. Provenance must show which model, version, tool call, and output were behind each action. Otherwise, incident response and audit lack the trace needed to reconstruct execution.

OpenHands illustrates a human-oversight model built for agentic software development. Its workflows include PR review, CI failure remediation, security scanning, and incident triage, while human oversight is concentrated at the review, approval, and merge step. That leaves the platform with work to do. It must control what agents do before a person reaches the approval point. Humanitec takes a different approach by positioning its orchestrator as the governed layer for developers and agents, wrapping existing Terraform and OpenTofu modules.

Agent actions can use the same audit and access-control surface as human actions or create a parallel, less-governed path. SBOMs, artifact signing, and dependency provenance remain baseline expectations when agents generate and merge code inside the perimeter.

How the compliance surface expands when AI is in scope for audit

AI-in-scope audits extend the evidence required from internal tooling. Organizations must document what data AI accessed, what decisions it influenced, who authorized deployment, and what human oversight existed. Those records depend on how the infrastructure is configured as well as how the application behaves. An approval step has limited audit value if the organization cannot connect it to the agent activity under review.

SOC 2 and HIPAA impose different requirements on that evidence and its handling. SOC 2 is voluntary and done because customers ask for it, and it covers network security controls and access management without formal rules that require BAAs or PHI-specific handling. HIPAA is mandatory for platforms touching PHI and adds signed BAAs throughout the data chain and specific breach-notification obligations. Neither framework subsumes the other. A healthcare SOC 2 Type 2 effort usually takes months of work to get ready, followed by a long observation period, so the architecture should be settled before the audit clock starts.

Compliance automation can cut the work of collecting evidence, but it cannot fix the design beneath it. Drata fits teams pursuing HIPAA and SOC 2 coverage together. Vanta offers broad integrations and continuous evidence collection but does not sign BAAs because its architecture avoids processing PHI. The question for the internal app platform remains whether it produces the evidence that the compliance tool needs.

The collection layer has a residency surface too. If a compliance platform stores payload logs, it enters the audit perimeter. Metadata and evidence pointers avoid that payload-storage exposure. International obligations can broaden the required mapping further, including DORA, NIS2, and the EU AI Act where applicable. The practical consequence is additional configuration and documentation work. Whether the organization can perform that work depends on who controls the infrastructure underneath.

Infrastructure ownership as the criterion that determines whether the other three can actually be satisfied

The organization’s ability to check and maintain residency promises, agent policies, and audit trails on its own depends on who owns the infrastructure. Vendor control over the boot path, patch cycle, or key management system limits that independence, regardless of contractual assurances. A dedicated tenant is not customer-owned infrastructure. Having a closed deployment also does not let the organization audit, modify, or fork its source code.

The ownership question has three linked layers. The runtime must execute on the organization’s infrastructure, whether that is an AWS, Azure, or GCP VPC, an on-premises data center, or an air-gapped enclave. Model weights and configuration must also fit the required boundary. Inputs, outputs, prompt context, and tool-call payloads need their own residency guarantees, with orchestration limited to metadata where required.

With a model-agnostic platform, workloads can move between models without the infrastructure being rebuilt, while a single-vendor model ties that choice to the platform. Cloud API access through a security-controlled proxy is a different arrangement from running model weights inside the boundary. The proxy does not make external inference local. For a strict no-payload-egress requirement, the model path must satisfy the same test as the rest of the stack.

Ownership also creates an operating obligation. Self-hosting needs at least one platform engineer for installation and configuration, realistically two to three for production SLA requirements, plus capacity for patching, upgrades, and incident response. An under-resourced deployment can create a larger compliance burden than a well-governed managed platform with strong contractual protections. A PaaS layer that automates cluster management, CVE patching, and autoscaling can reduce that staffing burden without surrendering infrastructure ownership. The advantage depends on the team’s ability to run the resulting system.

Evaluating specific platforms against the four-criterion matrix

Dify bundles a RAG pipeline, Prompt IDE, agent runtime, and monitoring in a self-hosted Docker Compose deployment. Its model options include OpenAI, Claude, Llama, and local models through Ollama, making it relevant to teams shipping internal AI apps without a dedicated ML infrastructure team. Model choice still determines the inference boundary. Its opinionated architecture also imposes structural limits when workflows do not match its model. The bundled deployment simplifies the starting point, but does not replace evaluation of payload paths, permissions, and evidence.

LangChain and LangGraph offer a different tradeoff through MIT-licensed open source, fine-grained control over state machines, and a large integration ecosystem. They are a good fit for engineering teams that build custom, production-grade agent applications. The organization owns runtime and infrastructure maintenance. Teams that do not regularly write Python or JavaScript face a steep learning curve, while teams that do gain more direct control over how their applications work. With that control, teams also have to put in place the governance that the matrix requires.

Flowise has a low-code visual node editor built on LangChain, is quick to set up with Docker, and has an Apache 2.0 community edition. It fits prototyping and internal tools rather than removing the need for infrastructure evaluation. Workday acquired Flowise in August 2025. That ownership change makes roadmap and licensing review relevant to assessing independence, even where deployment is self-hosted. A visual workflow editor and ownership of the full operating stack are different properties.

OpenHands is relevant when agentic software development is the central workload, with governance assessed around its review, approval, and merge model. Onyx, formerly Danswer, serves a narrower purpose through self-hosted search and chat over internal documents. It is model-agnostic, with the organization wiring up the LLM, and offers free open-source software with a paid cloud option. That makes it a candidate for enterprise search. It is not a substitute for evaluating a full agentic platform when broader autonomous workflows are required.

Humanitec wraps existing infrastructure modules and offers self-hosting for air-gapped and on-premises systems, so it fits when policy enforcement needs to happen at provisioning. Port has well-developed scorecards and dependency mapping, and its roadmap puts more focus on AI agents. It is relevant to organizations adding agentic capabilities to a developer portal over existing infrastructure. Those portal capabilities do not, by themselves, establish the payload boundary or ownership model required for a strict VPC deployment.

Running the matrix as a structured evaluation process, not a checklist

The review starts with what the team can run because that determines which forms of ownership can work. Can the organization operate Tier B, or does it need Tier A with strong contractual residency guarantees? If the workload requires Tier C, vendor connectivity is already disqualifying. A managed-VPC option stays in the running only if what its control plane actually does meets the required boundary.

Next, the residency review should request architecture documentation that distinguishes payload processing from metadata handling. General claims that data stays in the VPC are not enough. The agent review should then ask for the audit log schema for agent-initiated tool calls and establish whether agents share the human access-control and audit surface. The compliance review should map applicable frameworks to the evidence the platform produces. The review of what the organization can run must include any separate collection layer it needs to build.

These steps work well in this order, but their answers must also be tested together. A residency guarantee that the organization cannot independently verify is weak evidence for audit. Agent logs that send payloads out can weaken a deployment that otherwise keeps application data local. Where the platform includes inference workloads, token volume also affects the tradeoff between self-hosted weights and proxied API calls. That cost decision belongs at selection time, subject to the residency requirement.

For a small or growth-stage team, an infrastructure layer that runs in its own cloud account and automates cluster management, patching, autoscaling, and compliance controls can change what is practical to operate. The final selection test is whether the organization can audit, patch, and, if necessary, fork or replace every component in the stack. Where it cannot, that component’s vendor remains a single point of failure in the compliance model.