Infrastructure Drift Detection and Remediation in Terraform

Terraform's whole premise rests on one idea: the code is the source of truth. Three layers make that promise work, or break it. There's the .tf configuration, the state file that tracks what Terraform believes exists, and the actual resources sitting in AWS, GCP, or Azure. Drift is any gap between those three. For real teams running real infrastructure under real deadlines, that gap is not a risk you manage down to zero. It's a constant, and pretending otherwise is how teams end up surprised by their own cloud bill.

Here's why it's structurally unavoidable. Someone gets paged at 2 a.m., fixes a security group directly in the console because terraform apply takes four minutes and the outage doesn't wait, then tells themselves they'll clean it up tomorrow. Tomorrow doesn't come. Meanwhile Ansible or a security tool like AWS Config reaches into a resource Terraform provisioned and changes it directly through the cloud API, no state file involved. Cloud providers do this on their own too: managed databases get patched during maintenance windows, TLS defaults get forced, encryption settings flip without anyone asking. Auto-scaling groups replace instances constantly, and none of that ever touches a Terraform run.

Then there's the quieter failure: the pipeline itself stops working. Scalr measured this directly, tracking 121 broken version control connections across a single 30-day window (79 of them fully broken on paid accounts). Nobody notices right away, because a broken pipeline doesn't send an alert. It just stops producing evidence, and the gap between code and reality widens for weeks before anyone looks.

Drift doesn't announce itself. It piles up quietly until a plan fails, an audit turns up something nobody can explain, or a cost report shows a spike with no clear cause behind it. The skill here was never preventing drift; that's a losing fight. It's catching it fast, figuring out what actually matters, and fixing it without taking down something that's working.

What drift actually costs (in reliability, compliance, and cloud spend)

Start with the size of the problem. Flexera's 2026 State of the Cloud Report put global cloud waste at 29% of IaaS and PaaS spend, up from 27% in 2025. That's the first increase in five years, and the report ties the reversal largely to AI workload growth outrunning the governance meant to keep it in check.

Drift feeds that number directly, and here's the part most finance teams miss: you can't FinOps your way out of a problem your tooling can't see. A resource gets created outside Terraform, or an instance type gets bumped up manually during an incident and never scaled back down, and there's no record of it anywhere in the codebase. Most engineering teams already suspect they're over-provisioned by a wide margin; drift is exactly what makes that hard to measure, let alone fix, because the audit tool doesn't know the resource exists in the first place.

Reliability takes a hit too. A plan that should show zero changes suddenly proposes a dozen, because someone hand-edited a resource three weeks ago and forgot to mention it. Or worse: it works fine in staging, then breaks in production, and the root cause turns out to be a console change nobody documented back in March.

Compliance is where drift stops being an inconvenience and starts being a finding. SOC 2's CC8.1 control requires that every infrastructure change be approved, logged, and traceable back to a code commit. A manual change made outside Terraform fails that control by definition, no matter how good the intent behind it was. Teams that can't produce evidence of who approved a given infrastructure change six months back don't pass their audit. That's not a documentation gap. It's a drift problem wearing a documentation costume.

Cost pressure and audit pressure point the same direction here: detection isn't optional anymore. The next question is what your tooling actually sees, and what it doesn't.

What terraform plan catches and what it silently misses

terraform plan is the foundation, and it's a genuinely good one. It compares the state file against live cloud resources for anything Terraform already manages. Non-zero output means something's off between what's declared and what's real.

Two flags matter here specifically. terraform plan -refresh-only shows you what would change in the state file to match actual cloud state, without touching any infrastructure; apply -refresh-only is the follow-up that actually commits that update. It's a good first move when you want to see what changed before deciding what to do about it. Then there's terraform plan -detailed-exitcode, which returns exit code 2 when the diff isn't empty. Pipe that into a cron job that posts to Slack, and smaller fleets get reliable drift alerts for close to zero tooling cost.

Now the part that trips people up, and it's the single most misunderstood thing about terraform plan: it only checks resources Terraform already has in state. Anything created manually, outside Terraform entirely, with no state entry at all, is invisible to it. The tool doesn't check that resource and clear it; it never looks, full stop. That's how shadow infrastructure builds up over months without a single alert firing, and it's why teams who rely on terraform plan alone are flying blind on exactly the resources most likely to cause a problem.

The failure mode compounds this. If the job running your plans quietly stops, you don't get an error. You get nothing, which looks exactly like a clean environment until it very much isn't. That Scalr data on broken VCS connections isn't an edge case; it's a documented, recurring pattern on managed platforms that teams pay for.

None of this works reliably without remote state to begin with. Porter, a bring-your-own-cloud PaaS, handles remote state and CI/CD as part of its deployment layer for teams on AWS, GCP, or Azure. A.tfstate file sitting on one engineer's laptop is a single point of failure waiting for a bad afternoon. Remote, encrypted, versioned, locked, access-controlled state isn't a nice-to-have here. It's the floor, and any team not there yet should stop reading this and go fix that first.

Supplemental detection tools and when each fills a real gap

The blind spot native tooling leaves (unmanaged resources with no state entry) is exactly what a second layer of tooling exists to close. Tools that run scheduled comparisons against live cloud state can surface resources Terraform has never heard of.

On the open-source side, a few tools cover distinct needs, and they're not interchangeable. Driftctl is purpose-built specifically for drift detection, comparing cloud resource state directly against Terraform config, and it fits nicely as a lightweight CLI check inside CI/CD for teams that want something targeted rather than a whole platform. Terragrunt, if you're already using it as a Terraform wrapper, comes with drift detection built in and produces clean, readable plan output. Checkov takes a different angle: it's a policy-as-code scanner with over 1,000 built-in checks mapped to the CIS AWS Foundations Benchmark, SOC 2, HIPAA, and ISO 27001, and it outputs SARIF files that plug straight into GitHub Advanced Security. Custom policies in YAML or Python are supported too. Checkov earns its place wherever compliance scanning and drift detection are really the same conversation, which, for regulated teams, is most of the time.

On the managed side, the options scale by how much automation you actually want, and this is where teams tend to overreach. HCP Terraform (the product formerly called Terraform Cloud) runs automatic health checks every 24 hours after a successful run, about as low-friction as it gets if you're already in the HashiCorp ecosystem. Terraform Enterprise builds on that with configurable alerting, aimed at larger orgs with dedicated platform teams. Spacelift goes a step further and can trigger an automated terraform apply the moment it detects drift, restoring declared state on a cron schedule. That's a real capability, but wire it up to a stateful resource and you've built a machine that destroys production data on a timer. Scalr treats drift detection as a core platform feature, with reporting and remediation across both Terraform and OpenTofu, which matters for anyone managing governance across a lot of workspaces. ControlMonkey runs continuous comparison against desired state with a dashboard and one-click remediation, and env0 folds drift detection into its broader workflow automation.

Worth a mention on tooling lock-in: OpenTofu, the Linux Foundation's fork of Terraform released under MPL 2.0 after HashiCorp moved to the BSL 1.1 license in 2023, is running in production at a meaningful number of organizations by now, and most of the tools above support it. If tool lock-in is a live concern for the team, that's a real data point to weigh.

Picking between all of this comes down to a short list of real questions. How often does it check (on-demand versus continuous)? Does it catch unmanaged resources or only drift within existing state? Does it help you fix things or just tell you about them? How deep is the VCS integration, does it cover multiple clouds, and do you need policy enforcement bolted on? Tooling surfaces the drift. What you do with it is the harder part, and no tool on this list does that part for you.

Triaging detected drift before touching anything

Not every drift finding is a problem to solve. Sometimes it's the correct answer, and treating every diff as an emergency is its own kind of mistake. That 2 a.m. security group change made under incident pressure might represent exactly the state you want; automatically reverting it back to the old, broken config could take the service down all over again.

Before choosing a fix, ask a few pointed questions. Was the change intentional? Check incident logs, Slack threads, the cloud provider's own change history. Does the live state actually reflect something correct, or is it a mistake, or worse, something unauthorized? What's the blast radius if this gets reverted automatically, and is the resource stateful? Separately: does this violate a compliance requirement that has to be fixed no matter what business justification exists for it?

Sorting by risk level makes the next step obvious. Tag changes and minor config flags are low-risk and stateless; automate the fix without a second thought. Network configuration and security group changes sit in the middle; they need a human to look before anything gets applied. Database parameters, storage configuration, IAM policies: these are one-way doors. Revert carelessly and you can destroy data or lock yourself out of something you need.

The dangerous combination, and the one worth naming directly, is -auto-approve paired with a change to one of those one-way-door resources. That's the exact scenario where an automated remediation script quietly destroys a production database nobody meant to touch. If your pipeline can do that today, fix it before you read the next section.

Triage isn't a queue you work through when there's time. Every item that comes out of it should land on one of three labels: revert it, adopt it into code, or ignore it with a documented reason. No open loops.

The three remediation paths and how to execute each safely

Revert is for drift that's unauthorized, wrong, or a straight-up compliance violation. Run it through terraform apply after reading the plan output closely, not skimming it. For low-risk, stateless drift, this is safe to fully automate: the pipeline catches a non-empty diff on a scheduled plan, opens a pull request with the fix, and applies it once someone signs off. Never run -auto-approve anywhere near a stateful or one-way-door resource. That rule doesn't bend, and there's no exception worth making for the sake of convenience.

Adopt is for drift that turns out to be correct and should just become the new baseline in code. Use terraform import to bring a cloud resource that exists outside state into it, then update the .tf file so the configuration actually matches. terraform plan -refresh-only followed by apply -refresh-only updates the state file to reflect the accepted reality without changing any infrastructure. Every adopted change should go through a pull request; the import and the config update need to be reviewable and tied to a commit, same as any other change.

Ignore with justification is for drift that's legitimately owned by something other than Terraform: an auto-scaler that manages instance count, a security tool that flips an encryption flag on its own schedule. Handle it with a lifecycle { ignore_changes = [...] } block, and write down why it's there. Skip the documentation and that ignore block becomes invisible technical debt within a year, quietly masking drift that actually does matter. Review these blocks periodically. What was a reasonable exception in January can hide a real problem by December.

Direct state manipulation (terraform state mv, terraform state rm) is sometimes necessary and always risky. Back up the state file before touching it manually, every time, no exceptions.

Every one of these three paths (revert, adopt, ignore) should end in a commit. That's not just good hygiene. It's the audit trail compliance frameworks are going to ask for later.

Building drift detection into CI/CD so it runs without anyone remembering to run it

The core setup is simple, and there's no excuse for skipping it: schedule terraform plan -detailed-exitcode as a read-only job on a cron schedule, alert when it returns exit code 2, and keep it completely separate from whatever pipeline runs apply. It should never be able to trigger a change on its own.

Daily scheduled plans make sense for production. Every pull request should trigger one too, for any environment. The gap between scheduled runs is your actual detection window, so keep it tight; a weekly check on production infrastructure isn't detection, it's a very slow confirmation of damage already done.

Route the results somewhere people will actually see them. Slack or PagerDuty for immediate flags on production drift. A SIEM feed for the audit trail, since piping plan results into a SIEM strengthens the infrastructure-as-code evidence trail auditors and incident responders both rely on.

On the credentials side, drop static, long-lived keys entirely and use OIDC-based, short-lived federated credentials for both planning and apply runs. That alone closes one of the more common audit findings. Keep planning and production-apply permissions on separate roles too; whatever role can read state and run plans shouldn't also be the role that can apply changes to production. Least privilege at the CI layer, not just at the IAM layer.

Treat saved plan files like sensitive data, because they often contain credentials or resource metadata. Store and handle them under the same controls as the state file itself.

Watch the pipeline's pulse, not just its output. A dead pipeline produces zero alerts, and zero alerts looks identical to a clean environment until someone finds a resource that's been drifting for two months. Scalr's numbers on broken VCS connections are the clearest evidence available that this happens often, even on platforms teams pay to manage. For teams running workloads on a managed platform layer across AWS, GCP, or Azure, the CI/CD layer handling Terraform drift detection is a separate system from whatever runs the application itself. The cost and compliance argument for keeping both under active governance doesn't change.

Drift detection as continuous compliance evidence for SOC 2 and HIPAA audits

SOC 2's CC8.1 control, stated plainly: every infrastructure change needs approval, a log entry, and a trace back to a code commit. A manual change made outside Terraform fails that test automatically, regardless of how sound the reasoning behind it was. Drift detection is what catches that change before an auditor does, which is a much better place to find it.

There's a real difference between a Type I and a Type II audit here, and it's the difference that decides whether a manual, once-a-quarter drift check is worth anything at all. Type I is a snapshot, a single point in time. Type II asks for evidence that controls held up continuously, over a stretch of months. A drift detection pipeline running on schedule and logging its results every single day is exactly the kind of continuous evidence a Type II audit is built to check for; a spreadsheet someone updates before the audit is not.

HIPAA's requirements for state files (remote, encrypted, versioned, locked, logged, and access-controlled) happen to be the same baseline that makes drift detection work reliably in the first place. The compliance checklist and the operational checklist aren't two separate lists. They're the same one, and teams that treat them as separate end up doing the work twice.

Checkov's policy-as-code layer, with its 1,000-plus built-in policies mapped to CIS, SOC 2, HIPAA, and ISO 27001, can run right inside the plan pipeline and block a noncompliant configuration before it ever reaches apply, rather than surfacing it six months later during an audit. Teams on the HashiCorp stack have Sentinel as the equivalent policy enforcement layer.

Run this process well and the audit evidence builds itself along the way: git history showing every .tf change with an author, a reviewer, and a timestamp attached; scheduled plan logs proving infrastructure matched its declared state on any given date; pull request records showing each drift item got triaged and resolved with a reason written down. None of that requires extra work at audit time. It's just what a well-run pipeline leaves behind.