Infrastructure as Code Drift Detection: How to Monitor and Prevent Config Drift
Infrastructure drift occurs when the actual state of your infrastructure diverges from the state defined in your infrastructure as code. A developer manually changes a security group rule through the AWS console, a team member adds a DNS record outside of Terraform, or an automated process modifies a resource in a way that is not reflected in your IaC state files. Over time, these unmanaged changes accumulate, creating a growing gap between what you think your infrastructure looks like and what it actually looks like.
Drift is dangerous because it undermines the trust you have in your infrastructure as code. If your Terraform state does not accurately represent reality, you cannot safely plan changes, roll back deployments, or respond to incidents with confidence. This guide covers how to detect drift, prevent it, and remediate it when it occurs.
What Causes Infrastructure Drift
Infrastructure drift has several common causes. Manual changes through cloud consoles are the most frequent source. A developer troubleshoots an issue by modifying a resource directly, intending to commit the change to IaC later but forgetting to do so. Automated processes that modify resources outside of IaC are another source, including auto-scaling groups that create new instances, certificates that auto-renew with different parameters, and monitoring systems that add alerts through APIs.
Team collaboration issues also contribute to drift. Multiple team members manage infrastructure through different tools, and without a single source of truth, changes made through one tool are not reflected in others. Emergency changes made during incidents often bypass IaC processes entirely, creating drift that is only discovered later during planned changes.
Terraform Drift Detection
Terraform provides built-in drift detection through the terraform plan command. When you run terraform plan, Terraform compares the desired state in your configuration with the actual state of your infrastructure and reports any differences. This is the most straightforward way to detect drift, but it requires running manually or on a scheduled basis.
For automated drift detection, you can integrate terraform plan into your CI/CD pipeline to run on a schedule, such as daily or weekly. When drift is detected, the pipeline can create a ticket, send an alert, or automatically remediate the drift by running terraform apply.
Third-party tools like driftctl, also known as Driftctl, extend drift detection beyond what Terraform manages. They scan your cloud account for resources that are not managed by any IaC tool, helping you identify shadow IT and unmanaged resources.
CloudFormation Drift Detection
AWS CloudFormation provides native drift detection through the drift detection API. You can detect drift on individual stacks or on all stacks in a region. The CloudFormation console shows which resources in a stack have drifted, what the expected and actual values are, and when the drift was first detected.
Unlike Terraform, which requires running plan to detect drift, CloudFormation drift detection is a read-only operation that can run on a schedule without risk of modifying resources. This makes it easier to integrate into automated monitoring workflows.
Configuration Drift vs Configuration Compliance
Configuration drift refers to differences between your IaC and your actual infrastructure. Configuration compliance refers to whether your infrastructure meets security and organizational policies. A resource might not have drifted from its IaC definition but still be non-compliant because the IaC definition itself violates a policy.
Both drift detection and compliance checking are important. Drift detection ensures your IaC is accurate. Compliance checking ensures your IaC meets your standards. Together, they provide comprehensive infrastructure governance.
Preventing Infrastructure Drift
Prevention is more effective than detection. Strategies for preventing drift include making IaC the only way to modify infrastructure, removing console access or requiring approval for manual changes. Implementing break-glass procedures for emergencies that include automatic IaC updates. Using CI/CD pipelines that validate infrastructure state on every change. And educating teams about the importance of IaC discipline.
Tag-based governance helps by allowing you to identify resources that are not managed by IaC. Require all resources to have an IaC-managed tag, and set up automated scanning to flag untagged resources. The Deployxa Deployment Readiness Checker validates project configuration before deployment, catching inconsistencies that could lead to drift.
Automated Remediation
When drift is detected, the response should be automated. For non-critical drift, automatically apply the IaC state to bring the infrastructure back in line. For critical drift that could cause outages, alert the team and require manual review before remediation. For drift caused by approved processes, update the IaC state to reflect the actual state.
Automated remediation requires confidence in your IaC state and testing. Before automatically applying changes, run terraform plan to preview the impact, validate that the changes do not break dependent resources, and notify the team about what will change.
Infrastructure drift is an invisible problem that compounds over time. Detecting and preventing drift is essential for maintaining the reliability and security of your infrastructure. Use the Deployxa CI/CD Pipeline Builder to set up automated pipelines that include drift detection and remediation as part of your deployment workflow.