The Health Check System: How Deployxa Detects Unhealthy Apps

The Health Check System: How Deployxa Detects Unhealthy Apps

A container that is technically running but not functioning correctly is worse than a container that is down, because it silently degrades the user experience without triggering any alerts. Deployxa's health check system detects unhealthy apps before they affect users, by running continuous health checks and automatically rolling back unhealthy deployments. Here is how the system works.

The direct answer is that Deployxa's health check system has three components: the startup check (which runs when a container starts, to verify it is healthy before receiving traffic), the continuous check (which runs periodically, to detect health degradation), and the deployment check (which runs after a deployment, to verify the new version is healthy). All three use the 14-point readiness engine, which checks SSL, DNS, environment variables, health endpoints, container status, resource usage, database connectivity, and log errors. For more on the readiness engine, see our article on the 14-point readiness engine.

The Three Health Check Components

1. The startup check

When a container starts, the health check system runs the 14-point readiness engine to verify the container is healthy. If the check passes (grade A or B), the container is added to the load balancer and starts receiving traffic. If the check fails, the container is not added, and the deployment is aborted (or the previous version continues to serve traffic). This ensures that an unhealthy container never receives traffic.

2. The continuous check

The health check system runs the 14-point readiness engine periodically (default: every 60 seconds) on each running container. If the check fails (grade below B) for 3 consecutive checks, the container is marked as unhealthy, and an alert is sent. If the container is part of a blue/green deployment, the system can automatically roll back to the previous version. This ensures that health degradation is detected and addressed quickly.

3. The deployment check

After a deployment, the health check system runs the 14-point readiness engine on the new container for a monitoring period (default: 5 minutes). If the check fails during the monitoring period, the system automatically rolls back to the previous version. This ensures that a deployment that passes the startup check but degrades over time is detected and rolled back.

Step-by-Step: How a Health Check Works

Here is how a health check works for a typical container.

Step 1: The health check system calls the container's health endpoint

The system sends an HTTP request to the container's /health endpoint (configurable). If the endpoint returns a 200 status code, the check passes. If it returns a non-200 status code or times out, the check fails.

Step 2: The system checks the container's resource usage

The system checks the container's CPU usage, memory usage, and disk usage. If any resource is above a threshold (e.g., memory above 90 percent), the check produces a warning. If any resource is above a critical threshold (e.g., memory above 95 percent), the check fails.

Step 3: The system checks the container's logs for errors

The system scans the container's recent logs for error patterns (e.g., Error, Exception, FATAL). If the error rate is above a threshold (e.g., more than 10 errors per minute), the check produces a warning. If the error rate is above a critical threshold (e.g., more than 100 errors per minute), the check fails.

Step 4: The system checks the database connectivity

The system checks whether the container can connect to its database (by running a simple query, e.g., SELECT 1). If the query fails, the check fails. This catches database connectivity issues before they affect users.

Step 5: The system calculates the overall grade

The system calculates the overall grade (A, B, C, D, F) based on the individual check results. If all checks pass, the grade is A. If there are warnings, the grade is B or C. If there are failures, the grade is D or F.

Step 6: The system takes action based on the grade

If the grade is A or B, no action is taken. If the grade is C, a warning is logged. If the grade is D or F for 3 consecutive checks, the container is marked as unhealthy, and (if configured) the system rolls back to the previous version.

Common Pitfalls and Troubleshooting

The first pitfall is not having a health endpoint. If your app does not have a /health endpoint, the health check system cannot verify the app is functioning correctly. The fix is to add a /health endpoint that returns a 200 status code when the app is healthy. For more on health endpoints, see our article on the monitoring gap. The second pitfall is the health endpoint not checking dependencies. If the health endpoint always returns 200 (without checking the database, for example), the health check system cannot detect dependency failures. The fix is to make the health endpoint check critical dependencies (e.g., database connection, cache connection) and return a non-200 status code if any check fails. The third pitfall is the health endpoint being too slow. If the health endpoint takes too long to respond (e.g., because it runs a slow query), the health check system might time out, which causes a false failure. The fix is to make the health endpoint fast (e.g., use a simple SELECT 1 query, not a complex query). The fourth pitfall is false positives. If the health check thresholds are too sensitive, the system might mark healthy containers as unhealthy, which causes unnecessary rollbacks. The fix is to set conservative thresholds and to require multiple consecutive failures before taking action. The fifth pitfall is not testing the health check. If the health check is not tested, it might not detect real issues. The fix is to test the health check by simulating failures (e.g., stop the database, fill up the memory) and verifying the health check detects them.

Advanced Health Check Patterns

Beyond the basics, the health check system supports several advanced patterns. The first is custom health checks. In addition to the built-in 14-point readiness engine, the system supports custom health checks (e.g., "check if the Redis connection is alive", "check if the external API is reachable"). Custom health checks are defined via a configuration file and are run alongside the built-in checks. This is useful for apps with app-specific health requirements that the built-in checks do not cover.

The second pattern is health check intervals. The system supports configurable health check intervals (e.g., check every 10 seconds, 30 seconds, 60 seconds). A shorter interval detects issues faster but consumes more resources. A longer interval conserves resources but detects issues slower. The default interval (60 seconds) is a good balance for most apps.

The third pattern is health check timeouts. The system supports configurable health check timeouts (e.g., timeout after 5 seconds, 10 seconds, 30 seconds). If a health check takes longer than the timeout, it is considered failed. This prevents a slow health check from blocking the system.

The fourth pattern is health check retries. The system supports configurable health check retries (e.g., retry 3 times before failing). This prevents false failures due to transient issues (e.g., a temporary network blip). The default retry count (3) is sufficient for most apps.

The fifth pattern is health check weights. The system supports configurable health check weights (e.g., the database connectivity check is weighted higher than the SSL certificate check). This means critical checks have a bigger impact on the overall grade than non-critical checks. For example, a failed database connectivity check (weight 3) drops the grade more than a failed SSL certificate check (weight 1).

How Health Checks Integrate with the Auto-Scaling System

Health checks and the auto-scaling system work together to ensure your app is both healthy and scalable. The auto-scaling system uses health check results to determine whether to scale up or down. If a container is unhealthy (grade below B), the auto-scaling system does not route traffic to it and may restart it. If a container is healthy (grade A or B), the auto-scaling system routes traffic to it and may scale up (if traffic is high) or scale down (if traffic is low). For more on auto-scaling, see our article on the auto-scaling architecture.

Lessons Learned

Building the health check system taught us several lessons. First, the health endpoint is the most important check. Without a health endpoint, the system cannot verify the app is functioning correctly, which means issues go undetected. The fix is to always implement a /health endpoint that checks critical dependencies (e.g., database, cache) and returns a 200 status code only if all checks pass. Second, the health endpoint needs to be fast. If the health endpoint is slow (e.g., because it runs a slow query), the health check system might time out, which causes a false failure. The fix is to make the health endpoint fast (e.g., use SELECT 1 instead of a complex query). Third, false positives are worse than false negatives. A false positive (marking a healthy container as unhealthy) causes unnecessary restarts and rollbacks, which disrupts service. A false negative (marking an unhealthy container as healthy) allows issues to persist, but at least the service is not disrupted. The fix is to use conservative thresholds and to require multiple consecutive failures before taking action. Fourth, the health check system needs to be observable. Without visibility into the health check results, you cannot diagnose issues. The fix is to display the health check results in the dashboard and to expose them via the MCP server. Fifth, the health check system needs to be tested. If the health check system is not tested, it might not detect real issues. The fix is to test the health check system by simulating failures (e.g., stop the database, fill up the memory) and verifying the system detects them. For more on Deployxa's engineering, see our articles on the 14-point readiness engine and how we handle container restarts.

Conclusion: Detect Unhealthy Apps Before Users Do

Deployxa's health check system detects unhealthy apps before they affect users, by running continuous health checks and automatically rolling back unhealthy deployments. The three components (startup check, continuous check, deployment check) work together to ensure that your app is always healthy, even during deployments. For more on Deployxa's engineering, see our articles on the build cache architecture and how we handle container restarts. Learn about how we handle custom domains and the audit log system in our companion articles. Explore our free developer tools to speed up your workflow. Try Deployxa Drop for an instant live preview with zero signup.