How We Built Deployxa's Metrics Pipeline
Deployxa shows real-time CPU, memory, and traffic stats for your containers in the dashboard, so you can see how your app is performing at a glance. Building a metrics pipeline that collects, stores, and displays metrics for thousands of containers in real time is a significant engineering challenge. Here is how we built it, the architecture decisions we made, and the lessons we learned.
The direct answer is that Deployxa's metrics pipeline has four components: the metrics collector (which runs on each host and collects container metrics), the metrics aggregator (which receives metrics from all hosts and stores them in a time-series database), the query service (which serves metric queries from the dashboard and the MCP server), and the dashboard (which displays the metrics in real-time charts). The pipeline handles thousands of containers with 10-second granularity, which means you see your metrics within 10 seconds of them being generated. For more on real-time systems, see our article on how we built the logging pipeline.
The Architecture
The metrics pipeline has four main components:
1. The metrics collector
The metrics collector runs on each host and collects resource metrics (CPU usage, memory usage, network I/O, disk I/O) for each container at 10-second intervals. It uses cgroups (Linux control groups) to read the container's resource usage, which is accurate and efficient. The collector sends the metrics to the aggregator via gRPC.
2. The metrics aggregator
The metrics aggregator is a service that receives metrics from all hosts and stores them in a time-series database. We use Prometheus (for storage and querying) and Thanos (for long-term storage and global querying). The aggregator also computes derived metrics (e.g., request rate, error rate) from the raw metrics.
3. The query service
The query service is a service that serves metric queries from the dashboard and the MCP server. It translates high-level queries (e.g., "get CPU usage for app X for the past hour") into Prometheus queries and returns the results as JSON. The query service also handles aggregation (e.g., "get average CPU usage across all containers for app X") and downsampling (e.g., "downsample to 1-minute granularity for the past 24 hours").
4. The dashboard
The dashboard is a React app that displays the metrics in real-time charts. It polls the query service every 10 seconds (for real-time metrics) and uses a charting library (e.g., Chart.js or Recharts) to render the charts. The dashboard supports multiple time ranges (e.g., last 15 minutes, last hour, last 24 hours) and multiple metrics (e.g., CPU, memory, request count, error rate).
Step-by-Step: How a Metric Is Collected and Displayed
Here is how a metric (e.g., CPU usage) is collected and displayed in the dashboard.
Step 1: The collector reads the container's cgroup
The metrics collector reads the container's cgroup data (e.g., /sys/fs/cgroup/cpu/cpu_usage for CPU usage) at 10-second intervals. The cgroup data is accurate (it is the same data the Linux kernel uses for resource accounting) and efficient (it is a simple file read).
Step 2: The collector sends the metric to the aggregator
The collector sends the metric to the aggregator via gRPC. The metric includes the container ID, the app ID, the metric name (e.g., cpu_usage), the value (e.g., 75 percent), and the timestamp.
Step 3: The aggregator stores the metric in Prometheus
The aggregator receives the metric and stores it in Prometheus. Prometheus is a time-series database that is optimized for metrics storage and querying. The metric is stored with its labels (e.g., container_id, app_id), which allows for efficient querying.
Step 4: The dashboard polls the query service
The dashboard polls the query service every 10 seconds for the latest metrics. The query is: "get CPU usage for app X for the past 15 minutes, downsampled to 10-second granularity."
Step 5: The query service queries Prometheus
The query service translates the high-level query into a Prometheus query (e.g., avg(cpu_usage{app_id="X"}) by (time) with a 10-second step) and executes it.
Step 6: The dashboard displays the metric
The query service returns the results as JSON, and the dashboard renders them in a real-time chart. The chart updates every 10 seconds, showing the latest CPU usage.
Common Pitfalls and Troubleshooting
The first pitfall is cardinality explosion. If you store metrics with high-cardinality labels (e.g., user_id, request_id), the time-series database can explode in size, which degrades performance. The fix is to use low-cardinality labels (e.g., app_id, container_id) and to aggregate high-cardinality data before storing it. The second pitfall is storage cost. Storing metrics at 10-second granularity for months or years is expensive, which means you need a retention policy (e.g., keep 10-second data for 7 days, downsample to 1-minute data for 30 days, downsample to 1-hour data for 1 year). The fix is to use Thanos (or a similar tool) that supports automatic downsampling and retention. The third pitfall is query performance. Querying a large amount of metric data can be slow, which degrades the dashboard experience. The fix is to use Prometheus's query optimization (e.g., use recording rules for common queries) and to cache query results. The fourth pitfall is missing metrics. If the collector fails (e.g., due to a host issue), the metrics for that host are missing, which can cause gaps in the charts. The fix is to monitor the collector's health and to alert if metrics are missing. The fifth pitfall is metric accuracy. cgroup data is accurate for CPU and memory, but it might not capture the full picture (e.g., it does not capture the container's response time, which requires application-level instrumentation). The fix is to use a combination of system metrics (from cgroups) and application metrics (from OpenTelemetry instrumentation).
Performance: Handling Thousands of Containers
Handling metrics for thousands of containers requires careful attention to performance. The first optimization is batching. Instead of sending each metric individually, the collector batches metrics and sends them in a single gRPC call, which reduces the overhead per metric. The second optimization is compression. Metric batches can be compressed (e.g., with gzip or Snappy), which reduces the bandwidth usage. The third optimization is downsampling. For long time ranges (e.g., 30 days), the query service downsamples the data (e.g., from 10-second to 1-minute granularity), which reduces the amount of data to transfer and render. The fourth optimization is caching. The query service caches common queries (e.g., "CPU usage for app X for the past hour"), which reduces the load on Prometheus. The fifth optimization is parallelism. The query service can execute multiple queries in parallel (e.g., for the dashboard that shows multiple metrics), which reduces the total query time. For more on performance, see our article on the cost optimization engine.
How the Metrics Pipeline Integrates with the MCP Server
The metrics pipeline is exposed via the Deployxa MCP server as the deployxa_get_metrics tool. This means your AI assistant (in Cursor or Claude Desktop) can call the tool and receive metrics directly. For example, you can say "show me the CPU usage for my app for the past hour", and your AI assistant calls deployxa_get_metrics, receives the metrics, and displays them in the chat (or analyzes them to make recommendations). This is useful for monitoring and debugging, because you can inspect metrics without leaving your editor. For more on the MCP server, see our article on giving Cursor cloud superpowers.
Lessons Learned
Building the metrics pipeline taught us several lessons. First, cardinality is the enemy. High-cardinality labels can explode the storage and degrade the performance, which means you need to be careful about which labels you use. Second, retention is expensive. Storing metrics at high granularity for long periods is expensive, which means you need a downsampling and retention strategy. Third, query performance matters. A slow query service makes the dashboard feel slow, which means you need to optimize queries (e.g., use recording rules, cache results). Fourth, accuracy is important. Metrics need to be accurate (e.g., cgroup data for CPU and memory) and comprehensive (e.g., application-level metrics for response time and error rate). Fifth, testing is essential. The metrics pipeline is a complex system, which means it needs to be tested thoroughly (unit tests, integration tests, load tests). For more on testing, see our article on the testing void.
Advanced Metrics Patterns
Beyond the basics, metrics pipelines benefit from several advanced patterns. The first is custom metrics. In addition to system metrics (CPU, memory), you can collect custom metrics (e.g., request count, response time, error rate, business metrics like active users), which gives you more insight into your app's performance. Use OpenTelemetry to instrument your app and collect custom metrics. The second is alerting. Set up alerts on metric thresholds (e.g., "alert if error rate exceeds 1 percent", "alert if response time exceeds 1 second"), which lets you catch issues before they affect users. The third is dashboards. Build custom dashboards (e.g., via Grafana) that visualize your metrics, which makes it easier to spot trends and anomalies. The fourth is anomaly detection. Use anomaly detection algorithms (e.g., statistical outliers, machine learning models) to identify unusual metric patterns, which catches issues that threshold-based alerting misses. The fifth is SLO monitoring. Set Service Level Objectives (SLOs) for your app (e.g., "99.9 percent of requests should succeed", "95 percent of requests should complete in under 500ms") and monitor them, which ensures your app meets its reliability targets. For more on metrics, see our articles on the cost optimization engine and debugging from your IDE.
When Custom Metrics Are Not Needed
Custom metrics are not always needed. For simple apps (a single container, low traffic), the default system metrics (CPU, memory) are sufficient, and custom metrics add complexity without providing significant benefit. For hobby projects, the Deployxa dashboard's default metrics are sufficient, and custom instrumentation is overkill. For apps that do not need detailed performance monitoring (e.g., a static site), the default metrics are fine. For these apps, the default metrics are sufficient. The key is to match the metrics to the app's needs: for complex, high-traffic apps, custom metrics are valuable; for simple apps, the default is fine. For more on metrics, see our articles on the auto-scaling architecture and how we built the logging pipeline.
Conclusion: A Real-Time Metrics Pipeline at Scale
Deployxa's metrics pipeline collects, stores, and displays real-time metrics for thousands of containers, with 10-second granularity. Building it required careful attention to architecture (batching, downsampling, caching), performance (compression, parallelism, query optimization), and reliability (monitoring, alerting, testing). For more on Deployxa's engineering, see our articles on the auto-scaling architecture and how we built the logging pipeline. Learn about the container networking model and how we handle container isolation in our companion articles. Explore our free developer tools to speed up your workflow. Try Deployxa Drop for an instant live preview with zero signup.