How We Built Deployxa's Logging Pipeline
Deployxa streams real-time logs from thousands of containers to the dashboard, so developers can see their app's output as it happens. Building a logging pipeline that handles thousands of concurrent log streams, aggregates them, and delivers them to the dashboard in real time is a significant engineering challenge. Here is how we built it, the architecture decisions we made, and the lessons we learned.
The direct answer is that Deployxa's logging pipeline has four components: the log collector (which runs on each container and sends logs to the pipeline), the log aggregator (which receives logs from all containers and publishes them to Redis), the WebSocket server (which subscribes to Redis and streams logs to connected clients), and the dashboard (which displays the logs in real time). The pipeline handles thousands of concurrent log streams with sub-second latency, which means developers see their logs as they are generated. For more on real-time systems, see our article on how we built the dashboard.
The Architecture
The logging pipeline has four main components:
1. The log collector
The log collector runs on each container and captures the container's stdout and stderr output. It uses a lightweight agent (e.g., vector or fluent-bit) that reads the container's log file and sends the logs to the aggregator via HTTP or gRPC. The agent is configured to batch logs (to reduce the number of requests) and to add metadata (e.g., container ID, app ID, timestamp).
2. The log aggregator
The log aggregator is a service that receives logs from all containers and publishes them to Redis. It uses a pub/sub model, where each app has a Redis channel (e.g., logs:{app_id}), and logs are published to the channel. The aggregator also stores logs in a persistent store (e.g., Elasticsearch or Postgres) for historical queries.
3. The WebSocket server
The WebSocket server maintains WebSocket connections with the dashboard clients. When a client connects (e.g., to view logs for a specific app), the WebSocket server subscribes to the app's Redis channel and forwards log events to the client. When the client disconnects, the WebSocket server unsubscribes from the channel.
4. The dashboard
The dashboard is a React app that displays the logs in real time. It establishes a WebSocket connection to the WebSocket server, receives log events, and appends them to the log viewer. The log viewer supports search, filtering, and auto-scroll.
Step-by-Step: How a Log Event Flows
Here is how a log event flows from the container to the dashboard.
Step 1: The container generates a log
When your app calls console.log("Hello, World!"), the log is written to the container's stdout, which is captured by the log collector.
Step 2: The log collector sends the log to the aggregator
The log collector batches the log with other logs and sends the batch to the aggregator via HTTP. The batch includes the log message, the container ID, the app ID, and the timestamp.
Step 3: The aggregator publishes the log to Redis
The aggregator receives the batch, processes each log, and publishes it to the app's Redis channel (logs:{app_id}).
Step 4: The WebSocket server receives the log
The WebSocket server (which is subscribed to the Redis channel) receives the log event and forwards it to all connected clients that are viewing logs for that app.
Step 5: The dashboard displays the log
The dashboard receives the log event via the WebSocket and appends it to the log viewer. The log appears in the dashboard within sub-second latency.
Common Pitfalls and Troubleshooting
The first pitfall is log volume. A high-traffic app can generate thousands of log lines per second, which can overwhelm the pipeline. The fix is to batch logs (to reduce the number of requests), to sample logs (to reduce the volume), and to use a scalable message broker (like Redis or Kafka) that can handle high throughput. The second pitfall is log retention. Storing all logs indefinitely is expensive, which means you need a retention policy (e.g., keep logs for 30 days, then delete them). The fix is to use a time-series database (like Elasticsearch or TimescaleDB) that supports automatic retention policies. The third pitfall is log search. Searching through millions of log lines is slow without an index. The fix is to use a search engine (like Elasticsearch) that indexes logs and provides fast search. The fourth pitfall is WebSocket scaling. A single WebSocket server can handle thousands of connections, but for tens of thousands of connections, you need multiple WebSocket servers and a pub/sub system (like Redis) to distribute events. The fix is to use a Redis-backed pub/sub layer, which lets multiple WebSocket servers share events. The fifth pitfall is backpressure. If the dashboard client is slow (e.g., due to a slow network or a slow browser), the WebSocket server might queue up events, which can cause memory issues. The fix is to implement flow control (e.g., drop old events if the client is too slow) or to close the connection if the client is unresponsive.
Performance: Handling Thousands of Concurrent Log Streams
Handling thousands of concurrent log streams requires careful attention to performance. The first optimization is batching. Instead of sending each log individually, the log collector batches logs and sends them in a single request, which reduces the overhead per log. The second optimization is compression. Log batches can be compressed (e.g., with gzip), which reduces the bandwidth usage. The third optimization is sampling. For high-volume logs (e.g., debug logs from a high-traffic app), the log collector can sample (e.g., send 10 percent of logs) to reduce the volume. The fourth optimization is selective subscriptions. Instead of subscribing to all logs for all apps, the WebSocket server subscribes only to logs for apps the user is currently viewing, which reduces the number of events the server needs to forward. The fifth optimization is caching. For historical log queries (e.g., "show me the logs from 2 hours ago"), the aggregator caches the results, which reduces the load on the persistent store. For more on performance, see our article on the cost optimization engine.
How the Logging Pipeline Integrates with the MCP Server
The logging pipeline is exposed via the Deployxa MCP server as the deployxa_get_logs tool. This means your AI assistant (in Cursor or Claude Desktop) can call the tool and receive logs directly. For example, you can say "show me the last 100 lines of logs for my app", and your AI assistant calls deployxa_get_logs, receives the logs, and displays them in the chat. This is useful for debugging, because you can inspect logs without leaving your editor. For more on the MCP server, see our article on giving Cursor cloud superpowers.
Lessons Learned
Building the logging pipeline taught us several lessons. First, real-time is hard. WebSockets, pub/sub, and backpressure are all complex topics that require careful engineering. Second, log volume is higher than you expect. A single app can generate thousands of log lines per second, which means the pipeline needs to be designed for high throughput from the start. Third, log search is essential. Developers do not just want to see logs in real time; they also want to search historical logs, which means you need a search engine (like Elasticsearch) in addition to the real-time pipeline. Fourth, observability is critical. The logging pipeline itself needs to be observable (e.g., queue length, processing time, error rate), so you can identify and fix issues before they affect users. Fifth, testing is essential. The logging pipeline is a complex system, which means it needs to be tested thoroughly (unit tests, integration tests, load tests). For more on testing, see our article on the testing void.
Advanced Logging Patterns
Beyond the basics, logging pipelines benefit from several advanced patterns. The first is structured logging. Instead of plain-text logs, structured logging (e.g., JSON) makes logs easier to search, filter, and analyze, which speeds up debugging. The second is log levels. Using log levels (DEBUG, INFO, WARN, ERROR, FATAL) lets you control the verbosity of your logs, which reduces noise and focuses on important events. The third is log correlation. Adding a request ID (or trace ID) to each log entry lets you correlate logs across services, which is essential for debugging distributed systems. The fourth is log aggregation. For apps with multiple services, aggregating logs from all services into a single view (e.g., via Elasticsearch or Loki) makes it easier to search and analyze logs. The fifth is log alerting. Setting up alerts on log patterns (e.g., "alert if ERROR logs exceed 10 per minute") lets you catch issues before they affect users. For more on logging, see our articles on AI error handling failures and debugging from your IDE.
When Centralized Logging Is Not Needed
Centralized logging is not always needed. For small apps (a single container, low traffic), the container's stdout is sufficient, and centralized logging adds complexity without providing significant benefit. For hobby projects, the Deployxa dashboard's log viewer is sufficient, and a separate logging service is overkill. For apps that do not need long-term log retention (e.g., apps that only need logs for debugging, not for compliance), the default log retention is fine. For these apps, the default logging is fine. The key is to match the logging to the app's needs: for complex, high-traffic apps, centralized logging is valuable; for simple apps, the default is fine. For more on logging, see our articles on the metrics pipeline and the auto-scaling architecture.
Conclusion: A Real-Time Logging Pipeline at Scale
Deployxa's logging pipeline streams real-time logs from thousands of containers to the dashboard, with sub-second latency. Building it required careful attention to architecture (batching, pub/sub, WebSocket scaling), performance (compression, sampling, selective subscriptions), and reliability (retention, search, observability). For more on Deployxa's engineering, see our articles on how we built the dashboard and how we handle SSL at scale. Learn about the auto-scaling architecture and the metrics pipeline in our companion articles. Explore our free developer tools to speed up your workflow. Try Deployxa Drop for an instant live preview with zero signup.