MongoDB diagnostics tool preserves three months of history
A MongoDB diagnostics tool called mongo-health is designed to answer a familiar operational question: why was the database slow several days ago, even though no alert fired and nobody investigated at the time? Percona says the open-source tool retains detailed MongoDB diagnostic history for three months, evaluates it with 68 checks every hour and presents one overall verdict for each cluster.
The underlying evidence comes from Full Time Diagnostic Data Capture, or FTDC, MongoDB's built-in collection of operational measurements. Every mongod process, the program that runs a MongoDB database server, records its own metrics once per second in a diagnostic.data directory. That can reveal inefficient queries, delayed storage operations and network trouble that may not appear in the database log. Percona measured an Atlas M10 under production traffic retaining only two to five days of FTDC, meaning evidence from Thursday could already be unavailable when a team investigates on Monday.
Without automated analysis, an administrator must open the remaining files in a viewer and work through charts around the reported time. An obvious cause may appear among the first ten charts, but a less visible one might require reviewing another 40. The captured data can show a query reading 10 million documents for every result, storage taking 400 milliseconds per write for three minutes, or a replica-set member losing packets for 30 seconds. A replica set is a group of MongoDB servers that maintain copies of the same data for availability. The problem is not necessarily missing evidence, but the difficulty of finding the relevant signal before the files expire.
mongo-health keeps every second of FTDC for three months and reports what happened, which member was affected, how long it lasted and the associated chart. Percona says the software is licensed under Apache 2.0, runs in three Docker containers on a laptop or server and installs nothing on database hosts. Its GitHub instructions describe a four-command installation. In Percona's Atlas M10 test, the interface held 29.92 days of FTDC. If a cluster stops providing data, its state changes to “Not being watched” rather than continuing to display an outdated healthy verdict.
Individual findings contain a chart, a short suggestion about where to investigate and the incident's place in the cluster's history. An operator can mark a finding as fixed and add a note. It then leaves the active report, but a recurrence is identified as a repeat. The 68 checks cover work queues, available processing slots known as tickets, cache and memory, storage, replication, connections, queries, configuration and previous incidents. The project's README groups the checks, while docs/rules.md documents their thresholds.
Percona demonstrated five deliberately created faults on a three-member Percona Server for MongoDB 8.0 replica set and documented a sixth that occurred independently on an Atlas M10. In the first test, a query without a usable index began at 20:53. For ten minutes, the server examined roughly 10 million documents for each one returned, then repeated the behaviour at 21:54. The rule activates above a ratio of 100 to one. The operation did not fail and left nothing useful in the log, but the finding identified a likely missing index and the starting minute, allowing comparison with an application deployment.
A storage test began at 22:13 and lasted three minutes. Writes on all three members reached 400 milliseconds, far beyond the rule's 10-millisecond threshold. mongo-health classified it as the third such event on that cluster and linked the previous two. That history can distinguish an isolated slowdown from a recurring storage pattern.
The Atlas example instead pointed to the network. On 5 October at 08:25, one member retransmitted as many as 30 TCP segments per second for about 30 seconds. TCP segments are pieces of data sent across a network, and retransmission usually means they did not arrive successfully. The check starts at five retransmissions per second. Because the database server itself responded quickly, the evidence indicated packet loss in transit rather than a slow query, giving an operator a chart to attach to a support case.
Another laboratory workload repeatedly rewrote data in bulk. Checkpoints, periodic operations that make stored changes durable, normally completed in under a second but rose to between six and 11 seconds across all three members. This happened twice in ten minutes. The rule requires checkpoint times above one second for two minutes, preventing one isolated delay from becoming a finding. Percona notes that these brief freezes may not be evident in a throughput chart.
A separate test exposed a risk before users complained. The replica set handled about 250 writes per second, but each write replaced an entire 20 KB document and added about 20 KiB to the journal, MongoDB's record of changes used for recovery. The rule begins at 3.91 KiB per operation. At that rate, the oplog, the ordered history used to copy changes between replica-set members, would fill in one hour rather than one day. A secondary server falling an hour behind would then require a full resynchronisation. The useful response is to correct the update pattern before replication loses its recovery window.
For the final test, Percona reduced the WiredTiger cache to 128 MB and scanned a 1.9 GB collection. WiredTiger is MongoDB's storage engine, and its cache keeps frequently needed data in memory. For four minutes it fetched between 20,000 and 68,000 pages per second from disk, exceeding the rule's threshold of 10,000. The required data did not fit in memory. Rather than declaring memory alone to be the problem, the finding directs operators to examine disk latency and available read tickets before deciding whether the appropriate remedy is more memory or a more efficient query.
Collection methods depend on the deployment. A self-managed replica set needs a user assigned the clusterMonitor role, which grants access to monitoring information. The poller calls getDiagnosticData on every member at ten-second intervals. Atlas requires an API key with Project Read Only, described as its lowest role, and retrieves FTDC files through the Atlas Admin API. Operators can also upload diagnostic.data taken from a support bundle.
The tool is retrospective rather than an emergency alerting system. It reads FTDC once per hour, so a problem may be almost an hour old before appearing, and it never pages an operator. Existing live monitoring and alerts therefore remain necessary. It currently supports replica sets only, not sharded clusters, which distribute data across multiple groups of servers. Its rules come from Big Hole, and the repository documents each one.