Every release broke RDS. The team scaled the database, closed the ticket, and waited for it to happen again.
HarborLab manages disbursements, port costs, and vessel operations for 4,000 registered vessels, 5,000 port agents, and 35,000 port calls across global shipping. 20+ production microservices on AWS in eu-west-1, backed by RDS Aurora, Kubernetes with 46+ deployments, Auto Scaling Groups peaking at 24 instances. Grafana Cloud for observability. When a connection pool issue hits during peak hours, port agents closing operations across multiple time zones are waiting on delayed calculations.
The challenge
Deploy, watch RDS break, scale the database, repeat.
Multiple services shared one RDS instance. After every release, connections maxed out. The team deployed, watched RDS strain, scaled the database manually, waited for things to settle. Port cost calculations slowed. Agents across time zones waited. Nobody could tell which deployment did it.
The answer was always spread across Loki, CloudWatch, and GitHub. Engineers searched all three trying to figure out which service was draining connections. Never one place.
Any of the 20+ services could have caused it. Any recent change in any of them. Narrowing it down meant checking each one.
There was no way to connect a commit to a deployment to an RDS metric. The team had to reconstruct that chain manually every time. Nobody ever finished.
So they scaled RDS, resolved the symptom, and moved on. Next release, same thing.
Why they chose NOFire
HarborLab connected AWS, Kubernetes, and Grafana Cloud to NOFire. First incident after onboarding: 3 minutes to find the change, 15 minutes to resolve. They reverted the commit and the incident stopped recurring.
How they use NOFire, in practice
Commit to deployment to database spike. In one view.
What showed up in 3 minutes:
Which service was draining connections.
The commit that changed the connection pool configuration.
The deployment that rolled it out.
RDS connection metrics aligned with the deployment timeline.
Ongoing:
10,000+ changes tracked across EC2, RDS Aurora, ASGs, and Kubernetes. The context is there before anything breaks.
3 RDS failover events correlated with upstream changes. No manual timeline work.
35+ investigations across storage, CPU, latency, and pod alerts. Each one traced to the relevant change.
Inside an investigation
RDS connection pool exhausted after release. One service, one config change.
Routine release. Connections spiked, API calls started timing out. Two days of searching Loki, CloudWatch, and GitHub commits. NOFire matched the deployment event to a connection pool config change in one service. RDS metric spike lined up with the deployment. Team reverted, connections stabilized. Hasn't happened since.
A few more incidents
Low RDS storage alert (SEV2). Steady growth, no recent change.
SEV2 for critically low storage. NOFire checked the timeline: no deployments, no scaling events, no traffic changes. Storage had been growing steadily for weeks. The team investigated data retention instead of treating it as an acute incident.
Node CPU at 80% for 15 minutes. Scheduled batch job, not a problem.
Critical CPU alert on a production node. NOFire checked Kubernetes events, pod activity, deployment history. No recent changes. CPU pattern matched a scheduled reconciliation workload. Alert closed, suppression rule added.
RDS high CPU across multiple instances. Service scaled up earlier that day.
CPU alerts across RDS instances. Storage, memory, connections all normal. NOFire traced it to a write query surge from a service that had been scaled up hours earlier. The scaling event and the write surge were in the same timeline.
Results
The impact
Fixed the change. RDS has been stable since.
Days to 3 minutes: The team reverted the commit and deployed a fix the same day. What used to take days of searching across three tools was done before the on-call engineer finished reading the alert.
The change fixed, not the symptom: Previous cycles ended with scaling RDS and closing the ticket. NOFire showed the connection pool config change, the deployment, and the metric spike in one chain. The team fixed the actual problem.
10,000+ changes indexed before alerts fire: Deployments, scaling events, config changes. When something breaks, the relevant change is already there. On busy days, over 1,000 changes processed in 24 hours.
100+ hours back across 35+ investigations: Each one replaced the usual exercise of checking Loki, then CloudWatch, then GitHub. Answers show up in the same Slack channel where the alert fired.
“We found the exact change that kept breaking RDS and fixed it. Releases don't end with panic and manual scaling anymore.”Spyros Lamprinidis · CTO, HarborLab
“We used to spend days going through Loki, CloudWatch, and GitHub trying to figure out which change broke things. Now it's 3 minutes.”Stelis Panagiotakis · SRE Lead, HarborLab