A Solutions Architecture Investigation That Uncovered What Dashboards Can't

Savings plans cover the predictable spend, but ForUsAll's bill had layers of waste that only a hands-on infrastructure investigation could find. The Pump team assigned a dedicated solutions architect to ForUsAll's account who requested read-only IAM access to their AWS environment and laid out a structured investigation plan across three specific cost categories: AWS Backup, OpenSearch Service, and charged backup usage.

The investigation was methodical. For AWS Backup, the SA audited every backup plan, reviewing the rules for frequency, start windows, lifecycle to cold storage, and retention periods. They checked which resource types were attached to each plan (EBS, RDS, EFS, DynamoDB) and looked for cross-region or cross-account copy rules, which silently double or triple storage costs without appearing as a separate line item. The goal was to determine whether retention was longer than the business actually needed, whether anything cold-eligible was sitting in warm storage, and whether cross-region copies were inflating the bill.

For OpenSearch, the SA requested 30 days of CloudWatch metrics across eight dimensions: CPUUtilization (average and maximum), JVMMemoryPressure (average and maximum), FreeStorageSpace (minimum), SearchRate (average), IndexingRate (average), SearchLatency (average and p99), ClusterStatus (yellow/red maximum), and ThreadpoolWriteQueue/SearchQueue (maximum). These metrics together tell you whether a cluster is over-provisioned on compute, whether memory pressure leaves room for downsizing, how much storage is actually in use versus provisioned, and whether there is any queue pressure that would block a downsize. The SA also requested index-level data from the OpenSearch Dev Tools console to identify large or stale indexes that could move to UltraWarm or be deleted entirely.

For charged backup usage, the SA wrote and shared a CLI script that cross-referenced active EBS volume IDs against existing snapshots to surface orphans: snapshots tied to volumes that no longer existed but were still incurring daily storage charges. ForUsAll's DevOps engineer in Medellín ran the script and cleaned up a batch of orphaned snapshots immediately. The SA also reviewed RDS backup retention periods on each database instance and checked for manual DB snapshots, which persist indefinitely until explicitly deleted and are one of the most common sources of silent cost accumulation.

The zombie database discovery came directly out of this investigation. While drilling into RDS spend at the resource level, the SA noticed an Aurora Serverless v2 operations cluster in the pre-production account showing sustained IO activity and consistent daily spend that didn't correspond to any active workload. The cluster had been provisioned nearly three years earlier as part of a previous build cycle and had been running continuously ever since. At the account level, the spend blended in with legitimate pre-production costs. Only when filtered down to individual cluster ARNs did the orphaned instance stand out. Bhavani stopped it on the call. AWS keeps the data intact on a stopped cluster for seven days before requiring a decision, so there was no risk of data loss. The instance turned out to be completely unused.

Beyond finding waste, the SA provided architectural recommendations. ForUsAll had been restoring their production database into the staging environment by dumping it to S3 and re-uploading through GitHub Actions, burning both compute and CI/CD minutes on every refresh. The Pump SA documented a strategy for using Aurora read replicas and cross-account snapshot sharing to accomplish the same thing natively within AWS, eliminating the S3 intermediary and the GitHub Actions overhead entirely.