Cutting a Runaway Log Bill Without Going Blind

A log bill that triples in six months is a visibility problem that pricing happened to expose. Most teams find out the same way: someone in finance forwards a Datadog or Splunk invoice with a question mark in the subject line, and whichever on-call engineer gets pulled into the thread has no idea which service is generating the volume, let alone whether any of it gets used. Cutting that bill safely means an audit, done calmly. Figure out where the bytes come from, decide how long each kind of line needs to live, then cut what nobody will ever query again, and that order matters more than people give it credit for.

Cost dashboards lie by omission. They'll tell you logging spend went from $18,000 a month to $54,000, and stop right there, leaving out the part where a single retry loop in your payments service accounts for most of the jump. I've watched a team spend two weeks arguing about retention policy before anyone thought to check which service was actually driving the number. The first move, always, is a breakdown by source: service, environment, log level, individual logger or class name if the platform gives you that much resolution.

Datadog's Log Management, Splunk's Cloud Platform, and Elastic all expose some version of volume-by-index or volume-by-tag. Shops running the ELK stack or Loki can get there with a bit of query work against index metadata, though it takes longer and somebody has to actually want to do it. Whatever the tool, the exercise doesn't change: rank every service by bytes ingested per day over the trailing 30 days, and look for the outliers.

The distribution is almost never even, and it's not close. A handful of services, usually the ones with the noisiest retry logic or the most verbose framework defaults, account for a wildly disproportionate share of total volume. Spring Boot apps left on default logging configs are the classic offender, and so is any service still logging full HTTP request and response bodies for debugging that nobody switched off after the incident that made it necessary in the first place. Find those five or six services before touching anything else. That's where most of the savings hide, and it takes a fraction of the effort compared to nickel-and-diming every team's logging habits at once.

Log Levels Are a Policy Decision, Made Deliberately

Once the volume source is clear, the next question is what level it's logged at. DEBUG and TRACE lines have no business shipping to a paid ingestion pipeline in production, and they belong on a laptop, or in a short diagnostic window that turns itself off.

Most orgs never wrote this down as policy. Individual teams made their own calls, and plenty of those calls got made three years ago by an engineer who has since moved on, taking the reasoning with them. An audit has to check level distribution, not just raw volume. If a service ships 40% of its lines at DEBUG in production, that's a setting nobody revisited.

Fixing it is usually a config change, which is exactly why it goes first. Logback, Log4j2, Winston: all of the major logging frameworks support dynamic level thresholds without a redeploy. Set production defaults to INFO and above, then gate DEBUG behind a feature flag or a TTL override tied to an active incident. Do that much and you've often cut ingestion by a third before touching a single retention setting.

Retention Tiers: Matching Lifespan to Purpose

Here's where teams leave money on the table even after fixing the noisy services. Retention usually gets set as one global number, 30 days or 90 or a full year, applied uniformly no matter what the underlying data actually is. It should be tiered by how the data actually gets used.

A workable framework has four tiers. Hot, queryable storage for 7 to 14 days covers active debugging and incident response, the stuff engineers actually search while something's on fire. Warm storage for 30 to 90 days, often a cheaper indexed tier like Elasticsearch's frozen tier or Datadog's Flex Logs, covers compliance windows and the occasional "what happened last month" question. Cold archive, S3 or GCS with lifecycle rules pushing to Glacier or Coldline after 90 days, covers regulatory retention measured in years, where the working assumption is that nobody reads this again unless a lawyer asks. Then there's a fourth tier that isn't really a tier: delete. Access logs from staging, health check pings firing every 10 seconds that always say the same thing, debug traces from a feature that shipped and stabilized two quarters back — none of that earns a storage tier at all.

Mapping every log category to one of these forces a conversation most engineering orgs have never actually had: what is this data for, and who reads it. Security and compliance logs answer to auditors and regulators. PCI DSS and SOC 2 carry specific retention numbers, and those numbers should come from the compliance team, full stop, not from whoever happened to configure the index two years ago. Application debug logs answer to engineers mid-incident and lose nearly all their value the moment the incident closes. Business event logs, the ones feeding analytics or billing reconciliation, often need to live far longer than either.

Deciding What's Safe to Drop

The scariest part of any log-reduction project is convincing an engineering org that dropping data won't come back to bite them at 2 a.m. That fear is legitimate, though it's usually pointed at the wrong logs.

Ask one plain question about each high-volume line: has anyone queried this in the last 90 days? Splunk's search history and Datadog's log analytics will both answer directly if you ask them. Zero query hits over 90 days, no compliance requirement attached, and you've got a strong candidate for sampling or outright removal.

Health checks and synthetic monitoring pings are the easiest win in the whole exercise, and almost nobody argues once you point them out. A load balancer hitting /healthz every 5 seconds throws off more than 17,000 log lines a day per instance, and virtually none of them ever get read unless the check itself starts failing, in which case the failure is the only line anyone actually cares about. Sample the successful checks down to 1 in 100, keep 100% of the failures, and volume drops hard without losing any signal that matters.

Successful, low-stakes transaction logs are next, and this one takes more care. A payment service logging "charge succeeded" on every transaction is useful for reconciliation, and it's arguably necessary for it. But the raw log line is often the wrong tool for that job next to a proper event stream or a database record. If the same information already lives in a transactional store with its own retention policy, the log line is redundant, and redundant is precisely what an audit exists to catch.

Security events, authentication failures, authorization denials, anything tied to an active compliance framework, and error-level logs of any kind: leave these alone. These lines exist to catch the thing going wrong, which is the entire reason logging exists in the first place, and the instinct to cut costs stops well short of them, no exceptions.

The Discipline That Keeps It From Coming Back

None of this holds if it's a one-time cleanup, and it usually isn't, which is the part nobody wants to hear. Log volume creeps back the exact way it built up the first time: a new service ships on framework defaults, a debug flag stays on after an incident wraps, a new feature logs the full request payload "just for now" and nobody ever comes back to it. Six months out, the bill's right back where it started, and someone has to run the whole audit again.

The fix is a standing budget. Some teams set per-service ingestion quotas and alert when a service crosses 120% of its rolling 30-day average, the same pattern cloud cost anomaly detection uses for compute spend. Others bake a log-level review into the pull request template for anything touching a logger call in a high-traffic path. Neither move is glamorous, and neither gets celebrated the way a big cost-cutting win does, which is honestly a shame, because it's the boring habit that actually works. The difference shows up a year later, in whether the audit has to happen again in a panic or not at all.