I cut p95 from 360 ms to 120 ms without touching a single controller. The antagonist was a cache stampede that pushed every hot GET through service and database layers.
Duplicate work spiked CPU and filled queues while real users waited. I move cheap decisions to the filter chain so waste dies at the door.
The gain is steadier graphs and fewer retries when traffic jumps.
Cache at the door so hot reads never wake MVC
A small OncePerRequestFilter serves warm JSON directly when a cache entry is fresh. I stamp an ETag, respect If-None-Match, and return before Spring MVC allocates anything heavy.
Controllers stay asleep for happy-path reads, and more threads are free for real work.
@Component
class FastLaneCacheFilter extends OncePerRequestFilter {
@Override
protected void doFilterInternal(HttpServletRequest req,
HttpServletResponse res, FilterChain chain)
throws IOException, ServletException {
if (!"GET".equals(req.getMethod())) { chain.doFilter(req, res); return; }
var q = req.getQueryString();
var key = "v1:" + req.getRequestURI() + (q == null ? "" : "?"+q);
byte[] body = redisGet(key); // returns null when miss
if (body == null) { chain.doFilter(req, res); return; }
var etag = '"' + sha256Base64(body) + '"';
if (etag.equals(req.getHeader("If-None-Match"))) { res.setStatus(304); return; }
res.setHeader("ETag", etag);
res.setHeader("Cache-Control", "public,max-age=45,stale-while-revalidate=15");
res.setContentType("application/json");
res.getOutputStream().write(body);
}
}On our busiest endpoint, controller invocations fell by 68%, and CPU dropped 22% at the same QPS.
Two practical notes:
version your keys (v1:) so invalidation is precise, and keep TTL short on anything that embeds user-visible counts.
I shipped the first version without ETag and paid for it during a Friday burst—304s are worth the few bytes of hashing.
Collapse duplicate reads during bursts with single-flight
A miss still invites ten identical reads when a hot key turns cold. I coalesce them so one worker fetches, the others wait briefly and reuse the bytes.
+---------+ +---------------+ +-----------+
| Clients | --> | SingleFlight | --> | Spring MVC|
+---------+ | keyed wait | +-----------+
\ +---------------+ |
\-------> same-key callers wait <-----/This alone stopped a thundering herd after cache evictions when deploys rolled.
A keyed wait filter that completes fast on success
I park followers on a CompletableFuture<byte[]>.
The first caller either completes it with the serialized body or everyone falls through if it errors or stalls too long.
@Component
class SingleFlightFilter extends OncePerRequestFilter {
private final ConcurrentHashMap<String, CompletableFuture<byte[]>> in = new ConcurrentHashMap<>();
@Override
protected void doFilterInternal(HttpServletRequest req, HttpServletResponse res,
FilterChain chain) throws IOException, ServletException {
if (!"GET".equals(req.getMethod())) { chain.doFilter(req, res); return; }
var key = req.getRequestURI() + "?" + String.valueOf(req.getQueryString());
var fut = in.computeIfAbsent(key, k -> new CompletableFuture<>());
if (!fut.isDone() && fut.getNumberOfDependents() == 0) {
// This request is "winner": let MVC compute; a later hook will complete fut.
chain.doFilter(req, res);
return;
}
try {
byte[] bytes = fut.get(200, TimeUnit.MILLISECONDS);
res.setContentType("application/json");
res.getOutputStream().write(bytes);
} catch (Exception ex) {
chain.doFilter(req, res); // no amplification of failure
}
}
// Call this from ResponseBodyAdvice after JSON serialization:
void complete(String key, byte[] json) { in.computeIfPresent(key, (k,f) -> { f.complete(json); return f; }); }
}During a nightly price import, parallel DB calls for the same key collapsed to one, and p99 stopped spiking above 600 ms.
Wire a tiny ResponseBodyAdvice<byte[]> (or a HandlerInterceptor capturing the JSON) to call complete.
Bound the map with a small eviction after completion—leaking futures is an easy mistake here.
Shed load early with a small per-route token bucket
When the backend is already late, a fast 429 is kinder than another slow 200. I attach a light token bucket per route and leak tokens at the rate we can actually serve.
@Component
class ShedLoadFilter extends OncePerRequestFilter {
private final ConcurrentHashMap<String, AtomicLong> tokens = new ConcurrentHashMap<>();
private final long cap = 180; // burst
private final long refillEveryMs = 100; // tick
private final long refill = 6; // 60 rps per route
private volatile long last = System.currentTimeMillis();
@Override
protected void doFilterInternal(HttpServletRequest req, HttpServletResponse res,
FilterChain chain) throws IOException, ServletException {
var now = System.currentTimeMillis();
if (now - last >= refillEveryMs) {
tokens.values().forEach(a -> a.set(Math.min(cap, a.get() + refill)));
last = now;
}
var key = req.getMethod() + " " + req.getRequestURI();
var bucket = tokens.computeIfAbsent(key, k -> new AtomicLong(cap));
if (bucket.getAndDecrement() <= 0) {
res.setStatus(429);
res.setHeader("Retry-After", "1");
return;
}
chain.doFilter(req, res);
}
}Our 5xx rate flattened during a bad dependency day; latency for accepted requests tightened by 14% because fewer jobs waited in line.
Start conservative. Put buckets only on expensive routes first (fan-out, heavy joins, PDF generation).
Pair with a client retry budget so median latency stays boring even when traffic stutters.
Put the gates in the right order so wins stack
Position is everything. The gates must cut waste in the cheapest place possible.
[Netty/Tomcat] -> [FastLaneCache] -> [SingleFlight]
-> [ShedLoad] -> [Spring MVC] -> [Service] -> [DB]
^ quick 304 ^ one winner ^ only if tokenI learned the hard way that placing shedding before coalescing throws away cheap wins; you refuse work you could have served from the just-built body.
Be explicit about precedence so a future refactor does not reshuffle things. Tests that assert ordering save a surprise later.
Wire-up with explicit precedence and one hook point
I register filters with ascending order: cache first, coalescing next, shedding last. One hook captures successful JSON so single-flight has bytes to share.
@Configuration
class FilterOrder {
@Bean FilterRegistrationBean<FastLaneCacheFilter> cache(FastLaneCacheFilter f) {
var b = new FilterRegistrationBean<>(f); b.setOrder(10); return b;
}
@Bean FilterRegistrationBean<SingleFlightFilter> flight(SingleFlightFilter f) {
var b = new FilterRegistrationBean<>(f); b.setOrder(20); return b;
}
@Bean FilterRegistrationBean<ShedLoadFilter> shed(ShedLoadFilter f) {
var b = new FilterRegistrationBean<>(f); b.setOrder(30); return b;
}
}
@ControllerAdvice
class JsonTap implements ResponseBodyAdvice<byte[]> {
@Autowired SingleFlightFilter flight;
@Override public boolean supports(MethodParameter p, Class c) { return true; }
@Override public byte[] beforeBodyWrite(byte[] body, MethodParameter p, MediaType t,
Class c, ServerHttpRequest req, ServerHttpResponse res) {
var uri = ((ServletServerHttpRequest)req).getServletRequest();
var key = uri.getRequestURI() + "?" + String.valueOf(uri.getQueryString());
flight.complete(key, body);
return body;
}
}After this change, the same hardware handled peak traffic with 30% less CPU and steadier p95, and deploys stopped causing read spikes.
A tiny before/after that matched what we felt
Metric Before After
p95 latency 360 ms 120 ms
CPU (node) 78 % 56 %
5xx rate 0.8 % 0.2 %I am always wary of pretty numbers; these held across three peak windows and a synthetic load test.
The Simple Way Forward
The lesson is simple: push decisions forward. A cache answer at the door, a single winner on a cold miss, and a polite refusal when we are late — each cuts a class of waste before it gets expensive.
I keep each filter tiny, measurable, and ordered so they cooperate rather than fight.
Next step for me is tenant-aware buckets so a noisy neighbor never bends everyone's graphs; it is the same idea, just a narrower key.