Docker's HEALTHCHECK only updates daemon metadata that static nginx upstreams never poll, and while adding http_500 to proxy_next_upstream recovers read traffic, it doubles upstream latency on failing requests and risks duplicate mutations on non-idempotent routes before max_fails evicts the peer. Retries hide failures; ejection stops them.
mostly agree, and "retries hide failures, ejection stops them" is a good way to put it. one thing on the duplicate mutations though, nginx already guards that by default. since 1.9.13 it won't pass a POST, PATCH or LOCK to the next upstream once the request has been sent, unless you add non_idempotent to proxy_next_upstream. a 500 means it was sent, so a failing POST just goes back to the client as a 500.
which flips the risk around: no duplicates, but my test was all GETs, so the "client never saw an error" result only holds for reads. writes to that backend would still have failed until max_fails kicked it out. worth a line in the post.
Custom software & AI development company. We build production-ready AI integrations, mobile apps, and web platforms for businesses that want real results. lycore.com
Useful experiment, and the proxy_next_upstream finding is the part people miss. A Docker healthcheck only changes the status flag, so nginx never learns about it unless you add active checks or tune max_fails and fail_timeout. In my experience the fix is to treat the health endpoint and the proxy's passive failure rules as two separate layers. I'd also add a readiness-style endpoint that starts failing before the container is truly broken, so traffic drains before errors show up.
agreed on the two layers. one catch with the active checks option though, in open source nginx the health_check directive is NGINX Plus only, so on stock nginx the passive max_fails/fail_timeout rules are the only layer that can react at all. that's partly why the missing http_500 hurt so much in my test.
the readiness idea runs into the same wall. nginx never calls your readiness endpoint, it only sees real requests failing. so to drain on plain nginx you either need something that watches readiness and rewrites the upstream (drop the server, nginx -s reload), or a proxy that does active checks natively, Traefik or HAProxy for example.
Custom software & AI development company. We build production-ready AI integrations, mobile apps, and web platforms for businesses that want real results. lycore.com
That's a useful distinction: on stock nginx the only signal is real traffic failing, so the passive rules have to carry the whole load. A small watcher that polls readiness and rewrites the upstream file, then runs nginx -s reload, is a workable stopgap, though it adds its own lag and a failure mode of its own. If draining matters a lot, HAProxy or Traefik with native active checks sounds like the cleaner route. Thanks for spelling out the Plus-only detail.
yeah, that's where I landed too. one thing that makes Traefik a nice fit for this exact setup: its Docker provider reads the container's health status and leaves unhealthy containers out of routing, so the HEALTHCHECK that nginx ignored in my test actually gets wired to something. haven't tried it against this setup yet though.
and if anyone does go the watcher route, I'd mark the server down in the upstream file rather than deleting the line, and only reload after nginx -t passes. that keeps a bad rewrite from taking the whole proxy with it.
I measure AI coding claims in public. LGTM — Let's Get the Measurements: real runs, real numbers, receipts. Every article is an experiment I actually ran. Books & packs: aheadkit.gumroad.com
The http_500 detail is the one that stings — 502/503/504 is the line every tutorial copies, and a plain 500 is the failure your app is most likely to actually return, so the failover silently does nothing for exactly the case it exists for. Finding 4 is the part I'd want a follow-up on though: once the proxy retries quietly, edge monitoring goes blind. Did you end up alerting on upstream metrics, or is that a blind spot you've decided to live with?
honestly, I didn't wire up alerting in that test, finding 4 is me naming the gap, not closing it. the cheapest fix I know of doesn't need a metrics stack though. add $upstream_status and $upstream_addr to the nginx log_format. when a request gets retried, $upstream_status shows every attempt, so you get "500, 200" with the two backend addresses next to it, while $status still says 200.
then alert on any line where $upstream_status contains a 5xx, grouped by $upstream_addr. one backend failing every request it gets shows up straight away, even though clients never see an error. I haven't tested that end to end yet, so treat it as the plan, not a result.
Three things pointed at the same container and none of them listening to the other two is a very common setup, and the missing http_500 in proxy_next_upstream is the kind of detail that only shows up when someone breaks it on purpose.
When Ascendra Ventures answered a tender for an airport's generative AI platform, the premise was that passenger-facing AI is an uptime business first. The platform design was treated as production infrastructure from the start: Kubernetes, geographic redundancy and automated failover, auto-scaling for passenger peaks, Prometheus and Grafana alerting, and a support spine where critical incidents are answered in 30 minutes and resolved in an hour. In a design like that, a health status that the router never reads would quietly undo most of the work.
Did you end up adding the proxy side to your alerting as well, so a container that is unhealthy but still taking traffic actually pages someone?
not yet, no. the test stopped at finding the gap. the plan is the cheap version: add $upstream_status and $upstream_addr to the nginx log_format, since a retried request logs both attempts ("500, 200") even when the client just sees a 200. then alert on any 5xx in $upstream_status, grouped by backend address, so one container failing everything it gets shows up even while the retries hide it from users. I haven't run that end to end yet though, so it's a plan, not a result.
yeah, that's the whole post in one line. a status light nobody's wired to anything. the annoying part was that the proxy config looked fine, http_500 just wasn't in the list, and nothing complains about that until a backend starts throwing plain 500s.
http_500 missing from the list is the quiet failure. the config looks fine, nothing complains, then a backend starts throwing plain 500s and the healthcheck never noticed.
yep. and even with http_500 added, nginx only retries GETs and other idempotent requests by default, so a POST that hits the broken backend still gets the 500. the retry hides it for reads, not writes.
We're a place where coders share, stay up-to-date and grow their careers.
Docker's
HEALTHCHECKonly updates daemon metadata that static nginx upstreams never poll, and while addinghttp_500toproxy_next_upstreamrecovers read traffic, it doubles upstream latency on failing requests and risks duplicate mutations on non-idempotent routes beforemax_failsevicts the peer. Retries hide failures; ejection stops them.mostly agree, and "retries hide failures, ejection stops them" is a good way to put it. one thing on the duplicate mutations though, nginx already guards that by default. since 1.9.13 it won't pass a POST, PATCH or LOCK to the next upstream once the request has been sent, unless you add non_idempotent to proxy_next_upstream. a 500 means it was sent, so a failing POST just goes back to the client as a 500.
which flips the risk around: no duplicates, but my test was all GETs, so the "client never saw an error" result only holds for reads. writes to that backend would still have failed until max_fails kicked it out. worth a line in the post.
Useful experiment, and the proxy_next_upstream finding is the part people miss. A Docker healthcheck only changes the status flag, so nginx never learns about it unless you add active checks or tune max_fails and fail_timeout. In my experience the fix is to treat the health endpoint and the proxy's passive failure rules as two separate layers. I'd also add a readiness-style endpoint that starts failing before the container is truly broken, so traffic drains before errors show up.
agreed on the two layers. one catch with the active checks option though, in open source nginx the health_check directive is NGINX Plus only, so on stock nginx the passive max_fails/fail_timeout rules are the only layer that can react at all. that's partly why the missing http_500 hurt so much in my test.
the readiness idea runs into the same wall. nginx never calls your readiness endpoint, it only sees real requests failing. so to drain on plain nginx you either need something that watches readiness and rewrites the upstream (drop the server, nginx -s reload), or a proxy that does active checks natively, Traefik or HAProxy for example.
That's a useful distinction: on stock nginx the only signal is real traffic failing, so the passive rules have to carry the whole load. A small watcher that polls readiness and rewrites the upstream file, then runs nginx -s reload, is a workable stopgap, though it adds its own lag and a failure mode of its own. If draining matters a lot, HAProxy or Traefik with native active checks sounds like the cleaner route. Thanks for spelling out the Plus-only detail.
yeah, that's where I landed too. one thing that makes Traefik a nice fit for this exact setup: its Docker provider reads the container's health status and leaves unhealthy containers out of routing, so the HEALTHCHECK that nginx ignored in my test actually gets wired to something. haven't tried it against this setup yet though.
and if anyone does go the watcher route, I'd mark the server down in the upstream file rather than deleting the line, and only reload after nginx -t passes. that keeps a bad rewrite from taking the whole proxy with it.
The http_500 detail is the one that stings — 502/503/504 is the line every tutorial copies, and a plain 500 is the failure your app is most likely to actually return, so the failover silently does nothing for exactly the case it exists for. Finding 4 is the part I'd want a follow-up on though: once the proxy retries quietly, edge monitoring goes blind. Did you end up alerting on upstream metrics, or is that a blind spot you've decided to live with?
honestly, I didn't wire up alerting in that test, finding 4 is me naming the gap, not closing it. the cheapest fix I know of doesn't need a metrics stack though. add $upstream_status and $upstream_addr to the nginx log_format. when a request gets retried, $upstream_status shows every attempt, so you get "500, 200" with the two backend addresses next to it, while $status still says 200.
then alert on any line where $upstream_status contains a 5xx, grouped by $upstream_addr. one backend failing every request it gets shows up straight away, even though clients never see an error. I haven't tested that end to end yet, so treat it as the plan, not a result.
Three things pointed at the same container and none of them listening to the other two is a very common setup, and the missing http_500 in proxy_next_upstream is the kind of detail that only shows up when someone breaks it on purpose.
When Ascendra Ventures answered a tender for an airport's generative AI platform, the premise was that passenger-facing AI is an uptime business first. The platform design was treated as production infrastructure from the start: Kubernetes, geographic redundancy and automated failover, auto-scaling for passenger peaks, Prometheus and Grafana alerting, and a support spine where critical incidents are answered in 30 minutes and resolved in an hour. In a design like that, a health status that the router never reads would quietly undo most of the work.
Did you end up adding the proxy side to your alerting as well, so a container that is unhealthy but still taking traffic actually pages someone?
not yet, no. the test stopped at finding the gap. the plan is the cheap version: add $upstream_status and $upstream_addr to the nginx log_format, since a retried request logs both attempts ("500, 200") even when the client just sees a 200. then alert on any 5xx in $upstream_status, grouped by backend address, so one container failing everything it gets shows up even while the retries hide it from users. I haven't run that end to end yet though, so it's a plan, not a result.
the proxy_next_upstream bit is the real fix in here. a healthcheck that only reports and never reroutes is just a label.
yeah, that's the whole post in one line. a status light nobody's wired to anything. the annoying part was that the proxy config looked fine, http_500 just wasn't in the list, and nothing complains about that until a backend starts throwing plain 500s.
http_500 missing from the list is the quiet failure. the config looks fine, nothing complains, then a backend starts throwing plain 500s and the healthcheck never noticed.
yep. and even with http_500 added, nginx only retries GETs and other idempotent requests by default, so a POST that hits the broken backend still gets the 500. the retry hides it for reads, not writes.