Diagnosing intermittent 504 errors on the checkout API排查结算 API 间歇性 504 错误
A record of the experiments run to isolate and confirm the root cause.为定位并确认根本原因而开展的一系列实验记录。
Objective研究目标
Identify why roughly 3% of checkout requests return a 504 Gateway Timeout during peak hours, by running a series of controlled experiments to isolate and confirm the cause.通过开展一系列受控实验来定位并确认原因，查明为何在高峰时段约有 3% 的结算请求返回 504 网关超时。
Experiments实验
Chart 1.1图表 1.1 · 504 errors vs concurrent checkouts.504 错误数与并发结算量对比。
Errors begin once concurrency passes ~200 and rise sharply beyond it. The failure is load-dependent and reproducible.并发超过约 200 后开始出现错误，并随之急剧上升。该故障与负载相关且可复现。
- checkout-api < 50 mscheckout-api < 50 毫秒
- payment-service < 300 mspayment-service < 300 毫秒
- total < 2 s总计 < 2 秒
- checkout-api ≈ 41 mscheckout-api ≈ 41 毫秒
- payment-service ≈ 29,900 ms (DB wait)payment-service ≈ 29,900 毫秒（等待数据库）
- total ≈ 30 s → timeout总计 ≈ 30 秒 → 超时
{
"trace_id": "a1b2c3d4e5f6",
"spans": [
{ "service": "checkout-api", "duration_ms": 41 },
{ "service": "payment-service", "duration_ms": 29918, "wait": "db_pool" }
],
"total_ms": 30021
}
Almost all latency is in payment-service waiting on a saturated database connection pool. The pool is the bottleneck.几乎全部延迟都发生在 payment-service 等待已饱和的数据库连接池上。连接池即为瓶颈。
Overall Conclusion总体结论
Across the experiments, the 504 timeouts are caused by an undersized database connection pool in payment-service that saturates under peak load. The failure is load-dependent (Exp. 01) and localised to the payment-service DB pool (Exp. 02). Increasing pool capacity and adding backpressure is expected to resolve it.综合各项实验，504 超时的根本原因在于 payment-service 的数据库连接池配置过小，在高峰负载下被占满。该故障与负载相关（实验 01），并定位于 payment-service 的数据库连接池（实验 02）。提升连接池容量并引入背压机制有望予以解决。
Next Steps后续行动
- Increase the payment-service pool 3× and load-test at 2× peak.将 payment-service 连接池容量提升至 3 倍，并在 2 倍高峰下压测。
- Add an upstream timeout and retry-with-backpressure on checkout-api.在 checkout-api 上增加上游超时与带背压的重试机制。
- Re-run Experiment 01 to confirm zero timeouts after the change.重新执行实验 01，确认改动后超时归零。