Weekly Engineering Report

Summary

This week we completed the payment-service 1.14.0 rollout to 100% of production with no customer-facing incidents. p95 latency on the checkout path improved by ~18% (from 210ms to 172ms) and error rates remained below 0.2% across all regions.

Key highlights: - Connection pooling and timeout tuning reduced transient failures during peak load. - Canary analysis validated stability before full rollout. - Runbook updates shortened mean time to mitigate for payment-related alerts.

Focus areas for next week: - Ship SLO-based alert thresholds to reduce noise. - Address flaky refund idempotency test to unblock CI. - Plan production upgrade for Postgres 14.8 after final staging sign-off.

Open Issues

ID	Title	Severity	Status
PAY-231	Intermittent 504s on payment-service during checkout	High	Investigating
SEC-019	Update JWT dependency to 2.9.x to address CVE-2026-1234	High	Blocked
OPS-332	Flaky integration test test_refund_idempotency	Low	In Progress
INF-104	Upgrade NGINX Ingress Controller to 1.11	Medium	Planned
DATA-088	Backfill missing payment events for March 29-31	Medium	In Review

Changelog

Changes

payment-service: Add connection pooling and tune maxIdleConns (PR #842, commit abc1234)
payment-service: Reduce retry jitter and make timeouts configurable (PR #847, commit f1e2d3a)
api-gateway: Add circuit breaker thresholds for payment-service upstream (PR #851)
infra: Bump Postgres from 14.6 to 14.8 in staging; production rollout scheduled (PR #856)
tooling: Add canary analysis script for rollout verification (PR #859)
docs: Update runbook for payment incidents (PR #861)

git rev-parse --short HEAD
kubectl rollout status deploy/payment-service -n prod