Performance and reliability on a live e-commerce platform
Performance work on a live shop is different from performance work on a greenfield project: every change ships against real customers mid-checkout. These four stories are how I approach it — measure first, fix the actual bottleneck, and leave behind monitoring so the same class of problem cannot go undetected again.
One note on ownership before the stories. I architected this platform, so several of the flaws below were my own earlier trade-offs, revisited once real traffic exposed them; others predate the replatform, from the Laravel private app that ran alongside the original Shopify store. Production load is how you find out which decisions don't hold, and I think showing that honestly is more useful than pretending the system arrived perfect.
1. Storefront GraphQL response cache (June 2026)
The highest-traffic storefront query — the product grid powering collection pages — cost 150–600ms per cold request. I built a full-response Redis cache as route middleware: a strict eligibility allowlist (anonymous traffic only, page depth and filter-count limits, no long-tail search queries), tag-version invalidation via a single global counter bumped by model observers (O(1) writes, no key scanning), and a dedicated Redis database isolated from sessions and queues so a keyspace explosion could never cascade.
The part I care about most is what shipped alongside it: hit/miss/write-fail counters on every request, a 10-minute health check that emails on threshold breaches, a daily digest as an affirmative heartbeat, and a documented 2-minute rollback (remove one middleware line, reload php-fpm). 37 tests green before production.
Result: cache hits under 10ms versus 150–600ms cold, verified in production June 2026. Hit rate was 28% one hour after deploy and climbing as keys warmed.
flowchart LR
R["Storefront request"] --> E{"Eligible? anonymous + depth / filter limits"}
E -- "no" --> RES["GraphQL resolvers"]
E -- "yes" --> C{"Redis response cache"}
C -- "hit — under 10ms" --> OUT["Response"]
C -- "miss" --> RES
RES --> DB[("MySQL")]
RES --> W["Write-through with tag-versioned key"]
W --> OUT
OBS["Model observers"] -. "bump global tag version — O(1) invalidation" .-> C
The response-cache data flow: strict eligibility in front, tag-version invalidation behind, monitoring on every request.
2. Admin order search 4s → <50ms, and webhooks off the request path (April 2026)
Admin order search took 4 seconds. The query ran LIKE '%term%' across 9 columns with double-nested EXISTS subqueries and eager-loaded every order's items just to count them. I rewrote it: numeric input became an exact primary-key lookup, the subqueries became direct joins, counts came from withCount, and a migration added the missing composite indexes. 4s to under 50ms.
The same session fixed something worse. Shopify webhook handlers were making synchronous GraphQL calls back to Shopify (1–5 seconds each) inside the HTTP request. Under traffic spikes this exhausted PHP-FPM workers and webhook deliveries timed out — a silent order-loss mechanism, because Shopify permanently deletes a webhook subscription after enough consecutive failed deliveries, with no notification and no failed-job record on our side. I inverted the design: the handler now records only the Shopify entity ID and dispatches a queued job; the fetch happens in a Horizon worker. Webhook responses dropped to ~5ms. The mechanism was later proven real: after a separate weekend outage in May 2026, Shopify had silently deleted 8 of our 15 webhook subscriptions, freezing fulfillment-event flow until I re-registered them and ran backfills — so I added a daily idempotent self-heal job that re-registers all topics, capping any future silent drift at 24 hours.
flowchart LR
SH["Shopify webhook"] --> H["Handler: store entity ID only — respond in ~5ms"]
H --> QQ[("Queue")]
QQ --> W["Horizon worker"]
W -- "GraphQL fetch (1–5s), off the request path" --> SHAPI["Shopify API"]
W --> DB[("MySQL")]
HEAL["Daily self-heal job"] -. "re-registers all webhook topics" .-> SH
Non-blocking webhook ingestion: acknowledge fast, fetch async, self-heal the subscriptions daily.
3. A cache key that never hit: full MySQL saturation (July 2026)
In early July 2026, production stalled for ~15 minutes. It wasn't a crash — MySQL had 208 of 300 connections consumed. The slow-query log pointed at a public social-proof feed polled every 10–20 seconds by every visitor.
The feed had an 8-second cache designed to collapse all concurrent visitors into one DB query. That cache was silently dead: its key included a now()-derived 30-day clamp that advanced every second, so the key was unique per second and never hit. Every poll ran two queries with COALESCE(completed_at, created_at) in both WHERE and ORDER BY — un-indexable — scanning 174,278 rows per call (~2s each). Under storefront concurrency they stacked into saturation.
I shipped two independent defenses: quantizing the cache key into 30-minute buckets so the shared cache actually shares, and a virtual generated column (effective_paid_at) with a composite index, built online with zero table locking. Verified with EXPLAIN showing an indexed range scan, plus a regression test asserting the second call executes zero DB queries. The distilled rule went into the platform's documented pattern library: a cache whose key depends on a moving now() is not a cache.
4. Checkout dying at the network layer, the day after launch (June 2026)
The morning after our Swiss market launch (June 2026), checkouts started hanging for 30 seconds; customers retried up to 12 times, piling up orphaned pending orders. Code, firewall, and Stripe itself all checked out. One detail matters for what follows: the origin server was hosted with Shinjiru, a Southeast-Asian host, at the time, so Stripe's API traffic resolved to its nearest edge — AWS Singapore. Using mtr, packet-size probing, and tracepath, I proved the provider's upstream route to that edge was intermittently black-holed for large packets — TCP connected fine, then TLS handshakes hung after Client Hello, the classic PMTU black hole. The provider's eventual remediation was to lower the interface MTU to 1450 rather than repair the route.
I shipped a same-day workaround — pinning Stripe's API to its healthy US edge plus an iptables MSS clamp — which restored checkout within hours, then drove the provider escalation and verified their fix the next morning before rolling the pin back.
The outage had gone 14 hours undetected, and that bothered me more than the outage. The next day I deployed a 5-minute healthcheck cron for every third-party dependency (Stripe, PayPal, Klaviyo, Postmark) — written in dependency-free bash, deliberately outside the application framework, so it keeps alerting even when the application itself is down.
The safety net underneath
None of this happens safely without two foundations I put in place first. A k6 load-test baseline the week before the Swiss launch (June 2026): 24,440 requests over 2 minutes at ~198 RPS sustained with zero HTTP errors, proving PHP-FPM (not the database, which sat at 25% connection capacity) was the scaling ceiling. The same test exposed two mis-keyed rate limiters — one bucket shared by all anonymous traffic behind Cloudflare's NAT — whose exhaustion had been surfacing as masked "internal server error" responses; removing them eliminated the ~6% failure rate observed in test runs before the fix. In their place went a deliberate access policy: the frontend servers were whitelisted for unthrottled GraphQL calls, while every other source was rate-limited — protection aimed at abusers instead of at customers. And zero-downtime deploys (June 2026): I replaced the maintenance-mode deploy pipeline (90–180 seconds of 503s per deploy) with graceful php-fpm reloads plus OPcache timestamp validation across two different OS environments, verified with a continuous curl loop during live production deploys — all 200s. That change came with a standing rule, documented for every maintainer who followed: all migrations must be backwards-compatible, because old code runs against the new schema during every deploy window.
My reliability philosophy: find the real layer before fixing anything — I've traced "the site is slow" to a dead cache key, an un-indexable expression, exhausted FPM workers, and a broken network route, and each demanded a different fix. Prove fixes with evidence (EXPLAIN plans, load tests, curl probes), pair every fix with a detector so the failure class can't recur silently, and ship every risky change with a rehearsed rollback. Fast systems and reliable systems come from the same habit: refusing to guess.