writing
Notes from production.
Long-form posts about the systems I build and the ones I break. No listicles, no hot takes with a two-week shelf life.
Circuit breaker thresholds: 110 false trips a day from noise
A 10-request window at a 50% threshold, against a dependency that errors 5% of the time, opens on pure noise 110 times a day. Twenty requests: once in 100 days.
Oct 2026
Data transfer: 43% of the bill never leaves your account
Twenty terabytes to the internet costs $1,783 a month. The cross-AZ traffic and NAT gateways behind it cost $1,321 more, and nobody budgeted for either.
Oct 2026
Model routing: the break-even escalation rate nobody works out
Trying a cheap model first saves 60% at a 20% escalation rate, and nothing at 80%. That threshold is one minus the price ratio, and it decides the whole design.
Oct 2026
What a slow CI pipeline costs, and it is not the runner bill
Twelve engineers and an 18-minute pipeline lose 2,760 engineer-hours a year, against $1,590 of runner time. The compute is the smaller number by 104 times.
Oct 2026
Load shedding: the 70% utilisation rule is a single-worker rule
At 90% utilisation one worker turns a 40 ms request into 400 ms. Forty workers turn it into 44 ms. The advice everyone repeats depends on a number nobody states.
Oct 2026
When a Postgres index costs more than it saves, measured
Eight indexes cut insert throughput by 70% in my measurements, and one on the wrong column took heap-only updates from 89,555 to zero. Here is the ledger.
Oct 2026
A million embeddings in Postgres costs 15 GB, not 6 GB
Measured on pgvector 0.8.6: a million 1536-dimension vectors is 785 MB per 50,000 rows once the index and TOAST are counted. The estimate misses two thirds.
Oct 2026
On-call rotation sizing: the three numbers Google publishes
Twelve pages a week at six hours each is 72 engineer-hours. Across five people that is 36% of everyone's week, against a published ceiling of 25%.
Oct 2026
Cache stampedes: why TTL jitter alone will not save you
Forty requests a second on one key with a 1.5-second recomputation is sixty simultaneous queries the moment it expires. Jitter fixes the wrong half of that.
Oct 2026
Timeout budgets: splitting two seconds across four services
A two-second deadline across four hops with one retry each leaves 245 ms per attempt. The P99 of one hop is 300 ms. That timeout is not catching failures.
Oct 2026
Error budgets: how many deploys does your SLO actually pay for?
Three nines is 43 minutes a month. A bad deploy that takes 45 minutes to mitigate at 30% impact spends 14 of them. The rest is arithmetic nobody does.
Sep 2026
Kubernetes requests and limits: the numbers that bill and throttle
Requests decide what you pay for. Limits decide when the kernel stops you. Most teams set both to the same made-up number and pay twice for it.
Sep 2026
How I built Mandli: 49 party games, one codebase, free voice chat
One Expo codebase for Android and web, a server that owns every rule, and self-hosted LiveKit for voice. What worked, what broke, and measured numbers.
Sep 2026
Invalid JSON from a model: what the retry actually costs
Retrying schema failures costs less than you think and hides more than you expect. What strict mode guarantees, what it quietly does not, and what to validate.
Sep 2026
Batch inference: the half of your work that can wait
Every provider prices batch at half. The discount is not the variable — how much of your work tolerates a 24-hour ceiling is, and whether it covers the queue.
Sep 2026
The tool tax: what every MCP server costs on every call
Tool definitions ride on every request and are never deduplicated. What 58 tools cost per call, and why caching settles the bill but not the context window.
Sep 2026
LLM gateways: when you need one, and when you really don't
What an LLM gateway gives you, what it costs in fees and in latency, and a checklist for choosing between a library, a self-hosted proxy and a hosted one.
Sep 2026
Migrating off a monolith with the strangler fig
How to move off a legacy monolith without a rewrite: the facade, what order to move modules in, a wave planner, and the reasons migrations stall.
Sep 2026
Agent step budgets: why a longer loop costs so much more
An agent resends its history on every step, so cost grows with the square of the step count. What a task costs, and where to put the caps.
Sep 2026
Build it or buy it: the three-year cost of both
Auth, billing, search or notifications: what building really costs over three years once maintenance is counted, and the test that comes before the money.
Sep 2026
Freelancer, agency, fractional CTO or a full-time hire
Four ways to build or fix a product, compared on published rates: what each delivers in six months, what each costs an hour, and how each one fails.
Sep 2026
LLM evals: how many test cases before a result means anything
Your eval went from 80% to 85% on 40 cases. That is noise. The sample size maths for LLM evals, with a calculator and the intervals nobody prints.
Sep 2026
RAG, fine-tuning or long context: which one your data needs
Retrieval, fine-tuning and a million-token context window solve different problems. A cost comparison, the break-even corpus size, and when each one fails.
Sep 2026
What an hour of downtime actually costs a startup
Surveys put an hour of downtime above $300,000. Here is the arithmetic for a startup instead: interrupted revenue, SLA credits, people, support and churn.
Sep 2026
A technical due diligence checklist for a raise or an acquisition
What investors and acquirers check in your tech: architecture, security, data, delivery, bus factor and cost. A 20-question scorecard and what to fix first.
Sep 2026
Prompt caching: what it saves on Anthropic, OpenAI and Gemini
How prompt caching works on all three big providers: minimum prefixes, lifetimes, write premiums, prompt ordering, and the hit rate where caching starts paying.
Sep 2026
How many partitions does a Kafka topic need?
The formula is max(t/p, t/c), but the number you pick is close to permanent: partitions can be added and never removed, and adding them re-maps your keys.
Sep 2026
Rate limiting: the token bucket and its two rivals
A token bucket forgives a burst and still holds the average. Fixed windows let a client send double at the boundary. A simulator, and the headers to send back.
Sep 2026
Postgres connection pool sizing: why ten beats a hundred
Eight app instances with pools of ten is eighty connections to a database that can usefully run eight. The sizing formula, max_connections and pgbouncer.
Sep 2026
WhatsApp template categories: why one sentence doubles the bill
Meta's own test for marketing, utility and authentication templates, what gets re-categorised, and an exercise: guess the category before you see the price.
Sep 2026
How many WebSocket connections can one Node server hold?
File descriptors, memory, the event loop or the link: one of them stops you first. Measured memory per idle connection, and a calculator for your own box.
Sep 2026
Tail latency and fan-out: when p99 becomes everyone's problem
A backend that is slow 1% of the time makes 63% of user requests slow when one request waits on a hundred of them. The fan-out maths, and how hedging fixes it.
Sep 2026
Little's law: sizing worker pools, queues and concurrency
One formula sizes worker pools, connection pools and queues: L = λW. What it does and does not tell you, why the last worker matters most, and a calculator.
Sep 2026
SMS or WhatsApp for OTPs: what a code costs by country
Twilio's SMS rates against Meta's WhatsApp authentication rates in eight countries, the fallback nobody budgets for, and a calculator for your own volume.
Sep 2026
Exponential backoff and jitter: how to stop retry herds
Retries without jitter arrive in waves and stretch a ten-second outage into four and a half minutes. The four backoff formulas, a simulator, and what to set.
Sep 2026
WhatsApp Cloud API or a BSP: when building it yourself pays off
Meta charges the same per message either way. The real choice is a provider's cut against building it yourself. Current BSP prices and a break-even calculator.
Sep 2026
Composite SLA: why five good uptime numbers make one bad one
Every service a request touches multiplies into your uptime. A composite SLA calculator, the published numbers for AWS, GCP, Azure and Twilio, and what to promise.
Sep 2026
How much it costs to build an AI chatbot in 2026, line by line
Agency guides quote anything from $5,000 to $300,000. Here is where the money actually goes, line by line, with an estimator and the monthly bill after launch.
Sep 2026
LLM cost per user: the maths that decides an AI feature's margin
Tokens per call, calls per action, actions per user: a calculator for what an AI feature costs each user, and why the heaviest 10% decide whether the plan works.
Sep 2026
The latency budget of a voice AI agent, in milliseconds
People answer each other in about 200 ms. Voice agents aim for 800 and often land past 1,400. A stage-by-stage latency budget, with a calculator for your own stack.
Sep 2026
What an AI phone agent really costs per minute in 2026
One minute of an AI phone call, priced part by part: line, speech-to-text, model and voice. Where building beats Vapi, Retell and Bland, and where it doesn't.
Sep 2026
Idempotency: making every retry safe in a distributed system
A timeout does not mean the request failed. How idempotency keys stop retries creating duplicate charges and orders, what to store, and a calculator for the damage.
Mar 2026
LLM observability in production: tracing every call, token and cost
Trace model calls with OpenTelemetry's GenAI conventions, attribute cost per user at 2026 prices, and size trace storage before prompt payloads become most of it.
Mar 2026
Logging everything and nothing useful: fixing the noise problem
Health checks and string soup fill your logs and hide the line you need. A log volume calculator, sampling that keeps every failure, and canonical log lines.
Mar 2026
Microservices too early: the tax you pay before you need them
Every service on a request path multiplies failure and slow tails. A readiness scorecard, the distributed tax in numbers, and when a modular monolith is the answer.
Mar 2026
No backpressure: how a fast producer crashes a slow consumer
A queue with no ceiling grows at the difference between two rates until the process dies. The arithmetic, a simulator, and the four places the extra work can wait.
Mar 2026
Queue backlog: why it never drains, and how to get it back to zero
A queue drains only by the capacity above its incoming rate. Work out drain time, message wait and consumers needed, and where Kafka's partitions stop you.
Mar 2026
Nothing matches that yet. Try another category or clear the search.