One publish API in front of seven delivery channels, designed for more than a million notifications a day. The live model runs at exactly that rate: 11.6 publishes a second.
Live model11.6 publishes/s
Autopilotsteady traffic at the design rate.Any control takes over.
The service had to accept traffic quickly, isolate slow providers, preserve intent, and explain every outcome without coupling clients to channel-specific APIs.
The public surface is AWS SNS-style: publish once, then let durable delivery policy fan the intent out to selected destinations.
Delivery contract
A login OTP for user_42, sent to SMS and email at high priority. Each stage writes its own records, so intent, deliveries and attempts stay separate and every outcome can be explained.
01
The request names a template version, an audience, channels and a priority, plus an idempotency key.
Validation runs here. The answer is 202 as soon as intent is stored, before any provider is called, so a slow SMS gateway never becomes a slow API.
POST /notifications
Idempotency-Key: 9c1e-42
{ "template": "login-otp@v3", "to": "user_42",
"channels": ["sms", "email"], "priority": "high" }← 202 Accepted · n_7f3a · 3 ms
No records yet: nothing is stored until validation passes.
02
The notification row and an outbox event are written in one transaction.
A crash can't leave intent without the event that announces it, or the reverse. The template version is frozen on the row, so a later edit can't change this message.
| id | template | priority | state |
|---|---|---|---|
| n_7f3a | login-otp@v3 | high | accepted |
| event | subject | relayed |
|---|---|---|
| notification.accepted | n_7f3a | no |
03
The router reads the outbox and creates one delivery per channel, keyed n_7f3a:sms and n_7f3a:email.
Both join the high lane. From here each delivery has its own state and its own retries: SMS failing says nothing about email.
| event | subject | relayed |
|---|---|---|
| notification.accepted | n_7f3a | yes |
| id | channel | lane | state |
|---|---|---|---|
| d_1 | sms | high | queued |
| d_2 | high | queued |
04
A worker claims d_1, checks the tenant's SMS rate limit and renders the frozen template.
The attempt is recorded before the provider is called. If the worker dies mid-call, the claim expires and the stable key stops a second worker from sending a duplicate.
| id | channel | lane | state |
|---|---|---|---|
| d_1 | sms | high | sending |
| d_2 | high | queued |
| id | delivery | result | latency |
|---|---|---|---|
| a_1 | d_1 | in flight |
05
The SMS provider answers 503. The adapter classifies it as retryable, and d_1 waits on backoff.
Email is accepted with a provider reference in the meantime. A 400 would have been classified as permanent and sent d_1 straight to the dead-letter queue.
| id | channel | lane | state |
|---|---|---|---|
| d_1 | sms | high | retry · 1.2s |
| d_2 | high | sent |
| id | delivery | result | latency |
|---|---|---|---|
| a_1 | d_1 | 503 · retryable | 412 ms |
| a_2 | d_2 | 202 · em_88e1 | 186 ms |
06
The second SMS attempt is accepted, and a receipt confirms delivery later.
Every state change is also an event. Analytics read the stream for volume, latency, retry and DLQ rates, never the delivery tables the workers are writing to.
| id | channel | lane | state |
|---|---|---|---|
| d_1 | sms | high | delivered |
| d_2 | high | delivered |
| id | delivery | result | latency |
|---|---|---|---|
| a_1 | d_1 | 503 · retryable | 412 ms |
| a_2 | d_2 | 202 · em_88e1 | 186 ms |
| a_3 | d_1 | 200 · sms_51c | 298 ms |
| event | subject |
|---|---|
| attempt.failed | a_1 · transient |
| delivery.sent | d_2 · email |
| delivery.delivered | d_1 · sms |
| receipt | d_2 · email |
POST /notifications
Idempotency-Key: 9c1e-42
{ "template": "login-otp@v3", "to": "user_42",
"channels": ["sms", "email"], "priority": "high" }← 202 Accepted · n_7f3a · 3 ms
No records yet: nothing is stored until validation passes.
The orchestration layer works with a normalized delivery request. Adapters own provider credentials, payload translation, and response classification.
Customer-facing destinations with provider-specific payload rules.
HTML, text, subject, attachments
SMS
Segment-aware text delivery
Push
Device tokens and platform payloads
Approved templates and variables
Operational messages shaped for team collaboration tools.
Slack
Workspace and channel routing
Teams
Tenant-aware message cards
Machine-to-machine callbacks with signed payloads.
Webhook
HTTP delivery with response capture
Adding a provider changes one adapter, not the publish API, queue contract, retry policy, scheduler, or analytics pipeline.
The worker classifies every provider response before deciding whether to acknowledge, defer, or isolate the delivery.
claim(next_priority)
if duplicate(delivery_key):
acknowledge()
if limited(tenant, channel):
defer(until_capacity)
result = adapter.send(render(template_version))
if result.accepted:
record(provider_reference)
elif result.retryable:
schedule(backoff(attempt) + jitter)
else:
dead_letter(reason, payload_snapshot)delay = min(base x 2^attempt + jitter, cap)
Jitter keeps recovering workers from retrying the same provider at the same instant. A cap prevents old work from disappearing for too long.
Retry
Timeouts, temporary provider errors, and throttling responses.
Stop
Invalid destinations, rejected content, and permanent authentication failures.
A stable delivery key prevents a claimed message from producing duplicate provider requests.
Retry policy is explicit per channel and stops after the configured attempt budget.
The final payload, reason, and attempt history remain available for replay or diagnosis.
Scheduling, priority, and rate limiting meet before delivery. Work leaves only when it is both due and allowed.
High
Security, authentication, and time-sensitive alerts
Normal
Transactional product and account messages
Bulk
Campaign and high-volume informational traffic
Workers reserve capacity across lanes so sustained high-priority traffic cannot starve normal or bulk delivery indefinitely.
Future work stays durable with its due time. A scheduler promotes only due deliveries into active queues.
Limits can be keyed by tenant, channel, and provider to protect shared capacity and external quotas.
Queue depth absorbs bursts while worker concurrency scales independently for each channel.
Per-tenant controls stop one noisy publisher from consuming every delivery slot.
The live model shows the flow at volume. Here every delivery has a name and the pace is slow enough to follow: pick a scenario or publish your own, then break a provider and watch its deliveries retry and dead-letter.
A browser simulation of this design: three priority lanes, four workers and seven adapters, with modelled latencies.
Free workers claim 3 high : 2 normal : 1 bulk. An empty lane passes its turn on.
Nothing waiting. Transient failures are parked here until their backoff ends.
Empty. Permanent failures and exhausted retries land here with their reason.
1M+
notifications per day is a capacity target, not an average-load shortcut.
Daily totals hide bursts, provider slowdowns, and tenant hotspots. The queue and worker model absorbs each without extending publish latency.
Publish stays a short durable write. Channel worker pools expand independently as queue depth changes.
Durability wins over fragile exactly-once claims. Stable keys make repeated processing safe.
Ordering is preserved only where a recipient flow requires it, keeping unrelated deliveries parallel.
A queued notification keeps the selected template version, so later edits cannot change an in-flight message.