Queue-based fan-out across seven channels, designed to dispatch more than one million notifications every day.
Inspect the architecture
One ingress contract. Independent delivery machinery behind every route.
FAN-OUT / 7 CHANNELSThe service had to accept traffic quickly, isolate slow providers, preserve intent, and explain every outcome without coupling clients to channel-specific APIs.
The public surface is AWS SNS-style: publish once, then let durable delivery policy fan the intent out to selected destinations.
Delivery contract
A durable queue sits between acceptance and delivery. Every stage owns one responsibility, so provider latency never becomes API latency.
Validates the request, audience, template, schedule, and idempotency key.
Stores notification intent and an outbox event in the same transaction.
Moves due work into priority lanes without tying the API to provider latency.
Claim work, enforce limits, render content, and invoke the selected adapter.
Translate one internal contract into each provider-specific payload.
Records attempts, receipts, failures, latency, and dead-letter transitions.
The adapter returns a provider reference, the delivery is marked sent, and later receipts advance it to delivered when the channel supports confirmation.
Transient failures are deferred with backoff. Permanent failures and exhausted attempts are isolated in the dead-letter queue.
The orchestration layer works with a normalized delivery request. Adapters own provider credentials, payload translation, and response classification.
Customer-facing destinations with provider-specific payload rules.
HTML, text, subject, attachments
SMS
Segment-aware text delivery
Push
Device tokens and platform payloads
Approved templates and variables
Operational messages shaped for team collaboration tools.
Slack
Workspace and channel routing
Teams
Tenant-aware message cards
Machine-to-machine callbacks with signed payloads.
Webhook
HTTP delivery with response capture
Adding a provider changes one adapter, not the publish API, queue contract, retry policy, scheduler, or analytics pipeline.
The worker classifies every provider response before deciding whether to acknowledge, defer, or isolate the delivery.
claim(next_priority)
if duplicate(delivery_key):
acknowledge()
if limited(tenant, channel):
defer(until_capacity)
result = adapter.send(render(template_version))
if result.accepted:
record(provider_reference)
elif result.retryable:
schedule(backoff(attempt) + jitter)
else:
dead_letter(reason, payload_snapshot)delay = min(base x 2^attempt + jitter, cap)
Jitter keeps recovering workers from retrying the same provider at the same instant. A cap prevents old work from disappearing for too long.
Retry
Timeouts, temporary provider errors, and throttling responses.
Stop
Invalid destinations, rejected content, and permanent authentication failures.
A stable delivery key prevents a claimed message from producing duplicate provider requests.
Retry policy is explicit per channel and stops after the configured attempt budget.
The final payload, reason, and attempt history remain available for replay or diagnosis.
Scheduling, priority, and rate limiting meet before delivery. Work leaves only when it is both due and allowed.
High
Security, authentication, and time-sensitive alerts
Normal
Transactional product and account messages
Bulk
Campaign and high-volume informational traffic
Workers reserve capacity across lanes so sustained high-priority traffic cannot starve normal or bulk delivery indefinitely.
Future work stays durable with its due time. A scheduler promotes only due deliveries into active queues.
Limits can be keyed by tenant, channel, and provider to protect shared capacity and external quotas.
Queue depth absorbs bursts while worker concurrency scales independently for each channel.
Per-tenant controls stop one noisy publisher from consuming every delivery slot.
The data model keeps the original request stable while each channel accumulates its own attempts, receipts, and terminal state.
Notification
Audience, template version, variables, requested channels, priority, and schedule
Delivery
One channel destination with its current state and idempotency key
Attempt
Immutable provider request, response class, latency, and retry decision
Receipt or DLQ
Confirmed outcome, or the complete failure context retained for review
State transitions emit analytics events instead of making reporting queries compete with the delivery path.
1M+
notifications per day is a capacity target, not an average-load shortcut.
Daily totals hide bursts, provider slowdowns, and tenant hotspots. The queue and worker model absorbs each without extending publish latency.
Publish stays a short durable write. Channel worker pools expand independently as queue depth changes.
Durability wins over fragile exactly-once claims. Stable keys make repeated processing safe.
Ordering is preserved only where a recipient flow requires it, keeping unrelated deliveries parallel.
A queued notification keeps the selected template version, so later edits cannot change an in-flight message.