Clearance
Problem
While working through a payments serializer for an issue, I realized my understanding of how payment systems actually operate was kinda shallow. I could follow the immediate request path, but once the work became asynchronous, I did not have a strong mental model for what happened next.
Clearance was me pushing on that gap by shaping a payments system where retries, duplicate delivery, failure, and recovery matter more than the happy path.
Core decision
The transaction service writes the transaction, idempotency record, and outbox
event in one database transaction. A separate publisher drains pending outbox
rows with FOR UPDATE SKIP LOCKED and publishes them to Redpanda. Risk and
ledger consumers follow the same pattern: record the consumed event and their
database effects, then write the successor event in that transaction.
That removes the classic gap where the database commit succeeds but the event publish fails.
Reliability model
Clearance uses request idempotency keys, payload-conflict detection, stable event IDs, payload hashes, retries, dead-letter handling, correlation IDs, balanced immutable ledger entries, and account-keyed Kafka partitions.
Kafka delivery is still at least once. If a publish succeeds and the outbox row does not get marked, that event can be published again. Consumers make replay of the same event ID and bytes a successful no-op, while the same ID with different bytes is rejected. So the useful guarantee is effectively-once database effects for a stable event identity, not exactly-once Kafka and definitely not duplicate-free topics.
Failures that exhaust consumer retries are persisted with the exact original key, headers, payload bytes, and source coordinates before the Kafka dead-letter publish. The admin CLI can inspect and replay that record, requeue a dead-lettered outbox row, or prune old processed-event records in a bounded batch. Replay and pruning fail closed around the configured replay window, and every destructive operator action records a reason.
The repo also has typed Prometheus metrics, bounded labels, provisioned alerts, and a Grafana operations dashboard. More importantly, the CI failure suite uses a real Redpanda instance, deliberately takes it down, recovers the outbox, sends duplicates and malformed bytes, then checks that the ledger and durable recovery records still hold up.
Security model
The public HTTP surface uses separate static bearer values for transaction,
funding, and operator access. Comparisons use the constant-time primitive, request
bodies are capped, unknown JSON fields are rejected, correlation IDs are
allow-listed, SQL is parameterized, CORS is an explicit origin list, and Redis
rate limiting fails closed when its dependency is unavailable. Responses mask
internal errors and set no-store, nosniff, and deny framing.
The separation is useful, but it is not an identity system. One transaction credential can read any transaction ID it knows, and the operator credential can query across accounts. There are no customer identities, tenant claims, scoped service credentials, or policy engine here. The funding endpoint is also a trusted demo rail with an operator reason and idempotency controls, not proof that money moved through a bank or network.
Local service traffic to PostgreSQL, Redis, and Redpanda is unencrypted, and the optional metrics endpoint is not authenticated by the app. Compose binds the operational tools locally, but a real deployment would still need TLS, secret rotation, metrics authentication, broker authorization, and network isolation.
Tradeoffs
Honestly, this architecture would be overkill for the amount of business logic in Clearance, and that is kinda the point. The risk model is basically a threshold. Splitting it across services makes the product worse if the goal is to ship a tiny payment demo, but it makes the failure boundaries visible, which was the actual thing I wanted to learn.
The services share PostgreSQL. That keeps the important state transitions and operator views straightforward, and it lets each consumer atomically store its effect with the next outbox event. It also means these are focused process boundaries, not independently owned services with isolated data stores. If each service needed independent scaling or deployment ownership, that shared database would become the constraint pretty quickly.
At-least-once delivery is another deliberate trade. I could call the processed event table exactly once and make the diagram sound cleaner, but that would hide where it breaks. Stable IDs only make the database effect effectively once for the same bytes. They do not stop duplicate messages, and retention only stays safe while processed-event history outlives the replay window.
Recovery is CLI-first and retention needs an external scheduler. That keeps destructive operations out of the public HTTP surface, but it also means the system needs a trusted operator and a runbook. The single local Redpanda node and Compose environment prove application recovery, not broker high availability or production operations.
Outcome
The useful outcome is not that Clearance can approve a fake payment. The useful outcome is that I can send the same event twice, kill the broker between durable write and publication, inject garbage into a consumer, and still explain exactly what state survives, what retries, what gets quarantined, and how an operator recovers it without duplicating ledger effects.
It also changed how I think about reliability. An outbox does not magically make a system reliable, right? It moves the dangerous gap into a place you can inspect and retry. Then you still need stable identities, atomic consumer effects, payload-conflict detection, retention rules, observability, and a recovery path that does not make the incident worse.