Getting observability right in fintech payments in 2026

Your payment dashboards are green. A customer can STILL be charged twice.

That sounds contradictory, right? It is not.

Let’s say it is 00:05 on the first of the month. Payday. Your payment volume doubles, a PSP starts taking a little too long to answer, and a customer taps the spinning Pay button again. The gateway returns a 200. The orchestrator reports two successful attempts. Your authorization rate has not moved even a basis point. Nothing pages anybody.

But the customer has paid twice.

The cost is not just one refund. Support discovers the issue before engineering does. Finance later finds a settlement file that does not match the ledger. Your on-call engineer spends the next few hours opening five dashboards and trying to make eight partial stories agree. And at some point, an auditor asks the annoying but correct question: what did your systems actually do that night?

The mistake is thinking that healthy services mean a healthy payment. They do not. A payment is one business event crossing a bunch of asynchronous boundaries, and a dashboard that tells the truth about one service can still hide the truth about the money.

This article is about making that truth obvious. We will follow one payment through the stack, look at the failures service-level dashboards miss, and then see how a payment investigation should work when you need an answer before the morning settlement file.

TL;DR

1. A payment is not one request. It is a chain of handoffs.
2. Instrument the handoffs and preserve the identifiers, not just the services.
3. Keep money-path traces at full fidelity. They are evidence.
4. Keep card data out of telemetry before it leaves your network.

Alright, which brings me to my first point.

1. A payment is not a request

Most payment stacks converge on three tiers. Adyen’s write-up describes them nicely: a stateless edge where latency is important, an asynchronous accounting layer where correctness is important, and a batch reporting layer where throughput is important.

Now follow one card payment.

The app sends an idempotency key to the gateway. Card details go to your hosted capture, then come back as a token. The orchestrator owns the payment state machine and picks a processor adapter. The adapter calls the PSP. The PSP sends one answer now, the authorization response, and another answer later, through a webhook that lands in a queue. Your ledger then writes double-entry rows in integer cents. Overnight, a reconciliation job compares those rows with the settlement file.

That is not one request. That is a journey.

Think of it like a parcel changing hands between courier companies. Every courier can honestly tell you, “I handed it to the next person.” Very helpful. But none of those statements tells you where the parcel is now, whether it was handed over twice, or whether it arrived at the warehouse that is supposed to reconcile the inventory.

This is what a per-service dashboard does to a payment. It gives you a correct local statement, not a coherent global one.

And that is the error most teams make. They assume they have payment observability because every service has logs, metrics and a dashboard. What they actually have is service observability. Different thing.

2. What healthy dashboards cannot tell you

Let us look at three common failures.

Duplicate charges

A standard authorization-rate dashboard cannot see a duplicate charge. Both attempts can succeed. The gateway sees valid requests. The PSP sees valid authorizations. The orchestrator might see two independent payment attempts.

So the dashboard says: 100% success. Congratulations, you have successfully charged the customer twice.

What you need here is one span per PSP attempt, with the attempt number and idempotency key attached, plus a counter for payments charged more than once. The useful question is not “did the PSP succeed?” It is “how many successful PSP attempts belong to this one payment?”

Silent webhook loss

This one is worse because nothing has to fail loudly.

The PSP has authorized a payment. Now two different things can be wrong: the webhook reaches your queue but the consumer is stalled, or the webhook never reaches your queue at all. Error rate stays flat because no request returned a 500. Meanwhile, your payment state has stopped moving.

The age of the oldest unacknowledged event catches the first problem, stalled consumption and backlog. To catch a webhook lost before the queue, keep a timer or metric for payments awaiting PSP confirmation. A queue that is becoming old is telling you the truth. A request-error dashboard is not.

Ledger drift

The batch job runs overnight and prints done. Nice. It proves almost nothing.

What matters is the number of unmatched rows, grouped by break reason: PSP settled but ledger entry missing, ledger entry exists but settlement row missing, amount mismatch, currency mismatch. That number should belong to a person or team. Cron output belongs to nobody.

These three problems look unrelated when you see them as metrics. They become the same problem when you see them as a payment crossing boundaries: the story broke at a handoff.

3. Instrument the handoffs, not just the services

What does that actually mean?

Every log, trace and event related to a payment needs the identifiers that let you join the story back together: trace_id where it exists, payment_id, idempotency_key, merchant_id, psp_reference and attempt_number.

The important part is context propagation across the HTTP and queue hops you control. An external PSP webhook and an overnight batch job begin new traces. That is fine. Link them back to the payment with payment_id, psp_reference and the other business identifiers. A trace ID is useful inside one controlled flow; it is not magic that crosses every boundary.

You see, the point is not to create more telemetry. The point is to make the telemetry answer a business question: “What happened to this payment?”

Let’s say a support agent sends you a payment ID. You should be able to start there and see the client request, the orchestrator state transition, every PSP attempt, the webhook, the ledger rows and the reconciliation result. One path. One story. No detective wall with red string.

You can build that model with OpenTelemetry and your existing stack. Once the model is there, Osuite can join logs to traces on trace_id through log and trace correlation, then use the payment identifiers to move between trace roots. A log line becomes the entry point to the whole payment journey. This is the kind of boring correlation that humans do slowly at 00:40, with too many tabs open.

4. A small incident that becomes a very expensive morning

This is a fictional incident. The failure pattern is not.

Let us go back to payday.

The PSP’s p99 has moved past the 10-second timeout on your orchestrator’s HTTP client. The orchestrator retries. At the same time, the customer sees the Pay button spin for ten seconds and taps it again. This is what people do with a button that spins. The client makes a fresh idempotency key for every tap.

Nothing pages.

Authorization rate is normal because the retries succeed. Error rate is flat because a 200 is a 200. Each dashboard is correct about its own service. Collectively, they are useless.

At 00:40, support gets the first report: “I was charged twice.”

This is not made up as a pattern. Monzo’s postmortem of 29 July 2019 describes engineers looking at healthy metrics while card payments failed, until someone said they were seeing “conflicting evidence.” If you have worked on payments, you know this feeling. The metrics say one thing, the customer says another, and both are somehow true.

The old investigation is three terminals, a few dashboards, a queue browser, a database query and then a wait for the settlement file. Finance often gets there before engineering.

The better investigation begins with the payment, not the service.

An on-call engineer opens their IDE and asks /osuite-investigate customers reporting duplicate charges since midnight. The agent groups spans by payment_id, looks for payments with attempt_number greater than one and finds multiple 2xx responses from the PSP adapter under the same payment. Those attempts carry different idempotency keys from the same device fingerprint.

That is a candidate list, not proof of duplicate charges. A 2xx from the adapter means a successful attempt, not necessarily a second captured or settled charge. So the next check is the authoritative PSP charge or capture identifier and its confirmed status. If two confirmed charges map to one payment, now the cause is obvious: the client is regenerating the key on every tap, and the timeout is creating the tap. You do not need a 40-minute debate about whether the PSP is flaky. You can see the sibling attempts under one payment.

The proposed fix is still for a human to review: one idempotency key per payment intent, reused across taps, plus backoff with jitter. Stripe’s idempotency post has made the case for that pattern for years.

Two useful things fall out of this immediately.

First, confirmed duplicate charges become a refund-review queue; the payment IDs are investigation candidates until then. Support can start fixing the customer problem before settlement lands. Second, the duplicate-charge counter becomes a PromQL alert, so the next trend pages somebody before the first support ticket arrives.

That is the difference. You are not getting a prettier dashboard. You are getting the first true sentence about what happened, while it is still useful.

5. Keep the money path. Sample the rest.

Tracing people will now say, “But what about cost? Surely we sample the traces?”

Yes. But not all of them the same way.

A payment trace is not just a performance artifact. It is evidence: for debugging, customer disputes, reconciliation and sometimes compliance. If you sample away the payment that went wrong, you have saved a few bytes and removed the only useful record of the incident. That is bad design.

Razorpay’s engineers ruled out SaaS APM partly because a fintech needs to replay historical traces for debugging and compliance. The practical rule is simple: keep 100% of money paths and sample the rest.

This is not a claim that every trace in your system deserves full fidelity. Your health checks, static assets and noisy internal endpoints are not the same as a movement of money. Do not make the whole platform expensive because you need evidence for one path.

Osuite prices on stored GB rather than ingest. So if you keep every payment trace, that is a deliberate storage decision, not a budget fight in the middle of an incident. Its agents run in your IDE — four of them, unlimited on every tier, with $0 inference — to help with the correlation work. Faster context for the human. Not automatic triage.

If your services already emit OpenTelemetry, getting the data in is just repointing an exporter at ingest.<region>.osuite.io:443, not a re-instrumentation project. You can be live in under an hour.

6. How to NOT put card data in your telemetry

Back to the card boundary.

Card capture, the token vault and the PSP adapter touch the actual card number. Everything downstream should work on tokens. That separation is useful only if telemetry follows it.

If a PAN reaches your log store, the log store is now inside the cardholder data environment. Its indexes, its backups and every engineer who can query it are suddenly part of your security problem. No amount of “we will be careful” makes that a good architecture.

So drop the field at the collector, before anything leaves your network. log processing pipelines covers the processors that do this.

The opposite mistake is also common: teams are so worried about leaking a card number that they turn logging down to nothing. Then a payment incident happens and everybody is blind for a year. Do not do that either. Keep the evidence. Remove the card data.

The rest is the usual questionnaire, but it matters. Your telemetry stays in your account’s region and is never routed across regions (security and privacy). Access to production telemetry is per-user, and users can sign in through your own OIDC provider. Logs can sync to your own S3-compatible bucket on Enterprise plans, set up by the Osuite team on request (data retention). There is also fully managed on-prem for teams whose security review requires it.

The actual goal

The expensive part of a payment incident is rarely the first broken request. It is the silent time after it: before the support ticket, before you can state the cause, before refunds start, before finance tells you the settlement does not match your ledger.

You do not fix that by adding a sixth dashboard. You fix it by making the eight handoffs one queryable story, keeping the money path at full fidelity and making sure the story does not contain card data.

If this is your stack, and you want to walk through the hops, the boundary and what is worth keeping, Talk to us. Thanks for reading.