All articles Use case

Financial services: a payments cluster that stops on one broker

A worked health check on a made-up bank's self-managed Confluent Platform payments cluster: a replica setting that blocks writes, a consumer group falling behind, schemas with no compatibility rule, and no dead letter queue.

November 11, 2026 · 6 min read

Payments are the use of Kafka where small configuration choices have the largest consequences: a write that is refused stops a payment, and a consumer that falls behind delays a fraud check. This walk-through follows a health check on a made-up bank's payments platform, using the sample engagement that ships with InsightKafka.

Example. Fenwick Harbor Bank, its people and its clusters are made up; the configuration and findings are the product's sample data. Your cluster's right settings depend on its workload: treat the fixes here as illustrations.

The environment

The fictional bank runs self-managed Confluent Platform on AWS: a multi-AZ cluster dedicated to payments and a separate fraud-detection cluster. Because the bank operates its own brokers, it sets min.insync.replicas itself (managed services such as Confluent Cloud fix or limit that value). The payments cluster carries the transactions, settlements, notifications and audit topics, a compacted account-updates topic, and a dead letter queue for failed payments, with Debezium change capture from the payments database and an S3 archive sink.

The cluster view for a made-up payments cluster: production, dedicated, AWS, us-east-1, multi-AZ; 12 brokers, 1,536 partitions, 78 topics, 28 consumer groups, 6 connectors; and a topic table with partitions, replication factor, retention, cleanup policy, compression and use case.
The made-up payments cluster: its size, and each topic's partitions, replication, retention, cleanup and compression.

Finding 1 (critical): one broker restart stops payments

The payments cluster has min.insync.replicas=3 with a replication factor of 3. With acks=all, a write is accepted only when every in-sync replica required has it. Requiring all three means a rolling restart, or one broker failure, refuses writes on the affected partitions. The recommendation is to set it to 2: writes still need two copies, and the cluster tolerates one broker down.

Finding 2 (high): a consumer group falling behind

Consumer groups on the made-up payments cluster: payment processor, notification sender, audit archiver and DLQ retry processor, with members, total lag, state and assignment strategy; the audit archiver's lag is highlighted.
Consumer groups and their lag. The audit archiver has two members for six partitions and is falling behind.

The audit archiver has two consumers reading a six-partition topic, so each consumer carries three partitions and the group's lag grows. The fix is to scale the group to match the partition count; a consumer group cannot usefully have more members than partitions.

Findings 3 and 4 (high): schemas with no rule, failures with nowhere to go

  • The fraud model's feature schema has compatibility set to NONE, so a producer can publish a change that breaks the fraud scoring consumer. BACKWARD, Schema Registry's default, only accepts a new version that can read data written with the previous one, so consumers can be upgraded first. If producers usually change first, FORWARD or FULL is the setting that protects consumers still on the old version; pick the mode to match the order in which the teams release.
  • The fraud pipeline has no dead letter queue: a message the scorer cannot process is retried or dropped. A DLQ topic with the original message and the error keeps the pipeline moving and the failure visible.
Recommendations for the made-up bank: high-priority items to reduce min.insync.replicas to 2, scale the audit archiver group, set schema compatibility to BACKWARD and add a dead letter queue, each with category, effort and implementation steps.
The high-priority recommendations, each with its category, effort and implementation steps.
FindingSeverityRecommendationEffort
min.insync.replicas equals replication factorCriticalSet min.insync.replicas to 2Hours
Audit archiver group lagHighScale the group to the partition countDays
Schema compatibility NONEHighSet compatibility to BACKWARDHours
No dead letter queue on the fraud pipelineHighAdd a DLQ topic and route failures to itDays

Why 2, and what it costs

With a replication factor of 3 and min.insync.replicas=2, every write acknowledged with acks=all is on at least two brokers, and the partition keeps accepting writes with one broker down. With two brokers down, writes to the affected partitions are refused until one returns; that is the price of not accepting writes with a single copy. Setting it to 1 would keep writes flowing longer, but an acknowledged payment could then live on one broker only. Note that min.insync.replicas is both a broker default and a topic setting, so check the payments topics' own overrides, and on a managed service check which values the provider allows.

Checking each fix

FixHow to confirm it worked
min.insync.replicas to 2Restart one broker at a time and confirm producers with acks=all keep writing
Scale the audit archiverMembers up to the partition count; lag falls and stays down through a busy period
Schema compatibilityTry registering a deliberately incompatible version in a test subject and confirm it is rejected
Dead letter queueSend a malformed message in a test environment and confirm it lands on the DLQ with the error attached

Trade-offs in the consumer fix

Scaling to six members gives one partition each, and a seventh member would sit idle. If six is not enough, the alternative is more partitions, but for a keyed topic that changes which partition new messages for a key go to, so per-key ordering across the change is not guaranteed. For an audit archive that may be acceptable; for a ledger it may not. The DLQ is similar: Kafka Connect sink connectors have a built-in dead letter queue setting, but a custom consumer such as a fraud scorer needs the pattern built into the application.

How to decide the order

The order in the read-out follows three questions, in this sequence. Can it stop or lose payments today? That is the replica setting, so it goes first, in the next maintenance window. Can it break a release that is already scheduled? That is the schema compatibility, so it goes before the next fraud-model release. Everything else (the consumer scaling, the DLQ) is real but slower-moving, and goes into planned work with an owner and a date. The effort column helps break ties, not set the order.

Questions

Could InsightKafka have found this without the configuration files?

Partly. The cluster and topic views hold replication settings if they are entered or parsed from a questionnaire, but the rule-based check needs the broker file. That is why the preparation guide asks for every broker's configuration.

Is a score of 72 good for a payments platform?

It is “fair” on the dashboard's bands. For payments, the single critical finding matters more than the number; a cluster can score in the 70s and still have one setting that stops writes.

Example · Fenwick Harbor Bank

The read-out

The fictional bank's platform architect gets a score of 72, fair, and a short list. The critical replica setting is changed in the same week, during a maintenance window, and checked with a rolling restart. The schema compatibility change is made before the next fraud-model release. The consumer scaling and the DLQ go into the next sprint. At the next quarterly health check, the same method and the same scoring show what moved.

Next: a retailer's migration readiness assessment.

Run your next health check in InsightKafka

Bring one cluster's broker configuration, its topic and consumer-group listing and the questions your team keeps asking. We set up the engagement and walk through the findings with you.

Request a demo