Payments are the use of Kafka where small configuration choices have the largest consequences: a write that is refused stops a payment, and a consumer that falls behind delays a fraud check. This walk-through follows a health check on a made-up bank's payments platform, using the sample engagement that ships with InsightKafka.
The environment
The fictional bank runs self-managed Confluent Platform on AWS: a multi-AZ cluster dedicated to payments and a separate fraud-detection
cluster. Because the bank operates its own brokers, it sets min.insync.replicas itself (managed services such as Confluent
Cloud fix or limit that value).
The payments cluster carries the transactions, settlements, notifications and audit topics, a compacted account-updates topic, and a
dead letter queue for failed payments, with Debezium change capture from the payments database and an S3 archive sink.

Finding 1 (critical): one broker restart stops payments
The payments cluster has min.insync.replicas=3 with a replication factor of 3. With acks=all, a write is
accepted only when every in-sync replica required has it. Requiring all three means a rolling restart, or one broker failure, refuses
writes on the affected partitions. The recommendation is to set it to 2: writes still need two copies, and the cluster tolerates one
broker down.
Finding 2 (high): a consumer group falling behind

The audit archiver has two consumers reading a six-partition topic, so each consumer carries three partitions and the group's lag grows. The fix is to scale the group to match the partition count; a consumer group cannot usefully have more members than partitions.
Findings 3 and 4 (high): schemas with no rule, failures with nowhere to go
- The fraud model's feature schema has compatibility set to NONE, so a producer can publish a change that breaks the fraud scoring consumer. BACKWARD, Schema Registry's default, only accepts a new version that can read data written with the previous one, so consumers can be upgraded first. If producers usually change first, FORWARD or FULL is the setting that protects consumers still on the old version; pick the mode to match the order in which the teams release.
- The fraud pipeline has no dead letter queue: a message the scorer cannot process is retried or dropped. A DLQ topic with the original message and the error keeps the pipeline moving and the failure visible.

| Finding | Severity | Recommendation | Effort |
|---|---|---|---|
| min.insync.replicas equals replication factor | Critical | Set min.insync.replicas to 2 | Hours |
| Audit archiver group lag | High | Scale the group to the partition count | Days |
| Schema compatibility NONE | High | Set compatibility to BACKWARD | Hours |
| No dead letter queue on the fraud pipeline | High | Add a DLQ topic and route failures to it | Days |
Why 2, and what it costs
With a replication factor of 3 and min.insync.replicas=2, every write acknowledged with acks=all is on at least
two brokers, and the partition keeps accepting writes with one broker down. With two brokers down, writes to the affected partitions are
refused until one returns; that is the price of not accepting writes with a single copy. Setting it to 1 would keep writes flowing longer,
but an acknowledged payment could then live on one broker only. Note that min.insync.replicas is both a broker default and a
topic setting, so check the payments topics' own overrides, and on a managed service check which values the provider allows.
Checking each fix
| Fix | How to confirm it worked |
|---|---|
| min.insync.replicas to 2 | Restart one broker at a time and confirm producers with acks=all keep writing |
| Scale the audit archiver | Members up to the partition count; lag falls and stays down through a busy period |
| Schema compatibility | Try registering a deliberately incompatible version in a test subject and confirm it is rejected |
| Dead letter queue | Send a malformed message in a test environment and confirm it lands on the DLQ with the error attached |
Trade-offs in the consumer fix
Scaling to six members gives one partition each, and a seventh member would sit idle. If six is not enough, the alternative is more partitions, but for a keyed topic that changes which partition new messages for a key go to, so per-key ordering across the change is not guaranteed. For an audit archive that may be acceptable; for a ledger it may not. The DLQ is similar: Kafka Connect sink connectors have a built-in dead letter queue setting, but a custom consumer such as a fraud scorer needs the pattern built into the application.
How to decide the order
The order in the read-out follows three questions, in this sequence. Can it stop or lose payments today? That is the replica setting, so it goes first, in the next maintenance window. Can it break a release that is already scheduled? That is the schema compatibility, so it goes before the next fraud-model release. Everything else (the consumer scaling, the DLQ) is real but slower-moving, and goes into planned work with an owner and a date. The effort column helps break ties, not set the order.
Questions
Could InsightKafka have found this without the configuration files?
Partly. The cluster and topic views hold replication settings if they are entered or parsed from a questionnaire, but the rule-based check needs the broker file. That is why the preparation guide asks for every broker's configuration.
Is a score of 72 good for a payments platform?
It is “fair” on the dashboard's bands. For payments, the single critical finding matters more than the number; a cluster can score in the 70s and still have one setting that stops writes.
The read-out
The fictional bank's platform architect gets a score of 72, fair, and a short list. The critical replica setting is changed in the same week, during a maintenance window, and checked with a rolling restart. The schema compatibility change is made before the next fraud-model release. The consumer scaling and the DLQ go into the next sprint. At the next quarterly health check, the same method and the same scoring show what moved.


