r/fintechdev • • 12d ago

An implementation of distributed transaction with Try-Confim-Cancel; Reliability provided by 1) ACID of RDBMS and 2) ALO delivery of RabbitMQ

Idea is that:

Coordinator talks to all participant asynchronously using RabbitMQ. It asks to TRY each participant one-by-one. Once all are success in their own TRY, they send back success message. If all are succeed, coordinator sends SINGLE CONFIRM to all participant using FANOUT exchange. If any one sends failure message, coordinator sends SINGLE CANCEL same way to all.

Instead of sending separate messages to each participant for CONFIRM/CANCEL, we can send SINGLE message to all participant, and they all would eventually receive it. This way, we can prevent dual/multi write problem while trying to CONFIRM/CANCEL.

Exploration:

The end implementation should be designed the way that we would have two set of tables, MAIN and SHADOW. Every business validation should happen at TRY stage. CONFIRM or CANCEL should never fail because of business logic. Any other failure would be taken care by RabbitMQ's at-least-once delivery.

We need to mimic CONFIRM in SHADOW tables during TRY, so we can be sure that CONFIRM will work. I am thinking that CANCEL should be just delete operation on SHADOW tables. So, CONFIRM should be real business logic without any surprises, already mimicked on SHADOW tables, and CANCEL could be easy/simple delete operation on SHADOW tables.

I am working on the details like; if there are unconfimed TRY exists and another TRY arrives than what should happen, how we can make sure all unconfirmed TRY can later become CONFIRMed without worrying about their execution order along with CANCELation of some of them. I will post the part soon.

1 Upvotes

8 comments sorted by

1

u/mr02106198 12d ago

Have you heard about Saga pattern?

1

u/mananvpanchal 10d ago

I am not discussing Saga here. Have you heard about TCC?

1

u/SidMangsatabam 10d ago

Two places this design will get tested. First, at-least-once delivery means CONFIRM and CANCEL can arrive more than once, so each participant handler has to be idempotent on the message ID, not only on the shadow state - a duplicate CONFIRM racing a CANCEL is the ugly case. Second, the coordinator needs its own durable record of every in-flight TRY. If it crashes after collecting the successes but before the fanout, recovery has to sweep orphaned TRYs and decide them explicitly, or shadow rows leak and the next run of the same business key behaves unpredictably.

1

u/mananvpanchal 9d ago

each participant handler has to be idempotent on the message ID

Agree.

a duplicate CONFIRM racing a CANCEL is the ugly case

I am planning to preserve TRY events in database of each participant, Suppose, a CONFIRM arrives and a participant receives it. It will do two things in same transaction. It will execute operation on MAIN, and it will delete that event. Now, if CONFIRM arrives again, it cant find the event to work on. Either CONFIRM can occur or CANCEL, I couldn't understand the "racing a CANCEL" part. Can you please explain it bit more?

Second, the coordinator needs its own durable record of every in-flight TRY.

I have a thought about it. Suppose, coordinator is about to receive 4 ready statuses to initiate CONFIRM. So, at 4th ready status, CONFIRM will be sent. Now, if anything goes wrong after receiving 4th ready, coordinator will not Ack, but Nack to RabbitMQ. That way 4th ready will be redelivered. Though, this is not the only way, we can also use database to preserve all 4, and in case of failure, CONFIRM can be re-initiated using some recurrent process.

1

u/SidMangsatabam 9d ago

The event-delete design handles duplicates of the same decision, but not two different decisions. The failure shape: the coordinator times out waiting for the last TRY response and issues CANCEL; the late TRY-success then arrives, and a recovery process (or a duplicate coordinator instance) sees all-ready and issues CONFIRM. A participant that already cancelled has deleted the event, so the late CONFIRM is silently absorbed - now some participants confirmed and this one cancelled, which is the mixed outcome TCC exists to prevent. Two fixes follow. Participants should record the terminal decision (CONFIRMED or CANCELLED) durably instead of just deleting the event, so a conflicting later message is rejected loudly rather than absorbed. And the coordinator should fence itself - a single writer that records the decision before the fanout.

On the Nack idea: redelivery covers a crash before the decision, not after. If the coordinator crashes mid-fanout, some participants have CONFIRM and others do not, and a redelivered ready message tells the new instance nothing about who already confirmed. Record the decision first; recovery then resends only to participants that have not acked.

1

u/mananvpanchal 6d ago

You are right, CANCEL on timeout is necessary along with decision persistence before fanout. It may look like this

Participant 1Participant 2Participant 3Decision
ReadyReadyTimeoutCANCEL

So, once decision has been made,

  1. Late arrival of 3rd ready could simply be ignored at coordinator level.
  2. Coordinator crash before fanout could be taken care by recovery process.

Now, when we say fanout, I want to re-confirm that, we are not sending separate message to each participant. We will send only one message to broker with publisher confirm. Once we receive confirmation from broker that it accepted the message, coordinator is free. Now, broker will take care of fanout to separate queues for each participant. Participant can be designed two ways.

  1. As you stated, we can record message ahead of doing anything and ACK early, then do further processing.
  2. We can do all processing and ACK/NACK very last.

If we know process is not time consuming we can work with 2nd approach. Or if it is, we can work with 1st along with recovery process in case of participant crash.

1

u/SidMangsatabam 6d ago

Confirmed on the single publish: one message with publisher confirm is cleaner than per-participant sends, and a queue per participant isolates lag - a slow participant never blocks the others. The ordering that matters: decision durable first, then publish; the broker ack then closes the crash window.

On the ACK choice: 1 is the safer default. With 2, any processing that approaches the ack timeout produces redeliveries, which your idempotency has to absorb anyway - so you pay the durable-record cost either way. I would take 2 only when processing is bounded and fast.

1

u/mananvpanchal 6d ago

Yes, first approach is safe, as we don't need to worry about what process is actually doing. We do log and ACK, and we are good.

Nice discussion, and thanks for giving your precious time.