Preserve settlement decision across restart for pending escrows #1906
Problem
The current settlement flow works as expected while the process stays alive, but there is a fault-tolerance gap around restarts.
When an escrow is marked for settlement, only SettlementPending is persisted. We do not persist:
- why the settlement mark was created;
- whether settlement was enabled for that model at that moment;
- whether the settlement was triggered automatically or manually.
After a restart, reconciliation loads the current configuration and decides again whether the escrow should be settled.
This means the behavior before and after a restart can differ.
| Situation | Process stays up | After restart |
|---|---|---|
Settlement is already marked, then model M is disabled |
Drain still settles it because the flag is not checked again. | Reconcile reads the new flag, skips settlement, and keeps the pending mark. |
Model M was disabled when the escrow was retired |
Nothing is queued and no pending mark is created. | There is nothing to resume. Enabling M later does not settle that escrow. |
A pending mark exists, then M.settlement_enabled is omitted or M is removed from models |
An already queued settlement still runs. | Reconcile no longer sees the model override. If the global flag is enabled, it settles. |
Manual POST .../settle returned 409 because requests were still in flight |
Drain eventually settles it. | Reconcile treats it as an ordinary pending mark. If M is disabled, the manually requested settlement is never completed. |
Expected behavior
Once a settlement decision has been made and persisted, restarting the process should not change that decision based on newer configuration.
Reconciliation should be able to distinguish between:
- an escrow that was explicitly marked for settlement;
- an escrow that should only be settled if the current configuration allows it.
Manual settlement requests should also survive a restart and eventually complete once the escrow becomes drainable.
Possible approach
Persist enough context together with SettlementPending so reconciliation can resume the original settlement intent instead of recalculating it from the current model settings.
For example, the persisted state could include the settlement source/reason or the effective settlement decision at the time the mark was created.
Original posted by @a-kuprin https://github.com/gonka-ai/gonka/issues/1866#issuecomment-5921201295
🔄 Auto-synced from Issue #1906 every hour.