mirror of
https://github.com/simstudioai/sim.git
synced 2026-11-03 07:14:25 +08:00
Waleed
f9eaadfc70
The sweep treats failed as terminal — "the run that produced it has ended" — and that was true when processing ran inline. Since dispatch became asynchronous the worker writes failed and then rethrows, and the processing task retries up to three times, so failed is a resting state between attempts rather than a final one. The sweep gave it no grace at all, so it reclaimed a document mid-retry chain, deleted its embeddings, and re-dispatched with a fresh pass id: two live runs, two indexing passes, two bills, through the one state deliberately left unguarded. Aged from processingCompletedAt, which every failure write stamps, so it reads exactly when the last attempt ended rather than when the document was dispatched. Sized at the queue grace rather than a run-duration bound. What is being waited on between attempts is a re-queue behind the same global concurrency limit, not another run, so on a large backlog the next attempt starts hours after the previous one ended and a run-duration bound would still reclaim live work. The grace itself is now an expression rather than a constant — corpus over queue concurrency, times occupancy, times a contention factor — with each input documented where it can be re-measured. The previous 240 was derived from a 2,600-document corpus and sat below the drain time of the 7,730-document connector it was written for. Rounding is up: too small silently re-bills live work, too large only delays recovery of documents nothing is processing. Also withdraws the queue stamp when every dispatch in a batch fails, so a provably-undispatched document keeps next-sync recovery instead of waiting out a grace it never earned. Scoped to the batch's own ids, to rows still pending, and by compare-and-set on the exact stamp that call wrote, so a concurrent batch that re-stamped a document keeps its grace. Best-effort by design, while the stamp write still throws: a failed stamp means the grace cannot be promised and dispatching anyway is the unsafe direction, whereas a failed withdrawal only delays recovery and must not mask the dispatch error underneath it.