The MongoDB Race We Found Without Transactions

MongoDB multi-document transactions exist. We were not using them on this path — single-document updates, high throughput, "it should be fine."

It was not fine.

The symptom

A status field (think: PENDING → PROCESSING → DONE) occasionally jumped backward. Ops saw a record flip to DONE, then minutes later sit in PROCESSING again. No deploy. No manual edit. Just ghosts.

Logs showed two workers claiming the same work item within the same second.

How we detected it

1. Correlated traces — both workers logged workItemId + readStatus + writeStatus
2. A counter that did not add up — "started" events > "completed" events for the same id within a window
3. A reproduction — two parallel find → mutate → save loops against a local replica set

The smoking gun was the classic lost update:

• Worker A reads status: PENDING, version: 3
• Worker B reads the same
• A writes PROCESSING
• B writes PROCESSING (or worse, overwrites A's later fields) based on a stale read

Without a version check in the write filter, the second write wins silently.

Why "just use a transaction" was the wrong first move

Transactions would have papered over a design smell: we were doing read-modify-write in application memory for a hot single document. The fix that matched the access pattern was document-level optimistic concurrency.

The fix

Add a monotonic version (or reuse a well-understood field) and make the update conditional:

jsconst result = await collection.findOneAndUpdate(
  { _id: id, status: "PENDING", version: expectedVersion },
  {
    $set: { status: "PROCESSING", lockedBy: workerId, lockedAt: new Date() },
    $inc: { version: 1 },
  },
  { returnDocument: "after" }
);

if (!result) {
// someone else won the race — take the next item
return null;
}
```

Only one worker's filter matches. The other gets null and moves on. No lost update. No multi-doc transaction required for this path.

What we also changed

• Idempotent handlers: processing the same id twice must be safe
• Metrics: claim_conflict_total so races become visible, not folklore
• Alerts when conflict rate spikes after a deploy

Lesson

If your correctness depends on "I read it, therefore I may write it," you do not have correctness — you have a race with good intentions. Put the precondition in the query. Make conflict an explicit branch.