Async OCR Processing With S3, SignalR and Idempotent Callbacks
A document goes to an extraction service that answers whenever it feels like it, sometimes twice, sometimes not at all. Everyone watching the document should see the truth, and nobody should be charged for work that did not happen.
- Role
- Backend developer
- Where
- Vertex Special Technologies
- When
- 2025–2026
What it needed to do
Users upload a financial document — a bank statement, an invoice, a receipt — and get structured JSON back. The extraction itself is done by a separate service that takes as long as it takes and reports back when it is finished.
Everything hard about this sits in that gap. The upload returns immediately, the answer arrives minutes later over a callback, and in between the document has to have an honest status that everyone looking at it can see. Extraction is billed per page, so the accounting has to survive the same uncertainty.
What made it hard
The work happens elsewhere
The extraction service can be slow, unavailable, or wrong. None of that is controllable from this side, so the pipeline has to be built around a dependency that is allowed to fail.
Callbacks are not promises
A callback can arrive twice, arrive late, or describe a rerun of something already recorded. Treating each one as a fresh fact produces duplicate content and a document history nobody can trust.
Money is attached
Processing is charged per page. Charge before the work and a failure bills for nothing; charge after and a lost callback processes for free. Neither is acceptable when the number ends up on an invoice.
More than one person is watching
A document can be shared directly and it lives inside a workspace other people belong to. A status change is not news for one browser tab, it is news for everyone with a view of that document.
The path a document takes
- 1The client gets a pre-signed URL and uploads straight to object storage. The file never passes through the API.
- 2A job record is created with its own identifier, and credit for the expected page count is reserved rather than charged.
- 3The extraction service is health-checked before submission. Submitting into a service known to be down only manufactures a failure to handle later.
- 4Submission failures increment a retry counter and are logged per attempt. Past a configured limit the document fails for good and the reservation is released.
- 5Callbacks arrive against the job identifier and move the document through its states, each one recorded with its full payload.
- 6On completion the reservation is settled against the page count the service actually reported. On failure it is released.
Making a callback safe to receive twice
The same document can legitimately produce several completed callbacks. The first extraction is one. A reprocess is another. A user asking for a customised output shape is a third. They look almost identical, and treating them the same way is how a document ends up with three copies of its own first result.
So the payload distinguishes them, and the handler branches on that: a first result creates the document's content and schema, while a rerun or a customisation writes a new version that points at the one before it. The status change, the content, the schema and the audit record all happen inside one transaction, so a document is never left with new content and an old status.
Every callback is kept, not just applied
Each one is written to a status log with its entire payload. When a document ends up in a state nobody expects, the question “what did the extraction service actually send us, and when” had an answer. That log was worth more than any amount of defensive coding.
Writing that log is allowed to fail without failing the callback. An audit record is worth having and not worth rejecting a valid status update over.
The tradeoff: billing outside the transaction
A status update and a charge are both correct things to do. Making them atomic means a billing hiccup can silently discard the fact that a document finished.
The obvious instinct is to put the whole callback in one transaction: status, content, audit, billing. It is one unit of work, so make it one unit.
Billing was deliberately moved outside it. The status transaction commits first, and settlement runs after. If settlement then fails, the system has a finished document and an unsettled reservation — a discrepancy in a ledger that can be found and fixed, with the reservation still on record. The alternative is worse in a way that is harder to notice: a completed extraction rolled back because the wallet was momentarily unhappy, leaving the document stuck in processing forever while the user watches a spinner.
The reservation pattern is what makes that acceptable. Credit is held at submission, settled against the real page count at completion, and released on failure. Nothing is charged for work that did not happen, and nothing is processed for free because a callback went missing.
Storage first, database second
When a new version is written, the content goes to object storage before the row that points at it. Fail in that order and the cost is an orphaned object nobody references. Fail in the other order and a version row points at content that was never written, which is a document that cannot be opened.
Ask rather than guess
When the extraction service cannot identify a document's type, the document does not quietly become “other”. It moves to action required and waits for a person. On financial documents, a confident wrong answer is more expensive than an admission of uncertainty.
Live status for everyone who can see it
When a document changes state, the update is pushed over a websocket to the owner, to everyone the document is shared with directly, and to every member of the workspace it belongs to, as one deduplicated list.
That last part was not the first version. It is easy to build this so the person who uploaded the file sees the progress bar and everyone else refreshes and wonders. A shared document with a private status is not really shared.
Versions, and what revert actually means
Content and schema are versioned separately, each version recording what caused it — generated, reprocessed, customised, reverted — and pointing at the version it replaced. That chain is the document's history, and it is complete: every action that changed the document is in it, with who did it and when.
What is retained is the original extraction and the current content. Revert means going back to what the extraction service first produced, not stepping back one edit at a time. For a document whose value is the machine output plus a person's corrections, those are the two states anyone actually asks for, and keeping every intermediate copy would multiply storage for a version nobody requests.
It is worth being precise about that, because “version control” suggests more than it delivers here. The history is fully tracked. The content is kept at the two ends.
What I would do differently
Idempotency should not depend on the sender
The handler decides whether a callback is a rerun by reading a flag the extraction service sets. That works, and it puts correctness in the hands of the other side of the integration. A delivery identifier recorded on receipt and checked before doing anything would make a duplicate harmless regardless of what the sender claims about itself.
The retry ladder is one-sided
Submission retries when the extraction service will not accept the job. There is no equivalent for a job that is accepted and then never reports back — that document waits in processing until someone notices. A deadline on the job, with a sweep for anything past it, would close that gap.