> ## Documentation Index > Fetch the complete documentation index at: https://www.unsiloed.ai/docs/llms.txt > Use this file to discover all available pages before exploring further. # Review Handwritten Edits on Scanned Documents > Use /v2/extract to read pen markup on printed pages as tracked changes with a box for each mark, compare two reads to find the edits a person should check, and build a review screen on top. Printed documents come back covered in pen: an editor's corrections on a draft, a lawyer's markup on a contract, an archivist's notes on a typed letter. OCR tools read the printed text and either drop the handwriting or merge it into the sentence. Either way, the record of what changed disappears, and someone has to retype every edit by hand. The `/v2/extract` endpoint can keep that record. With the right schema, it returns each page as a tracked-changes transcript, with a bounding box for each detected change where a location is returned. Reading the page a second time and comparing the two reads shows which changes a person should check. This recipe explores those techniques through Markup Review, a small app in the Unsiloed cookbook repository, where a reviewer accepts or rejects each change against a close-up of the scan. ![Animation of a review: boxes appear over the handwritten changes on the scan, the unflagged changes are accepted in one step, and the reviewer checks three flagged changes against close-ups, picking the right read where the two disagree](https://mintcdn.com/unsiloed/YFMJsHwPu0BFXEGq/images/handwritten-review/review-demo.gif?s=6112403bf9d19cb8cd2faea5b0384cc3) ## What We'll Build We'll clone the app, review the sample letter that ships with it, and then process a scan of your own. The second half of the recipe, [How It Works](#how-it-works), walks through the code behind each technique: 1. Render each scan from the detailed image inside it, not at screen resolution. 2. Ask `/v2/extract` for a redline: the whole page as one transcript, with every handwritten change marked inline. 3. Read the page twice and pair each change with its bounding box. 4. Compare the two reads and flag the changes they disagree on, the ones that scored low, and the ones the model marked as unsure. 5. Apply the reviewer's decisions to produce the clean text. The numbered steps are what you run. How It Works explains the code. Each technique sits in its own short module, so you can lift the ones you need into your own pipeline. ## Step 1: Run the App The app is a small FastAPI server and a single HTML page, with no build step. ### 1.1 Get an Unsiloed API Key To get API access, [sign up on Unsiloed AI](https://cal.com/aman-mishra-p0ry57/15min). ### 1.2 Clone and Install You need Python 3.13 or higher. Clone the cookbook repository and install the app's dependencies: ```bash theme={null} git clone https://github.com/Unsiloed-AI/cookbook.git cd cookbook/handwritten-review python3 -m venv .venv source .venv/bin/activate pip install -r requirements.txt ``` Copy the example environment file: ```bash theme={null} cp .env.example .env ``` Open `.env` and replace the placeholder with your API key: ```bash .env theme={null} UNSILOED_API_KEY=your-api-key ``` ### 1.3 Start the Server Start the app: ```bash theme={null} uvicorn app:app --reload ``` Open [http://localhost:8000](http://localhost:8000). The app opens with a sample: the first page of an 1837 typed letter from William Ellery Channing to William Lloyd Garrison, covered in handwritten corrections. Its two reads are saved in the repository, so the sample costs no API calls. ## Step 2: Review the Sample Letter The scan on the left has a box over each detected change that has a location. The color shows the change type: red for a replacement or deletion, green for an insertion, and purple for a note. A dashed box is a mark that only the second read found. Click any box, or any change in the transcript, to open it in the review panel. Each change carries the flags that explain why it needs a look: * **Reads disagree:** the second read has a different version, shown under the flags * **Only the first read found this:** the second read has no matching mark, so the change may not be there at all * **Only the second read found this:** the first read's transcript has no matching mark, so the change is listed separately with a dashed box * **Low confidence:** the handwriting scored below 0.5 * **Model marked unsure:** Unsiloed set `needs_review` on the change itself * **No location returned:** the change has no box, so find it on the scan from its context words The review panel shows a close-up of the marked area, with the printed words, the handwriting, and the flags under it. When the reads disagree, the second read's version sits under the flags with a button to use it: ![The review panel for the 'this' change: a close-up of the scan with 'is' handwritten over the end of the typed word, the first read's reading 'thes' to 'this', a Reads disagree flag, and the second read's version 'this' to 'the' with a Use this reading button](https://mintcdn.com/unsiloed/YFMJsHwPu0BFXEGq/images/handwritten-review/disputed-change.jpg?fit=max\&auto=format\&n=YFMJsHwPu0BFXEGq\&q=85\&s=2d3a39b739ae6dd5b814971ca89eb1bb) Review the letter: **1. Accept the unflagged changes** Select **Accept unflagged** to accept every change with no flags. **2. Work through the flagged changes** For each one, compare the close-up with the handwriting. Correct the text or the change type if needed, or select **Use this reading** to take the second read's handwriting and change type (the printed words stay as the first read has them). Then select **Accept change** or **Reject change**. A mark that only the second read found has no place in the transcript, so it can't be accepted. Check it on the scan and select **Mark as checked**. If it's a real edit, correct the exported text by hand. The review log records it as checked, not applied. **3. Check the clean text** Switch the transcript to **Clean result** and read it against the scan once. The review lists detected changes only, so this catches any mark that neither read found. **4. Export** Select **Export** to download the clean text and a JSON log of every decision. The panel above shows the "this" change near the end of the letter. The scan has "is" handwritten over the end of the typed word, so the corrected word is "this". The first read has it as "thes → this", with no low score. The second read has "this → the", so the change is flagged instead of either version being applied automatically. The close-up shows the first read is right, so the reviewer keeps it and presses A. Once the changes are reviewed, **Clean result** shows the letter with every accepted edit applied, including "the meeting" and the inserted "cordial testimony": ![The transcript in Clean result view, reading 'If they try to mix up the meeting with Abolitionism, they will prevent a deep and cordial testimony against violence'](https://mintcdn.com/unsiloed/YFMJsHwPu0BFXEGq/images/handwritten-review/clean-result.jpg?fit=max\&auto=format\&n=YFMJsHwPu0BFXEGq\&q=85\&s=57aeffa3caaf5b24f42b428432a5bca1) ## Step 3: Review Your Own Scan Select **Upload scan** and choose a file. The app accepts PDFs and PNG, JPEG, TIFF, and WebP images, and processes every page. Each page makes two `/v2/extract` calls, one per read. The app shows its progress while Unsiloed reads the document: ![The app while a document is processing: a spinner and the message that Unsiloed is reading the page twice, which usually takes one to two minutes](https://mintcdn.com/unsiloed/YFMJsHwPu0BFXEGq/images/handwritten-review/upload-reading.jpg?fit=max\&auto=format\&n=YFMJsHwPu0BFXEGq\&q=85\&s=2fc1fc2cbe8da65e7d2140276f5883c7) When both reads are done, the document opens in the same review screen as the sample. Each upload is kept as plain files under `documents/`: the page images, the raw `/v2/extract` responses, the review built from them, and the decisions. Opening the raw responses next to the review is a quick way to see exactly what Unsiloed returned for each mark. ## How It Works Every technique lives in the `review` package, which has no dependency on the web app. The server in `app.py` runs it for each upload, and `frontend/index.html` only draws what the server returns, so the package is the part to carry into your own code. ### Render the Scan From Its Embedded Image Handwriting is much smaller and fainter than type, so a scan that looks fine on screen can be too coarse for a fine strike through a word to survive. Many PDF tools render pages at screen resolution and quietly throw away the detail in the scanned image inside the PDF. `review/prepare.py` renders each page from that embedded image instead: ```python review/prepare.py theme={null} def zoom_for(page: pymupdf.Page) -> float: """Zoom factor that renders a scanned page at the resolution of the image embedded in it.""" images = page.get_image_info() zoom = max(i["width"] for i in images) / page.rect.width if images else 300 / 72 zoom = max(zoom, 300 / 72) return min(zoom, MAX_LONG_SIDE / max(page.rect.width, page.rect.height)) ``` Small images, like a cropped photo of one paragraph, are upscaled by up to three times, so every page the app sends is at least 2,000 pixels on its long side. To keep uploads small, the app scales larger pages down to 2,500 pixels. The rendered pages then go into a new PDF with no text layer, so the app controls exactly what is uploaded and the request is the same whether the source was a PDF or an image. ### Ask for a Redline The `/v2/extract` endpoint fills in a JSON schema you supply, and the shape of the schema decides how useful the result is. For handwritten edits, the app asks for the whole page as one transcript with every change marked inline, rather than a separate set of fields for each edit. A transcript keeps each mark in the context of its sentence, so related marks stay together, and the printed text and the handwriting stay apart. The transcript uses [CriticMarkup](https://github.com/CriticMarkup/CriticMarkup-toolkit), a plain-text notation for tracked changes: | Markup | Meaning | Example | | - | - | - | | `{~~old~>new~~}` | Printed words struck through and replaced | `{~~preparations~>passionateness~~}` | | `{--old--}` | Printed words struck through with no replacement | `{--party of--}` | | `{++new++}` | Handwriting inserted with a caret or in a gap | `{++cordial testimony++}` | | `{>>note<<}` | A margin note, gloss, or proofreading mark | `{>>sp<<}` | The schema in `review/schema.json` asks for the transcript in a `redline` field and for a `changes` array with one entry per mark: ```json review/schema.json theme={null} { "type": "object", "properties": { "redline": { "type": "string", "description": "Transcribe ALL printed text of the document in reading order, exactly as printed (keep printed misspellings), and mark every handwritten markup inline using CriticMarkup: {~~printed words~>handwritten replacement~~} when handwriting replaces struck-through or marked printed words; {--printed words--} for printed words struck through with no replacement; {++handwritten words++} for handwriting inserted (caret, leader line, or written in a gap); {>>margin note, gloss or proofreading mark such as sp, cap, lc, paragraph<<} placed directly after the printed words it refers to. A struck-through word must stay inside the markup, never silently dropped. Copy handwriting verbatim with its spelling; write [?] for unreadable characters. Do not correct anything that is not marked. Underlining alone is a comment, not a deletion." }, "changes": { "type": "array", "description": "One entry per CriticMarkup tag in redline, same order.", "items": { "type": "object", "properties": { "kind": { "type": "string", "enum": [ "substitution", "deletion", "insertion", "comment" ] }, "printed_text": { "type": "string", "description": "The printed words struck through, replaced, or commented on. Empty for a pure insertion." }, "handwritten_text": { "type": "string", "description": "Verbatim handwriting. Empty for a pure deletion." }, "anchor": { "type": "string", "description": "3-6 printed words around the change, so a reviewer can find it." }, "needs_review": { "type": "boolean", "description": "true if the handwriting is hard to read, the span of the strike is unclear, or it is unclear what the mark refers to." } }, "required": [ "kind", "printed_text", "handwritten_text", "anchor", "needs_review" ], "additionalProperties": false } } }, "required": [ "redline", "changes" ], "additionalProperties": false } ``` The `redline` description holds the rules for the marks that are most often misread, like "A struck-through word must stay inside the markup, never silently dropped" and "Underlining alone is a comment, not a deletion". Unsiloed can attach a bounding box to each field it returns, but the transcript string gets a single box covering the whole page, so the per-mark boxes come from the `changes` array. The app uses `/v2/extract` rather than `/parse` for this. Parsing is built to return the text of the page, so it applies or drops the handwritten edits instead of recording them. ### Read Each Page Twice The redline is a single string, so the pipeline extracts each page on its own: every page goes into a one-page PDF, and each one-page PDF is read twice. The reviews for each page are then merged, with the page number added to each change's ID and box. Each read is one extraction job. The request leaves the model on the default `gamma` tier and sets `enable_citations=true`, which turns on bounding boxes: ```python review/extract.py theme={null} def submit(pdf: Path) -> str: """Start an extraction job and return its ID.""" with open(pdf, "rb") as f: response = requests.post( f"{BASE_URL}/v2/extract", headers=headers(), files={"pdf_file": ("document.pdf", f, "application/pdf")}, data={"model": MODEL, "schema_data": SCHEMA, "enable_citations": "true"}, timeout=180, ) response.raise_for_status() return response.json()["job_id"] ``` On the hardest marks, two reads of the same page can differ. A disagreement between reads is a strong signal that a change needs a person, and it catches cases that a single confidence score doesn't. The `/v2/extract` endpoint also has a `high_accuracy_mode` option that cross-checks uncertain fields and reconciles them before returning a result. Reading twice is the alternative for a review workflow: instead of being resolved automatically, each disagreement goes to the reviewer with both versions side by side. The pipeline runs both reads in parallel: ```python review/extract.py theme={null} def read_twice(pdf: Path) -> list[dict]: """Run two independent extractions of the same PDF in parallel.""" with ThreadPoolExecutor(2) as pool: return list(pool.map(read_once, [pdf, pdf])) ``` ### Pair Each Change With Its Box Every field in an extraction result is wrapped in an envelope with the value, two scores, and a citation. Here's the `printed_text` field of the "preparations" change from the sample: ```json theme={null} { "score": { "grounding_score": 0.97, "extraction_score": 0.97 }, "value": "preparations", "citation": { "bbox": [ 493, 251, 599, 268 ], "page": 1, "page_width": 633.3599853515625, "page_height": 840.0 } } ``` The `bbox` is in PDF points, measured from the top-left corner of the page. The app converts it to fractions of the page, which the review screen can draw over an image of any size: ```python review/redline.py theme={null} def box(citation): """Convert a citation's point coordinates to fractions of the page, which is all the viewer needs.""" if not citation: return None x0, y0, x1, y1 = citation["bbox"] w, h = citation["page_width"], citation["page_height"] return {"page": citation.get("page", 1), "left": x0 / w, "top": y0 / h, "width": (x1 - x0) / w, "height": (y1 - y0) / h} ``` To build the review, `review/redline.py` splits the redline into plain text and changes with one regular expression per CriticMarkup tag, then pairs each change in the transcript with its entry in the `changes` array by matching the change type and text. That gives each change a box and a score. A change with no matching entry is flagged **No location returned**, and the reviewer finds it on the scan from its context words. Use the scores to decide what to review first, not to accept or reject changes. Faint or cramped handwriting scores low even when the reading is right: "passionateness" is correct in the sample, and it scored 0.10. The app flags anything below 0.5 and never rejects a change for scoring low. ### Compare the Reads The first read's transcript becomes the document. Each of its changes is matched against the second read by type, printed words, and handwriting, after normalizing case, punctuation, and words hyphenated across a line break: ```python review/redline.py theme={null} def compare(first: list[dict], second: list[dict]) -> None: """Flag changes the reads disagree on, and show the second read's version next to them. A first-read change with an identical second-read change agrees. Any other first-read change is disputed. Each leftover second-read change is then attached to the nearest disputed change as its alternative: for example, the first read's "thes → this" and the second read's "this → the" sit at the same spot, so the reviewer sees both. A leftover with nothing nearby is a mark only the second read saw, and is listed separately. """ unmatched = list(second) disputed = [] for change in first: match = next((c for c in unmatched if key(c) == key(change)), None) if match: unmatched.remove(match) else: disputed.append(change) ``` A change with no match is disputed. The function then looks for a second-read change near the same position in the printed text and attaches it as the second read's version, which is what **Use this reading** applies. A second-read change with nothing nearby is a mark only the second read saw, so it's listed separately for the reviewer to check on the scan. ### Apply the Reviewer's Decisions A decision records the reviewer's verdict and any corrections: the handwriting, the change type, and, for an insertion, whether it goes before or after the printed words. The clean text is the transcript with each change replaced by what its decision produces: ```python review/redline.py theme={null} def apply(change: dict, decision: dict) -> str: """The text a change contributes to the clean result. A decision is {"state", "handwriting", "kind", "before"}. The reviewer can correct the handwriting, and the change type, for example when a caret insertion was read as a replacement. Only accepted changes alter the printed text. """ printed = change["printed"] if decision.get("state") != "accepted": return printed kind = decision.get("kind") or change["kind"] handwriting = decision.get("handwriting", change["handwriting"]).strip() if kind == "substitution": return handwriting if kind == "deletion": return "" if kind == "insertion" and printed: return f"{handwriting} {printed}" if decision.get("before") else f"{printed} {handwriting}" if kind == "insertion": return handwriting return printed # notes never change the text ``` A change that's pending or rejected keeps the printed text, so an unreviewed edit never reaches the clean text by accident. The export pairs that text with a log of every change: the printed words, the handwriting, the decision, where an insertion was placed, whether the reviewer edited it, its flags, and its box. ## Where Handwritten Marks Are Hard to Read Unsiloed places a box on the handwriting for each detected change it returns a location for, so any review step built on these results can show a person the exact mark. What a mark means is still a judgment call, and these are the cases that need the closest look, even when they aren't flagged: * **Marks near a caret:** a caret next to a word can look like a strike, so a listed deletion may have no line through it * **Long strikes:** a strike that runs across several words, or onto the next line, can be read as covering fewer words than it does * **Carets before a word:** an insertion before a word can be read as a replacement of that word, which is why the app lets the reviewer change a change's type and placement * **Margin notes:** an archival or reviewer's note in the margin can be read as an insertion, which would put it in the body text * **Marks neither read found:** the review lists detected changes only, so the final clean text still needs one read against the scan This is why the app routes every change past a reviewer, and treats **Accept unflagged** as a shortcut rather than a substitute for review. The reviewer's job becomes checking each change against a close-up, instead of transcribing the page and comparing it by hand. ## Where to Take This Next Markup Review covers one workflow from scan to clean text. The techniques inside it carry over to other documents and other pipelines: * **Change what the redline captures.** The schema is the lever. Add your reviewers' conventions to the `redline` description in `review/schema.json`, like proofreading marks or legal shorthand, or point the same approach at contract markups and forms amended by hand. * **Put the changes where your team already works.** CriticMarkup is plain text, so the parsed changes can become tracked changes in your own editor or document system, instead of a separate review screen. * **Read twice wherever a mistake is costly.** The two-read comparison isn't specific to handwriting. Run any extraction twice and send only the fields the reads disagree on to a person, or set `high_accuracy_mode=true` when you'd rather the service reconcile uncertain fields itself. * **Show the evidence with every value.** The citation boxes work for any `/v2/extract` field, so any human review step can show a close-up of the source next to each value, and an audit trail can keep that crop with the decision. > **[Extraction](/docs/document-processing/extraction/extraction.md)** > > How schemas drive extraction, including nested objects and arrays. > **[API Reference](/docs/api-reference/extraction/extract-data.md)** > > The full request and response specs for `/v2/extract`. This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.