Review Handwritten Edits on Scanned Documents

Printed documents come back covered in pen: an editor’s corrections on a draft, a lawyer’s markup on a contract, an archivist’s notes on a typed letter. OCR tools read the printed text and either drop the handwriting or merge it into the sentence. Either way, the record of what changed disappears, and someone has to retype every edit by hand. The /v2/extract endpoint can keep that record. With the right schema, it returns each page as a tracked-changes transcript, with a bounding box for each detected change where a location is returned. Reading the page a second time and comparing the two reads shows which changes a person should check. This recipe explores those techniques through Markup Review, a small app in the Unsiloed cookbook repository, where a reviewer accepts or rejects each change against a close-up of the scan.

Animation of a review: boxes appear over the handwritten changes on the scan, the unflagged changes are accepted in one step, and the reviewer checks three flagged changes against close-ups, picking the right read where the two disagree

What We’ll Build

We’ll clone the app, review the sample letter that ships with it, and then process a scan of your own. The second half of the recipe, How It Works, walks through the code behind each technique:

Render each scan from the detailed image inside it, not at screen resolution.
Ask /v2/extract for a redline: the whole page as one transcript, with every handwritten change marked inline.
Read the page twice and pair each change with its bounding box.
Compare the two reads and flag the changes they disagree on, the ones that scored low, and the ones the model marked as unsure.
Apply the reviewer’s decisions to produce the clean text.

The numbered steps are what you run. How It Works explains the code. Each technique sits in its own short module, so you can lift the ones you need into your own pipeline.

Step 1: Run the App

The app is a small FastAPI server and a single HTML page, with no build step.

1.1 Get an Unsiloed API Key

To get API access, sign up on Unsiloed AI.

1.2 Clone and Install

You need Python 3.13 or higher. Clone the cookbook repository and install the app’s dependencies:

git clone https://github.com/Unsiloed-AI/cookbook.git
cd cookbook/handwritten-review

python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

Copy the example environment file:

cp .env.example .env

Open .env and replace the placeholder with your API key:

.env

UNSILOED_API_KEY=your-api-key

1.3 Start the Server

Start the app:

uvicorn app:app --reload

Open http://localhost:8000. The app opens with a sample: the first page of an 1837 typed letter from William Ellery Channing to William Lloyd Garrison, covered in handwritten corrections. Its two reads are saved in the repository, so the sample costs no API calls.

Step 2: Review the Sample Letter

The scan on the left has a box over each detected change that has a location. The color shows the change type: red for a replacement or deletion, green for an insertion, and purple for a note. A dashed box is a mark that only the second read found. Click any box, or any change in the transcript, to open it in the review panel. Each change carries the flags that explain why it needs a look:

Reads disagree: the second read has a different version, shown under the flags
Only the first read found this: the second read has no matching mark, so the change may not be there at all
Only the second read found this: the first read’s transcript has no matching mark, so the change is listed separately with a dashed box
Low confidence: the handwriting scored below 0.5
Model marked unsure: Unsiloed set needs_review on the change itself
No location returned: the change has no box, so find it on the scan from its context words

The review panel shows a close-up of the marked area, with the printed words, the handwriting, and the flags under it. When the reads disagree, the second read’s version sits under the flags with a button to use it:

The review panel for the 'this' change: a close-up of the scan with 'is' handwritten over the end of the typed word, the first read's reading 'thes' to 'this', a Reads disagree flag, and the second read's version 'this' to 'the' with a Use this reading button

Review the letter:

Accept the unflagged changes

Select Accept unflagged to accept every change with no flags.

Work through the flagged changes

For each one, compare the close-up with the handwriting. Correct the text or the change type if needed, or select Use this reading to take the second read’s handwriting and change type (the printed words stay as the first read has them). Then select Accept change or Reject change.A mark that only the second read found has no place in the transcript, so it can’t be accepted. Check it on the scan and select Mark as checked. If it’s a real edit, correct the exported text by hand. The review log records it as checked, not applied.

Check the clean text

Switch the transcript to Clean result and read it against the scan once. The review lists detected changes only, so this catches any mark that neither read found.

Export

Select Export to download the clean text and a JSON log of every decision.

The panel above shows the “this” change near the end of the letter. The scan has “is” handwritten over the end of the typed word, so the corrected word is “this”. The first read has it as “thes → this”, with no low score. The second read has “this → the”, so the change is flagged instead of either version being applied automatically. The close-up shows the first read is right, so the reviewer keeps it and presses A. Once the changes are reviewed, Clean result shows the letter with every accepted edit applied, including “the meeting” and the inserted “cordial testimony”:

The transcript in Clean result view, reading 'If they try to mix up the meeting with Abolitionism, they will prevent a deep and cordial testimony against violence'

Step 3: Review Your Own Scan

Select Upload scan and choose a file. The app accepts PDFs and PNG, JPEG, TIFF, and WebP images, and processes every page. Each page makes two /v2/extract calls, one per read. The app shows its progress while Unsiloed reads the document:

The app while a document is processing: a spinner and the message that Unsiloed is reading the page twice, which usually takes one to two minutes

When both reads are done, the document opens in the same review screen as the sample. Each upload is kept as plain files under documents/: the page images, the raw /v2/extract responses, the review built from them, and the decisions. Opening the raw responses next to the review is a quick way to see exactly what Unsiloed returned for each mark.

How It Works

Every technique lives in the review package, which has no dependency on the web app. The server in app.py runs it for each upload, and frontend/index.html only draws what the server returns, so the package is the part to carry into your own code.

Render the Scan From Its Embedded Image

Handwriting is much smaller and fainter than type, so a scan that looks fine on screen can be too coarse for a fine strike through a word to survive. Many PDF tools render pages at screen resolution and quietly throw away the detail in the scanned image inside the PDF. review/prepare.py renders each page from that embedded image instead:

review/prepare.py

def zoom_for(page: pymupdf.Page) -> float:
    """Zoom factor that renders a scanned page at the resolution of the image embedded in it."""
    images = page.get_image_info()
    zoom = max(i["width"] for i in images) / page.rect.width if images else 300 / 72
    zoom = max(zoom, 300 / 72)
    return min(zoom, MAX_LONG_SIDE / max(page.rect.width, page.rect.height))

Small images, like a cropped photo of one paragraph, are upscaled by up to three times, so every page the app sends is at least 2,000 pixels on its long side. To keep uploads small, the app scales larger pages down to 2,500 pixels. The rendered pages then go into a new PDF with no text layer, so the app controls exactly what is uploaded and the request is the same whether the source was a PDF or an image.

Ask for a Redline

The /v2/extract endpoint fills in a JSON schema you supply, and the shape of the schema decides how useful the result is. For handwritten edits, the app asks for the whole page as one transcript with every change marked inline, rather than a separate set of fields for each edit. A transcript keeps each mark in the context of its sentence, so related marks stay together, and the printed text and the handwriting stay apart. The transcript uses CriticMarkup, a plain-text notation for tracked changes:

Markup	Meaning	Example
`{~~old~>new~~}`	Printed words struck through and replaced	`{~~preparations~>passionateness~~}`
`{--old--}`	Printed words struck through with no replacement	`{--party of--}`
`{++new++}`	Handwriting inserted with a caret or in a gap	`{++cordial testimony++}`
`{>>note<<}`	A margin note, gloss, or proofreading mark	`{>>sp<<}`

The schema in review/schema.json asks for the transcript in a redline field and for a changes array with one entry per mark:

review/schema.json

{
  "type": "object",
  "properties": {
    "redline": {
      "type": "string",
      "description": "Transcribe ALL printed text of the document in reading order, exactly as printed (keep printed misspellings), and mark every handwritten markup inline using CriticMarkup: {~~printed words~>handwritten replacement~~} when handwriting replaces struck-through or marked printed words; {--printed words--} for printed words struck through with no replacement; {++handwritten words++} for handwriting inserted (caret, leader line, or written in a gap); {>>margin note, gloss or proofreading mark such as sp, cap, lc, paragraph<<} placed directly after the printed words it refers to. A struck-through word must stay inside the markup, never silently dropped. Copy handwriting verbatim with its spelling; write [?] for unreadable characters. Do not correct anything that is not marked. Underlining alone is a comment, not a deletion."
    },
    "changes": {
      "type": "array",
      "description": "One entry per CriticMarkup tag in redline, same order.",
      "items": {
        "type": "object",
        "properties": {
          "kind": {
            "type": "string",
            "enum": [
              "substitution",
              "deletion",
              "insertion",
              "comment"
            ]
          },
          "printed_text": {
            "type": "string",
            "description": "The printed words struck through, replaced, or commented on. Empty for a pure insertion."
          },
          "handwritten_text": {
            "type": "string",
            "description": "Verbatim handwriting. Empty for a pure deletion."
          },
          "anchor": {
            "type": "string",
            "description": "3-6 printed words around the change, so a reviewer can find it."
          },
          "needs_review": {
            "type": "boolean",
            "description": "true if the handwriting is hard to read, the span of the strike is unclear, or it is unclear what the mark refers to."
          }
        },
        "required": [
          "kind",
          "printed_text",
          "handwritten_text",
          "anchor",
          "needs_review"
        ],
        "additionalProperties": false
      }
    }
  },
  "required": [
    "redline",
    "changes"
  ],
  "additionalProperties": false
}

The redline description holds the rules for the marks that are most often misread, like “A struck-through word must stay inside the markup, never silently dropped” and “Underlining alone is a comment, not a deletion”. Unsiloed can attach a bounding box to each field it returns, but the transcript string gets a single box covering the whole page, so the per-mark boxes come from the changes array. The app uses /v2/extract rather than /parse for this. Parsing is built to return the text of the page, so it applies or drops the handwritten edits instead of recording them.

Read Each Page Twice

The redline is a single string, so the pipeline extracts each page on its own: every page goes into a one-page PDF, and each one-page PDF is read twice. The reviews for each page are then merged, with the page number added to each change’s ID and box. Each read is one extraction job. The request leaves the model on the default gamma tier and sets enable_citations=true, which turns on bounding boxes:

review/extract.py

def submit(pdf: Path) -> str:
    """Start an extraction job and return its ID."""
    with open(pdf, "rb") as f:
        response = requests.post(
            f"{BASE_URL}/v2/extract",
            headers=headers(),
            files={"pdf_file": ("document.pdf", f, "application/pdf")},
            data={"model": MODEL, "schema_data": SCHEMA, "enable_citations": "true"},
            timeout=180,
        )
    response.raise_for_status()
    return response.json()["job_id"]

On the hardest marks, two reads of the same page can differ. A disagreement between reads is a strong signal that a change needs a person, and it catches cases that a single confidence score doesn’t. The /v2/extract endpoint also has a high_accuracy_mode option that cross-checks uncertain fields and reconciles them before returning a result. Reading twice is the alternative for a review workflow: instead of being resolved automatically, each disagreement goes to the reviewer with both versions side by side. The pipeline runs both reads in parallel:

review/extract.py

def read_twice(pdf: Path) -> list[dict]:
    """Run two independent extractions of the same PDF in parallel."""
    with ThreadPoolExecutor(2) as pool:
        return list(pool.map(read_once, [pdf, pdf]))

Pair Each Change With Its Box

Every field in an extraction result is wrapped in an envelope with the value, two scores, and a citation. Here’s the printed_text field of the “preparations” change from the sample:

{
  "score": {
    "grounding_score": 0.97,
    "extraction_score": 0.97
  },
  "value": "preparations",
  "citation": {
    "bbox": [
      493,
      251,
      599,
      268
    ],
    "page": 1,
    "page_width": 633.3599853515625,
    "page_height": 840.0
  }
}

The bbox is in PDF points, measured from the top-left corner of the page. The app converts it to fractions of the page, which the review screen can draw over an image of any size:

review/redline.py

def box(citation):
    """Convert a citation's point coordinates to fractions of the page, which is all the viewer needs."""
    if not citation:
        return None
    x0, y0, x1, y1 = citation["bbox"]
    w, h = citation["page_width"], citation["page_height"]
    return {"page": citation.get("page", 1), "left": x0 / w, "top": y0 / h, "width": (x1 - x0) / w, "height": (y1 - y0) / h}

To build the review, review/redline.py splits the redline into plain text and changes with one regular expression per CriticMarkup tag, then pairs each change in the transcript with its entry in the changes array by matching the change type and text. That gives each change a box and a score. A change with no matching entry is flagged No location returned, and the reviewer finds it on the scan from its context words. Use the scores to decide what to review first, not to accept or reject changes. Faint or cramped handwriting scores low even when the reading is right: “passionateness” is correct in the sample, and it scored 0.10. The app flags anything below 0.5 and never rejects a change for scoring low.

Compare the Reads

The first read’s transcript becomes the document. Each of its changes is matched against the second read by type, printed words, and handwriting, after normalizing case, punctuation, and words hyphenated across a line break:

review/redline.py

def compare(first: list[dict], second: list[dict]) -> None:
    """Flag changes the reads disagree on, and show the second read's version next to them.

    A first-read change with an identical second-read change agrees. Any other first-read change
    is disputed. Each leftover second-read change is then attached to the nearest disputed change
    as its alternative: for example, the first read's "thes → this" and the second read's
    "this → the" sit at the same spot, so the reviewer sees both. A leftover with nothing nearby
    is a mark only the second read saw, and is listed separately.
    """
    unmatched = list(second)
    disputed = []
    for change in first:
        match = next((c for c in unmatched if key(c) == key(change)), None)
        if match:
            unmatched.remove(match)
        else:
            disputed.append(change)

A change with no match is disputed. The function then looks for a second-read change near the same position in the printed text and attaches it as the second read’s version, which is what Use this reading applies. A second-read change with nothing nearby is a mark only the second read saw, so it’s listed separately for the reviewer to check on the scan.

Apply the Reviewer’s Decisions

A decision records the reviewer’s verdict and any corrections: the handwriting, the change type, and, for an insertion, whether it goes before or after the printed words. The clean text is the transcript with each change replaced by what its decision produces:

review/redline.py

def apply(change: dict, decision: dict) -> str:
    """The text a change contributes to the clean result.

    A decision is {"state", "handwriting", "kind", "before"}. The reviewer can correct the
    handwriting, and the change type, for example when a caret insertion was read as a
    replacement. Only accepted changes alter the printed text.
    """
    printed = change["printed"]
    if decision.get("state") != "accepted":
        return printed
    kind = decision.get("kind") or change["kind"]
    handwriting = decision.get("handwriting", change["handwriting"]).strip()
    if kind == "substitution":
        return handwriting
    if kind == "deletion":
        return ""
    if kind == "insertion" and printed:
        return f"{handwriting} {printed}" if decision.get("before") else f"{printed} {handwriting}"
    if kind == "insertion":
        return handwriting
    return printed  # notes never change the text

A change that’s pending or rejected keeps the printed text, so an unreviewed edit never reaches the clean text by accident. The export pairs that text with a log of every change: the printed words, the handwriting, the decision, where an insertion was placed, whether the reviewer edited it, its flags, and its box.

Where Handwritten Marks Are Hard to Read

Unsiloed places a box on the handwriting for each detected change it returns a location for, so any review step built on these results can show a person the exact mark. What a mark means is still a judgment call, and these are the cases that need the closest look, even when they aren’t flagged:

Marks near a caret: a caret next to a word can look like a strike, so a listed deletion may have no line through it
Long strikes: a strike that runs across several words, or onto the next line, can be read as covering fewer words than it does
Carets before a word: an insertion before a word can be read as a replacement of that word, which is why the app lets the reviewer change a change’s type and placement
Margin notes: an archival or reviewer’s note in the margin can be read as an insertion, which would put it in the body text
Marks neither read found: the review lists detected changes only, so the final clean text still needs one read against the scan

This is why the app routes every change past a reviewer, and treats Accept unflagged as a shortcut rather than a substitute for review. The reviewer’s job becomes checking each change against a close-up, instead of transcribing the page and comparing it by hand.

Where to Take This Next

Markup Review covers one workflow from scan to clean text. The techniques inside it carry over to other documents and other pipelines:

Change what the redline captures. The schema is the lever. Add your reviewers’ conventions to the redline description in review/schema.json, like proofreading marks or legal shorthand, or point the same approach at contract markups and forms amended by hand.
Put the changes where your team already works. CriticMarkup is plain text, so the parsed changes can become tracked changes in your own editor or document system, instead of a separate review screen.
Read twice wherever a mistake is costly. The two-read comparison isn’t specific to handwriting. Run any extraction twice and send only the fields the reads disagree on to a person, or set high_accuracy_mode=true when you’d rather the service reconcile uncertain fields itself.
Show the evidence with every value. The citation boxes work for any /v2/extract field, so any human review step can show a close-up of the source next to each value, and an audit trail can keep that crop with the decision.

Extraction

How schemas drive extraction, including nested objects and arrays.

API Reference

The full request and response specs for /v2/extract.

Recipes

Spreadsheet Automation

Review Handwritten Edits on Scanned Documents

What We’ll Build

Step 1: Run the App

1.1 Get an Unsiloed API Key

1.2 Clone and Install

1.3 Start the Server

Step 2: Review the Sample Letter

Step 3: Review Your Own Scan

How It Works

Render the Scan From Its Embedded Image

Ask for a Redline

Read Each Page Twice

Pair Each Change With Its Box

Compare the Reads

Apply the Reviewer’s Decisions

Where Handwritten Marks Are Hard to Read

Where to Take This Next

Extraction

API Reference

​What We’ll Build

​Step 1: Run the App

​1.1 Get an Unsiloed API Key

​1.2 Clone and Install

​1.3 Start the Server

​Step 2: Review the Sample Letter

​Step 3: Review Your Own Scan

​How It Works

​Render the Scan From Its Embedded Image

​Ask for a Redline

​Read Each Page Twice

​Pair Each Change With Its Box

​Compare the Reads

​Apply the Reviewer’s Decisions

​Where Handwritten Marks Are Hard to Read

​Where to Take This Next

Extraction

API Reference

What We’ll Build

Step 1: Run the App

1.1 Get an Unsiloed API Key

1.2 Clone and Install

1.3 Start the Server

Step 2: Review the Sample Letter

Step 3: Review Your Own Scan

How It Works

Render the Scan From Its Embedded Image

Ask for a Redline

Read Each Page Twice

Pair Each Change With Its Box

Compare the Reads

Apply the Reviewer’s Decisions

Where Handwritten Marks Are Hard to Read

Where to Take This Next