Why RAG and AI Agents Need an IDP Ingestion Layer
Extracted document text can be incomplete, outdated, or unsuitable for a workflow. An IDP ingestion layer checks coverage, provenance, and workflow requirements before RAG or agents use it.
A retrieval-augmented generation (RAG) assistant reports a $12,000 renewal price from a supplier price sheet. The indexed text contains that introductory price but omits the final page, which says renewals use a higher rate. Before making the document searchable, the application needed to check that the result covered all eight pages.
Producing extracted text doesn't establish that the result contains everything a downstream workflow needs.
RAG systems and AI agents therefore need checks between intelligent document processing (IDP) and consumption. We'll call this responsibility the ingestion layer. It can live inside your existing ingestion code rather than in a separate service. It decides whether the result is complete, what its fields mean, which source revision produced it, and which workflows may use it.
Processing Success Is Not the Same as Approval
A parser's status describes a processing job. It doesn't answer every question the downstream application cares about.
Consider three separate decisions:
- Processing outcome: Did the requested pages finish processing?
- Record validity: Does the result have the expected shape, field states, and resolvable evidence?
- Workflow eligibility: Does this result meet the requirements for retrieval or a renewal tool?
Collapsing those decisions into one success flag hides useful failure states. A result can be structurally valid while accurately reporting that page eight failed. It can be complete enough for broad document search but still lack the renewal date required by an agent that changes a contract. Conversely, a low-confidence field may be useful to a reviewer even when automation must reject it.
The ingestion layer gives each decision a name and keeps it attached to the result. It separates “the document was processed” from “this result meets the workflow's requirements.”
Where Unsiloed Fits in the Ingestion Architecture
Unsiloed handles the document-understanding step. Its Parse API converts PDFs, scans, images, Word documents, and presentations into ordered Markdown and layout-aware segments with source coordinates. Its Extract API fills a JSON Schema with typed values, per-field confidence scores, and optional page-and-bounding-box citations.
Those outputs give an ingestion layer better material to evaluate than flat OCR text:
- Preserved headings, tables, reading order, and page locations for RAG
- Schema-shaped fields for application workflows
- Confidence signals for review policies
- Citations that connect a value to its source region
Unsiloed produces the document representation. Your application still owns the source registry, acceptance rules, active index, and agent permissions. This separation is useful because the same Unsiloed result can serve different consumers without forcing them to share the same risk tolerance.
For retrieval, you might use Parse output as the basis for chunks while retaining page and segment coordinates. For a workflow that needs supplier name, renewal date, and annual fee, you might use Extract with a schema that defines those fields. The ingestion layer then checks whether the document and the result meet the requirements of each consumer.
This also clarifies what the layer is not. It isn't another model prompt, and it doesn't improve an incorrect extraction by declaring it valid. It converts document-processing signals into explicit application decisions.
Keep Each Result Tied to the File That Produced It
A supplier can upload a corrected pricing.pdf while extraction of the previous upload is still running. If both jobs write their results using the same filename or document ID, the older job can finish last and overwrite the corrected values.
Give each upload a revision ID and attach it to every extraction result. Before publishing a result, check that it belongs to the revision the application has selected. Keep a late result from an older revision in history, but don't let it replace the selected result.
Four linked identities make that decision auditable:
- Document identity: Groups every version of the supplier price sheet
- Source revision: Identifies the immutable bytes in one upload
- Processing result: Records what one extraction attempt produced from that revision
- Publication: Records which accepted result a consumer may currently use
The application, rather than job completion order, decides which revision becomes current.
The same relationship keeps evidence tied to the correct source revision and location. A citation that opens page two of the latest file is misleading if the value came from page two of an older file. Source identity, page number, coordinates, and original evidence text need to travel together through normalization and chunking.
Also record the parser or extractor configuration that produced each result. Changing a requested schema, chunking strategy, or adapter can change the output even when the source file stays the same.
Replace the Whole Document Version at Once
When a corrected document arrives, don't expose its chunks as they finish processing. Otherwise, a retrieval query can combine part of the new document with part of the old one and produce an answer that matches neither version.
Keep the last complete, approved version searchable while building and validating its replacement separately. Once the replacement is complete, publish the whole version at once. If processing fails, leave the previous approved version in place and surface that it is out of date.
Missing Data Needs More Than null
An application loses useful information when it collapses every missing value to null. A missing annual fee can mean the completed document doesn't contain one, the relevant page is unreadable, processing skipped a page, or the extraction schema never requested the field.
When the available evidence supports it, define application states that preserve those differences and lead to an appropriate response. For example:
| Field State | What It Means | Appropriate Response |
|---|---|---|
| Present | Extraction produced a candidate value | Check its type, evidence, and policy |
| Not found | A complete search didn't locate the requested value | Preserve the absence; don't substitute zero |
| Unreadable | The relevant content couldn't be resolved | Request review or a better source |
| Conflicting | Multiple candidates remain unresolved | Preserve the candidates for review |
| Not requested | The extraction configuration omitted the field | Run an extraction that requests it |
| Unknown | The result gives no reliable reason for the absence | Preserve the uncertainty and inspect it |
This is where structured JSON helps. The application can represent value, state, evidence, and provenance separately instead of passing an ambiguous null downstream.
The layer should only assign a state it can support. If page eight never finished, it can't conclude that the whole document lacks a renewal clause. It can only say that the result is incomplete.
RAG and Agents Need Different Acceptance Rules
A complete document can be searchable even when it lacks a value needed to update a renewal record. Each workflow needs acceptance rules for the information it uses.
A RAG index may require every requested page, resolvable source locations, and an approved source revision. These rules can keep incomplete or mixed document content from becoming retrievable.
An agent tool usually needs stricter field-level rules. A tool that updates renewal records might require the supplier, fee, currency, period, renewal date, and evidence for every value. If one field is unresolved, the document can remain searchable while the tool stays blocked.
A review interface has a different purpose. It should receive partial and uncertain results with their failure reasons intact. Hiding rejected records makes the pipeline harder to debug and leaves operators with no recovery path.
Keep confidence scores attached to the fields they describe. Decide which scores require review separately for each workflow. Set thresholds using labeled examples from representative documents, then measure both errors among accepted results and the volume routed to review.
The backend that updates the renewal record should enforce these requirements when the tool runs. A prompt telling the agent to avoid unapproved results doesn't prevent the backend from receiving one.
The Ingestion Layer Makes Failures Operable
Recording why a result was rejected makes failures easier to investigate. Operators can distinguish a missing page from an unresolved field or an outdated source revision.
Track rejections by reason and processing configuration. Compare review volume with errors found in accepted records to see whether extraction quality or an acceptance rule needs attention.
The layer can't prove that every extracted value is correct. It can preserve uncertainty, provenance, revision state, and permissions before an AI system sees the result.
A Document Layer and an Ingestion Layer Solve Different Problems
The document layer determines what the file contains. The ingestion layer determines what your application may do with that result.
Unsiloed supplies layout-aware Markdown for retrieval and schema-shaped fields, confidence scores, and citations for structured workflows. The application-owned layer retains the source revision, checks completeness, applies consumer-specific policy, and publishes accepted results without mixing versions.
Together, these checks help keep incomplete, outdated, or unresolved results out of workflows that require reliable data. Try Unsiloed on your documents to see the structured output your ingestion layer can evaluate before it reaches RAG or an AI agent.
