Document Processing
Turn files into clean, typed elements with provenance: choose a parser tier per file, keep tables as structure, clean without deleting content, dedupe without dropping new versions, and gate the output.
Last updated
After this section you can
- Choose a parser tier for each file from its cost, licence and where it runs
- Keep reading order, tables and each element’s page and position through parsing, and clean without deleting content
- Explain why a near-duplicate re-send replaces the old version, and gate parser output before it is indexed
03
Document Processing: Parse to Structure
The vector can only be as good as the text you feed it.
Key idea
Every later stage inherits what parsing keeps: a swapped column, a flattened table or a deleted clause is invisible to chunking, embedding and retrieval. Parse to typed elements, clean by tested rules, gate the index.
One page in, typed elements out; furniture dropped
| Element and box | Text |
|---|---|
1 section_header50, 90, 285, 124 | 4. Refunds |
2 text50, 138, 285, 400 | Refunds are accepted… |
3 text50, 414, 285, 640 | Table 1 shows how long… |
4 table310, 90, 545, 330 | <table> 4 rows × 2 columns |
5 picture310, 346, 545, 590 | no text; its caption is 6 |
6 caption310, 598, 545, 632 | Figure 2. How a refund is… |
page_header page_footer | furniture: repeats on every page, dropped |
Illustrative Acme page; boxes in points.