Skip to main content

What is DocParsingBench?

DocParsingBench is an open-source document parsing benchmark from SoMark. It includes a public dataset, a scoring CLI, and visual analysis tools. It compares a model’s Markdown output against a human annotation of the same page and reports text, formula, table, and overall DPB scores. Any model that produces Markdown can be compared on the same dataset and metrics.

GitHub repository

View the source code, scoring method, and public leaderboard.

Hugging Face dataset

Download the original page images and reference Markdown.

ModelScope dataset

Access the dataset through ModelScope.
For installation and scoring, see Run an evaluation.

Dataset coverage

The dataset contains 1,400 manually annotated pages from business workflows. It covers Chinese, English, and mixed-language pages, with single-column, two-column, three-column, and mixed layouts. The pages preserve conditions such as scan noise, overlapping stamps, and blurred characters. Dataset card

Scoring metrics

The toolkit segments the Markdown into text, display formulas, and tables, then pairs segments one to one within each category. Corresponding content can still match when a model places segments in a different order. Unmatched segments receive zero similarity. The general preset assigns 50% to text, 30% to display formulas, and 20% to tables. It redistributes weights across the categories present when a category is absent from both reference and prediction. The default configuration filters images and skips pages whose reference Markdown contains \smiles{...}, so the scored page count can differ from the dataset size. Example configuration

Formula scoring with FastCDM

FastCDM is the library DocParsingBench uses by default to score display formulas. You can also call it directly from Python. It renders reference and predicted LaTeX with KaTeX in Chrome, matches characters by symbol and position, and returns F1, recall, and precision. DocParsingBench uses F1 for its display formula score. FastCDM repository In the project’s published test of 1,000 formulas, the original CDM with TeX Live and ImageMagick took 608 seconds. FastCDM with Chrome and KaTeX took 97 seconds, a 6.26× speedup for that formula evaluation. The LaTeX commands you can render depend on KaTeX support. Benchmark results · KaTeX command support

Inspect errors behind the scores

The visual analysis tool puts the original page, reference Markdown, predicted Markdown, and scores in one view. Filter by industry, subindustry, or sample to see whether a low score came from paragraphs, formulas, or tables. You can also combine batch runs into an interactive HTML leaderboard and sort by overall score or any component. Official interface example The published leaderboard reports 0.8776 DPB for OXR-1.0, with 0.9060 for text, 0.8576 for formulas, and 0.8004 for tables. Official leaderboard The toolkit uses the Apache License 2.0. The dataset uses ODC-BY. Dataset license