> ## Documentation Index
> Fetch the complete documentation index at: https://docs.somark.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Overview

> Evaluate Markdown output on real business documents, compare text, formula, and table scores, and inspect individual errors.

## What is DocParsingBench?

**DocParsingBench** is an open-source document parsing benchmark from SoMark. It includes a public dataset, a scoring CLI, and visual analysis tools. It compares a model's Markdown output against a human annotation of the same page and reports text, formula, table, and overall DPB scores. Any model that produces Markdown can be compared on the same dataset and metrics.

> [card] [GitHub repository](https://github.com/SoMarkAI/DocParsingBench)
>
> View the source code, scoring method, and public leaderboard.

> [card] [Hugging Face dataset](https://huggingface.co/datasets/SoMarkAI/DocParsingBench)
>
> Download the original page images and reference Markdown.

> [card] [ModelScope dataset](https://modelscope.cn/datasets/SoMark/DocParsingBench)
>
> Access the dataset through ModelScope.

For installation and scoring, see [Run an evaluation](https://docs.somark.ai/en/open-source-tools/docparsingbench/usage.md).

## Dataset coverage

The dataset contains **1,400 manually annotated pages** from business workflows. It covers Chinese, English, and mixed-language pages, with single-column, two-column, three-column, and mixed layouts. The pages preserve conditions such as scan noise, overlapping stamps, and blurred characters. [Dataset card](https://huggingface.co/datasets/SoMarkAI/DocParsingBench)

| Domain | Example documents | Challenges you can evaluate |
| - | - | - |
| Finance | Brokerage research and company annual reports | Dense column layouts and complex tables |
| Legal | Contract clauses and legal documents | Dense footnotes, headers, and footers |
| Research | Papers, patents, and programming textbooks | Mixed columns, formulas, and code blocks |
| Manufacturing | Operating procedures, forms, and receipts | Blurry scans and handwritten fields |
| Education | English, chemistry, and mathematics textbooks | Answer choices, blanks, and formulas |

## Scoring metrics

The toolkit segments the Markdown into text, display formulas, and tables, then pairs segments one to one within each category. Corresponding content can still match when a model places segments in a different order. Unmatched segments receive zero similarity.

| Component | Default metric | What it measures |
| - | - | - |
| Text | `1 - NED` | Compares text by normalized edit distance, which also scores inline formulas by default |
| Display formulas | CDM, implemented by FastCDM | Character and position matching after formula rendering |
| Tables | TEDS | Table structure and cell content |

The general preset assigns **50%** to text, **30%** to display formulas, and **20%** to tables. It redistributes weights across the categories present when a category is absent from both reference and prediction. The default configuration filters images and skips pages whose reference Markdown contains `\smiles{...}`, so the scored page count can differ from the dataset size. [Example configuration](https://github.com/SoMarkAI/DocParsingBench/blob/main/config.example.yaml)

### Formula scoring with FastCDM

**FastCDM** is the library DocParsingBench uses by default to score display formulas. You can also call it directly from Python. It renders reference and predicted LaTeX with KaTeX in Chrome, matches characters by symbol and position, and returns F1, recall, and precision. DocParsingBench uses F1 for its display formula score. [FastCDM repository](https://github.com/SoMarkAI/FastCDM)

In the project's published test of 1,000 formulas, the original CDM with TeX Live and ImageMagick took 608 seconds. FastCDM with Chrome and KaTeX took 97 seconds, a **6.26×** speedup for that formula evaluation. The LaTeX commands you can render depend on KaTeX support. [Benchmark results](https://github.com/SoMarkAI/FastCDM#cdm-vs-fastcdm) · [KaTeX command support](https://katex.org/docs/support_table)

## Inspect errors behind the scores

The visual analysis tool puts the original page, reference Markdown, predicted Markdown, and scores in one view. Filter by industry, subindustry, or sample to see whether a low score came from paragraphs, formulas, or tables. You can also combine batch runs into an interactive HTML leaderboard and sort by overall score or any component. [Official interface example](https://github.com/SoMarkAI/DocParsingBench/blob/main/assets/visual-analysis.jpeg)

The published leaderboard reports **0.8776** DPB for OXR-1.0, with **0.9060** for text, **0.8576** for formulas, and **0.8004** for tables. [Official leaderboard](https://github.com/SoMarkAI/DocParsingBench#evaluation-leaderboard)

The toolkit uses the [Apache License 2.0](https://github.com/SoMarkAI/DocParsingBench/blob/main/LICENSE). The dataset uses ODC-BY. [Dataset license](https://huggingface.co/datasets/SoMarkAI/DocParsingBench)


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.