Skip to main content

Install and configure

This example uses Python 3.12 to install dpb and its FastCDM dependency in a virtual environment.

Score one page first

Create two text-only Markdown files in the repository root. The reference says 120 USD, while the prediction reads 100 USD. Use the configuration file you just copied.
The terminal summary includes Files=1, DPB=0.9286, and Text=0.9286. The overall and text scores are equal because this page contains only text. In single-page.result.json, see summary for the scores and matches for match details.

Prepare references and model output

Download the dataset using the Hugging Face CLI installed with the toolkit. images/ contains the source pages, and markdowns/ contains the corresponding reference Markdown.
Run your model on the pages in images/ and save each page’s Markdown directly under predictions/model-a/. For example, save the prediction for education_chemistry_00001.png as predictions/model-a/education_chemistry_00001.md. The evaluator pairs only .md files with identical names at the top level of both directories.

Run the evaluation and check results

To use CDM, the default formula metric, install Node.js, Chrome, and a compatible ChromeDriver. Set chromedriver_path in config.yaml to your local driver path. ChromeDriver installation guide
For a batch run, the terminal displays Files, DPB, Text, Formula, Table, and FormulaRenderFailures. The JSON output contains aggregate scores in summary and page scores and segment matches in reports. Use the same page list and scoring configuration when comparing models. Check Files, the per-page reports, and the formula rendering failure count to confirm that the runs cover the same pages. The default configuration skips pages whose reference annotation contains \smiles{...}.

Score individual formulas with FastCDM

You can run the Python example below in a DocParsingBench environment. For formula scoring alone, run python -m pip install fastcdm in a virtual environment and use the browser and driver setup above. Replace the example path with your local ChromeDriver path before running it.
Recall is the share of reference characters that matched. Precision is the share of predicted characters that matched. F1 combines both measures. To inspect the character matches as an image, set visualize=True and read the fourth return value. FastCDM usage

Inspect differences page by page

When a page scores poorly, use dpb segment-report to export its reference, prediction, normalized Markdown, and segment breakdown. Replace the filename on the last line with the page you want to inspect.
In repository revision 90bf1fb, dpb visualize passes a css launch argument that Gradio 4.x rejects. If you encounter this error, inspect the per-page JSON or the segment report. Visual analysis launch code

Generate an interactive leaderboard

Save each model’s batch results as *.result.json files in results/ and run the following command. Open results/leaderboard.html to filter by industry and sort by DPB or component score.
For segment inspection, summary charts, and more configuration options, see the project documentation.