> ## Documentation Index
> Fetch the complete documentation index at: https://docs.somark.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Run an evaluation

> Install the DocParsingBench CLI, score Markdown in batches, inspect page results, and generate an interactive leaderboard.

## Install and configure

This example uses Python 3.12 to install `dpb` and its FastCDM dependency in a virtual environment.

```bash theme={null}
git clone https://github.com/SoMarkAI/DocParsingBench.git docparsingbench
cd docparsingbench
python3.12 -m venv .venv
source .venv/bin/activate
python -m pip install .
cp config.example.yaml config.yaml
```

## Score one page first

Create two text-only Markdown files in the repository root. The reference says `120 USD`, while the prediction reads `100 USD`. Use the configuration file you just copied.

```bash theme={null}
printf '%s\n' 'Total: 120 USD' > reference.md
printf '%s\n' 'Total: 100 USD' > prediction.md

dpb eval \
  --gt reference.md \
  --pred prediction.md \
  --config config.yaml \
  --out single-page.result.json
```

The terminal summary includes `Files=1`, `DPB=0.9286`, and `Text=0.9286`. The overall and text scores are equal because this page contains only text. In `single-page.result.json`, see `summary` for the scores and `matches` for match details.

## Prepare references and model output

Download the dataset using the Hugging Face CLI installed with the toolkit. `images/` contains the source pages, and `markdowns/` contains the corresponding reference Markdown.

```bash theme={null}
hf download SoMarkAI/DocParsingBench --repo-type dataset \
  --local-dir data/DocParsingBench
```

Run your model on the pages in `images/` and save each page's Markdown directly under `predictions/model-a/`. For example, save the prediction for `education_chemistry_00001.png` as `predictions/model-a/education_chemistry_00001.md`. The evaluator pairs only `.md` files with identical names at the top level of both directories.

## Run the evaluation and check results

To use CDM, the default formula metric, install Node.js, Chrome, and a compatible ChromeDriver. Set `chromedriver_path` in `config.yaml` to your local driver path. [ChromeDriver installation guide](https://github.com/SoMarkAI/FastCDM/blob/main/docs/chromedriver_installation.md)

```bash theme={null}
dpb eval \
  --gt data/DocParsingBench/markdowns \
  --pred predictions/model-a \
  --config config.yaml \
  --out results/model-a.result.json
```

For a batch run, the terminal displays `Files`, `DPB`, `Text`, `Formula`, `Table`, and `FormulaRenderFailures`. The JSON output contains aggregate scores in `summary` and page scores and segment matches in `reports`.

Use the same page list and scoring configuration when comparing models. Check `Files`, the per-page reports, and the formula rendering failure count to confirm that the runs cover the same pages. The default configuration skips pages whose reference annotation contains `\smiles{...}`.

## Score individual formulas with FastCDM

You can run the Python example below in a DocParsingBench environment. For formula scoring alone, run `python -m pip install fastcdm` in a virtual environment and use the browser and driver setup above. Replace the example path with your local ChromeDriver path before running it.

```python theme={null}
from fastcdm import FastCDM

evaluator = FastCDM(chromedriver="/path/to/chromedriver")
try:
    f1, recall, precision = evaluator.compute(
        gt="x^{2} + y^{2} = z^{2}",
        pred="x^{2} + y = z^{2}",
    )
    print(f"F1={f1:.4f}, recall={recall:.4f}, precision={precision:.4f}")
finally:
    evaluator.close()
```

Recall is the share of reference characters that matched. Precision is the share of predicted characters that matched. F1 combines both measures. To inspect the character matches as an image, set `visualize=True` and read the fourth return value. [FastCDM usage](https://github.com/SoMarkAI/FastCDM#usage)

## Inspect differences page by page

When a page scores poorly, use `dpb segment-report` to export its reference, prediction, normalized Markdown, and segment breakdown. Replace the filename on the last line with the page you want to inspect.

```bash theme={null}
dpb segment-report \
  --gt data/DocParsingBench/markdowns \
  --pred predictions/model-a \
  --out results/segment-report.md \
  education_chemistry_00001.md
```

> [!NOTE]
>
> In repository revision `90bf1fb`, `dpb visualize` passes a `css` launch argument that Gradio 4.x rejects. If you encounter this error, inspect the per-page JSON or the segment report. [Visual analysis launch code](https://github.com/SoMarkAI/DocParsingBench/blob/90bf1fbccc6d9967a9f9981bd5589eb5a8d9d6cd/docparsingbench/visualize/vis_result.py#L347)

## Generate an interactive leaderboard

Save each model's batch results as `*.result.json` files in `results/` and run the following command. Open `results/leaderboard.html` to filter by industry and sort by DPB or component score.

```bash theme={null}
dpb leaderboard-html \
  --gt data/DocParsingBench/markdowns \
  --results results \
  --output results/leaderboard.html
```

For segment inspection, summary charts, and more configuration options, see the [project documentation](https://github.com/SoMarkAI/DocParsingBench#usage).


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.