> ## Documentation Index
> Fetch the complete documentation index at: https://docs.somark.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# mJev

> Ask multiple questions about the same image, video, or audio and use mJev to choose from the answers you define.

**mJev** is an open-source tool for multimodal inference. Give it a media file and shared context, then define candidate answers for each question. It returns the selected answer and each candidate's score for use in classification, annotation, or further processing.

> [card] [GitHub repository](https://github.com/SoMarkAI/mJev)
>
> View the README, examples, and source code.

> [card] [mJev-Qwen3-VL-4B-RLCD](https://huggingface.co/SoMarkAI/mJev-Qwen3-VL-4B-RLCD)
>
> Explore and download the model fine-tuned from Qwen3-VL-4B-Instruct.

## Six-second video example

The project used `Qwen3-VL-4B-Instruct` with the Transformers backend to answer three questions about a six-second animation. The commands below use the same model revision, inputs, and sampling settings as the published run.

| Question | Model choice | Candidate probability |
| - | - | -: |
| How does the red ball move? | From left to right | 71.34% |
| Which object moves upward? | The blue square | 97.29% |
| Which movement happens first? | The red ball moves right | 89.62% |

[Original video](https://github.com/SoMarkAI/mJev/blob/main/examples/motion-demo/motion.mp4) · [Full output](https://github.com/SoMarkAI/mJev/blob/main/examples/motion-demo/recorded-output.json)

## Prepare your environment

Prepare Linux, Python 3.11 or later, a compatible NVIDIA driver, and system FFmpeg. The project's published inference record for the six-second video used one 24 GB NVIDIA GPU.

```bash theme={null}
git clone https://github.com/SoMarkAI/mJev.git
cd mJev
python3 -m venv .venv
source .venv/bin/activate
python -m pip install torchcodec==0.11.0+cpu --index-url https://download.pytorch.org/whl/cpu
python -m pip install -e '.[hf-vl]' huggingface_hub

hf download Qwen/Qwen3-VL-4B-Instruct \
  --revision ebb281ec70b05090aa6165b016eac8ec08e71b17 \
  --local-dir models/Qwen3-VL-4B-Instruct
```

The CPU build of `torchcodec` decodes video, and PyTorch runs inference on the GPU.

## Run the video example

Run this command from the repository root. The example JSON contains the video path, sampling settings, and three questions. The command uses GPU `0`, enables prefix caching, and processes the three questions in a batch.

```bash theme={null}
CUDA_VISIBLE_DEVICES=0 python demo_hf.py \
  --model models/Qwen3-VL-4B-Instruct \
  --input examples/motion-demo/input.json \
  --mode causal --numerics stable --projection full \
  --prefix-cache --question-batch-size 3 \
  --output outputs/motion-demo.json
```

When the command finishes, open `outputs/motion-demo.json` to inspect the results. [Video example input](https://github.com/SoMarkAI/mJev/blob/main/examples/motion-demo/input.json)

To run the same video with the fine-tuned mJev model, download `SoMarkAI/mJev-Qwen3-VL-4B-RLCD` and point `--model` to its local directory. Set a different `--output` path so you can compare the two results. [Fine-tuned model download and quickstart](https://huggingface.co/SoMarkAI/mJev-Qwen3-VL-4B-RLCD#quick-start)

## Use your own media

Save the following JSON as `task.json`, place `clip.mp4` in the same directory, and point `--input` to `task.json`. `media_path` is relative to the input JSON file. For reproducible video results, set the sampling rate and the maximum number of frames.

```json theme={null}
{
  "modality": "video",
  "media_path": "clip.mp4",
  "context": "Look at the vehicles in the frame and their direction of motion.",
  "video_options": { "fps": 2, "min_pixels": 3136, "max_pixels": 100352, "max_frames": 16 },
  "questions": [
    {
      "id": "vehicle",
      "question": "What kind of vehicle is the main moving object?",
      "candidates": ["Bicycle", "Car", "Bus", "Cannot tell"]
    },
    {
      "id": "direction",
      "question": "In which direction does it mostly move in the frame?",
      "candidates": ["Left", "Right", "Another direction", "Cannot tell"]
    }
  ]
}
```

For an image, set `"modality": "image"`, point `media_path` to the image, and remove `video_options`. Use `audio` for audio alone or `audio_video` to analyze video and its audio track together. Both audio modes use the Qwen3-Omni model. See the [Transformers deployment guide](https://github.com/SoMarkAI/mJev/blob/main/docs/hf.md#demo) for dependencies and startup instructions.

## Read the results

The output is a JSON array with one entry per question. `decision` contains the selected candidate. `candidates` lists every option with its `raw_logit` and `probability`. The probabilities are normalized over the supplied candidates for that question, and `probability_sum` reports their total.

Candidate answers must be distinct, nonempty strings. The Transformers demo limits the full input to 4,000 tokens by default and returns an error if it exceeds the limit. To check the input length and candidate labels before loading the model weights, add `--check-only` and omit `--output` to see the results in your terminal.

When scores tie, mJev selects the earlier candidate in input order.

## Models and inputs

| Model | Supported inputs | Use cases |
| - | - | - |
| `mJev-Qwen3-VL-4B-RLCD` | Images or video, plus text questions and options | Identify visual content, actions, and event order |
| `Qwen3-Omni-30B-A3B-Instruct` | Images, audio, video, or video with audio, plus text questions and options | Detect speech or make a choice using sound and visuals together |

Both models use the same format for questions and candidates.

* Compare the two model families, dependencies, and supported inputs. [Model guide](https://github.com/SoMarkAI/mJev/blob/main/docs/models.md)
* For vLLM, follow the project's pinned version and GPU configuration. [vLLM deployment guide](https://github.com/SoMarkAI/mJev/blob/main/docs/vllm.md)

## Public evaluation

* **Accuracy**. On the same 195 held-out questions from `mJev-Compositional-VQA`, the base model answered 152 correctly (77.95%) and the fine-tuned mJev model answered 156 correctly (80.00%). [Model card and results](https://huggingface.co/SoMarkAI/mJev-Qwen3-VL-4B-RLCD#evaluation)
* **Prefix caching**. On one 24 GB NVIDIA GPU, the project tested 16 questions about the same video. Caching reduced total latency from 32.62 s to 6.29 s and increased peak allocated GPU memory from 15.80 GiB to 20.60 GiB. The project also recorded results for shorter inputs and single questions. [Cache benchmark conditions](https://github.com/SoMarkAI/mJev/blob/main/docs/cache_scaling.md)

The code uses the [Apache License 2.0](https://github.com/SoMarkAI/mJev/blob/main/LICENSE). Download the models separately and see their model cards for licensing terms.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.