Metadata-Version: 2.1
Name: llama_cpp_python
Version: 0.4.1
Summary: Efficient Python bindings for the llama.cpp library
Author-Email: Andrei Betlen <abetlen@gmail.com>, JamePeng <jame_peng@sina.com>
Maintainer-Email: JamePeng <jame_peng@sina.com>
License: MIT
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Project-URL: Homepage, https://github.com/JamePeng/llama-cpp-python
Project-URL: Issues, https://github.com/JamePeng/llama-cpp-python/issues
Project-URL: Documentation, https://github.com/JamePeng/llama-cpp-python/wiki
Project-URL: Changelog, https://github.com/JamePeng/llama-cpp-python/blob/main/CHANGELOG.md
Project-URL: FAQ, https://github.com/JamePeng/llama-cpp-python?tab=readme-ov-file#faq
Requires-Python: >=3.9
Requires-Dist: typing-extensions>=4.8.0
Requires-Dist: numpy<=2.3.2,>=1.21.6
Requires-Dist: diskcache>=5.6.2
Requires-Dist: jinja2>=3.1.0
Requires-Dist: Pillow>=9.5.0
Provides-Extra: server
Requires-Dist: uvicorn>=0.22.0; extra == "server"
Requires-Dist: fastapi>=0.100.0; extra == "server"
Requires-Dist: pydantic-settings>=2.0.1; extra == "server"
Requires-Dist: sse-starlette>=1.6.1; extra == "server"
Requires-Dist: starlette-context<0.4,>=0.3.6; extra == "server"
Requires-Dist: PyYAML>=5.1; extra == "server"
Provides-Extra: test
Requires-Dist: pytest>=7.4.0; extra == "test"
Requires-Dist: httpx>=0.24.1; extra == "test"
Requires-Dist: scipy>=1.10; extra == "test"
Requires-Dist: fastapi>=0.100.0; extra == "test"
Requires-Dist: sse-starlette>=1.6.1; extra == "test"
Requires-Dist: starlette-context<0.4,>=0.3.6; extra == "test"
Requires-Dist: pydantic-settings>=2.0.1; extra == "test"
Requires-Dist: huggingface-hub>=0.23.0; extra == "test"
Provides-Extra: all
Requires-Dist: llama_cpp_python[server,test]; extra == "all"
Description-Content-Type: text/markdown

<p align="center">
  <img src="docs/icon.png" alt="llama-cpp-python logo" width="300">
</p>

# Efficient Python Bindings for [`llama.cpp`](https://github.com/ggml-org/llama.cpp) library

[![Tests](https://github.com/JamePeng/llama-cpp-python/actions/workflows/test.yaml/badge.svg?branch=main)](https://github.com/JamePeng/llama-cpp-python/actions/workflows/test.yaml)
![GitHub Tag](https://img.shields.io/github/v/tag/JamePeng/llama-cpp-python)
[![PyPI - License](https://img.shields.io/pypi/l/llama-cpp-python)](https://pypi.org/project/llama-cpp-python/)
[![PyPI - Downloads](https://static.pepy.tech/badge/llama-cpp-python/month)](https://pepy.tech/projects/llama-cpp-python)
[![GitHub Downloads](https://img.shields.io/github/downloads/JamePeng/llama-cpp-python/total.svg?label=GitHub%20Downloads)](https://github.com/JamePeng/llama-cpp-python/releases)

Efficient Python bindings for **ggml-org's** [`llama.cpp`](https://github.com/ggml-org/llama.cpp) library.
This package provides:

- Low-level access to C API via `ctypes` interface.
    - [Low-level API](#low-level-api)
    - [Low-level tutorial and examples](examples/low_level_api/README.md)
    - [llama_cpp_lib](https://github.com/JamePeng/llama-cpp-python/blob/main/llama_cpp/llama_cpp.py)
    - [mtmd_cpp_lib](https://github.com/JamePeng/llama-cpp-python/blob/main/llama_cpp/mtmd_cpp.py)
    - [ggml_cpp_lib](https://github.com/JamePeng/llama-cpp-python/blob/main/llama_cpp/_ggml.py)
        - *Note: Synchronize ggml's ctypes calls as needed, but won't fully implement it, because most of it is called at the lower level in the upstream llama.cpp.*
- High-level Python API for text completion
    - OpenAI-like API and Type([llama_types.py](https://github.com/JamePeng/llama-cpp-python/blob/main/llama_cpp/llama_types.py))
    - [High-level API](#high-level-api)
    - [Continuing Assistant Responses (Prefill)](https://github.com/JamePeng/llama-cpp-python#continuing-assistant-responses-prefill)
    - [Dynamic LoRA Routing & Control Vectors (Multi-Tenant Serving)](https://github.com/JamePeng/llama-cpp-python#dynamic-lora-routing--control-vectors-multi-tenant-serving)
        - [Dynamic LoRA Example](https://github.com/JamePeng/llama-cpp-python#dynamic-lora-example)
        - [Control Vector Injection (Representation Engineering)](https://github.com/JamePeng/llama-cpp-python#control-vector-injection-representation-engineering)
    - [Sampling Configuration & Usage (LlamaSamplingParams)](https://github.com/JamePeng/llama-cpp-python#sampling-configuration--usage-llamasamplingparams)
        - [How to use the ReasoningBudgetSampler](https://github.com/JamePeng/llama-cpp-python#reasoning-budget-first-reasoning-block)
    - [Speculative Decoding](https://github.com/JamePeng/llama-cpp-python#speculative-decoding)
        - [MTP speculative decoding](https://github.com/JamePeng/llama-cpp-python#mtp-speculative-decoding)
        - [DFlash, DFlash2, and DSpark speculative decoding](https://github.com/JamePeng/llama-cpp-python#dflash-dflash2-and-dspark-speculative-decoding)
        - [N-gram speculative decoding](https://github.com/JamePeng/llama-cpp-python#n-gram-speculative-decoding)
    - [Multi-modal Models Support](https://github.com/JamePeng/llama-cpp-python#multi-modal-models)
        - Support Models Lists
        - [Introducing Generic MTMD Chat Handler](https://github.com/JamePeng/llama-cpp-python#generic-mtmd-chat-handler)
        - [Loading a Local Video With Generic MTMD](https://github.com/JamePeng/llama-cpp-python#loading-a-local-video-with-generic-mtmd)
        - [Loading a Local Image With Qwen3VL(Thinking/Instruct)](https://github.com/JamePeng/llama-cpp-python#loading-a-local-image-with-qwen3vlthinkinginstruct)
        - [Speech Recognition With Qwen3-ASR (Speech-to-Text)](https://github.com/JamePeng/llama-cpp-python#speech-recognition-with-qwen3-asr-speech-to-text)
        - [Speech Synthesis With MTMD (Text-to-Speech)](#speech-synthesis-with-mtmd-text-to-speech)
        - [Comprehensive Omni MultiModal Example: Gemma-4 (Vision + Audio + Video + Text)](https://github.com/JamePeng/llama-cpp-python#comprehensive-omni-multimodal-example-gemma-4-vision--audio--video--text)
    - [Embeddings & Reranking (GGUF)](https://github.com/JamePeng/llama-cpp-python#embeddings--reranking-gguf)
        - [1. Text Embeddings (Vector Search)](https://github.com/JamePeng/llama-cpp-python#1-text-embeddings-vector-search)
        - [2. Reranking (Cross-Encoder Scoring)](https://github.com/JamePeng/llama-cpp-python#2-reranking-cross-encoder-scoring)
        - [3. Normalization](https://github.com/JamePeng/llama-cpp-python#3-normalization)
- [FAQ](https://github.com/JamePeng/llama-cpp-python#faq)

The project's maintained documentation is available in [docs/wiki](docs/wiki/index.md)
and published to the [project wiki](https://github.com/JamePeng/llama-cpp-python/wiki).
It follows the LLM Wiki approach, and contributions to its maintenance are welcome :)


## Discussions

Starting March 2026, I am excited to announce that we have officially enabled the **Discussions** tab for `llama-cpp-python`!

You can access it right here: [GitHub Discussions](https://github.com/JamePeng/llama-cpp-python/discussions).

**Why Discussions? & Updates on Documentation**
As the project has evolved, some older documentation pages have become bloated or outdated. For timely updates alongside the maintained wiki:

* **New Feature Releases:** Moving forward, whenever a new feature is rolled out, I will publish a dedicated standalone article in the Discussions section. These posts will include detailed explanations, usage guides, and important caveats.
* Discussions provide interactive release notes and usage articles alongside the maintained [`docs/wiki`](docs/wiki/index.md) reference.

**Join the Community**
I warmly welcome all of you to use this new space. Let's build together:

* 💬 **Discuss & Share:** Have a question, an idea, or a cool use case? Share it with the community!
* 🛠️ **Maintain & Test:** Help us test new features, troubleshoot issues, and collaboratively maintain the repository.
* 📚 **Learn & Grow:** I hope everyone can benefit from this project, learn from each other, and gain valuable insights.

Thank you for your continuous support!

## Installation

For a structured source-install and backend build guide, see [docs/wiki/install.md](https://github.com/JamePeng/llama-cpp-python/blob/main/docs/wiki/install.md).

Requirements:

  - Python 3.9+
  - C compiler
      - Linux: gcc or clang
      - Windows: [`Visual Studio 2022 Build Tools`](https://download.visualstudio.microsoft.com/download/pr/6efb3484-905b-485c-8b5f-9d3a5f39e731/07908cd6d91e75b8ea4339d8f2cfa6e8d8bb4fd706af7b918ae391cd6fc2a066/vs_BuildTools.exe) or `MinGW`
      - MacOS: Xcode
  - CMake 3.21+
  - Git

To install the package, run:
- Method 1:
    ```bash
    pip install "llama-cpp-python @ git+https://github.com/JamePeng/llama-cpp-python.git"
    ```
- Method 2:
    ```bash
    git clone https://github.com/JamePeng/llama-cpp-python --recursive
    cd llama-cpp-python
    python -m pip install -U pip
    pip install .
    ```

This will also build `llama.cpp` from source and install it alongside this python package.

If this fails, add `--verbose` to the `pip install` see the full cmake build log.

### Installation Configuration

`llama.cpp` supports a number of hardware acceleration backends to speed up inference as well as backend specific options. See the [llama.cpp build docs](https://github.com/ggml-org/llama.cpp/blob/master/docs/build.md) for a full list.

All `llama.cpp` cmake build options can be set via the `CMAKE_ARGS` environment variable or via the `--config-settings / -C` cli flag during installation.

<details open>
<summary>Environment Variables</summary>

```bash
# Linux and Mac
CMAKE_ARGS="-DGGML_BLAS=ON -DGGML_BLAS_VENDOR=OpenBLAS" \
  pip install "llama-cpp-python @ git+https://github.com/JamePeng/llama-cpp-python.git"
```

```powershell
# Windows powershell
$env:CMAKE_ARGS = "-DGGML_BLAS=ON -DGGML_BLAS_VENDOR=OpenBLAS"
pip install "llama-cpp-python @ git+https://github.com/JamePeng/llama-cpp-python.git"
```

```command prompt
# Windows command prompt
set CMAKE_ARGS = "-DGGML_BLAS=ON -DGGML_BLAS_VENDOR=OpenBLAS"
pip install "llama-cpp-python @ git+https://github.com/JamePeng/llama-cpp-python.git"
```
</details>

**Sanity Checking**  
Use this line to check if installation was successful before moving further.  
`python.exe -c "from llama_cpp import Llama; print('llama-cpp import OK')"`

<details>
<summary>CLI / requirements.txt</summary>

They can also be set via `pip install -C / --config-settings` command and saved to a `requirements.txt` file:

```bash
pip install --upgrade pip # ensure pip is up to date
pip install "llama-cpp-python @ git+https://github.com/JamePeng/llama-cpp-python.git" \
  -C cmake.args="-DGGML_BLAS=ON;-DGGML_BLAS_VENDOR=OpenBLAS"
```

```txt
# requirements.txt

llama-cpp-python -C cmake.args="-DGGML_BLAS=ON;-DGGML_BLAS_VENDOR=OpenBLAS"
```

</details>

### Supported Backends

Below are some common backends, their build commands and any additional environment variables required.

<details open>
<summary>CUDA</summary>

Installing a CUDA-supported version requires the `CUDA Toolkit` environment to be installed first.

**Note: Please select and install according to your system environment and local graphics card model to ensure that the compilation is based on the optimal local environment.**

See here: https://developer.nvidia.com/cuda-toolkit-archive

More Information see: https://github.com/ggml-org/llama.cpp/blob/master/docs/build.md#cuda

Then, set the `GGML_CUDA=on` environment variable before installing:

```bash
# Linux
CMAKE_ARGS="-DGGML_CUDA=on" pip install "llama-cpp-python @ git+https://github.com/JamePeng/llama-cpp-python.git"
```

```powershell
# Windows
$env:CMAKE_ARGS = "-DGGML_CUDA=on"
pip install "llama-cpp-python @ git+https://github.com/JamePeng/llama-cpp-python.git"
```

**Programmatic Dependent Launch (PDL)** is available on compatible CUDA
toolkits and Hopper-class or newer GPUs (compute capability 9.0 or later; Ada
is not included). The bundled backend compiles PDL support automatically when
the CUDA toolchain supports it. `GGML_CUDA_PDL=0` disables it at runtime;
leaving the variable unset enables it when the selected kernel supports PDL.

**Pre-built Wheel (New)**

It is also possible to install a pre-built wheel with CUDA support. Make sure your system meets the following requirements:

- CUDA version: 12.4, 12.6, 12.8, or 13.1
- Python version: 3.10, 3.11, 3.12, 3.13, or 3.14
- Starting with `0.3.39-preview`, Windows and Linux x64 wheels are built with `GGML_BACKEND_DL` and `GGML_CPU_ALL_VARIANTS`.

This means CPU backends are shipped as dynamically loaded runtime libraries under:

```text
site-packages/llama_cpp/lib
```

Supported CPU backend variants may include:

* `ggml-cpu-x64`
* `ggml-cpu-sse42`
* `ggml-cpu-sandybridge`
* `ggml-cpu-ivybridge`
* `ggml-cpu-piledriver`
* `ggml-cpu-haswell`
* `ggml-cpu-skylakex`
* `ggml-cpu-cannonlake`
* `ggml-cpu-cascadelake`
* `ggml-cpu-cooperlake`
* `ggml-cpu-icelake`
* `ggml-cpu-alderlake`
* `ggml-cpu-sapphirerapids`
* `ggml-cpu-zen4`

The old `Basic` and `AVX2` wheel variants are no longer required for the new dynamic-backend wheels. GGML can load the compatible CPU backend at runtime, which improves CPU instruction-set compatibility across different x64 machines.

Before `0.3.39-preview`:

* `Basic`: compiled without AVX instructions for maximum compatibility.
* `AVX2`: compiled with AVX2 instructions for newer CPUs.

Check the releases page:
https://github.com/JamePeng/llama-cpp-python/releases

</details>

<details>
<summary>OpenBLAS (CPU)</summary>

To install with OpenBLAS, set the `GGML_BLAS` and `GGML_BLAS_VENDOR` environment variables before installing:

```bash
CMAKE_ARGS="-DGGML_BLAS=ON -DGGML_BLAS_VENDOR=OpenBLAS" pip install "llama-cpp-python @ git+https://github.com/JamePeng/llama-cpp-python.git"
```
</details>

<details>
<summary>OpenVINO</summary>

### Install OpenVINO Runtime

Follow the guide to install OpenVINO Runtime from an archive file: [Linux](https://docs.openvino.ai/2026/get-started/install-openvino/install-openvino-archive-linux.html) | [Windows](https://docs.openvino.ai/2026/get-started/install-openvino/install-openvino-archive-windows.html)

- **Linux:**

    <details>
    <summary>📦 Click to expand OpenVINO installation from an archive file on Ubuntu</summary>
    <br>

    ```bash
    wget https://raw.githubusercontent.com/ravi9/misc-scripts/main/openvino/ov-archive-install/install-openvino-from-archive.sh
    chmod +x install-openvino-from-archive.sh
    ./install-openvino-from-archive.sh
    ```

    Verify OpenVINO is initialized properly:
    ```bash
    echo $OpenVINO_DIR
    ```
    </details>

### Supported Devices

OpenVINO backend supports the following hardware:

- Intel CPUs
- Intel GPUs (integrated and discrete)
- Intel NPUs

Although OpenVINO supports a wide range of [Intel hardware](https://docs.openvino.ai/2026/about-openvino/release-notes-openvino/system-requirements.html), the llama.cpp OpenVINO backend has been validated specifically on AI PCs such as the Intel® Core™ Ultra Series 1 and Series 2.

More Information see: https://github.com/ggml-org/llama.cpp/blob/master/docs/backend/OPENVINO.md

To install with OpenVINO, set the `GGML_OPENVINO=ON` environment variable before installing:

```bash
# Linux
source /opt/intel/openvino/setupvars.sh
# Windows
"C:\Program Files (x86)\Intel\openvino_2026.0\setupvars.bat"
# Build
CMAKE_ARGS="-DGGML_OPENVINO=ON" pip install "llama-cpp-python @ git+https://github.com/JamePeng/llama-cpp-python.git"
```
</details>

<details>
<summary>Metal</summary>

On MacOS, Metal is enabled by default(`GGML_METAL=ON`). Using Metal makes the computation run on the GPU.

To disable the Metal build at compile time use the `CMAKE_ARGS="-DGGML_METAL=OFF"` cmake option.

In Python, `n_gpu_layers=0` disables model-layer offload. It does not guarantee
that every operation avoids the GPU. Build with `GGML_METAL=OFF` when a CPU-only
Metal-free package is required.

```bash
pip install "llama-cpp-python @ git+https://github.com/JamePeng/llama-cpp-python.git"
```

**Pre-built Wheel (New)**

It is also possible to install a pre-built wheel with Metal support. As long as your system meets some requirements:

- CPU Arch: arm64
- MacOS Version is 11.0 or later
- Python Version is 3.10, 3.11, 3.12, 3.13 or 3.14

Check the releases page:
https://github.com/JamePeng/llama-cpp-python/releases

</details>

<details>
<summary>HIP (ROCm)</summary>

  - <details>
    <summary>Linux ROCm</summary>

    This provides GPU acceleration on HIP-supported AMD GPUs. Make sure to have ROCm installed.

    You can download it from your Linux distro's package manager or from here: [ROCm Quick Start (Linux)](https://rocm.docs.amd.com/projects/install-on-linux/en/latest/tutorial/quick-start.html#rocm-install-quick).

    To install with HIP / ROCm support for AMD cards, set the `GGML_HIP=ON` environment variable before installing:

    ```bash
    CMAKE_ARGS="-DGGML_HIP=ON -DGPU_TARGETS=gfx1030" pip install "llama-cpp-python @ git+https://github.com/JamePeng/llama-cpp-python.git"
    ```
    Note: `GPU_TARGETS` is optional, omitting it will build the code for all GPUs in the current system.

    More details see here: https://github.com/ggml-org/llama.cpp/blob/master/docs/build.md#hip

    </details>

  - <details>
    <summary>Windows ROCm</summary>

    > **Note:** Install TheRock ROCm, activate your venv, then run in PowerShell. Replace `gfx1200` with your GPU architecture.

    ```powershell
    cmd /c '"C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Auxiliary\Build\vcvars64.bat" >nul 2>&1 && set' | ForEach-Object { if ($_ -match '^([^=]+)=(.*)$') { [System.Environment]::SetEnvironmentVariable($matches[1], $matches[2], 'Process') } }

    rocm-sdk init

    $ROCM_DEVEL = "$env:VIRTUAL_ENV\Lib\site-packages\_rocm_sdk_devel"
    $ROCM_CORE  = "$env:VIRTUAL_ENV\Lib\site-packages\_rocm_sdk_core"
    $ROCM_GFX   = (Get-Item "$env:VIRTUAL_ENV\Lib\site-packages\_rocm_sdk_libraries_gfx*").FullName

    $env:HIP_PATH          = $ROCM_DEVEL
    $env:ROCM_PATH         = $ROCM_DEVEL
    $env:HIP_DEVICE_LIB_PATH = "$ROCM_CORE\lib\llvm\amdgcn\bitcode"
    $env:PATH              = "$ROCM_DEVEL\bin;$ROCM_DEVEL\lib\llvm\bin;$ROCM_GFX\bin;$env:PATH"
    $env:CMAKE_GENERATOR   = "Ninja"
    $env:HIP_PLATFORM      = "amd"
    $env:CC                = "$ROCM_DEVEL\lib\llvm\bin\clang.exe"
    $env:CXX               = "$ROCM_DEVEL\lib\llvm\bin\clang++.exe"
    $env:HIP_CLANG_PATH    = "$ROCM_DEVEL\lib\llvm\bin"

    $R = $ROCM_DEVEL -replace '\\', '/'
    $env:CMAKE_ARGS = "-DGGML_HIP=ON -DGGML_HIPBLAS=on -DGPU_TARGETS=gfx1200 -DCMAKE_HIP_ARCHITECTURES=gfx1200 -DCMAKE_C_COMPILER=`"$R/lib/llvm/bin/clang.exe`" -DCMAKE_CXX_COMPILER=`"$R/lib/llvm/bin/clang++.exe`" -DHIP_LIBRARIES=`"$R/lib/amdhip64.lib`" -DCMAKE_PREFIX_PATH=`"$R`""

    pip install "llama-cpp-python @ git+https://github.com/JamePeng/llama-cpp-python.git" --no-cache-dir
    ```

    </details>

</details>

<details>
<summary>Vulkan</summary>

- For Windows User: Download and install the [`Vulkan SDK`](https://vulkan.lunarg.com/sdk/home#windows) with the default settings.

- For Linux User:
    * First, follow the official LunarG instructions for the installation and setup of the Vulkan SDK in the [Getting Started with the Linux Tarball Vulkan SDK](https://vulkan.lunarg.com/doc/sdk/latest/linux/getting_started.html) guide.

    * After completing the first step, ensure that you have used the `source` command on the `setup_env.sh` file inside of the Vulkan SDK in your current terminal session. Otherwise, the build won't work. Additionally, if you close out of your terminal, you must perform this step again if you intend to perform a build. However, there are ways to make this persistent. Refer to the Vulkan SDK guide linked in the first step for more information about any of this.

- For Mac User:
    * Generally, follow LunarG's [Getting Started with the MacOS Vulkan SDK](https://vulkan.lunarg.com/doc/sdk/latest/mac/getting_started.html) guide for installation and setup of the Vulkan SDK. There are two options of Vulkan drivers on macOS, both of which implement translation layers to map Vulkan to Metal. They can be hot-swapped by setting the `VK_ICD_FILENAMES` environment variable to point to the respective ICD JSON file. Check the box for "KosmicKrisp" during the LunarG Vulkan SDK installation.

    * Set environment variable for the LunarG Vulkan SDK after installation (and optionally add to your shell profile for persistence):
        ```bash
        source /path/to/vulkan-sdk/setup-env.sh
        ```

More Information see: https://github.com/ggml-org/llama.cpp/blob/master/docs/build.md#vulkan

Then install with Vulkan support by set the `GGML_VULKAN=on` environment variable before installing:

```bash
CMAKE_ARGS="-DGGML_VULKAN=on" pip install "llama-cpp-python @ git+https://github.com/JamePeng/llama-cpp-python.git"
```

</details>

<details>
<summary>SYCL</summary>

### Supported OS

| OS      | Status  | Verified                                       |
|---------|---------|------------------------------------------------|
| Linux   | Support | Ubuntu 22.04, Fedora Silverblue 39, Arch Linux |
| Windows | Support | Windows 11                                     |

### Intel GPU

SYCL backend supports Intel GPU Family:

- Intel Data Center Max Series
- Intel Flex Series, Arc Series
- Intel Built-in Arc GPU
- Intel iGPU in Core CPU (11th Generation Core CPU and newer, refer to [oneAPI supported GPU](https://www.intel.com/content/www/us/en/developer/articles/system-requirements/intel-oneapi-base-toolkit-system-requirements.html#inpage-nav-1-1)).

For OpenCL backend requirements and configuration, see the bundled [OpenCL guide](vendor/llama.cpp/docs/backend/OPENCL.md).

More Information see here: https://github.com/ggml-org/llama.cpp/blob/master/docs/backend/SYCL.md

To install with SYCL support, set the `GGML_SYCL=on` environment variable before installing:

```bash
# Export relevant ENV variables
source /opt/intel/oneapi/setvars.sh
# Option 1: Use FP32 (recommended for better performance in most cases)
CMAKE_ARGS="-DGGML_SYCL=on -DCMAKE_C_COMPILER=icx -DCMAKE_CXX_COMPILER=icpx" pip install "llama-cpp-python @ git+https://github.com/JamePeng/llama-cpp-python.git"
# Option 2: Use FP16
CMAKE_ARGS="-DGGML_SYCL=on -DCMAKE_C_COMPILER=icx -DCMAKE_CXX_COMPILER=icpx -DGGML_SYCL_F16=ON" pip install "llama-cpp-python @ git+https://github.com/JamePeng/llama-cpp-python.git"
```
</details>

<details>
<summary>RPC</summary>

To install with RPC support, enable `GGML_RPC` in the CMake build:

```bash
CMAKE_ARGS="-DGGML_RPC=ON" pip install "llama-cpp-python @ git+https://github.com/JamePeng/llama-cpp-python.git"
```

On Windows PowerShell:

```powershell
$env:CMAKE_ARGS = "-DGGML_RPC=ON"
pip install "llama-cpp-python @ git+https://github.com/JamePeng/llama-cpp-python.git"
```

You can download a prebuilt `ggml-rpc-server` from the official
[llama.cpp releases](https://github.com/ggml-org/llama.cpp/releases). Choose an
archive for the server host's operating system, architecture, and backend
(for example, CUDA or Vulkan), and keep the executable with its matching ggml
backend libraries and required runtime libraries. Use a release compatible
with the llama.cpp revision in this Python package. For example, from the
directory containing the server and its libraries on Windows:

```powershell
.\ggml-rpc-server.exe --host 127.0.0.1 --port 50052 --device CUDA0
```

Use the server's reachable IP address in place of `127.0.0.1` when Python runs
on a different host. The server prints its exposed device names at startup;
omit `--device` to let it select available accelerators. Then select its
devices when loading the model:

```python
from llama_cpp import Llama

llm = Llama(
    model_path="model.gguf",
    rpc_servers=["192.168.1.50:50052"],
    rpc_local_devices=[],  # do not offload model layers to local GPUs
    n_gpu_layers=-1,
)
```

`rpc_servers` also accepts a comma-separated string. RPC devices are listed
before local GPUs when interpreting `tensor_split` and `main_gpu`; each remote
server may expose more than one device. When supplied, `tensor_split` must have
one value per selected RPC or local GPU device. The model file remains on the
Python host. Keep the RPC server on a trusted network; llama.cpp's RPC protocol
does not provide authentication or encryption.

Set `rpc_local_devices=[]` to use only the remote devices, or pass local GPU
names such as `["CUDA0"]` to select and order the local devices added after the
remote ones. `LLAMA_SPLIT_MODE_ROW` is not supported with RPC. Each server is
probed before registration, and repeated use of an endpoint reuses its
process-wide vendor registration. Registrations remain available until the
Python process exits because ggml does not expose a safe way to destroy them
while models may hold device pointers. The Python adapter caps distinct
endpoints used in one process at `GGML_RPC_MAX_SERVERS` (16); restart Python
to switch beyond that limit. A server that disconnects between the probe and
the native call can still cause ggml to terminate the process. For images and
other multimodal inputs, see [Multimodal Models with RPC](#multimodal-models-with-rpc).
If a configured server is unavailable at startup, model loading fails instead
of silently switching to local GPUs.
</details>


### Install Notes
<details>
<summary> Optimization Options (Optional)</summary>

> **💡 Tip:** If you want to save compilation time, you can skip building of llama.cpp with the standalone examples, tools, tests, and server by adding the following flags, as they are not required for Python bindings:

```bash
-DLLAMA_BUILD_EXAMPLES=OFF \
-DLLAMA_BUILD_TOOLS=OFF \
-DLLAMA_BUILD_TESTS=OFF \
-DLLAMA_BUILD_SERVER=OFF
```
</details>

<details>
<summary> CUDA compiler warning suppression is optional</summary>
CUDA nvcc compiler may print many template-related warnings from ggml-cuda, such as:

```bash
warning #177-D
warning #221-D
warning #550-D
```

These usually generate a huge amount of noisy diagnostics rather than build blockers. They constantly flood logs and consume CPU printing performance.

For cleaner CI/local logs, you can pass:

```bash
-DCMAKE_CUDA_FLAGS="--diag-suppress=177 --diag-suppress=221 --diag-suppress=550"
```
</details>

<details>
<summary> Notes for `GGML_BACKEND_DL` + `GGML_CPU_ALL_VARIANTS` builds</summary>
When building wheels with `GGML_BACKEND_DL=ON` and `GGML_CPU_ALL_VARIANTS=ON`,
GGML CPU backends are built as separate dynamic libraries, such as:

`GGML_BACKEND_DL` requires `BUILD_SHARED_LIBS=ON`, and
`GGML_CPU_ALL_VARIANTS` requires `GGML_BACKEND_DL=ON`. Use
`GGML_NATIVE=OFF` for this portable dynamic CPU backend layout.

```text
ggml-cpu-x64.dll
ggml-cpu-haswell.dll
ggml-cpu-alderlake.dll
ggml-cpu-zen4.dll
```
These backend libraries must be packaged together under:

```text
site-packages/llama_cpp/lib
```

The runtime must also explicitly load them with:

```text
ggml_backend_load_all_from_path()
```

### Windows notes

For full x64 CPU variant coverage, `LLVM/Clang` is recommended. `MSVC` may skip some variants such as `zen4`, `cooperlake`, or `sapphirerapids`.

If `GGML_OPENMP=ON` is used, the LLVM OpenMP runtime must also be packaged next to the backend DLLs:

```text
libomp140.x86_64.dll
```

Without this file, `ggml-cpu-*.dll` may fail to load dynamically at runtime.

### Wheel packaging checklist

* Enable `GGML_BACKEND_DL=ON`
* Enable `GGML_CPU_ALL_VARIANTS=ON`
* Enable `BUILD_SHARED_LIBS=ON`
* Use `GGML_NATIVE=OFF` for portable wheels
* Install all `ggml-cpu-*` backend libraries into `llama_cpp/lib`
* Package required runtime dependencies such as `libomp140.x86_64.dll`
* Remove development-only files such as `.lib`, `cmake/`, and `pkgconfig/`

</details>

### Upgrading and Reinstalling

To upgrade and rebuild `llama-cpp-python` add `--upgrade --force-reinstall --no-cache-dir` flags to the `pip install` command to ensure the package is rebuilt from source.

## Low-level API

[Low-level tutorial and examples](examples/low_level_api/README.md)

The low-level API exposes direct [`ctypes`](https://docs.python.org/3/library/ctypes.html)
bindings for the current native APIs:

- [`llama_cpp/llama_cpp.py`](llama_cpp/llama_cpp.py) mirrors
  [`llama.h`](vendor/llama.cpp/include/llama.h).
- [`llama_cpp/mtmd_cpp.py`](llama_cpp/mtmd_cpp.py) mirrors the multimodal
  `mtmd` API.
- [`llama_cpp/_ggml.py`](llama_cpp/_ggml.py) contains the supported `ggml`
  backend bindings.

The low-level examples cover streaming generation, chat templates, a
reason/action tool loop, and model quantization. Start with the tutorial in
[`examples/low_level_api`](examples/low_level_api):

```bash
python examples/low_level_api/generate.py -m path/to/model.gguf -p "Explain why the sky is blue:" --n-ctx 4096 --max-tokens 256 --n-gpu-layers auto --verbosity info
```

Run an example with `-h` for its complete options. The shared runtime
demonstrates the current model/context lifecycle, vocabulary-based
tokenization, `llama_batch` decoding, sampler chains, dynamic backend loading,
and native logging configuration.

## High-level API

[API Reference](docs/wiki/core/Llama.md)

The [`Llama`](docs/wiki/core/Llama.md)
class manages GGUF model loading, chat formatting, tokenization, generation, and
the native llama.cpp context.

Install `huggingface_hub` to download GGUF models directly from Hugging Face:

```bash
pip install --upgrade huggingface_hub
```

### Quick Start

[`Llama.from_pretrained`](docs/wiki/core/Llama.md)
downloads the selected file into the Hugging Face cache. Modern GGUF models
usually include their chat template, so no explicit `chat_format` is needed.
This example uses the compact
[Qwen3-0.6B-GGUF](https://huggingface.co/Qwen/Qwen3-0.6B-GGUF) model.

```python
from llama_cpp import Llama

llm = Llama.from_pretrained(
    repo_id="Qwen/Qwen3-0.6B-GGUF",
    filename="Qwen3-0.6B-Q8_0.gguf",
    n_ctx=0,
    verbose=False,
)

response = llm.create_chat_completion(
    messages=[
        {
            "role": "user",
            "content": "Explain speculative decoding in simple terms. /no_think",
        }
    ],
    max_tokens=512,
    temperature=0.7,
    top_p=0.8,
    top_k=20,
    min_p=0.0,
    present_penalty=1.5,
)

print(response["choices"][0]["message"]["content"])
```

`n_ctx=0` uses the context length stored in the model metadata. Model layer
offloading and file loading default to `n_gpu_layers="auto"` and
`load_mode=LLAMA_LOAD_MODE_AUTO`, allowing llama.cpp to select suitable behavior
for the available hardware. Use `verbose=False` for
quiet error-only native logging, or set `verbosity` from `0` to `5` for finer
control; `verbosity` takes precedence when both are provided.

### Loading a Local GGUF

Pass `model_path` when the model is already available locally:

```python
from llama_cpp import Llama

llm = Llama(
    model_path="/path/to/Qwen3-0.6B-Q8_0.gguf",
    n_ctx=0,
)
```

Advanced loading can be configured explicitly when required:

```python
import llama_cpp
from llama_cpp import Llama

llm = Llama(
    model_path="/path/to/Qwen3-0.6B-Q8_0.gguf",
    n_ctx=0,
    n_gpu_layers="all",
    load_mode=llama_cpp.llama_load_mode.LLAMA_LOAD_MODE_MMAP,
    verbosity=3,  # llama.cpp-style informational logs
)
```

See the [Llama class guide](docs/wiki/core/Llama.md) for loading modes, GPU
offloading, context configuration, KV cache types, and multimodal projection
models.

### Streaming Chat Completion

Set `stream=True` to iterate over OpenAI-compatible chat completion chunks:

```python
stream = llm.create_chat_completion(
    messages=[
        {
            "role": "user",
            "content": "Write a detailed technical explanation of how incremental UTF-8 decoding and cross-token stop-sequence detection work in a streaming text generator."}],
    max_tokens=512,
    stream=True,
)

for chunk in stream:
    print(chunk["choices"][0]["delta"].get("content", ""), end="", flush=True)
```

### More High-level Features

The same `Llama` instance also supports:

- raw text completion with `create_completion()` or `llm(...)`
- JSON and JSON Schema constrained output through `response_format`
- OpenAI-compatible tool calling through `tools` and `tool_choice`
- typed OpenAI v1 responses with `create_chat_completion_openai_v1()`
- embeddings and reranking through `LlamaEmbedding`
- speculative decoding through `SpecConfig`
- multimodal GGUF models through `mmproj_path`

See the [Llama guide](docs/wiki/core/Llama.md) for the complete API and focused
examples, or continue below for advanced features.

---

## Continuing Assistant Responses (Prefill)

`llama-cpp-python` supports native **Assistant Prefill** for seamless message continuation. You can now simply use the `assistant_prefill=True` parameter in the `create_chat_completion` function.

With a compatible chat handler, this appends the final partial assistant message
to the rendered conversation prompt. The continuation depends on the model and
chat template.

```python
from llama_cpp import Llama

llm = Llama(model_path="path/to/model.gguf")

# An interrupted/partial conversation
messages = [
    {"role": "user", "content": "What are the first 5 planets in the solar system?"},
    {"role": "assistant", "content": "The first 5 planets in our solar system are:\n1. Mercury\n2."}
]

# Seamlessly continue the generation
response = llm.create_chat_completion(
    messages=messages,
    max_tokens=50,
    assistant_prefill=True # <--- Enables seamless continuation
)

prefilled_text = messages[-1]["content"]
# Append the generated continuation to the supplied assistant prefix.
generated_text = response["choices"][0]["message"]["content"]

print(prefilled_text + generated_text)
```

---

## Dynamic LoRA Routing & Control Vectors (Multi-Tenant Serving)

Historically, `llama-cpp-python` only supported "static loading" where a LoRA was permanently baked into the context during initialization. Switching personas required reloading the entire model or duplicating it in VRAM.

`llama-cpp-python` now supports **Just-In-Time (JIT)** dynamic adapter routing. Instead of statically binding a single LoRA to a model during initialization (which locks the instance to a single task), you can now preload multiple adapters into VRAM and seamlessly apply them on-the-fly per request.

Loaded adapters share the base model and can be selected for sequential requests.
Adapter changes are not guaranteed to be zero-cost, and a `Llama` instance does
not provide concurrent request isolation. Serialize requests on each instance
and finish or close a stream before switching adapters.

### Dynamic LoRA Example

```python
from llama_cpp import Llama

# 1. Load the pure base model once
llm = Llama(model_path="path/to/llama-3-8b.gguf")

# 2. Preload multiple LoRAs into VRAM
llm.load_lora("python_coder", "path/to/python-coder-lora.gguf")
llm.load_lora("translator", "path/to/spanish-translator-lora.gguf")

# 3. User A: Coding task with the coder LoRA
response_a = llm.create_chat_completion(
    messages=[{"role": "user", "content": "Write a fast inverse square root in C."}],
    active_loras=[{"name": "python_coder", "scale": 1.0}]
)

# 4. User B: Translation task with the translator LoRA
response_b = llm.create_chat_completion(
    messages=[{"role": "user", "content": "Explain quantum physics in Spanish."}],
    active_loras=[{"name": "translator", "scale": 0.85}] # Apply at 85% strength
)

# 5. User C: General Query (Automatically wipes graph weights for a clean base model state)
response_c = llm.create_chat_completion(
    messages=[{"role": "user", "content": "What is the capital of France?"}]
)

# 6. Cleanup (Optional: manually free VRAM for specific LoRAs)
llm.unload_lora("python_coder")
```

### Control Vector Injection (Representation Engineering)

In addition to LoRA, the API supports dynamic injection of **Control Vectors (CVec)**. This allows you to steer the model's behavior, emotion, or alignment by directly modifying the activation values at specific hidden layers, without needing `.gguf` weight files.

```python
response = llm.create_chat_completion(
    messages=[{"role": "user", "content": "Tell me a story about a futuristic city."}],
    control_vector={
        "data": [...],         # A flattened 1D list of floats representing the vector
        "layer_start": 15,     # Apply starting from this layer (inclusive)
        "layer_end": 32        # Apply up to this layer (inclusive)
    }
)
```
*Note(JamePeng): Ensure your `data` array length exactly matches `embedding_length * layer_end`. The C++ backend maps the buffer continuously starting from layer 1, so early skipped layers must be zero-padded in your array.*

---

##  Sampling Configuration & Usage (LlamaSamplingParams)

The `Llama` class provides extensive control over the `llama.cpp` sampling chain during text generation. You can configure state-of-the-art sampling algorithms, dynamic temperature, and advanced repetition penalties directly via the `generate`, `create_completion`, or `__call__` methods.

### Core Sampling Parameters

These are the most common parameters used to control the randomness and focus of the model's output.

* **`temperature`** (`float`, default: `0.80`): Controls the randomness of the generation. Higher values (e.g., `1.0`) make output more random, while lower values (e.g., `0.2`) make it more deterministic. Set to `<= 0.0` for greedy decoding.
* **`top_k`** (`int`, default: `40`): Limits the next token selection to the K most probable tokens. Set to `<= 0` to use the full vocabulary size.
* **`top_p`** (`float`, default: `0.95`): Nucleus sampling. Limits selection to a cumulative probability of P. Set to `1.0` to disable.
* **`min_p`** (`float`, default: `0.05`): Minimum P sampling. Drops tokens with a probability less than `min_p` relative to the most likely token. Set to `0.0` to disable.
* **`typical_p`** (`float`, default: `1.0`): Locally typical sampling. Adjusts probabilities based on the entropy of the distribution. Set to `1.0` to disable.


### Advanced & Experimental Samplers

* **XTC (Exclude Top Choice)**: Removes the most likely tokens to force the model to take creative alternative paths.
* **`xtc_probability`** (`float`, default: `0.0`): The chance for token removal. `0.0` disables XTC.
* **`xtc_threshold`** (`float`, default: `0.1`): The minimum probability threshold for a token to be considered for removal.


* **Dynamic Temperature**: Adjusts the temperature dynamically based on the entropy of the current token distribution.
* **`dynatemp_range`** (`float`, default: `0.0`): The range of the dynamic temperature. `0.0` disables it.
* **`dynatemp_exponent`** (`float`, default: `1.0`): Controls how entropy maps to temperature.


* **`top_n_sigma`** (`float`, default: `-1.0`): Limits selection to tokens with pre-softmax logits within $n * \sigma$ of the max logit. Set to `-1.0` to disable.
* **Adaptive-P**: Dynamically adjusts the target probability using an exponential moving average (EMA).
* **`adaptive_target`** (`float`, default: `-1.0`): The target probability (0.0 to 1.0). Negative values disable it.
* **`adaptive_decay`** (`float`, default: `0.9`): The EMA decay rate (0.0 to 0.99).


### Target Entropy (Mirostat)

Mirostat actively maintains a target entropy (`tau`) during generation to prevent text from becoming too boring or too chaotic.

* **`mirostat_mode`** (`int`, default: `0`): `0` = disabled, `1` = Mirostat 1.0, `2` = Mirostat 2.0.
* **`mirostat_tau`** (`float`, default: `5.0`): The target cross-entropy (surprisal) value.
* **`mirostat_eta`** (`float`, default: `0.1`): The learning rate used to update the algorithm's internal state.


### Repetition Penalties

* **Standard Penalties**:
* **`repeat_penalty`** (`float`, default: `1.0`): General penalty for repeated tokens. `1.0` = disabled.
* **`frequency_penalty`** (`float`, default: `0.0`): Penalty based on the absolute frequency of a token in the prompt.
* **`present_penalty`** (`float`, default: `0.0`): Flat penalty applied if a token is present anywhere in the context.
* **`presence_penalty`** (`Optional[float]`, default: `None`): Compatibility alias accepted by `create_completion()`, `__call__()`, `create_chat_completion()`, and the OpenAI-compatible server. `present_penalty` remains the primary parameter and takes precedence when set to a non-default value.
* **`penalty_last_n`** (`int`, default: `64`): The number of recent tokens to consider for standard penalties. `0` = disabled, `-1` = full context size.


* **DRY (Don't Repeat Yourself)**: An advanced exponential penalty specifically designed to break exact repeating sequences.
* **`dry_multiplier`** (`float`, default: `0.0`): The multiplier for the penalty. `0.0` disables DRY.
* **`dry_base`** (`float`, default: `1.75`): The base value for the exponential penalty.
* **`dry_allowed_length`** (`int`, default: `2`): Sequences extending beyond this length receive the penalty.
* **`dry_penalty_last_n`** (`int`, default: `64`): Tokens to scan for repetitions. `0` = disabled, `-1` = context size.
* **`dry_seq_breakers`** (`list[str]`, default: `["\n", ":", "\"", "*"]`): Tokens that reset the DRY sequence matching.


### Constraints & Callbacks

* **`logit_bias`** (`Dict[int, float]`, optional): Manually boost or penalize specific token IDs.
* **`grammar`** (`LlamaGrammar`, optional): Force the model to generate text matching a specific BNF-like grammar (e.g., valid JSON).
* **`logits_processor`** (`LogitsProcessorList`, optional): Custom Python callbacks to modify the logits tensor in-place before sampling.
* **`stopping_criteria`** (`StoppingCriteriaList`, optional): Custom Python callbacks to halt generation based on the current sequence or scores.


### Reasoning Budget (First Reasoning Block)

`llama-cpp-python` provides a generic reasoning-budget sampler for models that expose their thinking content with visible start/end tags. It controls only the **first visible reasoning block** in the generated output. After that block naturally ends or is forcibly closed, the sampler switches to passthrough mode and later reasoning tags are ignored.

This feature is intentionally model-agnostic. It does not infer model families, inspect chat templates, or guess thinking tags. If a model uses tags other than `<think>...</think>`, pass the correct `reasoning_start` and `reasoning_end` explicitly.

| Parameter | Default | Description |
| --- | --- | --- |
| `reasoning_budget` | `-1` | Token budget for the first visible reasoning block. `-1` disables the sampler, `0` forces an immediate end after the block starts, and `N > 0` allows at most `N` generated tokens inside the block. |
| `reasoning_start` | `"<think>"` | Token/text sequence that marks the beginning of the first reasoning block. |
| `reasoning_end` | `"</think>"` | Token/text sequence that naturally ends the reasoning block. When the budget is exhausted, the sampler forces this sequence. |
| `reasoning_budget_message` | `None` | Optional message inserted before `reasoning_end` when the budget is exhausted. |
| `reasoning_start_in_prompt` | `False` | Set to `True` only when the prompt/chat template has already inserted `reasoning_start`, so the sampler should start counting from the first generated token. |
| `reasoning_start_max_tokens` | `32` | Safety window for non-reasoning outputs. If `reasoning_start` is not generated within this many output tokens, the sampler becomes a no-op. Set to `None` to wait indefinitely. |

Basic usage with the default `<think>...</think>` tags:

```python
response = llm.create_chat_completion(
    messages=[{"role": "user", "content": "Solve this carefully."}],
    max_tokens=1024,
    reasoning_budget=256,
    reasoning_budget_message="\n[reasoning budget exhausted]\n",
    # You can also inject a natural-language transition before reasoning_end:
    # reasoning_budget_message="\n...Wait, I have been thinking long enough. Let me start answering the user's question.\n",
)
```
When the budget is exhausted, the sampler forces: `reasoning_budget_message` + `reasoning_end`

For Mistral-style thinking tags, pass the tags explicitly:

```python
response = llm.create_chat_completion(
    messages=[{"role": "user", "content": "Solve this carefully."}],
    max_tokens=1024,
    reasoning_budget=256,
    reasoning_start="[THINK]",
    reasoning_end="[/THINK]",
)
```

For Gemma4 channel-style thinking, adjust the start and end markers to match the visible channel tags:

```python
response = llm.create_chat_completion(
    messages=[{"role": "user", "content": "Solve this carefully."}],
    max_tokens=1024,
    reasoning_budget=256,
    reasoning_start="<|channel>",
    reasoning_end="<channel|>",
)
```

Use `reasoning_start_in_prompt=True` when the prompt or chat template has already inserted the reasoning start tag. In that case, the sampler will not see the start tag during generation, so it must start directly in `COUNTING` state from the first generated token. This is suitable for thinking models or handlers that prefill the assistant prefix with a thinking tag, for example:

```text
<|im_start|>assistant\n<think>\n
```

Example:

```python
response = llm.create_chat_completion(
    messages=[{"role": "user", "content": "Solve this carefully."}],
    max_tokens=1024,
    reasoning_budget=256,
    reasoning_start="<think>",
    reasoning_end="</think>",
    reasoning_start_in_prompt=True,
)
```

`reasoning_start_in_prompt` is **not** a generic "thinking enabled" switch. It should only be set when the final prompt already contains `reasoning_start` before generation begins. For templates that merely enable thinking but still expect the model to generate the start tag itself, keep `reasoning_start_in_prompt=False`.

When `verbose=True`, high-level reasoning-budget transitions are printed to stderr, such as initialization, start-tag detection, budget exhaustion, forced ending, and final passthrough.

### 🛠️ Usage Example

You can pass these parameters directly when calling the model to generate text.

```python
from llama_cpp import Llama

# Load the model
model = Llama(model_path="path/to/your/model.gguf")

# Generate text with advanced sampling
response = model.create_completion(
    prompt="The secret to a happy life is",
    max_tokens=100,
    # Adjust core randomness
    temperature=0.85,
    top_p=0.90,
    min_p=0.05,
    # Prevent the model from repeating specific phrases
    dry_multiplier=0.8,
    dry_base=1.75,
    dry_allowed_length=3,
    # Standard repetition penalty
    repeat_penalty=1.1,
    penalty_last_n=256,
)

print(response["choices"][0]["text"])

```

---

## Speculative Decoding

[`llama-cpp-python`](https://github.com/JamePeng/llama-cpp-python) provides a stateful speculative-decoding path aligned with
the `begin -> process -> draft -> accept` lifecycle used by `llama.cpp`.
Configure it with `speculative=SpecConfig(...)`; the older `draft_model=` API is
deprecated and is kept only for compatibility with stateless draft callbacks.

> **Version availability:** MTP speculative decoding is available starting with
> `0.3.48`. DFlash, DFlash2, and DSpark speculative decoding are available
> starting with `0.3.49`.

The current implementation supports one sequence (`seq_id=0`) and provides
five usable modes. MTP is text-only. N-gram engines can consume a completed
MTMD prefill; proposals stop before negative media ledger IDs. The DFlash family
can process token or embedding batches at the lower level, but MTMD chat still
rejects MTP and DFlash-family engines.

| Mode | `SpeculativeType` | Draft source |
|---|---|---|
| MTP | `DRAFT_MTP` | Target model NextN/MTP heads or an external MTP GGUF |
| DFlash / DFlash2 | `DRAFT_DFLASH` | External block-diffusion draft GGUF; DFlash2 is detected from selector metadata |
| DSpark | `DRAFT_DSPARK` | External DFlash-family GGUF with Markov/confidence heads |
| N-gram K | `NGRAM_MAP_K` | Previous matching positions in verified token history |
| N-gram K4V | `NGRAM_MAP_K4V` | Up to four cached continuations per n-gram key, matching `llama.cpp` |

Eagle3, draft-simple, and the other n-gram variants appear in
`SpeculativeType` for `llama.cpp` API compatibility but do not yet have Python
engines.

DFlash, DFlash2, and DSpark share `LlamaDFlashDecoding`: target-layer features
are passed through the fused draft decode path, with attention configuration
selected from the sidecar metadata. All three require `draft_model_path`. DFlash2 uses the same
`DRAFT_DFLASH` type and is selected automatically when the sidecar reports a
non-zero `dflash.selector_top_k`. The requested `draft_n_max` is clamped to the
draft GGUF's trained block size; benchmark several values on the deployment
hardware because a longer block is not always faster.

For the full API and lifecycle reference, see
[Llama Speculative Decoding](https://github.com/JamePeng/llama-cpp-python/blob/main/docs/wiki/modules/LlamaSpeculative.md).
For a runnable selector-based workflow, see
[DFlash2 Speculative Decoding](https://github.com/JamePeng/llama-cpp-python/blob/main/docs/wiki/examples/dflash2-speculative-decoding.md).

### MTP speculative decoding

**Available since:** `0.3.48`

The built-in and external MTP paths have been tested with the Qwen3.5,
Qwen3.6, and Qwen3.8 model families. External MTP has also been tested with a
`gemma4` target paired with a compatible `gemma4-assistant` GGUF. Other model
families may work when their GGUF tensors are compatible, but they have not yet
been validated.

#### Built-in MTP

When the target GGUF contains compatible NextN/MTP tensors, omit
`draft_model_path`. `Llama` automatically enables loading the target MTP
layers.

```python
from llama_cpp import Llama
from llama_cpp.llama_speculative import SpecConfig, SpeculativeType

llm = Llama(
    model_path="path/to/model-with-mtp.gguf",
    n_batch=512,
    n_gpu_layers="all",
    speculative=SpecConfig(
        spec_type=SpeculativeType.DRAFT_MTP,
        draft_n_max=2,
        draft_p_min=0.0,
    ),
)
```

#### External MTP model

Set `draft_model_path` when the MTP tensors are stored in a separate compatible
GGUF. Target and draft vocabularies and output embedding dimensions must match.
For a `gemma4 + gemma4-assistant` pair, the engine automatically links the
assistant to the target context and follows its shared-KV, same-position draft
workflow. This tested path is currently text-only, like the other stateful MTP
modes.

```python
llm = Llama(
    model_path="path/to/target.gguf",
    n_batch=512,
    n_gpu_layers="all",
    speculative=SpecConfig(
        spec_type=SpeculativeType.DRAFT_MTP,
        draft_model_path="path/to/mtp.gguf",
        draft_n_max=2,
        draft_n_gpu_layers="all",
        draft_backend_sampling=True,
    ),
)
```

For Qwen3.8 27B, testing so far suggests `draft_n_max=2` as the best starting
point. This is not a universal optimum: GPU, backend, quantization, prompt,
sampling settings, and whether MTP is built in or external can change the
result. Run the included benchmark and choose the fastest stable value for the
actual deployment environment.

The verification batch contains `[id_last, draft...]`, so the maximum draft
length must not exceed `n_batch - 1`. Longer drafts only help when their
additional acceptance outweighs verification and rollback cost.

### DFlash, DFlash2, and DSpark speculative decoding

**Available since:** `0.3.49`

DFlash, DFlash2, and DSpark require a compatible external draft GGUF. DFlash
generates a non-causal mask block. DFlash2 adds a selector lattice whose
candidate IDs are walked with predecessor-dependent transition scores. DSpark
uses the same block path with additional Markov and acceptance-confidence
heads.

```python
llm = Llama(
    model_path="path/to/target.gguf",
    n_ctx=8192,
    n_batch=512,
    n_gpu_layers="all",
    speculative=SpecConfig(
        spec_type=SpeculativeType.DRAFT_DFLASH,  # or DRAFT_DSPARK
        # DFlash2 uses the same type and is detected from selector metadata.
        draft_model_path="path/to/dflash-dflash2-or-dspark.gguf",
        draft_n_max=7,
        draft_p_min=0.0,
        draft_n_gpu_layers="all",
        # Used by DFlash v1/DSpark; remains inactive for DFlash2.
        draft_backend_sampling=True,
    ),
)
```

The effective draft length is clamped to the block size recorded in the draft
GGUF. For DFlash, `draft_p_min` filters draft-token probability. For DFlash2,
it filters the selected transition probability within the selector row. For
DSpark, it filters predicted acceptance confidence. Backend sampling is useful
for DFlash v1 and DSpark with large vocabularies. DFlash2 reads its compact
selector output directly, requests unmasked NextN rows, and does not activate
the backend vocabulary sampler even when `draft_backend_sampling=True`.

Testing covers compatible Qwen3.6 DFlash and Qwen3.8 DSpark pairs, plus
`Qwen3.8-27B-Q5_K_M.gguf` with the compatible
`Qwen3.8-27B-DFlash2-Q8_0.gguf` sidecar. The Qwen3.8 DFlash2 pair completed
greedy baseline comparisons with matching output tokens. Other compatible GGUF
pairs may work but have not yet been validated here.

Treat `draft_n_max=7` as a benchmark starting point rather than a universal
default. Compare several draft lengths with the supplied example and inspect
acceptance, target decode/sync time, rollback count, and final throughput
together.

### N-gram speculative decoding

N-gram decoding is model-free and works best for repeated JSON, tables, code,
templates, and boilerplate. It does not require a second GGUF model.

```python
from llama_cpp import Llama
from llama_cpp.llama_speculative import SpecConfig, SpeculativeType

llm = Llama(
    model_path="path/to/model.gguf",
    n_ctx=4096,
    n_batch=512,
    n_gpu_layers="all",
    speculative=SpecConfig(
        spec_type=SpeculativeType.NGRAM_MAP_K,
        ngram_size_n=8,
        ngram_size_m=16,
        ngram_min_hits=1,
    ),
)

response = llm.create_chat_completion(
    messages=[
        {
            "role": "user",
            "content": "Write a Python script using sqlite3 with repeated CRUD classes.",
        }
    ]
)
```

Use `SpeculativeType.NGRAM_MAP_K4V` to cache continuations directly:

```python
speculative = SpecConfig(
    spec_type=SpeculativeType.NGRAM_MAP_K4V,
    ngram_size_n=8,
    ngram_size_m=16,
    ngram_min_hits=1,
    ngram_max_entries_per_key=4,
)
```

`SpecConfig` follows the `llama.cpp` n-gram defaults (`N=12`, `M=48`). The
best values are workload-dependent. `N=8, M=16` is a conservative starting
point; longer drafts such as `M=32` or `M=48` can be faster for highly
repetitive output. Always benchmark against ordinary decoding.

For hybrid or recurrent targets, n-gram rejection needs target checkpoints:

```python
llm = Llama(
    model_path="path/to/hybrid-model.gguf",
    speculative=speculative,
    ctx_checkpoints=16,
    checkpoint_on_device=True,
)
```

### Runtime statistics

With `verbose=True`, `Llama.generate` prints calls, acceptance, phase timings,
checkpoint activity, rollbacks, TTFT, and sustained generation speed. The same
values are available programmatically after a generation:

```python
stats = llm.last_speculative_stats
print(stats["draft_token_acceptance_rate"])
print(stats["mean_accepted_length"])
print(stats["generation_tokens_per_second"])
print(stats["checkpoint_restore_seconds"])
```

Use the included examples for repeatable comparisons:

```bash
# Ordinary vs built-in/external MTP
python -m examples.high_level_api.high_level_api_mtp_speculative -h

# Ordinary vs external DFlash, DFlash2, or DSpark
python -m examples.high_level_api.high_level_api_dflash_dspark_speculative -h

# N-gram N x M scans and cross-method benchmarks
python -m examples.benchmark.benchmark_speculative -h
```

### Notes and limitations

* Speculative decoding does not skip target-model verification. Low acceptance
  can make it slower than ordinary decoding.
* Greedy speculative and ordinary runs can diverge because verification uses a
  different batch shape and may change floating-point tie-breaking. The
  DFlash/DFlash2/DSpark benchmark reports the first divergent generated token.
* Stateful engines currently support one sequence. MTMD prefill supports
  `NGRAM_MAP_K` and `NGRAM_MAP_K4V`; MTP and DFlash-family engines remain
  unsupported by the MTMD chat handler. High-level verification also requires
  native positions consistent with the token cursor.
* A speculative reset clears target and draft state together. Fresh speculative
  text requests do not reuse the ordinary cross-request prefix cache. Loading a
  `LlamaState` does not restore draft state; start a full-prompt request with
  `reset=True`. See the [state reuse guide](docs/wiki/features/caching.md).
* `draft_model=` and `LlamaDraftModel` are legacy compatibility APIs. New code
  should use `speculative=SpecConfig(...)`.
* Close `Llama` explicitly in long-running applications to release an external
  draft model and its context deterministically.

---

## Multi-modal Models

`llama-cpp-python` provides MTMD-based multimodal input for compatible GGUF
models. Depending on the model and its multimodal projector (`mmproj`), inputs
can include images, audio, and video in addition to text. Video is implemented
as timestamped image-frame sampling through the llama.cpp MTMD helper and
therefore requires a vision-capable projector plus `ffmpeg` and `ffprobe`.

For audio output, `MTMDAudioGenerator` provides non-streaming speech synthesis
with Qwen3-TTS Base and Pocket TTS. See [Text-to-Speech](#speech-synthesis-with-mtmd-text-to-speech)
for supported models, downloadable weights, and examples.

Below are the supported multi-modal models and their respective chat handlers (Python API) and chat formats (Server API).

| Model | MTMD chat handler | `chat_format` |
|:--- |:--- |:--- |
| [llava-v1.5-7b](https://huggingface.co/mys/ggml_llava-v1.5-7b) | `Llava15ChatHandler` | `llava-1-5` |
| [llava-v1.6-34b](https://huggingface.co/cjpais/llava-v1.6-34B-gguf) | `Llava16ChatHandler` | `llava-1-6` |
| [moondream2](https://huggingface.co/vikhyatk/moondream2) | `MoondreamChatHandler` | `moondream2` |
| [nanollava](https://huggingface.co/abetlen/nanollava-gguf) | `NanoLlavaChatHandler` | `nanollava` |
| [llama-3-vision-alpha](https://huggingface.co/abetlen/llama-3-vision-alpha-gguf) | `Llama3VisionAlphaChatHandler` | `llama-3-vision-alpha` |
| [minicpm-v-2.6](https://huggingface.co/openbmb/MiniCPM-V-2_6-gguf) | `MiniCPMv26ChatHandler` | `minicpm-v-2.6`, `minicpm-v-4.0` |
| [minicpm-v-4.5](https://huggingface.co/openbmb/MiniCPM-V-4_5-gguf) | `MiniCPMv45ChatHandler` | `minicpm-v-4.5` |
| [minicpm-v-4.6](https://huggingface.co/openbmb/MiniCPM-V-4.6-gguf) | `MiniCPMV46ChatHandler` | `minicpm-v-4.6` |
| [gemma3](https://huggingface.co/unsloth/gemma-3-27b-it-GGUF) | `Gemma3ChatHandler` | `gemma3` |
| [gemma4](https://huggingface.co/unsloth/gemma-4-26B-A4B-it-GGUF) | `Gemma4ChatHandler` | `gemma4` |
| [glm4.1v](https://huggingface.co/unsloth/GLM-4.1V-9B-Thinking-GGUF) | `GLM41VChatHandler` | `glm4.1v` |
| [glm4.6v](https://huggingface.co/unsloth/GLM-4.6V-Flash-GGUF) | `GLM46VChatHandler` | `glm4.6v` |
| [granite-docling](https://huggingface.co/ibm-granite/granite-docling-258M-GGUF) | `GraniteDoclingChatHandler` | `granite-docling` |
| [lfm2-vl](https://huggingface.co/LiquidAI/LFM2-VL-3B-GGUF) | `LFM2VLChatHandler` | `lfm2-vl` |
| [lfm2.5-vl](https://huggingface.co/LiquidAI/LFM2.5-VL-1.6B-GGUF) | `LFM25VLChatHandler` | `lfm2.5-vl` |
| [deepseek-ocr](https://huggingface.co/JamePeng2023/DeepSeek-OCR-2-GGUF) | `MTMDChatHandler` | `None` |
| [mineru2.5-pro](https://huggingface.co/JamePeng2023/MinerU2.5-Pro-2605-1.2B-GGUF) | `Qwen25VLChatHandler` | `qwen2.5-vl` |
| [paddleocr-vl-1.5](https://huggingface.co/JamePeng2023/PaddleOCR-VL-1.5-GGUF) | `PaddleOCRChatHandler` | `paddleocr` |
| [qwen2.5-vl](https://huggingface.co/unsloth/Qwen2.5-VL-3B-Instruct-GGUF) | `Qwen25VLChatHandler` | `qwen2.5-vl` |
| [qwen3-asr](https://huggingface.co/JamePeng2023/Qwen3-ASR-1.7B-GGUF) | `Qwen3ASRChatHandler` | `qwen3-asr` |
| [qwen3-vl](https://huggingface.co/unsloth/Qwen3-VL-8B-Thinking-GGUF) | `Qwen3VLChatHandler` | `qwen3-vl` |
| [qwen3.5](https://huggingface.co/unsloth/Qwen3.5-27B-GGUF) | `Qwen35ChatHandler` | `qwen3.5` |
| [qwen3.6](https://huggingface.co/unsloth/Qwen3.6-35B-A3B-GGUF) | `Qwen35ChatHandler` | `qwen3.6` |
| [qwen3.8](https://huggingface.co/unsloth/Qwen3.8-27B-GGUF) | `GenericMTMDChatHandler` | `qwen3.8` |
| [step3-vl](https://huggingface.co/JamePeng2023/Step3-VL-10B-GGUF) | `Step3VLChatHandler` | `step3-vl` |

The table identifies known handlers, not a guarantee that every listed model
supports every modality. The loaded `mmproj` determines image/audio capability,
and video additionally requires vision support, an MTMD video-enabled build,
available ffmpeg tools, and a chat template that renders a video marker.

Then you'll need to load the multimodal projection model (`mmproj`) together with the main language model.

Starting from `0.3.41-preview`, new multimodal implementations are recommended to use the updated interfaces in `llama_multimodal`. For backward compatibility, the legacy `llama_chat_format` path is still retained, but may be deprecated in future versions.

The parameter `clip_model_path` has been renamed to `mmproj_path` to better reflect its purpose and align with llama.cpp's multimodal projection model naming convention. New code should use `mmproj_path` exclusively.

### Generic MTMD Chat Handler (Recommended)

For multimodal GGUF models that already include a valid `tokenizer.chat_template`, you can use the generic MTMD handler through `mmproj_path`.

This is especially useful for newer multimodal models that have not yet received a dedicated Python chat handler. The generic handler renders the model-provided Jinja chat template, then normalizes rendered media placeholders or media URLs into the canonical llama.cpp MTMD media marker, usually `<__media__>`, before calling `mtmd_tokenize`.

> **Note:** `GenericMTMDChatHandler` is intended as a flexible fallback for template-driven multimodal models. Because different model families may use different media ordering rules, reasoning switches, stop tokens, or special template variables, some models may still require a dedicated chat handler. Please test carefully and report issues if you encounter incorrect prompts, missing media markers, or mismatched media counts.

```python
from llama_cpp import Llama

# Model and multimodal projection paths
MODEL_PATH = r"path/to/model.gguf"
MMPROJ_PATH = r"path/to/mmproj.gguf"

llm = Llama(
    model_path=MODEL_PATH,
    mmproj_path=MMPROJ_PATH,
    n_gpu_layers=-1,
    n_ctx=10240,
    verbose=True,
    verbosity=2,
    chat_handler_kwargs={
        "verbose": True,
    },
)

response = llm.create_chat_completion(
    messages=[
        {
            "role": "user",
            "content": [
                {
                    "type": "image_url",
                    "image_url": {
                        "url": "path/to/image.jpg",
                    },
                },
                {
                    "type": "text",
                    "text": "Describe this image in detail.",
                },
            ],
        }
    ]
)

print(response["choices"][0]["message"]["content"])
```

### Multimodal Models with RPC

Use `rpc_servers` for the language model and `mmproj_path` for its matching
multimodal projector. This example uses Qwen3.5 and sends a local PNG image as
a data URI:

```python
import base64
from pathlib import Path

from llama_cpp import Llama

image_data = base64.b64encode(Path("image.png").read_bytes()).decode("ascii")
image_url = f"data:image/png;base64,{image_data}"

llm = Llama(
    model_path="Qwen3.5-9B-MTP-Q8_0.gguf",
    mmproj_path="mmproj-Qwen3.5-9B-MTP-BF16.gguf",
    rpc_servers=["127.0.0.1:50052"],
    rpc_local_devices=[],
    n_gpu_layers=-1,
    n_ctx=2048,
    n_batch=256,
    chat_handler_kwargs={
        "extra_template_arguments": {"enable_thinking": False},
    },
)
try:
    response = llm.create_chat_completion(
        messages=[{
            "role": "user",
            "content": [
                {"type": "image_url", "image_url": {"url": image_url}},
                {"type": "text", "text": "What is in this image?"},
            ],
        }],
        max_tokens=64,
    )
    print(response["choices"][0]["message"]["content"])
finally:
    llm.close()
```

The projector is initialized on the first multimodal request. A successful
`Llama(...)` call alone does not verify that the `mmproj` loaded. Also,
`rpc_local_devices=[]` selects devices for the language model only. The MTMD
projector chooses its own backend; with `use_gpu=True` (the default), it uses
the first GPU in the process-wide ggml registry, which can be a local GPU.
Set `chat_handler_kwargs={"use_gpu": False}` to run the projector on the local
CPU. The current high-level handler does not expose an explicit RPC device
selection for the projector.

#### Chat Template Resolution Order

When `mmproj_path` is passed directly to `Llama`, llama-cpp-python constructs a
`GenericMTMDChatHandler` and uses the model's `tokenizer.chat_template` metadata.
If the model does not provide a usable template, the handler falls back to its
built-in MTMD template.

To supply a template explicitly, instantiate `GenericMTMDChatHandler` yourself
and pass it through `chat_handler`:

```python
from pathlib import Path

from llama_cpp import Llama
from llama_cpp.llama_multimodal import GenericMTMDChatHandler

chat_template = Path("path/to/chat_template.jinja").read_text(encoding="utf-8")

llm = Llama(
    model_path=r"path/to/model.gguf",
    chat_handler=GenericMTMDChatHandler(
        chat_format=chat_template,
        mmproj_path=r"path/to/mmproj.gguf",
        verbose=False,
    ),
    n_gpu_layers=-1,
    n_ctx=4096,
)
```

#### Passing Extra Template Arguments

Some model chat templates expose optional Jinja variables such as `enable_thinking`, `add_vision_id`, or model-specific media token switches. Further details can be obtained by analyzing the chat templates provided in `chat_template.jinja` or `tokenizer_config.json` for each model.

You can pass those values through `chat_handler_kwargs["extra_template_arguments"]`:

```python
from llama_cpp import Llama

# Model and multimodal projection paths
MODEL_PATH = r"path/to/model.gguf"
MMPROJ_PATH = r"path/to/mmproj.gguf"

llm = Llama(
    model_path=MODEL_PATH,
    mmproj_path=MMPROJ_PATH,
    n_gpu_layers=-1,
    n_ctx=10240,
    verbose=False,
    verbosity=1,
    chat_handler_kwargs={
        "extra_template_arguments": {
            "enable_thinking": True,
        },
        "verbose": False,
    },
)
...
```

The values inside `extra_template_arguments` are passed directly into the Jinja template render call.

For models that already have a dedicated handler, you can still instantiate that handler directly:

```python
from llama_cpp import Llama
from llama_cpp.llama_multimodal import PaddleOCRChatHandler

MODEL_PATH = r"path/to/model.gguf"
MMPROJ_PATH = r"path/to/mmproj.gguf"

llm = Llama(
    model_path=MODEL_PATH,
    chat_handler=PaddleOCRChatHandler(
        mmproj_path=MMPROJ_PATH,
    ),
    n_gpu_layers=-1,    # Use all available GPU layers
    n_ctx = 0,          # Context window size
    n_batch=2048,
)
...
```

Use `GenericMTMDChatHandler` when the model-provided `tokenizer.chat_template` already works correctly. Prefer a dedicated handler when the model requires custom prompt construction, special reasoning behavior, custom stop tokens, OCR/ASR-specific handling, or non-standard media ordering.


**Note**: Multi-modal models also support tool calling and JSON mode.


## Loading a Local Video With Generic MTMD

MTMD video input requires a vision-capable `mmproj` and an MTMD build compiled
with video support. The default build enables video support and launches the
system `ffprobe` and `ffmpeg` executables to inspect the video and sample RGB
frames. Download them from the [official FFmpeg download page](https://ffmpeg.org/download.html),
then place both tools on `PATH`, or pass the directory that contains them through
`video_ffmpeg_bin_dir`.

```python
from llama_cpp import Llama

MODEL_PATH = r"path/to/model.gguf"
MMPROJ_PATH = r"path/to/mmproj.gguf"
VIDEO_PATH = r"path/to/video.mp4"

video_options = {
    # Start low: MTMD expands the sampled frames during tokenization.
    "video_fps_target": 1.0,
    # Insert text timestamps such as [0m5.00s]; <= 0 disables them.
    "video_timestamp_interval_ms": 5000,
    "batch_max_tokens": 1024,
    # Optional when ffmpeg and ffprobe are already available on PATH:
    # "video_ffmpeg_bin_dir": r"path/to/ffmpeg/bin",
}

llm = Llama(
    model_path=MODEL_PATH,
    mmproj_path=MMPROJ_PATH,
    n_gpu_layers="auto",
    n_ctx=32768,
    n_batch=2048,
    chat_handler_kwargs=video_options,
    verbose=True,
)

response = llm.create_chat_completion(
    messages=[
        {
            "role": "user",
            "content": [
                {
                    # Gemma 4 templates expect the `video` schema.
                    "type": "video",
                    "video": VIDEO_PATH,
                },
                {
                    "type": "text",
                    "text": (
                        "Describe the main scenes and actions in chronological "
                        "order, including approximate timestamps."
                    ),
                },
            ],
        }
    ],
    max_tokens=512,
)

print(response["choices"][0]["message"]["content"])
llm.close()
```

Video preprocessing options are passed through `chat_handler_kwargs`:

| Option | MTMD default | Description |
|:--- |:--- |:--- |
| `video_fps_target` | `4.0` | Desired sampled FPS. Values `<= 0` use the video's native FPS and can be very expensive. |
| `video_ffmpeg_bin_dir` | `None` | Directory containing both `ffmpeg` and `ffprobe`. `None` searches `PATH`. |
| `video_timestamp_interval_ms` | `5000` | Interval between timestamp text chunks. Values `<= 0` disable timestamps. |
| `batch_max_tokens` | `1024` | Maximum number of MTMD media tokens processed in one decode batch. |

Advanced callers can instead pass a complete `mtmd_helper_init_opt` ctypes
structure. Do not combine that structure with individual `video_*` options.

Use `{"type": "video_url", "video_url": {"url": ...}}` only when the
model's Jinja chat template explicitly supports `video_url`. For example,
Gemma 4's default template expects `{"type": "video", "video": ...}`; using
`video_url` with that template produces no media marker and fails marker-count
validation.

The helper samples visual frames only. A video's audio track is ignored, so a
vision-only model does not need audio-input support. Video understanding is
therefore based on sampled frames, timestamps, and model-specific temporal frame
merging rather than native audio/video stream modeling.

> **Resource note:** The current helper reads the input video into memory and
> expands all sampled frames during MTMD tokenization. There is no automatic
> duration, frame-count, or total-video-token budget. Begin with short videos
> and a low FPS; long or high-resolution videos can consume substantial RAM and
> exceed the model context window.

For a configurable command-line version, including `-h` setup guidance, see
[`examples/high_level_api/mtmd_video_chat.py`](examples/high_level_api/mtmd_video_chat.py).


## Loading a Local Image With Qwen3VL(Thinking/Instruct)

<summary>This script demonstrates how to load a local image, encode it as a base64 Data URI, and pass it to a local Qwen3-VL model (with the 'force_reasoning' parameter enabled for thinking model, disabled for instruct model) for processing using the llama-cpp-python library.</summary><br>


**Example Code**: <details>

```python
# Import necessary libraries
from llama_cpp import Llama
from llama_cpp.llama_multimodal import Qwen3VLChatHandler
import base64
import os

# --- Model Configuration ---
# Define the path to the main model file
MODEL_PATH = r"./Qwen3-VL-8B-Thinking-F16.gguf"
# Define the path to the multi-modal projector file
MMPROJ_PATH = r"./mmproj-Qwen3-VL-8b-Thinking-F16.gguf"

# --- Initialize the Llama Model ---
llm = Llama(
    model_path=MODEL_PATH,
    # Set up the chat handler for Qwen3-VL, specifying the projector path
    chat_handler=Qwen3VLChatHandler(
      mmproj_path=MMPROJ_PATH,
      force_reasoning=True,  # Note: Some models use `enable_thinking` as a switch variable. See the comments in the corresponding model's chathandler for details.
      image_min_tokens=1024, # Note: Qwen3-VL models require at minimum 1024 image tokens to function correctly on bbox grounding tasks
    ),
    n_gpu_layers=-1,  # Offload all layers to the GPU
    n_ctx=10240,      # Set the context window size
    swa_full=True,
)

# Comprehensive MIME type mapping (updated as of 2025)
# Based on Pillow 10.x+ "Fully Supported" (Read & Write) formats
# Reference: IANA official media types + common real-world usage
# See: https://pillow.readthedocs.io/en/stable/handbook/image-file-formats.html
_IMAGE_MIME_TYPES = {
    # Most common formats
    '.png':  'image/png',
    '.jpg':  'image/jpeg',
    '.jpeg': 'image/jpeg',
    '.gif':  'image/gif',
    '.webp': 'image/webp',

    # Next-generation formats
    '.avif': 'image/avif',
    '.jp2':  'image/jp2',
    '.j2k':  'image/jp2',
    '.jpx':  'image/jp2',

    # Legacy / Windows formats
    '.bmp':  'image/bmp',
    '.ico':  'image/x-icon',
    '.pcx':  'image/x-pcx',
    '.tga':  'image/x-tga',
    '.icns': 'image/icns',

    # Professional / Scientific imaging
    '.tif':  'image/tiff',
    '.tiff': 'image/tiff',
    '.eps':  'application/postscript',
    '.dds':  'image/vnd-ms.dds',
    '.dib':  'image/dib',
    '.sgi':  'image/sgi',

    # Portable Map formats (PPM/PGM/PBM)
    '.pbm':  'image/x-portable-bitmap',
    '.pgm':  'image/x-portable-graymap',
    '.ppm':  'image/x-portable-pixmap',

    # Miscellaneous / Older formats
    '.xbm':  'image/x-xbitmap',
    '.mpo':  'image/mpo',
    '.msp':  'image/msp',
    '.im':   'image/x-pillow-im',
    '.qoi':  'image/qoi',
}

def image_to_base64_data_uri(
    file_path: str,
    *,
    fallback_mime: str = "application/octet-stream"
) -> str:
    """
    Convert a local image file to a base64-encoded data URI with the correct MIME type.

    Supports 20+ image formats (PNG, JPEG, WebP, AVIF, BMP, ICO, TIFF, etc.).

    Args:
        file_path: Path to the image file on disk.
        fallback_mime: MIME type used when the file extension is unknown.

    Returns:
        A valid data URI string (e.g., data:image/webp;base64,...).

    Raises:
        FileNotFoundError: If the file does not exist.
        OSError: If reading the file fails.
    """
    if not os.path.isfile(file_path):
        raise FileNotFoundError(f"Image file not found: {file_path}")

    extension = os.path.splitext(file_path)[1].lower()
    mime_type = _IMAGE_MIME_TYPES.get(extension, fallback_mime)

    if mime_type == fallback_mime:
        print(f"Warning: Unknown extension '{extension}' for '{file_path}'. "
              f"Using fallback MIME type: {fallback_mime}")

    try:
        with open(file_path, "rb") as img_file:
            encoded_data = base64.b64encode(img_file.read()).decode("utf-8")
    except OSError as e:
        raise OSError(f"Failed to read image file '{file_path}': {e}") from e

    return f"data:{mime_type};base64,{encoded_data}"

# --- Main Logic for Image Processing ---

# 1. Create a list containing all image paths
image_paths = [
    r'./scene.jpeg',
    r'./cat.png',
    r'./network.webp',
    # Add more image paths here if needed
]

# 2. Create an empty list to store the message objects (images and text)
images_messages = []

# 3. Loop through the image path list, convert each image to a Data URI,
#    and add it to the message list as an image_url object.
for path in image_paths:
    data_uri = image_to_base64_data_uri(path)
    images_messages.append({"type": "image_url", "image_url": {"url": data_uri}})

# 4. Add the final text prompt at the end of the list
images_messages.append({"type": "text", "text": "Describes the images."})

# 5. Use this list to build the chat_completion request
res = llm.create_chat_completion(
    messages=[
        {"role": "system", "content": "You are a highly accurate vision-language assistant. Provide detailed, precise, and well-structured image descriptions."},
        # The user's content is the list containing both images and text
        {"role": "user", "content": images_messages}
    ]
)

# Print the assistant's response
print(res["choices"][0]["message"]["content"])

```

</details>

## Speech Recognition With Qwen3-ASR (Speech-to-Text)

The `Qwen3ASRChatHandler` is specifically designed for the Qwen3 Automatic Speech Recognition (ASR) models. Unlike standard multimodal models, this handler aggregates system prompts for instructions and automatically extracts audio data from the user's message, ignoring any user text.

> **⚠️ Important Note on Quantization:** > For Qwen3-ASR models, it is highly recommended to use the **BF16** version of the multimodal projector (`mmproj`). Other quantizations are known to cause severe audio degradation.

**Example Code**: <details>

```python
from llama_cpp import Llama
from llama_cpp.llama_multimodal import Qwen3ASRChatHandler
import base64
import os

# 1. Define paths to the model and the BF16 multimodal projector
MODEL_PATH = r"./Qwen3-ASR-1.7B-BF16.gguf"
MMPROJ_PATH = r"./mmproj-Qwen3-ASR-1.7b-BF16.gguf"

# 2. Initialize the Llama model with the dedicated ASR handler
llm = Llama(
    model_path=MODEL_PATH,
    chat_handler=Qwen3ASRChatHandler(
        mmproj_path=MMPROJ_PATH,
        verbose=False,
    ),
    n_gpu_layers=-1,
    n_ctx=10240,
    verbose=False,
    verbosity=0
)

# 3. Helper function to encode audio files into OpenAI-compatible payloads
_MEDIA_MIME_TYPES = {
    '.wav':  ('audio', 'wav'),
    '.mp3':  ('audio', 'mp3'),
}

def build_media_payload(file_path: str) -> dict:
    """Reads a local audio file and converts it into the LLM input structure."""
    if not os.path.isfile(file_path):
        raise FileNotFoundError(f"Media file not found: {file_path}")

    extension = os.path.splitext(file_path)[1].lower()
    media_category, mime_or_format = _MEDIA_MIME_TYPES.get(extension, ('unknown', 'application/octet-stream'))

    if media_category == 'unknown':
        print(f"Warning: Unknown extension '{extension}'.")

    # Read and Base64 encode the file
    with open(file_path, "rb") as f:
        encoded_data = base64.b64encode(f.read()).decode("utf-8")

    if media_category == 'audio':
        return {
            "type": "input_audio",
            "input_audio": {
                "data": encoded_data,
                "format": mime_or_format
            }
        }
    else:
        return {"type": "text", "text": f"[Attached unsupported file: {file_path}]"}


# ========================
# Main Inference Section
# ========================

media_paths = ["./audio/test.wav"]
user_content = [build_media_payload(path) for path in media_paths]

# 4. Generate the transcription
response = llm.create_chat_completion(
    messages=[
        {
            "role": "system",
            "content": (
                "You are an advanced multilingual Speech-to-Text model. "
                "Accurately transcribe the audio into text in its original spoken language. "
                "You should ignore background noise, filler words, and stutters where possible, "
                "and format the final output with correct grammar and capitalization."
            )
        },
        {
            "role": "user",
            "content": user_content
        }
    ],
    temperature=1.0,
    top_p=0.95,
    top_k=64,
    max_tokens=10240,
)

print(f"Transcribe: {response['choices'][0]['message']['content']}")

```

#### How it works:

* **`input_audio` Schema:** The script reads the local `.wav` or `.mp3` file, encodes it in Base64, and wraps it in an OpenAI-compatible `"type": "input_audio"` dictionary.
* **System Prompt:** Because the Qwen3-ASR template strips out user text, all instructions (like translation requests or formatting rules) **must** be placed in the `"system"` role.

</details>

### Speech Synthesis With MTMD (Text-to-Speech)

Use `MTMDAudioGenerator` with a dedicated `Llama` instance and a matching
audio-generation `mmproj`. TTS uses `create_speech()` rather than a chat handler
or server `chat_format`.

| Model | Python API | Reference audio | Language |
|:--- |:--- |:--- |:--- |
| [Qwen3-TTS-12Hz-Base-GGUF](https://huggingface.co/JamePeng2023/Qwen3-TTS-12Hz-Base-GGUF) | `MTMDAudioGenerator` | Optional speaker reference | Selectable, including Chinese, English and Japanese |
| Pocket TTS | `MTMDAudioGenerator` | Required | Determined by the language-pack weights |

The Qwen repository currently provides **1.7B Base** backbone and mmproj files
in BF16, F16 and Q8_0. The example below uses the tested BF16 pair. Download both
files and replace `/path/to/model/` with their local directory.

```python
from contextlib import closing

from llama_cpp import Llama, LLAMA_POOLING_TYPE_NONE
from llama_cpp.llama_multimodal import MTMDAudioGenerator

with closing(Llama(
    model_path="/path/to/model/qwen3-TTS-12Hz-1.7B-Base-BF16.gguf",
    embeddings=True,
    pooling_type=LLAMA_POOLING_TYPE_NONE,
    n_ctx=4096,
    n_gpu_layers=-1,  # Use 0 to disable model-layer offload.
)) as llama:
    with MTMDAudioGenerator(
        mmproj_path="/path/to/model/mmproj-qwen3-TTS-12Hz-1.7B-Base-BF16.gguf",
        use_gpu=True,
        flash_attn=None,  # AUTO; True enables FA, False disables it.
    ) as generator:
        audio = generator.create_speech(
            llama=llama,
            text="Hello, welcome to speech synthesis.",
            language="en",
            speaker_reference="/path/to/reference.wav",
            seed=42,
        )
        audio.save("output.wav")
```

Use `speaker_reference="/path/to/reference.wav"` for a reference voice, or omit
it for Qwen without a reference;
encoded audio bytes, URLs and data URIs are also accepted. For Pocket, provide
a reference and omit `language`. Qwen currently uses only the reference speaker
embedding: `ref_text`-conditioned full cloning, CustomVoice and preset speaker
IDs are not supported. The FA option controls mmproj independently of the backbone.

The result contains complete WAV or raw float32 PCM audio. Invalid audio raises
an error; `finish_reason="length"` means the generation-step limit was reached
and speech may be incomplete. There is no streaming output yet.

- [TTS API guide and limitations](docs/wiki/examples/audio/audio-tts.md)
- [CLI examples: reference voices, multilingual and batch synthesis](examples/high_level_api/mtmd_tts.py)
- [Streamlit playground: upload, recording, playback and downloads](examples/streamlit_tts/README.md)

## Comprehensive Omni MultiModal Example: Gemma-4 (Vision + Audio + Video + Text)

Below is a complete example showing how to route image, audio, and video files into one request. Images use Data URIs, audio uses `input_audio`, and videos use local paths so the MTMD video helper can sample frames with FFmpeg without Base64-encoding the entire file.

Video input requires a vision-capable projector, an MTMD build compiled with video support, and `ffmpeg` plus `ffprobe`. Start with a low sampling rate such as 1 FPS because every sampled frame consumes context tokens and preprocessing memory.

> **⚠️ IMPORTANT: GEMMA-4 MODEL CAPABILITIES & LIMITATIONS**
> * **Gemma4 E2B / E4B:** Supports Vision + Audio + Video + Text. `enable_thinking` **MUST** be `True`(default).
> * **Gemma4 31B / 26BA4B:** Supports Vision + Video + Text (Audio is NOT supported). `enable_thinking` can be toggled (`True` or `False`).

```python
from llama_cpp import Llama
from llama_cpp.llama_multimodal import Gemma4ChatHandler
import base64
import os

# Model and multimodal projection paths
MODEL_PATH = r"/path/to/Gemma-4-E4B-It-BF16.gguf"
# BF16 mmproj is required for audio. Other quantizations are known to have degraded performance.
MMPROJ_PATH = r"/path/to/mmproj-Gemma-4-E4B-It-BF16.gguf"

# Initialize the Llama model with multimodal and video support
# Note: Since we are using E4B here, enable_thinking MUST be True, and audio is supported.
llm = Llama(
    model_path=MODEL_PATH,
    chat_handler=Gemma4ChatHandler(
        mmproj_path=MMPROJ_PATH,
        enable_thinking=True,  # MUST be True for E2B/E4B models
        video_fps_target=1.0,  # Start low; sampled frames consume context tokens
        video_timestamp_interval_ms=5000,
        # video_ffmpeg_bin_dir=r"/path/to/ffmpeg/bin",  # Omit when tools are on PATH
        batch_max_tokens=1024,
        verbose=True,          # Enable Debug Info
    ),
    n_gpu_layers=-1,
    n_ctx=32768,               # Video usually needs more context than a single image
    verbose=True,              # Enable Debug Info
)

# 1. Extend the MIME dictionary to support audio formats
_MEDIA_MIME_TYPES = {
    # ------ Image formats ------
    '.png':  ('image', 'image/png'),
    '.jpg':  ('image', 'image/jpeg'),
    '.jpeg': ('image', 'image/jpeg'),
    '.gif':  ('image', 'image/gif'),
    '.webp': ('image', 'image/webp'),
    '.bmp':  ('image', 'image/bmp'),

    # ------ Audio formats ------
    '.wav':  ('audio', 'wav'),    # OpenAI standard usually uses raw format names for audio
    '.mp3':  ('audio', 'mp3'),
    # '.flac': ('audio', 'flac'),

    # ------ Video formats ------
    '.mp4':  ('video', 'video/mp4'),
    '.mkv':  ('video', 'video/x-matroska'),
    '.mov':  ('video', 'video/quicktime'),
    '.webm': ('video', 'video/webm'),
    '.avi':  ('video', 'video/x-msvideo'),
}

def build_media_payload(file_path: str) -> dict:
    """
    Convert a local image, audio file, or video into a Gemma 4 content item.
    """
    if not os.path.isfile(file_path):
        raise FileNotFoundError(f"Media file not found: {file_path}")

    extension = os.path.splitext(file_path)[1].lower()
    media_category, mime_or_format = _MEDIA_MIME_TYPES.get(extension, ('unknown', 'application/octet-stream'))

    if media_category == 'unknown':
        raise ValueError(f"Unsupported media extension: {extension} ({file_path})")

    if media_category == 'video':
        # Gemma 4 expects type="video". Keep the path intact so MTMD can pass it
        # to ffprobe/ffmpeg and decode sampled frames on demand.
        return {
            "type": "video",
            "video": os.path.abspath(file_path),
        }

    # Images and audio use inline Base64 payloads.
    with open(file_path, "rb") as f:
        encoded_data = base64.b64encode(f.read()).decode("utf-8")

    # 2. Return the appropriate dictionary structure based on the media type
    if media_category == 'image':
        # Image format: Data URI (OpenAI compatible)
        data_uri = f"data:{mime_or_format};base64,{encoded_data}"
        return {
            "type": "image_url",
            "image_url": {"url": data_uri}
        }

    elif media_category == 'audio':
        # Audio format: input_audio (OpenAI compatible)
        return {
            "type": "input_audio",
            "input_audio": {
                "data": encoded_data,
                "format": mime_or_format
            }
        }
    else:
        # Fallback for unsupported formats
        return {"type": "text", "text": f"[Attached unsupported file: {file_path}]"}


def run_inference(media_paths: list, text_prompt: str):
    """
    Helper function to dynamically build the payload and run inference.
    """
    # 3. Build the user_content list
    user_content = []

    # Automatically parse each file and append to the payload
    for path in media_paths:
        payload = build_media_payload(path)
        user_content.append(payload)

    # Append the final text instruction
    user_content.append({
        "type": "text",
        "text": text_prompt
    })

    print(f"\n--- Running Inference with {len(media_paths)} media file(s) ---")

    # 4. Send to the model for inference
    response = llm.create_chat_completion(
        messages=[
            {"role": "system", "content": """
            You are a highly capable multimodal assistant that can process text, images, audio, and video.

            """}, # Note: Audio Supported ONLY by Gemma4 E2B / E4B.
            {"role": "user", "content": user_content}
        ],
        temperature=1.0,
        top_p=0.95,
        top_k=64,
        max_tokens=8192,
    )

    print("\n[Model Response]:")
    print(response["choices"][0]["message"]["content"])
    print("-" * 60)


# ==============================================================================
# Main Inference Examples
# Uncomment the example block you wish to execute.
# ==============================================================================

# --- Example A: Image + Audio (Full Multimodal) ---
# Note: Supported ONLY by Gemma4 E2B / E4B.
run_inference(
    media_paths=[r"/path/to/test.png", r"/path/to/test.wav"],
    text_prompt="Introduce the content by combining the images and converting the audio to text."
)

# --- Example B: Image Only (Vision + Text) ---
# Note: Supported by all Gemma4 variants (E2B, E4B, 31B, 26BA4B).
# run_inference(
#     media_paths=[r"/path/to/test.png"],
#     text_prompt="Describe the contents of this image in detail."
# )

# --- Example C: Audio Only (Audio + Text) ---
# Note: Supported ONLY by Gemma4 E2B / E4B.
# run_inference(
#     media_paths=[r"/path/to/test.wav"],
#     text_prompt="Transcribe this audio and summarize the main points."
# )

# --- Example D: Video Only (Video + Text) ---
# Requires an MTMD video-enabled build plus ffmpeg and ffprobe.
# run_inference(
#     media_paths=[r"/path/to/test.mp4"],
#     text_prompt="Describe the main events in this video and include timestamps."
# )

# --- Example E: Image + Audio + Video (Omni Multimodal) ---
# Note: Audio support requires Gemma4 E2B/E4B and a compatible BF16 mmproj.
# run_inference(
#     media_paths=[
#         r"/path/to/test.png",
#         r"/path/to/test.wav",
#         r"/path/to/test.mp4",
#     ],
#     text_prompt="Analyze the media together and explain how their contents relate."
# )
```


---

## Embeddings & Reranking (GGUF)

`LlamaEmbedding` provides embedding defaults and a reranking helper on top of
the shared `Llama.embed()` implementation.

### Key Features:
* **Batch Processing:** Pack inputs into decode batches within token and sequence
  limits. Results accumulate in Python memory; large datasets should be split
  into application-level batches.
* **Native Reranking:** Built-in support for Cross-Encoder models (outputting relevance scores instead of vectors).
* **Optimized Performance:** Utilizes Unified KV Cache for parallel encoding of multiple documents.
* **Chat Template Support:** Support for rerank templates has been introduced (via `llama_model_chat_template(model, b"rerank")`), which can automatically populate the query and document into a specific format.

### Support Embeddings & Rerank Model:


|  Model             |  Type     |  Link                                                  |  Status      |
|--------------------|-----------|--------------------------------------------------------|--------------|
|`bge-m3`| Embedding |[bge-m3-GGUF](https://huggingface.co/gpustack/bge-m3-GGUF) |  Useful ✅  |
|`jina-embeddings-v2-base-zh`| Embedding |[jina-embeddings-v2-base-zh-GGUF](https://huggingface.co/gpustack/jina-embeddings-v2-base-zh-GGUF)  |  Useful ✅  |
|`jina-embeddings-v3`| Embedding |[jina-embeddings-v3-GGUF](https://huggingface.co/second-state/jina-embeddings-v3-GGUF) |  Useful ✅  |
|`bge-reranker-v2-m3`|   Rerank  |[bge-reranker-v2-m3-GGUF](https://huggingface.co/gpustack/bge-reranker-v2-m3-GGUF) |  Useful ✅  |
|`qwen3-reranker`|   Rerank  |[Qwen3-Reranker-GGUF](https://huggingface.co/JamePeng2023/Qwen3-Reranker-GGUF) |  Useful ✅  |

#### TODO(JamePeng): Needs more extensive testing with various embedding and rerank models. :)

### 1. Text Embeddings (Vector Search)

To generate embeddings, use the `LlamaEmbedding` class. It automatically configures the model for vector generation.

```python
from llama_cpp.llama_embedding import LlamaEmbedding

# Initialize the model (automatically sets embeddings=True)
llm = LlamaEmbedding(
    model_path="path/to/bge-m3.gguf",
    n_gpu_layers=-1,
    # Use the embedding model's default pooling for one vector per input.
    n_seq_max=128,  # Maximum independent sequences in one decode batch
)

# 1. Simple usage (OpenAI-compatible format)
response = llm.create_embedding("Hello, world!")
print(response['data'][0]['embedding'])

# 2. Batch processing (High Performance)
# Inputs are packed into native batches; returned vectors remain in Python memory.
documents = ["Hello, world!", "Goodbye, world!", "Llama is cute."] * 100
embeddings = llm.embed(documents) # Returns a list of lists (vectors)

print(f"Generated {len(embeddings)} vectors.")
```

> **Parallel batch capacity:** `n_seq_max` controls how many independent
> sequence IDs may coexist in one decode batch; it is not the total number of
> documents accepted by `embed()`. For batch embedding, set it high enough for
> the desired number of short documents per batch. The batcher flushes when
> token or sequence capacity is reached; `n_seq_max=1` processes inputs
> sequentially. Larger values can use more context resources.
> `LLAMA_POOLING_TYPE_NONE` returns one vector per token, adding a token
> dimension to each input's output; it does not return one pooled document vector.

**Advanced Output Formats:**
You can request raw arrays or cosine similarity matrices directly:

```python
from llama_cpp.llama_embedding import LlamaEmbedding

# Initialize the model (automatically sets embeddings=True)
llm = LlamaEmbedding(model_path="path/to/bge-m3.gguf", n_gpu_layers=-1)

# Returns a list of vectors without the dictionary wrapper
vector = llm.create_embedding("Text", output_format="array")

# Returns a similarity matrix (A @ A.T) in the response
# Note: Requires numpy installed
response = llm.create_embedding(
    ["apple", "fruit", "car"],
    output_format="json+"
)
print(response["cosineSimilarity"])
```

### 2. Reranking (Cross-Encoder Scoring)

Reranking models (like `bge-reranker`) take a **Query** and a list of **Documents** as input and output a relevance score (scalar) for each document.

> **Important:** You must explicitly set `pooling_type` to `LLAMA_POOLING_TYPE_RANK` (4) when initializing the model.

```python
import llama_cpp
from llama_cpp.llama_embedding import LlamaEmbedding

# Initialize a Reranking model
ranker = LlamaEmbedding(
    model_path="path/to/qwen3-reranker-0.6b-q8_0.gguf",
    pooling_type=llama_cpp.LLAMA_POOLING_TYPE_RANK,  # Crucial for Rerankers!
    n_gpu_layers=-1,
    n_ctx=0
)

query = "What causes Rain?"
docs = [
    "Clouds are made of water droplets...", # Relevant
    "To bake a cake you need flour...",     # Irrelevant
    "Rain is liquid water in the form of droplets..." # Highly Relevant
]

# Calculate relevance scores
# Logic: Constructs inputs like "[BOS] query [SEP] doc [EOS]" automatically
scores = ranker.rank(query, docs)

# Result: List of floats (higher means more relevant)
print(scores) 
# e.g., [0.0011407170677557588, 5.614783731289208e-05, 0.7173627614974976] -> The 3rd doc is the best match
```

### 3. Normalization

The `embed` method supports various mathematical normalization strategies via the `normalize` parameter.

| Normalization modes | $Integer$ | Description         | Formula |
|---------------------|-----------|---------------------|---------|
| NORM_MODE_NONE | $-1$      | none                |
| NORM_MODE_MAX_INT16 | $0$       | max absolute int16  | $\Large{{32760 * x_i} \over\max \lvert x_i\rvert}$
| NORM_MODE_TAXICAB | $1$       | taxicab             | $\Large{x_i \over\sum \lvert x_i\rvert}$
| NORM_MODE_EUCLIDEAN | $2$       | euclidean (default) | $\Large{x_i \over\sqrt{\sum x_i^2}}$
| NORM_MODE_PNORM | $>2$      | p-norm              | $\Large{x_i \over\sqrt[p]{\sum \lvert x_i\rvert^p}}$

Mode `0` rescales floating-point values; it does not cast to int16 or compress
storage. `NORM_MODE_PNORM` equals `6`; any integer greater than `2` selects that
p-norm. L2 normalization makes dot products equivalent to cosine similarity.
`LlamaEmbedding` defaults to L2, while `Llama.embed()` defaults to raw output.
Rank outputs are not normalized.

```python
from llama_cpp.llama_embedding import (
  LlamaEmbedding,
  NORM_MODE_NONE,
  NORM_MODE_MAX_INT16,
  NORM_MODE_TAXICAB,
  NORM_MODE_EUCLIDEAN
)

# Initialize the model (automatically sets embeddings=True)
llm = LlamaEmbedding(model_path="path/to/bge-m3.gguf", n_gpu_layers=-1)

# Taxicab (L1)
vec_l1 = llm.embed("text", normalize=NORM_MODE_TAXICAB)

# Default is Euclidean (L2) - Standard for vector databases
vec_l2 = llm.embed("text", normalize=NORM_MODE_EUCLIDEAN)

# Scale to a maximum absolute value of 32760; output remains floating point.
vec_scaled = llm.embed("text", normalize=NORM_MODE_MAX_INT16)

# Raw Output (No Normalization) - Get the raw floating point values from the model
embeddings_raw = llm.embed(["search query", "document text"], normalize=NORM_MODE_NONE)
```

### Using the standard `Llama` class

The standard `Llama` class owns the shared embedding implementation.
Initialize it with `embeddings=True`, then call `embed()` for
raw results or `create_embedding()` for an OpenAI-compatible response.
`LlamaEmbedding` remains a convenient specialized interface because it enables
embedding-oriented defaults and provides the `rank()` helper. Once embedding
execution begins, it resets existing generation state and cleans up again on
exit. Use separate instances if a live generation context must be preserved.

```python
llm = llama_cpp.Llama(
    model_path="path/to/model.gguf",
    embeddings=True,
    n_batch=512,
    n_seq_max=8,
    kv_unified=True,
)

# OpenAI-compatible response; normalize=True selects L2 normalization.
response = llm.create_embedding(["query", "document"], normalize=True)

# Raw vectors. Integer normalization modes are also supported.
vectors = llm.embed(["query", "document"], normalize=2)
```

---

## Docker image

See here: https://github.com/JamePeng/llama-cpp-python/tree/main/docker#cuda_simple

## OpenAI Compatible Web Server (Deprecated)

`llama-cpp-python` offers a web server which aims to act as a drop-in replacement for the OpenAI API.
This allows you to use llama.cpp compatible models with any OpenAI compatible client (language libraries, services, etc).

To install the server package and get started:

```bash
pip install 'llama-cpp-python[server] @ git+https://github.com/JamePeng/llama-cpp-python.git'
python3 -m llama_cpp.server --model models/7B/llama-model.gguf
```

For a source build with the CUDA backend on a POSIX shell:

```bash
CMAKE_ARGS="-DGGML_CUDA=on" pip install 'llama-cpp-python[server] @ git+https://github.com/JamePeng/llama-cpp-python.git'
python3 -m llama_cpp.server --model models/7B/llama-model.gguf --n_gpu_layers 35
```

Navigate to [http://localhost:8000/docs](http://localhost:8000/docs) to see the OpenAPI documentation.

To bind to `0.0.0.0` to enable remote connections, use `python3 -m llama_cpp.server --host 0.0.0.0`.
Similarly, to change the port (default is 8000), use `--port`.

You probably also want to set the prompt format. For chatml, use

```bash
python3 -m llama_cpp.server --model models/7B/llama-model.gguf --chat_format chatml
```

That will format the prompt according to how model expects it. You can find the prompt format in the model card.
For possible options, see [llama_cpp/llama_chat_format.py](llama_cpp/llama_chat_format.py) and look for lines starting with "@register_chat_format".

If you have `huggingface-hub` installed, you can also use the `--hf_model_repo_id` flag to load a model from the Hugging Face Hub.

```bash
python3 -m llama_cpp.server --hf_model_repo_id Qwen/Qwen2-0.5B-Instruct-GGUF --model '*q8_0.gguf'
```

### Web Server Features

The following links describe the upstream Python server. This fork retains the
server as a deprecated component; newer Python features may not be exposed there.

- [Local Copilot replacement](https://llama-cpp-python.readthedocs.io/en/latest/server/#code-completion)
- [Function Calling support](https://llama-cpp-python.readthedocs.io/en/latest/server/#function-calling)
- [Vision API support](https://llama-cpp-python.readthedocs.io/en/latest/server/#multimodal-models)
- [Multiple Models](https://llama-cpp-python.readthedocs.io/en/latest/server/#configuration-and-multi-model-support)

## Documentation

Documentation is maintained in [`docs/wiki`](docs/wiki) and published to the
[project wiki](https://github.com/JamePeng/llama-cpp-python/wiki).
If you find any issues with the documentation, please open an issue or submit a PR.

## Development

This package is under active development and I welcome any contributions.

To get started, clone the repository and install the package in editable / development mode:

```bash
git clone https://github.com/JamePeng/llama-cpp-python --recursive
cd llama-cpp-python

# Upgrade pip (required for editable mode)
pip install --upgrade pip

# Install with test dependencies
pip install -e ".[test]"

# if you want to use the fastapi / openapi server
pip install -e '.[server]'

# to install all optional dependencies
pip install -e '.[all]'

# to clear the local build cache
make clean
```

Run the tests; model-backed checks use the paths described in
[tests/README.md](tests/README.md). Unconfigured model tests are skipped locally;
Actions requires its configured models.

```bash
pytest
```

There's a `Makefile` available with useful targets.
A typical workflow would look like this:

```bash
make build
make test
```

You can also test out specific commits of `llama.cpp` by checking out the desired commit in the `vendor/llama.cpp` submodule and then running `make clean` and `pip install -e .` again. Any changes in the `llama.h` API will require
changes to the `llama_cpp/llama_cpp.py` file to match the new API (additional changes may be required elsewhere).

## FAQ

### Are there pre-built binaries / binary wheels available?

The recommended installation method is to install from source as described above.
The reason for this is that `llama.cpp` is built with compiler optimizations that are specific to your system.
Using pre-built binaries would require disabling these optimizations or supporting a large number of pre-built binaries for each platform.

That being said there are some pre-built binaries available through the Releases as well as some community provided wheels.

In the future, I would like to provide pre-built binaries and wheels for common platforms and I'm happy to accept any useful contributions in this area.

### How does this compare to other Python bindings of `llama.cpp`?

I originally wrote this package for my own use with two goals in mind:

- Provide a simple process to install `llama.cpp` and access the full C API in `llama.h`and `mtmd.h` from Python

- Provide a high-level Python API that can be used as a drop-in replacement for the OpenAI API so existing apps can be easily ported to use `llama.cpp`

- Provide a high-throughput, relatively low-latency Python library by continuously optimizing (reducing unnecessary CPU processing or algorithm tuning) and accepting feedback (issues or pull requests), making loading and running GGUF files via Python simpler and more controllable.

- Provides clearer code comments and error code analysis feedback in llama.cpp, based on common usage feedback and code execution flow, to help more users who are learning LLM through this project understand the project's operation and subsequent feedback optimization.

### OSError: libcudart.so.XX/cudart64_XX.dll: cannot open shared object file: No such file or directory
This error is primarily caused by the following reasons:

- Missing Installation or Configuration: The CUDA Toolkit is either not installed, or the environment variables were not correctly configured after installation, preventing the system from locating the required dynamic link libraries. You can try running `nvidia-smi` or `nvcc` in your terminal to check if they output results correctly.

- Version Mismatch: The CUDA Toolkit environment is installed and configured, but it does not match the CUDA version of the pre-compiled llama-cpp-python wheel you are using. For example, your local environment might be running CUDA 12.1, but you installed a version compiled for CUDA 12.6.

- Recommendation (Build from Source): It is recommended to fully configure your local CUDA Toolkit environment (ensuring the PATH for dynamic libraries is set and the nvcc compiler is recognized). Then, clone the code and compile it locally. Remember to enable the -DGGML_CUDA=on CMake option during compilation. This ensures the installation achieves maximum compatibility with your local system.

### FileNotFoundError: Could not find module (like ggml.dll, ggml-cpu.dll, ggml-cuda.dll)

**Step 1:** Locate the `lib` folder of the `llama-cpp-python` library within your current Python runtime environment: `Python3XX\Lib\site-packages\llama_cpp\lib\`

**Step 2:** Verify that the missing DLL mentioned in the error is correctly present in this directory. Developers often have multiple Python environments locally, or projects like ComfyUI may use embedded virtual environments. Please ensure that you are installing the library and running the code in the exact same environment.

This error is primarily caused by the following reasons:

1. **Environment Mismatch:** The Python environment used for installation is different from the one being used for execution.

2. **Instruction Set Incompatibility:** Regarding `ggml.dll` and `ggml-cpu.dll`, the instruction sets (such as AVX) supported by the pre-compiled version may be incompatible with your local processor. (This typically manifests as `OSError: [WinError -1073741795] Windows Error 0xc000001d` after execution).

3. **CUDA Version Mismatch:** Regarding `ggml-cuda.dll`, the CUDA version of the pre-compiled library does not match your local CUDA Toolkit version (e.g., a mismatch between CUDA 12.X and CUDA 13.X). It is recommended to fully configure your local CUDA Toolkit environment (ensuring the PATH for dynamic libraries is set and the nvcc compiler is recognized). Then, clone the code and compile it locally.

### Why are libraries compiled by other authors only around 100MB, while your pre-compiled versions are 300MB or larger?

My GitHub Actions workflow is configured to compile against multiple supported CUDA compute architectures for each CUDA version I maintain.

For example:

- **CUDA 13.1 and CUDA 12.8:** currently target architectures from SM75 (Turing) up to SM120a / SM121a (Blackwell generation, depending on CUDA support).
- **CUDA 12.4 and CUDA 12.6:** currently target architectures from SM70 (Volta) up to SM90 (Hopper).

Libraries from other authors are often smaller because they may only compile for a single architecture, such as RTX 30 series (`SM86`) or RTX 40 series (`SM89`). To maximize compatibility, these wheels include CUDA kernels for a wider range of GPUs. You only need to choose the wheel that matches your installed CUDA version.

 - **Updated 2026-05-16 / 2026-05-17:** Starting with `0.3.39-preview`, Windows wheels support the `GGML_BACKEND_DL` + `GGML_CPU_ALL_VARIANTS` runtime layout. CPU backend libraries such as `ggml-cpu-*.dll` are packaged under `site-packages/llama_cpp/lib` and loaded dynamically at runtime. This allows GGML to select a compatible CPU backend automatically, reducing the need for separate `Basic` / `AVX2` wheel variants.

 - Note: for full x64 CPU variant coverage on Windows, LLVM/Clang builds are preferred. MSVC may skip some variants such as `zen4`, `cooperlake`, or `sapphirerapids` due to compiler intrinsic support limitations.

### Quick tips for develop/user (continuously updated):

* 1. I've determined that `llama_cpp.server` is currently in a semi-deprecated state (meaning it won't be maintained unless absolutely necessary, and I might even consider deleting or separating it to reduce the library size). I highly recommend using the `llama-server` program maintained by the upstream `llama.cpp` project, which offers a lower-level implementation, more frequent maintenance and optimization, and more reliable API calls.

* 2. Regarding AMD and Intel graphics cards, AMD can use ROCm as the primary backend, while Intel's Sycl will encounter some compilation difficulties. I consistently recommend using the Vulkan backend for these two types of graphics cards for greater efficiency and stability, because the upstream `llama.cpp` Vulkan backend is actively maintained by many developers, generally allowing you to enjoy new feature optimizations and bug fixes earlier and faster.

* 3. If you are using hybrid multimodal model for building ComfyUI nodes or running single-turn API wrappers where you do not need multi-turn state rollbacks, simply initialize your Llama instance with `ctx_checkpoints=0`:

        ```python
        llm = Llama(
            model_path="./Qwen3.5-VL-9B.gguf",
            mmproj_path="./mmproj.gguf",
            n_ctx=4096,
            ctx_checkpoints=0  # <-- SET THIS TO 0 TO ENABLE ZERO-LATENCY FAST PATH
        )
        ```


### Any suggestions, contributions, and modifications to this package will be directed toward building a user-friendly, efficient, and secure Python library.

## License

This project is licensed under the terms of the MIT license.
