🐙 Google Turboquant

Google public Turboquant, which is KV-Cache smallest and fast enought.
Reduce 16 bit data into 3-4bit only, it frees up to 83% of the memory typically consumed by long prompts I self-build this fork llama-cpp-turboquant and have a first try.

docker-compose.yaml

services:
  llama-cpp:
    image: llama-cpp:turboquantsturboquant
    container_name: llama-cpp
    ports:
      - 8080:8080
    privileged: true
    ipc: host
    volumes:
      - /mnt/data/files/models:/app/models
    command: >
      -m /app/models/unsloth/Qwen3.6-35B-A3B-MTP-GGUF/Qwen3.6-35B-A3B-UD-IQ4_NL.gguf
      --port 8080 --host 0.0.0.0 
      -t 12 -c 262144
      --parallel 1 --no-context-shift --no-mmap --jinja 
      -b 4096 -ub 4096
      -ngl 999 --cpu-moe
      -ctk turbo3 -ctv turbo2
      --spec-type draft-mtp --spec-draft-n-max 3
      --kv-unified --cache-ram 0
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities:
                - gpu
    restart: unless-stopped

---

Result

Original TurboQuant token/s 29 32 RAM 26G 20G vRAM 11G 10G

Speed loose ~10%, but RAM consumption lower.