🤖 Why user GPU cannot support coding agent ?

Common user GPU only have 8-16G VRAM.
When the model not fully loaded to VRAM, they have to offload into RAM and processing data with CPU.
This cause local AI system super slow, token generation speed only reach 1-5 token/s.
This paper is writing about test cases and looking for the best configuration for self-host agent.

On my test case.
I run llama.cpp with Qwen3.6-35B-A3B-MTP-GGUF (IQ4_XS).
Then public the API key to use vscode + pi.dev agent.

Test case 1:

services:
  llama-cpp:
    image: sandichhuu/llama-cpp:cu132
    container_name: llama-cpp
    ports:
      - 8080:8080
    environment:
      - GGML_CUDA_GRAPH_OPT=1
      - LD_LIBRARY_PATH=/usr/local/nvidia/lib64:/usr/local/nvidia/lib:/usr/lib/x86_64-linux-gnu
    privileged: true
    ipc: host
    volumes:
      - /mnt/data/files/models:/app/models
    command: >
      -m /app/models/unsloth/Qwen3.6-35B-A3B-MTP-GGUF/Qwen3.6-35B-A3B-UD-IQ4_NL.gguf
      --mmproj /app/models/unsloth/Qwen3.6-35B-A3B-MTP-GGUF/mmproj-BF16.gguf
      --port 8080 --host 0.0.0.0
      -t 12 -c 262144 --parallel 1 -fa on
      --cache-type-k q4_0 --cache-type-v q4_0 
      --n-cpu-moe 35 --no-context-shift
      -b 4096 -ub 2048 
      --no-mmap --direct-io --fit off --jinja --no-cache-prompt --cache-ram 0
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities:
                - gpu
    restart: unless-stopped

The llama.cpp process consume 16G of RAM and 11G of VRAM.
And graph help me understand how it's working.

On the case trigger data layers on RAM -> CPU will working.
andthen trigger data layers on VRAM -> GPU will working.
This mean we need a GPU rigs enough VRAM to full load model into.