A colleague recently asked me what it would take to run a production-grade multimodal model entirely on local hardware — no API keys, no rate limits, no data leaving the building. Not a toy. Something that reads images, reasons step by step, calls tools natively, and fits on a single GPU.

The answer, as of this month, is Muse Glimmer.

What Muse Glimmer Actually Is

Muse Glimmer is Meta’s 30-billion-parameter open-source multimodal model. It is dense (not a mixture-of-experts), decoder-only, with a built-in vision encoder. It reads text and images, produces text output, and reasons through problems in a private chain-of-thought before answering.

The model is distilled from Muse Spark — Meta’s larger proprietary multimodal model — rather than trained from scratch. That distillation matters. You get the reasoning quality of a larger model compressed into a 30B footprint that a single GPU can serve.

The weights ship under Apache 2.0. Full stop. No research-only license, no gated access, no “we’ll let you know.” You download them, you run them, you own the inference.

Architecture: What’s Under the Hood

┌─────────────────────────────────────────────────────────┐   
│                   MUSE GLIMMER 30B                      │   
│                                                         │   
│  ┌──────────────────┐    ┌───────────────────────────┐  │   
│  │  Vision Encoder  │    │  Text Decoder             │  │ 
│  │  (built-in)      │───▶│  (dense, decoder-only)    │  │ 
│  └──────────────────┘    │  128K context window      │  │ 
│         ▲                │  GQA: 2 KV heads          │  │ 
│    ┌────┴────┐           │  Sliding-window attn      │  │
│    │  Image  │           │  (3/4 layers)             │  │
│    │  Input  │           └───────────────────────────┘  │   
│    └─────────┘                                          │   
└─────────────────────────────────────────────────────────┘

Three architectural choices make Muse Glimmer practical for local serving:

  1. Grouped-query attention with 2 KV heads. The KV cache stays small relative to the model size. This is what makes 128K context feasible on a 24 GB card.

  2. Sliding-window attention on three-quarters of the layers. Only one-quarter of the layers hold a full-length cache. The rest use a local window. This is an enormous memory win.

  3. Integrated vision encoder with a separate projector. Images go through the vision encoder, get projected into the model’s token space via a vision projector (mmproj), and the text decoder handles them alongside regular tokens. No separate CLIP model to manage.

The tokenizer is a tiktoken-style BPE with a 202,048-token vocabulary. It includes special tokens for the chat format (<|start|>, <|message|>, <|eot|>) and an <|image|> sentinel for multimodal input.

The Prompting Model: ATEM and Reasoning Strength

Muse Glimmer’s prompting format is more structured than most open-source models. It uses explicit role markers, turn separators, and a recipient system that separates private reasoning from user-facing output.

How Turns Work

Every turn opens with <|start|>, names its role, opens content with <|message|>, and ends with <|eot|>. Assistant turns can target a specific recipient:

  • to=user — the normal reply (default)
  • to=self — private chain-of-thought reasoning
  • to=<tool_name> — a tool call directed at a named tool

The model reasons by default. Before answering, it writes a to=self turn with its internal thinking, then emits the to=user turn with the final answer. This is not optional — it’s baked into the format.

Reasoning Strength

You can control how much the model reasons with four levels:

Level Behavior
low Minimal reasoning, faster responses
medium Balanced reasoning
high Default. Thorough reasoning.
xhigh Maximum reasoning depth

Set it server-wide or per-request:

prompt = tokenizer.apply_chat_template(
    messages, tokenize=False, add_generation_prompt=True,
    reasoning_strength="medium"
)

In llama.cpp, use --chat-template-kwargs '{"reasoning_strength":"low"}' for server-wide control, or pass chat_template_kwargs per request.

The critical detail: reasoning tokens count against max_tokens. If you set max_tokens too low, the model clips mid-thought and returns empty content with finish_reason: "length". Give reasoning workloads generous headroom.

Tool Calling

Muse Glimmer uses a native format called ATEM for tool calls. You pass OpenAI-style function schemas, and the model emits ATEM blocks in assistant to=<tool_name> turns:

<|start|>assistant to=get_weather<|message|><atem:function_calls>
<atem:invoke name="get_weather">
<atem:parameter name="city">Tokyo</atem:parameter>
</atem:invoke>
</atem:function_calls><|eot|>

One tool call per turn. No parallel tool calls. When served behind an OpenAI-compatible endpoint (vLLM or llama.cpp), the server parses ATEM and exposes standard tool_calls in the JSON response. You never touch ATEM directly.

Quantization: Running on a Single GPU

Muse Glimmer ships pre-quantized GGUF checkpoints. No conversion needed.

File Size Purpose
muse-glimmer-30B-kquant-17gb.gguf 16.8 GB Fixed K-quant. Start here.
muse-glimmer-30B-kquant-dynamic.gguf 19.7 GB Dynamic K-quant. Higher quality.
mmproj-kquant.gguf 1.4 GB Vision projector (required for images)
dflash-kquant.gguf 1.6 GB DFlash draft model (optional)

VRAM Budget

The K-Quant-17GB checkpoint with the vision projector and a full 131,072-token context fits in 19 GiB:

Component VRAM
Text model 15.6 GiB
Vision projector 1.3 GiB
KV cache + compute buffers 2.1 GiB
Total 19.0 GiB

That fits a 24 GB card with headroom. The KV cache stays small because of the GQA and sliding-window architecture — only a quarter of the layers hold a full-length cache.

Add the DFlash drafter for speculative decoding and you’re at roughly 20.5 GiB. Still fits.

Speculative Decoding: DFlash

DFlash is Muse Glimmer’s block-diffusion speculative decoding. A small draft model proposes continuation tokens, and the target model verifies them in bulk. More than one token lands per verification step.

This helps most on reasoning traces, where single-request latency dominates. The draft model is 1.6 GB as GGUF.

# llama.cpp with DFlash
./build/bin/llama-server \
  -m muse-glimmer-30B-kquant-17gb.gguf \
  -md dflash-kquant.gguf \
  --spec-type draft-dflash \
  -ngld 99 --spec-draft-n-max 4 \
  ... # other flags

A startup warning about [spec] failed to measure draft model memory is harmless — the draft loads and serves normally.

Deployment: Four Runtimes

Runtime Best For Hardware Notes
vLLM Production serving, high throughput NVIDIA GPU OpenAI-compatible HTTP
SGLang Concurrent user serving NVIDIA, Apple silicon OpenAI-compatible HTTP
llama.cpp Local/mixed CPU/GPU CPU, NVIDIA/AMD, Metal HTTP + CLI
ExecuTorch AOT export, edge deployment CUDA, Apple silicon Single program with draft

llama.cpp: The Full Setup

Muse Glimmer support landed in llama.cpp release b10353. Older builds refuse these checkpoints with unknown model architecture: 'muse-glimmer'.

# Download checkpoints
pip install -U huggingface_hub
hf download meta-models/Muse-Glimmer-30B-GGUF --local-dir ./muse-glimmer \
  --include "muse-glimmer-30B-kquant-17gb.gguf" \
  --include "mmproj-kquant.gguf"

# Start the server
./build/bin/llama-server \
  -m ./muse-glimmer/muse-glimmer-30B-kquant-17gb.gguf \
  --mmproj ./muse-glimmer/mmproj-kquant.gguf \
  -a muse-glimmer \
  -ngl 99 -c 131072 -np 1 \
  --host 127.0.0.1 --port 8080 --api-key <your-key> \
  --jinja \
  --chat-template-kwargs '{"reasoning_strength":"low"}'

The Flags That Matter

Flag Why It Matters
--jinja Applies the Muse Glimmer template from the GGUF. Without it, tool calling and reasoning separation break entirely.
-a muse-glimmer Sets the API model name. Without it, the model alias is the checkpoint path.
-np N Concurrent slots. Context is divided N ways. -np 4 with -c 131072 gives each slot 32,768 tokens.
-ngl 99 Offload all layers to GPU.
--chat-template-kwargs Server-wide reasoning_strength. Default is high.

Context Per Slot: The Silent Killer

This is the most important operational detail. n_ctx_slot — not -c — bounds a single generation. Getting this wrong fails silently: a generation that runs out of room produces no answer and no error. A batch job or eval reports a worse number and gives you nothing to debug.

Muse Glimmer reasons at length. A single generation can approach 32,768 tokens on its own. To keep concurrency without shrinking the slot:

-c 524288 -np 4        # 131,072 per slot, 4-way concurrent

Stale Metadata Fix

If the server reports exceeds the training context ... - capping, the GGUF’s context_length metadata is stale. Fix it:

python gguf-py/gguf/scripts/gguf_set_metadata.py <model>.gguf muse-glimmer.context_length 131072

CLI Without the Server

# Text only
./build/bin/llama-cli -m model.gguf -ngl 99 -c 32768 --jinja -st

# With images
./build/bin/llama-mtmd-cli -m model.gguf --mmproj mmproj.gguf \
  -ngl 99 -c 32768 --jinja --image photo.png -p "Describe this image."

-st = single-turn (answers once and exits). Both CLIs require --jinja. Both print thinking traces inline — --reasoning-format only separates them in the server’s JSON response.

What to Avoid

  1. Don’t manually construct the chat template. Use apply_chat_template. Muse Glimmer’s format has details the template handles for you.

  2. Don’t use AutoTokenizer for images. Use AutoProcessor — it handles both text tokenization and image preprocessing.

  3. Don’t set add_generation_prompt=False for inference. The model needs the cue to start generating.

  4. Don’t cut max_tokens short. Reasoning traces can be multi-thousand tokens. Cutting them mid-thought gives you empty output.

  5. Don’t use --chat-template-file in llama.cpp. There is no upstream Muse Glimmer template file. --jinja uses the template embedded in the GGUF, which is what wires up the ATEM parser.

  6. Don’t use reasoning_effort in llama.cpp. It’s not implemented. Use chat_template_kwargs.reasoning_strength.

  7. Don’t assume reasoning can be turned off. The template opens the thinking channel unconditionally. reasoning_strength: low is the minimum.

  8. Don’t ignore n_ctx_slot. A slot that’s too small silently swallows your output.

The Launch Ecosystem

Muse Glimmer launched with support from AMD, Arm, Dell, Fireworks AI, Hugging Face, Intel, llama.cpp, LM Studio, NVIDIA, Ollama, OpenRouter, SGLang/RadixArk, Together AI, Unsloth, and vLLM/Inferact. That is a broad launch — the model is not tied to a single serving stack or hardware vendor.

The Bottom Line

Muse Glimmer fills a gap that has existed since the open-source LLM ecosystem matured: a genuinely multimodal model with strong reasoning, open weights, and first-class support for local deployment. The 30B parameter count is the sweet spot — large enough for serious reasoning, small enough to fit on consumer hardware with quantization.

If you are running agentic workflows that need image understanding, tool calling, and chain-of-thought reasoning — and you need the data to stay on your hardware — this is the model to evaluate first.

The combination of K-Quant GGUF checkpoints, DFlash speculative decoding, and llama.cpp’s mature serving infrastructure means you can go from zero to a working multimodal inference server in under an hour. No API keys. No cloud dependency. No data leaving the building.

[[Muse Glimmer - Meta Open-Source 30B Multimodal Model]]