Measured on device (edge-compat, granite-4.2-3b-int4): Galaxy S26 · LiteRT-LM 0.16.0 · GPU · decode 15.6 tok/s · prefill 235 tok/s · TTFT 940 ms · all 1783 ops delegated (2026-08-31); Mac Studio M4 Max · LiteRT-LM 0.16.0 · GPU · decode 85.5 tok/s · prefill 1240 tok/s · TTFT 237 ms (2026-08-31); Galaxy S26 · LiteRT-LM 0.16.0 · CPU · decode 9.8 tok/s · prefill 22 tok/s · TTFT 9.28 s (2026-08-31). Record: https://github.com/john-rocky/edge-compat/blob/main/cards/granite-4.2-3b-int4/CARD.md

Measured on device (edge-compat, granite-4.2-3b-int8): Galaxy S26 · LiteRT-LM 0.16.0 · GPU · decode 9.2 tok/s · prefill 241 tok/s · TTFT 960 ms · all 1783 ops delegated (2026-08-31); Mac Studio M4 Max · LiteRT-LM 0.16.0 · GPU · decode 70.7 tok/s · prefill 1208 tok/s · TTFT 244 ms (2026-08-31); Galaxy S26 · LiteRT-LM 0.16.0 · CPU · decode 7.0 tok/s · prefill 51 tok/s · TTFT 4.20 s (2026-08-31). Record: https://github.com/john-rocky/edge-compat/blob/main/cards/granite-4.2-3b-int8/CARD.md

granite-4.2-3b — LiteRT-LM

Try it in your browser: the LiteRT-LM Web chat demo loads granite-4.2-3b_int4.litertlm (2.19 GB download) and runs it with WebGPU.

ibm-granite/granite-4.2-3b converted to the LiteRT-LM (.litertlm) format for on-device inference with Google's LiteRT-LM runtime. Requires litert-lm ≥ 0.16.

Granite 4.2-3B is IBM's compact reasoning model (3.66B dense, 40 layers, hidden 2560, GQA 40:8, untied embeddings, 100k vocab), released August 2026. It works problems inside <think>…</think> before answering, and these bundles carry that machinery: the generation prompt pre-fills the think opener the way IBM's own chat template does, and a thought channel is declared in the bundle metadata so the runtime separates reasoning from the answer and honors a thinking budget.

File	Recipe	Size
granite-4.2-3b_int4.litertlm	int4 blockwise-32 + OCTAV on linears, int8 embedding	2.19 GB
granite-4.2-3b_int8.litertlm	int8 dynamic on linears + embedding	3.76 GB

The int4 file is the one that fits a phone (smaller, and faster on every GPU we measured); int8 is the quality build — it is within 1 GSM8K point of bf16, where int4 costs 11 (see Accuracy below). Pick by what your deployment can hold.

Correctness

Both files score 8/8 on an 8-question sanity gate on both CPU and GPU (Apple M4 Max, litert-lm 0.16.0 runtime lineage), with every answer arriving as a clean final answer through the thought channel — no reasoning leakage, no degeneration. The bf16 PyTorch reference scores 8/8 on the same questions under the same template.

On a Galaxy S26 (SM-S942Q, Snapdragon SM8850, Adreno) both files generate correctly on GPU and CPU, with full OpenCL delegation — 1783/1783 nodes on every prefill signature and decode, zero rejected ops — and the runtime separates the reasoning on-device exactly as intended: the answer arrives as a labeled thought block followed by the clean final answer.

On an iPhone 17 Pro the int4 file passes the same gate on both backends (on-device byte count verified against the source file): Metal GPU 7/8 (engine created at the bundle's full 4096-token budget, init 13.0 s; the one miss is the rhyme line at the end of the composite 8-question prompt — the model answers that question correctly when asked on its own, on every other platform) and CPU 8/8 (init 5.5 s). Available process memory never dropped below ~4.2 GB during either leg.

Accuracy — GSM8K

GSM8K, greedy, 0-shot chain-of-thought, max_tokens 2048 (a reasoning model truncated mid-thought never reaches its answer), n=100, identical prompt, harness and answer extraction for every row; the LiteRT rows ran on the litert-lm 0.16.1 engine (CPU backend), scored on the text after the thought channel:

Configuration	GSM8K
PyTorch bf16 (reference, MPS)	91.0%
LiteRT int8	90.0% (−1)
LiteRT int4 (block-32)	80.0% (−11)

int8 is at parity. int4 costs 11 points on this model — noticeably more than the −4 the same recipe cost the non-thinking granite-4.1-3b of the identical shape. The obvious escape was measured and closed: a block-128 build of the same weights scores 81.0% on the same protocol (a wash), so the cost is a property of int4 on this model, not of the block size. If your deployment can host the 3.76 GB int8 file, it is the better model; the int4 file is the one that fits a phone, and this table is the price.

Usage

litert-lm run ./granite-4.2-3b_int4.litertlm --prompt "What is the capital of France? Answer in one word."

# GPU
litert-lm run ./granite-4.2-3b_int4.litertlm --backend gpu --cache no --prompt "..."

Notes for a reasoning model:

• Give it a generous output budget (≥ 2048 tokens). The model thinks before it answers; truncated mid-thought it produces no final answer at all.
• The bundle declares the thought channel (<think>…</think>), so runtimes that expose ThinkingConfig / a thinking budget can cap or read the reasoning separately; the streamed answer contains only the final response.
• The bundle uses ChatML role markers (<|im_start|>…<|im_end|>) with a 4096-token KV budget and six prefill signatures (1024, 256, 64, 16, 4, 1) — six rather than eleven because every exported signature is charged engine memory whether or not it is called, a lesson measured on this model family's 4.1 ship, where an eleven-signature build was killed by iOS during Metal engine initialisation.
• Replies that take over 10 minutes. On litert-lm 0.18.x and 0.19.0, a blocking send_message that is still generating after 10 minutes fails with DEADLINE_EXCEEDED (a fixed timeout with no setting), and the generation it leaves running can crash the process. For long replies on a slow backend (long thinking in particular), lower max_output_tokens or use the streaming send_message_async, which has no such limit.

Performance

litert-lm benchmark (litert-lm 0.16.0), Apple M4 Max, -p 256 -d 256 --runs 3 --cache no --max-num-tokens 1024, quiet machine, serialized, ≥300 s rest before each GPU reading:

File	Backend	Prefill (256)	Decode	TTFT	Init
int4	GPU (Metal)	1240 tok/s	85.5 tok/s	0.24 s	4.4 s
int4	CPU	106 tok/s	21.5 tok/s	2.94 s	6.9 s
int8	GPU (Metal)	1208 tok/s	70.7 tok/s	0.24 s	3.5 s
int8	CPU	264 tok/s	21.1 tok/s	2.61 s	23.0 s

Init is the Init time the CLI prints; that value includes both Init Total and the phases inside it, so it counts engine creation twice (LiteRT-LM #4057).

Two patterns worth knowing, both consistent with the granite-4.1-3b conversion of the same shape: int4 beats int8 on the GPU (+21% decode on top of the 1.7× size reduction), while int8 wins CPU prefill (2.5×) — XNNPACK's large matmuls outrun the blockwise dequant.

Galaxy S26 (SM-S942Q, Snapdragon SM8850, Adreno; litert_lm_advanced_main from the litert-lm v0.16.0 release kit, 205-token prompt with --benchmark, 2 runs per cell, ranges shown; a reasoning model decodes its own full response, so decode-turn lengths vary):

File	Backend	Prefill (205)	Decode	TTFT	Init
int4	GPU (OpenCL)	235–241 tok/s	12.3–15.6 tok/s	0.93–0.94 s	13.6–15.7 s
int4	CPU	22–42 tok/s	9.5–9.8 tok/s	5.0–9.3 s	3.8–8.4 s
int8	GPU (OpenCL)	239–241 tok/s	9.2 tok/s	0.96–0.97 s	5.4–5.8 s
int8	CPU	51–95 tok/s	7.0 tok/s	2.3–4.2 s	0.6 s

The int4 GPU decode range's low end is the second back-to-back run (thermal); its high end is the rested reading. The shape matches the other platforms: int4 wins GPU decode 1.7× over int8, GPU wins TTFT ~5–10× over CPU, and peak process RSS stayed at 1.4 GB (int4 GPU) / 4.3 GB (int8 CPU) with no OOM on the 12 GB device.

Conversion notes

Converted with litert-torch 0.9.3 / litert-converter 0.4.0 / ai-edge-quantizer 0.9.0 / litert-lm-builder 0.16.1, from a pristine released stack — no patched checkout (reproduction script: hf-to-litertlm).

• The think opener is pre-filled by the bundle's prompt template. IBM's chat template ends the generation prompt with <|im_start|>assistant\n<think>\n, and the bundle's assistant prefix reproduces that exactly. This matters at export time: the converter derives the assistant prefix from an assistant history message, so a template that only opens <think> in its generation branch silently ships without the scaffold — the reasoning discipline is then left to the quantized model's own choice.
• A thought channel is declared in the metadata (<think> / </think>). Without it the runtime streams raw reasoning into the answer and silently ignores any thinking budget. The exporter does not emit it on this path; it was added post-export with the weights verified byte-identical.
• A Jinja chat template is bundled (files uploaded 2026-09-15). It is the canonical models/granite4_2 form: ChatML roles with the always-present system turn, IBM's <tool_call>/<function=…> XML for tool calls, enable_thinking (default true) choosing between the pre-opened <think> and the closed <think></think> generation prompt, and past assistant turns rendered as <think></think> + answer, as IBM's template renders history. The files before this date carried only the structured prefix/suffix; on LiteRT-LM 0.16 and 0.17 their re-rendered history dropped the assistant turns, which is why long conversations drifted (LiteRT-LM #3445). Weights and tokenizer are byte-identical to the earlier files; only the metadata section changed.
• No start token in the metadata. granite-4.2's tokenizer declares <s> as BOS but never prepends it (the tokenizer's post-processor adds nothing), so the converter's unconditional start_token write was suppressed. The 4.1 lesson, different mechanism: a prepended token the model never saw in that position degrades answers.
• Embedder externalised so the 100352×2560 input-embedding table sits in its own section, clear of the ~2 GiB single-section mmap ceiling on iOS. (4.2's embeddings are untied; the lm_head stays in the main graph.)
• Tokenizer embedded as the upstream tokenizer.json (HF tokenizer section), not a SentencePiece conversion — byte-level BPE survives intact.
• Quantization: int4 is blockwise-32 with OCTAV clipping on the linears and int8 on the embedding; int8 is dynamic per-channel on linears and embedding.

Raspberry Pi 5 (CPU)

Measured on a Raspberry Pi 5 Model B Rev 1.1 (8 GB, Raspberry Pi OS 64-bit) with litert-lm benchmark 0.16.1: CPU backend, 4 threads, 256 prefill + 256 decode tokens, --cache memory (the compile cache lives and dies with the process, so every invocation compiles the model from scratch; nothing is reused between runs), one warm-up plus one timed iteration per invocation, 3 invocations per file with cooldown in between. Values are the median across invocations (min–max in parentheses). No thermal throttling occurred during these runs (vcgencmd get_throttled stayed 0x0). Every file listed produced coherent text in a real generation on this backend before its numbers were recorded.

File	Prefill (tok/s)	Decode (tok/s)	TTFT	Peak RSS
granite-4.2-3b_int4.litertlm	11.7 (11.7–11.8)	2.1 (2.1–2.1)	24.6 s	3.4 GB
granite-4.2-3b_int8.litertlm	12.8 (12.8–13.0)	1.7 (1.7–1.7)	20.6 s	4.7 GB

Known runtime issues

External weights (since 2026-10-30). In granite-4.2-3b_int4.litertlm and granite-4.2-3b_int8.litertlm, the weights sit in TFLiteWeights sections at the end of the file (LiteRT's external weights) instead of inside the model graph. On Android, the GPU path needs LiteRT-LM 0.19.0 or later: the GPU engine of the v0.18.0 and v0.18.1 Android builds crashes (SIGSEGV) on such a file (LiteRT-LM#3986). The earlier files with the weights inside the graph stay at revision 31108ea.

With thinking off, from the second turn of a conversation on, LiteRT-LM puts the previous answer's stop token into the KV cache with a zero embedding, because these files hold the embedding table in its own section and the decoder takes embeddings, so later answers in a conversation can differ from the answer the model gives to the same messages in one prompt (read in the source of LiteRT-LM 0.16.0 to 0.18.1; measured with 0.18.1 on WeVisOne-VL-2B and Qwen2.5-Coder-1.5B-Instruct, not on these files). With thinking on, the runtime rebuilds the cache from the re-rendered history at each turn, so the second turn is free of it when the conversation opens with a user message; the same source shows a zero embedding at another token from the third turn on (not measured). One question per conversation avoids both, as does a new conversation that starts with the earlier messages and the new question.

On LiteRT-LM releases before 0.17.0 (read in the source of 0.10.1 to 0.16.1), an answer whose last punctuation mark is a token of its own, such as a . or ? after a word, arrives without that mark: these files carry stop sequences of <|im_end|> and a newline with each of 13 characters in front (12 punctuation marks and a space), so the runtime holds the mark back to see whether <|im_end|> and a newline follow, the answer ends on <|im_end|> alone, and the held mark is never written. LiteRT-LM 0.17.0 and later write the held mark (StopTokenStreamFilter in runtime/core/tasks.cc, read in the source of 0.17.0 to 0.18.1). Not measured on these files: this follows from the runtime's source read with these files' stop list and tokenizer; the drop itself was seen with LiteRT-LM 0.16.1 on a Galaxy S26, on another bundle whose stop sequences have the same form.

From a conversation preface (the system message, initial messages, tools and extra context a conversation is created with) that has tools and no messages, LiteRT-LM renders these files' chat template with an empty message list, and the template reads messages[0] without checking that the list is non-empty, so the render fails with Failed to apply template: undefined value. This happens in two places: when the preface is prefilled at creation (prefillPrefaceOnInit = true in Kotlin, prefillPrefaceOnInit: true on the web, SetPrefillPrefaceOnInit(true) in C++; the default is false), where creating the conversation fails with that error, and in the preface render (renderPrefaceIntoString() in Kotlin and Swift, RenderPrefaceIntoString() in C++, litert_lm_conversation_render_preface_to_string in the C API). The same happens with extra context (such as enable_thinking) and no messages, and in the preface render with an empty preface (run in the runtime on one Qwen3 template; the other templates rendered with minijinja 2.14 and jinja2). Read in the source of LiteRT-LM 0.17.0, 0.18.1 and main; measured with LiteRT-LM 0.18.1 (the litert-lm package from PyPI) through its C API on these files' template text. The default path, where the first message is rendered together with the preface, is not affected: the Python API and the CLI of 0.18.1 expose neither the flag nor the preface render, and Google AI Edge Gallery sets neither (its source as read on 2026-10-25). A system message or any initial message in the conversation config avoids it.

License and changes

Distributed under Apache-2.0 (inherited from the base model). Changes from the original work: weights converted from safetensors bf16 to LiteRT flatbuffers and quantized as described above; tokenizer and chat template repackaged into the .litertlm bundle (thinking pre-fill and thought-channel metadata as described). This repository is a community conversion and is not affiliated with IBM.