Instructions to use tevfikk/granite-docling-258M-bitnet with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use tevfikk/granite-docling-258M-bitnet with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="tevfikk/granite-docling-258M-bitnet")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("tevfikk/granite-docling-258M-bitnet")
model = AutoModelForMultimodalLM.from_pretrained("tevfikk/granite-docling-258M-bitnet", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use tevfikk/granite-docling-258M-bitnet with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "tevfikk/granite-docling-258M-bitnet"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "tevfikk/granite-docling-258M-bitnet",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/tevfikk/granite-docling-258M-bitnet

SGLang

How to use tevfikk/granite-docling-258M-bitnet with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "tevfikk/granite-docling-258M-bitnet" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "tevfikk/granite-docling-258M-bitnet",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "tevfikk/granite-docling-258M-bitnet" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "tevfikk/granite-docling-258M-bitnet",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use tevfikk/granite-docling-258M-bitnet with Docker Model Runner:
```
docker model run hf.co/tevfikk/granite-docling-258M-bitnet
```

Granite-Docling-258M-BitNet-1.58b

A ternary (BitNet b1.58) edition of IBM's granite-docling-258M document-conversion VLM (Idefics3): the 30-layer language decoder runs with weights in {-1, 0, +1} (one scale per output row), trained by quantisation-aware distillation from the BF16 original. A pure-C99 runtime runs it on CPU without Python.

The C runtime was renamed. What this card previously called eif-runtime is now bitllm.c (https://github.com/tevfik/bitllm.c).

Compatibility: the .eifm stores the vision tower as INT8 and needs bitllm.c commit b94202e or newer (a07bbe8: INT8 / INT4 quantised ViT weights; b94202e: aspect-ratio tile grid like the HF processor, --prompt honoured, greedy decoding by default with --image). Older commits mis-handle wide table crops and ignore the task prompt.

Status: step 5500 release. Good on clean English pages, invoices and OTSL tables (cell text); table structure, dense pages and Turkish body text are still weak (see Results and Limitations).

Files

File	Size	Use
`model.safetensors`	728 MB	Training checkpoint (step 5500): latent FP32 master weights of the decoder + SubLN norms + vision tower + connector. Load it with `modeling_granite_docling_bitnet.py`.
`modeling_granite_docling_bitnet.py`	7 KB	Standalone loader (torch + transformers only) that reproduces the trained ternary forward pass.
`granite_docling_bitnet_dense.eifm`	264 MB	Model for `bitllm.c`: ternary decoder (2-bit packed), INT8 embeddings, INT8 per-row ViT projections, FP32 connector/norms.
`tokenizer.bin`	1.5 MB	Byte-level BPE tokenizer for `bitllm.c`.
tokenizer / processor / config files		Standard Hugging Face files.

Results

The benchmark numbers below were measured on this release (step 5500, 2026-10-19, the published files through the bundled PyTorch loader) against the original BF16 model on the same inputs. The real-document table further down was measured on the previous checkpoint (step 3000) and was not repeated for step 5500. Scripts: models/vlm/granite_docling/eval_docling.py, eval_plugin_docs.py in the edge-intelligence repository.

Benchmarks (greedy decoding)

FinTabNet and PubTables-1M samples come from the datasets' test splits (training used the train splits); DocLayNet pages come from its test split and DocLayNet is not among the training sources listed below. Charts are random samples of the SynthChartNet train split, which is also a training source, so that row is not guaranteed unseen. n = 30 per row, 10 pages. The FinTabNet set is the same one used for the step 3000 numbers (F1 0.816, structure 0.866); PubTables, charts and pages are fresh seeded samples (fetch_eval_data.py), so their step 3000 values (0.739 / 0.890 / 0.347) are not exactly comparable. The step 1126 column is from older samples.

Task	Metric	BF16 teacher	This model	Earlier version of this model (step 1126)
FinTabNet tables (`Convert table to OTSL.`)	cell-text F1	0.943	0.801	0.376
FinTabNet tables	structure similarity	0.987	0.800	0.495
FinTabNet tables	exact structure match	0.867	0.200	0.000
PubTables-1M tables	cell-text F1	0.953	0.769	0.14
PubTables-1M tables	structure similarity	0.963	0.755	–
Charts (SynthChartNet)	DocTags similarity	0.938	0.919	0.87
Charts	number F1	0.701	0.632	–
DocLayNet pages (`Convert this page to docling.`)	word recall	0.883	0.461	0.11

Table structure is the weak point: cell texts are mostly right, but rows/columns are often misaligned (exact structure match 20% on FinTabNet vs 87% for the teacher).

Real documents through `gleann-plugin-docs` (bitllm.c, CPU; step 3000 checkpoint)

The plugin's convert was run on the documents in its testdata; the markdown is scored against the document's own text layer (word recall = share of the reference words that appear in the output). scanned_report.pdf has no text layer; its reference is the digital twin (benchmark_report.pdf, page 1). The other two cases force the model onto digital pages.

Document	Earlier version (step 1126)	Step 3000, INT8 ViT (shipped format)	Step 3000, INT4 ViT
Scanned report page (1 page, English)	0.095	0.968	0.847
Digital report forced through the VLM (2 pages)	0.350	0.978	0.957
Dense bilingual (Turkish/English) academic paper, 2 pages	0.018	0.093	0.031
same pages, BF16 teacher (PyTorch)		0.416

The shipped .eifm uses INT8 vision weights. INT4 (g128 + Hadamard rotation) is about 40 MB smaller but visibly worse (garbled words, repetition loops on dense text) and is not shipped; docling_to_eif.py --vision-weights int4-g128-had produces it.

Size and speed (CPU, 20-core ARM, 10 threads, INT8 ViT, `bitllm.c`)

	BF16 original	This model
Whole model file	728 MB (safetensors)	264 MB (`.eifm`)
Language decoder projections (106 M weights)	212 MB	26.5 MB ternary (2-bit packed)
Vision tower + connector	187 MB	114 MB
Peak memory (one page, 50 generated tokens)		~0.4 GB
One page (13 tiles): vision encoding / prompt prefill / decoding		~7-9 s / 2.6 s / ~62 tokens/s

The vision encoder is the largest single cost of a page. With bitllm.c commit cef6d8a or newer the INT8 ViT runs real INT8 matrix products (Armv8.6 i8mm), about 3x faster than the FP32 path (ViT ~4.6 s vs 13-15 s for a 17-tile page). Numbers above the table are from an older commit; a whole page now takes roughly 12-16 s on an idle 20-core ARM (ViT 4-5 s, prefill ~3 s, decode ~4 s), and timings vary a lot when other processes share the cores. On x86 or older ARM cores without i8mm the ViT falls back to the slower path.

Limitations

Turkish is only partly supported. Phase 6 added Turkish synthetic pages: on a Turkish test page the headings are now read (with occasional wrong letters and one invalid UTF-8 byte), but a dense body paragraph still collapses into a repeating garbled loop. The step 3000 numbers on the two-page bilingual paper (teacher 0.42, model 0.05 in PyTorch / 0.09 in the C runtime) were not re-measured for step 5500. Use another OCR for Turkish body text (see "Which path to use" below).
Dense pages (two-column papers, small print) lose most of the text: page word recall 0.46 vs 0.88 for the teacher. Keep the generation cap generous and check outputs for early stops.
Tables: structure is often wrong even when the cell text is right; merged cells are the common failure.
Specific symbols: < inside a cell is written as <s> (observed in both the PyTorch and the C runtime, so it is the model).
Occasional invalid UTF-8 from the decoder on special characters (lone continuation bytes); sanitise outputs.
No confidence estimate. The model does not signal when it is wrong.
Training and evaluation used greedy decoding; sampling was not evaluated.

Which path to use

Input	Recommendation
Clean English scan, invoice, single-column page	This model (`GLEANN_DOCLING_VLM=1`): word recall 0.97 on a scanned report, vs 0.10 without a VLM.
Table crop (`Convert table to OTSL.`)	This model: cell text mostly right (F1 0.77-0.80), structure often wrong.
Dense two-column papers, Turkish/diacritic text	Not this model (recall 0.09 on a dense bilingual paper). `gleann-plugin-docs`' ONNX page parser (layout + OCR, ~0.5-1.2 s/page) reached 0.68 recall on the same paper.
Digital PDFs	Text layer (`pdftotext`), no model needed.

Architecture


Base	`ibm-granite/granite-docling-258M` (`Idefics3`), 257.6 M parameters
Vision	SigLIP2 ViT: 12 layers, hidden 768, 16×16 patches, 512×512 tiles (86.4 M); frozen, identical to the original
Connector	pixel-shuffle (×4) + linear projection 12 288 → 576 (7.1 M); trained in the student
Decoder	Llama-style, 30 layers, hidden 576, 9 heads / 3 KV heads, MLP 1536, vocab 100 352, tied embeddings (164.1 M; 106.2 M ternary projection weights + 57.8 M embedding)
Page = 13 tiles	12 local tiles + 1 global overview = 832 visual tokens

Quantisation

Weights (decoder, 210 projections): W_ternary = clip(round(W / γ), -1, +1) · γ, γ = mean(|W_row|), one scale per output row. Stored latent in FP32 in model.safetensors, 2-bit packed + FP32 row scales in the .eifm.
SubLN: o_proj and down_proj apply a trained RMSNorm to their input (60 norms), as in BitNet b1.58.
Activations: training simulated 8-bit per-token absmax activations (straight-through estimator). At inference both the loader and bitllm.c keep activations in floating point.
Embeddings / LM head: INT8 per-row in the .eifm. Vision weights: INT8 per-row in the .eifm (lossless in our tests: table F1 0.946 vs 0.950); the connector stays FP32.
Tried, not used: one scale per 128 weights plus a Walsh-Hadamard rotation (Bonsai-style). Switching the trained model to it and training both variants for 100 steps: KL to the teacher 5.55 vs 0.57, table F1 0.00 vs 0.58, so the per-row quantiser was kept. bitllm.c supports that layout for models trained with it from scratch.

Training

Quantisation-aware distillation from the frozen BF16 teacher. Loss = 0.25·CE (assistant tokens) + 0.35·KL (T = 2, full vocabulary)

0.15·hidden-state cosine + 0.25·logit cosine. Vision features are computed once by the teacher's tower and shared.

Data mix of the final phase (Phase 6): DoclingMatix 30 %, Turkish Procedural Synthetics 20 % (invoices, receipts, bilateral agreements, 2-column papers, OTSL tables), SynthChartNet 14 %, FinTabNet-OTSL 10 %, PubTables-1M-OTSL 10 %, SynthCodeNet 8 %, SynthFormulaNet 8 %.
Phases 1-4: Early pipeline exploration.
Phase 5 (steps 1 127 - 3 000): Corrected distillation alignment, full answers (up to 3 072 tokens), OTSL tables integration.
Phase 6 (steps 3 001 - 5 500, this release): 2 500 additional distillation steps (~160 k pages) aimed at multilingual alignment and at BPE byte-level corruption on Turkish diacritics (not fully solved, see Limitations) (ş, ğ, ç, ö, ü, ı, İ). 17 497 zero-replay dynamic procedural Turkish pages were generated and trained on.
Phase 6 settings: micro-batch 4 × 16 accumulation (64 pages per step), 8-bit PagedAdamW, LR 5e-5 with cosine decay (floor 2.5e-5), ~2.0-2.5 pages/s. Final micro-batch loss ≈ 0.79 (KD 1.43, CE 0.96, CosSim 0.882, FeatSim 0.815).
Hardware: NVIDIA GB10 (Grace-Blackwell, unified memory), VRAM peak 26.6 GB.

Usage

Hugging Face (`transformers`)

model.safetensors holds latent weights. Plain Idefics3ForConditionalGeneration.from_pretrained would run them unquantised and without SubLN, which is a different (and worse) model. Use the bundled loader:

import sys
from huggingface_hub import snapshot_download
from PIL import Image

path = snapshot_download("tevfikk/granite-docling-258M-bitnet")
sys.path.insert(0, path)
from modeling_granite_docling_bitnet import load_granite_docling_bitnet, convert_page

model, processor = load_granite_docling_bitnet(path)           # cuda if available, else cpu; bf16
page = Image.open("page.png").convert("RGB")
print(convert_page(model, processor, page, "Convert this page to docling.", max_new_tokens=1024))
# tables: convert_page(model, processor, table_image, "Convert table to OTSL.")

The loader quantises the weights on the target device once; its logits and generations are identical to the training code path (tested). Needs torch, transformers (tested with 5.x) and safetensors.

C99, no Python: `bitllm.c`

git clone https://github.com/tevfik/bitllm.c.git && cd bitllm.c   # commit b94202e or newer
cmake -B build -DCMAKE_BUILD_TYPE=Release && cmake --build build -j
./build/bitllm-run --model granite_docling_bitnet_dense.eifm --tokenizer tokenizer.bin \
    --image page.png --prompt "Convert this page to docling." --max-tokens 1024 --threads 10
# other tasks: --prompt "Convert table to OTSL."  (table crops)   --prompt "Convert chart to table."

--image implies greedy decoding (-T 0 -r 1.0); sampling or a repetition penalty corrupts DocTags/OTSL.

The DocTags/OTSL text follows the --- Output Streaming --- line on stdout (the banner and statistics go to stdout too; progress logs go to stderr). PDFs have to be rendered to images first (e.g. pdftoppm -r 150 -png).

Documents end to end: `gleann-plugin-docs`

gleann-plugin-docs embeds bitllm.c and converts PDF/DOCX/XLSX/PPTX/images into markdown and a section graph for gleann. Digital PDFs use the text layer; scanned pages use this model:

GLEANN_DOCLING_VLM=1 GLEANN_DOCLING_MODEL=/path/to/granite_docling_bitnet_dense.eifm \
    ./bin/gleann-plugin-docs convert scan.pdf

Reproduce

python models/vlm/granite_docling/eval_docling.py --data <eval pickles> --checkpoint <step_3000.pt>   # benchmarks
python models/vlm/granite_docling/eval_plugin_docs.py --plugin <bin> --docs <testdata> --model name=<dir>   # real documents
python -m eif_tools.convert.llm.docling_to_eif --model-dir <pkg> --output model.eifm --vision-weights int8 --tokenizer-out tokenizer.bin

Citation & credits

Base model: IBM Granite-Docling 258M (Apache-2.0); Docling datasets by IBM / ds4sd and HuggingFaceM4.
BitNet b1.58: The Era of 1-bit LLMs (Microsoft Research).
Training framework: edge-intelligence. C runtime: bitllm.c (formerly eif-runtime).

Downloads last month: 238

Safetensors

Model size

0.3B params

Tensor type

F32

BF16

Model tree for tevfikk/granite-docling-258M-bitnet

Base model

ibm-granite/granite-docling-258M

Finetuned

(15)

this model

Datasets used to train tevfikk/granite-docling-258M-bitnet

Paper for tevfikk/granite-docling-258M-bitnet

The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits

Paper • 2402.17764 • Published Feb 27, 2024 • 632