Instructions to use unsloth/Llama-3.2-1B-Instruct with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use unsloth/Llama-3.2-1B-Instruct with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="unsloth/Llama-3.2-1B-Instruct")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("unsloth/Llama-3.2-1B-Instruct")
model = AutoModelForCausalLM.from_pretrained("unsloth/Llama-3.2-1B-Instruct", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use unsloth/Llama-3.2-1B-Instruct with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "unsloth/Llama-3.2-1B-Instruct"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "unsloth/Llama-3.2-1B-Instruct",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/unsloth/Llama-3.2-1B-Instruct

SGLang

How to use unsloth/Llama-3.2-1B-Instruct with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "unsloth/Llama-3.2-1B-Instruct" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "unsloth/Llama-3.2-1B-Instruct",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "unsloth/Llama-3.2-1B-Instruct" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "unsloth/Llama-3.2-1B-Instruct",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Unsloth Desktop
Docker Model Runner
How to use unsloth/Llama-3.2-1B-Instruct with Docker Model Runner:
```
docker model run hf.co/unsloth/Llama-3.2-1B-Instruct
```

A DSpark draft model for Llama-3.2-1B-Instruct (1.5x - 2.7x higher throughput)

by rasyosef - opened 18 days ago

Discussion

rasyosef

18 days ago

Llama-3.2-1B-Instruct-speculator.dspark

Model: https://huggingface.co/rasyosef/Llama-3.2-1B-Instruct-speculator.dspark

A DSpark draft model for speculative decoding with unsloth/Llama-3.2-1B-Instruct as the verifier, trained with speculators. The drafter proposes 4 tokens at a time and the verifier checks them in one forward pass, so output is identical to running the verifier alone — a lossless speedup. In single-stream decoding it delivers 1.5x - 2.7x higher throughput on average.

Training code: rasyosef/train-dspark-draft-models.

Trained on 25,000 samples. Still a small run by drafter standards — acceptance should keep improving with more data.

Usage

vLLM loads the verifier automatically from the config — don't pass it separately.

vllm serve rasyosef/Llama-3.2-1B-Instruct-speculator.dspark --port 8000 --gpu-memory-utilization 0.75

Then query the OpenAI-compatible endpoint at http://localhost:8000/v1.

Details

2 Qwen3 layers (hidden size 2048, intermediate size 8192, 32 attention heads over 8 KV heads, sliding-window attention with a 2048-token window), ~0.25B params, bfloat16. Block size 4, draft vocabulary reduced to 32,000, aux hidden-state layers 2/5/8/11/14, confidence head with Markov (rank 256).

Trained for 4 epochs at lr 6e-4 (AdamW) on 25,000 Magpie prompts regenerated by the verifier itself, split 96/4 into train and validation, with a {"ce": 0.1, "tv": 0.9} loss. Sequence length 1280, up to 160 anchors per sample. Verifier hidden states were pulled on demand from a running vLLM server during training and deleted after use rather than staged to disk up front. speculators 0.8.0.dev207, vLLM 0.28.0, transformers 5.15.1, torch 2.13.0.

Evaluation

evaluate.py throughput across the nine RedHatAI/speculator_benchmarks subsets. acceptance_length is mean tokens committed per verification step, including the bonus token — floor 1.0, ceiling 5.0 at block size 4.

subset	drafts	acceptance_length	pos_0	pos_1	pos_2	pos_3
HumanEval	22,861	2.698	0.358	0.249	0.164	0.103
math_reasoning	5,163	2.385	0.315	0.193	0.105	0.055
tool_call	39,412	2.256	0.351	0.202	0.103	0.053
writing	16,936	2.105	0.302	0.174	0.080	0.038
question	15,292	2.023	0.283	0.145	0.077	0.041
qa	7,839	1.772	0.210	0.090	0.042	0.018
rag	4,502	1.766	0.209	0.093	0.037	0.016
summarization	12,901	1.550	0.200	0.066	0.017	0.004
translation	2,199	1.528	0.184	0.058	0.015	0.001

Weighted across all subsets: 2.161 over 127,105 verification steps, up from 1.776 in the previous 5,000-sample run. Every subset improved; the largest gains are on HumanEval, tool_call and math_reasoning (all roughly +0.5), the smallest on qa and rag (about +0.19).

Acceptance is still highest where the verifier's next token is most predictable — code, math, structured tool calls — and lowest on translation and summarization. tool_call now nearly matches HumanEval at the first position (0.351 vs 0.358) but trails it overall, because code stays predictable deeper into the block while tool-call arguments diverge sooner.

Limitations

25,000 training samples is still a modest budget, so acceptance would improve with more data. Works only with Llama-3.2-1B-Instruct and is not usable as a standalone model. Real-world speedup depends on your traffic mix, and because verification is lossless, the verifier's own behavior and biases carry through unchanged.

yosefw/Qwen3-0.6B-DSpark — same pipeline with Qwen/Qwen3-0.6B as the verifier.

License

Llama 3.2 Community License, inherited from the verifier. The speculators training code is Apache-2.0.

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment

unsloth
/

Llama-3.2-1B-Instruct