Text Generation
GGUF
edge0
Mixture of Experts
edge-inference
latent-moe
4-bit precision
agent
coding
cuda
conversational
Instructions to use Edge0/Edge0-145B-NVFP4 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use Edge0/Edge0-145B-NVFP4 with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf Edge0/Edge0-145B-NVFP4:BF16 # Run inference directly in the terminal: llama cli -hf Edge0/Edge0-145B-NVFP4:BF16
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf Edge0/Edge0-145B-NVFP4:BF16 # Run inference directly in the terminal: llama cli -hf Edge0/Edge0-145B-NVFP4:BF16
Build from source code
git clone https://github.com/ggml-org/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-app # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama serve -hf Edge0/Edge0-145B-NVFP4:BF16 # Run inference directly in the terminal: ./build/bin/llama cli -hf Edge0/Edge0-145B-NVFP4:BF16
Use Docker
docker model run hf.co/Edge0/Edge0-145B-NVFP4:BF16
- Llama new
- LM Studio
- Jan
- vLLM
How to use Edge0/Edge0-145B-NVFP4 with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "Edge0/Edge0-145B-NVFP4" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "Edge0/Edge0-145B-NVFP4", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker
docker model run hf.co/Edge0/Edge0-145B-NVFP4:BF16
- Ollama
How to use Edge0/Edge0-145B-NVFP4 with Ollama:
ollama run hf.co/Edge0/Edge0-145B-NVFP4:BF16
- Unsloth Desktop
- Pi
How to use Edge0/Edge0-145B-NVFP4 with Pi:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf Edge0/Edge0-145B-NVFP4:BF16
Configure the model in Pi
# Install Pi: npm install -g @earendil-works/pi-coding-agent # Add to ~/.pi/agent/models.json: { "providers": { "llama-cpp": { "baseUrl": "http://localhost:9931/v1", "api": "openai-completions", "apiKey": "none", "models": [ { "id": "Edge0/Edge0-145B-NVFP4:BF16" } ] } } }Run Pi
# Start Pi in your project directory: pi
- Docker Model Runner
How to use Edge0/Edge0-145B-NVFP4 with Docker Model Runner:
docker model run hf.co/Edge0/Edge0-145B-NVFP4:BF16
- Lemonade
How to use Edge0/Edge0-145B-NVFP4 with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull Edge0/Edge0-145B-NVFP4:BF16
Run and chat with the model
lemonade run user.Edge0-145B-NVFP4-BF16
List all available models
lemonade list
- Hermes Agent
How to use Edge0/Edge0-145B-NVFP4 with Hermes Agent:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf Edge0/Edge0-145B-NVFP4:BF16
Configure Hermes
# Install Hermes: curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash hermes setup # Point Hermes at the local server: hermes config set model.provider custom hermes config set model.base_url http://127.0.0.1:9931/v1 hermes config set model.default Edge0/Edge0-145B-NVFP4:BF16
Run Hermes
hermes
- Atomic Chat
- OpenClaw
How to use Edge0/Edge0-145B-NVFP4 with OpenClaw:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf Edge0/Edge0-145B-NVFP4:BF16
Configure OpenClaw
# Install OpenClaw: npm install -g openclaw@latest # Register the local server and set it as the default model: openclaw onboard --non-interactive --mode local \ --auth-choice custom-api-key \ --custom-base-url http://127.0.0.1:9931/v1 \ --custom-model-id "Edge0/Edge0-145B-NVFP4:BF16" \ --custom-provider-id llama-cpp \ --custom-compatibility openai \ --custom-text-input \ --accept-risk \ --skip-health
Run OpenClaw
openclaw agent --local --agent main --message "Hello from Hugging Face"
docs: say where strata-vision comes from (not the Edge0 source tree)
Browse files
README.md
CHANGED
|
@@ -447,9 +447,19 @@ and then serves OpenAI- and Anthropic-compatible endpoints:</p>
|
|
| 447 |
</details>
|
| 448 |
<details open>
|
| 449 |
<summary><b>Images</b> 路 the vision encoder in this folder</summary>
|
| 450 |
-
<p><code>mmproj-BF16.gguf</code> is the vision encoder
|
| 451 |
-
|
| 452 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 453 |
<pre><code class="language-json">"exe": "/path/to/strata",
|
| 454 |
"vision": {
|
| 455 |
"exe": "/path/to/strata-vision",
|
|
@@ -460,14 +470,14 @@ next to <code>args</code>:</p>
|
|
| 460 |
}
|
| 461 |
</code></pre>
|
| 462 |
<ul>
|
| 463 |
-
<li><code>strata-vision</code> ships with the engine build, in the same folder as the engine.</li>
|
| 464 |
<li><code>"model"</code> is the text backbone this folder already uses.</li>
|
| 465 |
<li><code>"gpu": false</code> runs the encoder on the CPU and takes no VRAM. <code>true</code> runs it on the card and
|
| 466 |
keeps about 1.4 GB free for it, so the expert cache gets that much less.</li>
|
| 467 |
<li><code>"max_tokens"</code> caps how many tokens one picture becomes. More keeps more detail and takes longer.</li>
|
| 468 |
</ul>
|
| 469 |
<p>Add <code>--vision</code> to <code>"args"</code> as well. The server adds it by itself when the config has a
|
| 470 |
-
<code>vision</code> section and prints a line saying so; writing it keeps the two in step.
|
|
|
|
| 471 |
<p>The log says <code>images on</code> once the encoder is up. Then send a picture: <code>/image /path/to.png</code>
|
| 472 |
followed by the question in the terminal chat, or an <code>image_url</code> part over the OpenAI API. That part takes a
|
| 473 |
<code>data:</code> URL, an <code>http(s)</code> URL or a local file path.</p>
|
|
|
|
| 447 |
</details>
|
| 448 |
<details open>
|
| 449 |
<summary><b>Images</b> 路 the vision encoder in this folder</summary>
|
| 450 |
+
<p><code>mmproj-BF16.gguf</code> is the vision encoder: it turns a picture into the embedding rows the language model
|
| 451 |
+
reads, and the language model itself is unchanged. Reading pictures also needs a small helper program,
|
| 452 |
+
<code>strata-vision</code>. That program is <strong>not</strong> in this repository and the Edge0 source tree does not
|
| 453 |
+
build it, so it comes from one of these two places:</p>
|
| 454 |
+
<ul>
|
| 455 |
+
<li><strong>The prebuilt engine archive.</strong> <code>setup.py</code> and <code>strata launch</code> download one and
|
| 456 |
+
unpack <code>strata</code> and <code>strata-vision</code> side by side, so an install that ran with <code>--vision</code>
|
| 457 |
+
already has the helper next to the engine.</li>
|
| 458 |
+
<li><strong>Building it.</strong> Take a Strata source checkout (the <code>tools/vision</code> project lives there, not
|
| 459 |
+
in Edge0) and run <code>cmake --build build-vision --target strata-vision</code>, then put the result next to
|
| 460 |
+
<code>strata</code>.</li>
|
| 461 |
+
</ul>
|
| 462 |
+
<p>Then two entries in <code>model.json</code>, next to <code>args</code>:</p>
|
| 463 |
<pre><code class="language-json">"exe": "/path/to/strata",
|
| 464 |
"vision": {
|
| 465 |
"exe": "/path/to/strata-vision",
|
|
|
|
| 470 |
}
|
| 471 |
</code></pre>
|
| 472 |
<ul>
|
|
|
|
| 473 |
<li><code>"model"</code> is the text backbone this folder already uses.</li>
|
| 474 |
<li><code>"gpu": false</code> runs the encoder on the CPU and takes no VRAM. <code>true</code> runs it on the card and
|
| 475 |
keeps about 1.4 GB free for it, so the expert cache gets that much less.</li>
|
| 476 |
<li><code>"max_tokens"</code> caps how many tokens one picture becomes. More keeps more detail and takes longer.</li>
|
| 477 |
</ul>
|
| 478 |
<p>Add <code>--vision</code> to <code>"args"</code> as well. The server adds it by itself when the config has a
|
| 479 |
+
<code>vision</code> section and prints a line saying so; writing it keeps the two in step. Without the helper beside the
|
| 480 |
+
engine, a request that carries a picture fails with <code>this server was started without the vision encoder</code>.</p>
|
| 481 |
<p>The log says <code>images on</code> once the encoder is up. Then send a picture: <code>/image /path/to.png</code>
|
| 482 |
followed by the question in the terminal chat, or an <code>image_url</code> part over the OpenAI API. That part takes a
|
| 483 |
<code>data:</code> URL, an <code>http(s)</code> URL or a local file path.</p>
|