onwx 1.0.0: LFM2.5 24B-A2B (converted on your PC, two-block int4, chunked prompts ~430 tok/s)
Browse files- CHANGELOG.md +5 -0
- CHANGELOG_en.md +5 -0
- README.md +3 -1
- README_en.md +3 -1
- onwx/__init__.py +1 -1
- onwx/app.py +10 -0
- onwx/bin/lfx_glue.dll +0 -0
- onwx/bin/liblfx_glue.so +0 -0
- onwx/dl.py +4 -0
- onwx/i18n.json +10 -2
- onwx/kit.py +79 -0
- onwx/manager.py +4 -0
- onwx/npu_lfx.py +477 -0
- onwx/web/manage.html +12 -3
- onwx/xdna.py +15 -9
- onwx/xdna_moe.py +42 -8
- pyproject.toml +1 -1
- tools/lfx_glue/build.bat +4 -0
- tools/lfx_glue/build.sh +4 -0
- tools/lfx_glue/lfx_glue.cpp +621 -0
CHANGELOG.md
CHANGED
|
@@ -5,6 +5,11 @@
|
|
| 5 |
日付は日本時間です。各版は Hugging Face の `ryugyosoft/onwx` にタグ(`v0.2.0` 以降)付きで残っています。
|
| 6 |
onwx ウィンドウは、新しい版があるとこのファイルから変更点を表示します(`## v…` の見出しで区切っています)。
|
| 7 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 8 |
## v0.9.0 (2026-10-16)
|
| 9 |
- **Qwen3.6 35B-A3B が少ないメモリでも動くように(エキスパートを SSD から読む)**:MoE のエキスパートを SSD に置いたまま、ルーターが選んだものだけを読み込み、最近使ったものをメモリに残します(onw と同じ方式。OS のファイルキャッシュを通さない直接読み込みで、次の層で使いそうなものは先読みします)。設定の「MoE のエキスパートのメモリ」で量を選べます(自動・すべてメモリ・4〜12 GB)。自動は、モデル全体がメモリの 7 割に収まればすべてメモリに置きます。Ryzen AI Max+ 395 で 4 GB にすると、使用メモリは約 20 GB → 約 7 GB、生成は約 20 → 約 15 tok/s、プロンプトの読み込みは 1,000 トークンで 3.1 → 3.5 秒です。
|
| 10 |
- **Qwen3.6 35B-A3B が画像を読めるように**:画像のエンコーダー(ViT)は Qwen3.5 と同じく CPU で動きます(モデルの更新が必要です)。
|
|
|
|
| 5 |
日付は日本時間です。各版は Hugging Face の `ryugyosoft/onwx` にタグ(`v0.2.0` 以降)付きで残っています。
|
| 6 |
onwx ウィンドウは、新しい版があるとこのファイルから変更点を表示します(`## v…` の見出しで区切っています)。
|
| 7 |
|
| 8 |
+
## v1.0.0 (2026-10-22)
|
| 9 |
+
- **LFM2.5 24B-A2B(Liquid AI の MoE)をモデル一覧に追加**:64 エキスパートから 4 つを選ぶ 24B のモデル(1 トークンあたり 2.4B)を、全部 NPU で動かします。onwx は Liquid AI の重みを配布しません:ダウンロードを選ぶと、変換用のキットと元のモデル(約 48 GB)を Hugging Face から取ってきて、この PC で int4 に変換します(ダウンロードと変換で約 20〜30 分、Ryzen AI Max+ 395 で約 23 分。終わると元のファイルは消します。LFM Open License v1.0 が適用されます)。モデル一覧ではボタンが「ダウンロードして変換」になり、押すと元のモデルの大きさ(約 48 GB)、変換中に要る空き(約 65 GB)、変換後の大きさ(約 17 GB)、かかる時間を確かめてから始めます。
|
| 10 |
+
- LFM2 は int4 の誤差に弱い(B×x、C×y の掛け算のゲート)ので、エキスパート以外と、エキスパートの down を int4 の 2 枚重ね(1 枚目の誤差をもう 1 枚の int4 で表す)にしました。fp32 と比べた KL は 0.33 → 0.10。生成は約 20 tok/s、使用メモリは約 19 GB です。
|
| 11 |
+
- LFM2.5 のプロンプトはまとめて読みます:行列の計算は NPU が int4 の重みから直接行い(全部のプログラムが 1 つの NPU 構成を共有)、注意機構・短い畳み込み・ルーティングは CPU(新しいライブラリ lfx_glue)が行います。1,000 トークンで約 2.4 秒、4,000 トークンで約 8.5 秒(1 トークンずつの約 20 倍)。
|
| 12 |
+
|
| 13 |
## v0.9.0 (2026-10-16)
|
| 14 |
- **Qwen3.6 35B-A3B が少ないメモリでも動くように(エキスパートを SSD から読む)**:MoE のエキスパートを SSD に置いたまま、ルーターが選んだものだけを読み込み、最近使ったものをメモリに残します(onw と同じ方式。OS のファイルキャッシュを通さない直接読み込みで、次の層で使いそうなものは先読みします)。設定の「MoE のエキスパートのメモリ」で量を選べます(自動・すべてメモリ・4〜12 GB)。自動は、モデル全体がメモリの 7 割に収まればすべてメモリに置きます。Ryzen AI Max+ 395 で 4 GB にすると、使用メモリは約 20 GB → 約 7 GB、生成は約 20 → 約 15 tok/s、プロンプトの読み込みは 1,000 トークンで 3.1 → 3.5 秒です。
|
| 15 |
- **Qwen3.6 35B-A3B が画像を読めるように**:画像のエンコーダー(ViT)は Qwen3.5 と同じく CPU で動きます(モデルの更新が必要です)。
|
CHANGELOG_en.md
CHANGED
|
@@ -5,6 +5,11 @@
|
|
| 5 |
Dates are Japan time. Every version is kept on Hugging Face `ryugyosoft/onwx` with a tag (from `v0.2.0` on).
|
| 6 |
The onwx window shows what changed from this file when there is a newer version (sections start at `## v…`).
|
| 7 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 8 |
## v0.9.0 (2026-10-16)
|
| 9 |
- **Qwen3.6 35B-A3B in less memory (experts read from the SSD)**: the MoE's experts stay on the SSD; only the ones the router picks are read, and the recently used ones stay in memory (as in onw: unbuffered reads that bypass the OS file cache, the experts the next layer will likely use read ahead). The setting "Memory for MoE experts" picks how much (auto, all in memory, 4-12 GB); auto keeps them all when the whole model fits in 70% of the memory. On a Ryzen AI Max+ 395 with 4 GB: memory ~20 GB -> ~7 GB, generation ~20 -> ~15 tok/s, a 1,000-token prompt 3.1 -> 3.5 s.
|
| 10 |
- **Qwen3.6 35B-A3B reads images**: the image encoder (ViT) runs on the CPU as for Qwen3.5 (the model needs an update).
|
|
|
|
| 5 |
Dates are Japan time. Every version is kept on Hugging Face `ryugyosoft/onwx` with a tag (from `v0.2.0` on).
|
| 6 |
The onwx window shows what changed from this file when there is a newer version (sections start at `## v…`).
|
| 7 |
|
| 8 |
+
## v1.0.0 (2026-10-22)
|
| 9 |
+
- **LFM2.5 24B-A2B (Liquid AI's MoE) in the model list**: a 24B model that picks 4 of 64 experts (2.4B a token), all on the NPU. onwx does not distribute Liquid AI's weights: the download fetches a conversion kit and the original model (~48 GB) from Hugging Face and converts it to int4 on your PC (download and conversion ~20-30 minutes, ~23 on a Ryzen AI Max+ 395; the original files are deleted after; the LFM Open License v1.0 applies). In the model list its button reads "Download and convert" and asks first, showing the original model's size (~48 GB), the free disk the conversion needs (~65 GB), the converted size (~17 GB) and the time.
|
| 10 |
+
- LFM2 is sensitive to the int4 error (its multiplicative gates B*x, C*y), so everything but the experts, and the experts' down projections, are two stacked int4 blocks (the second holding the first one's error). KL against fp32 0.33 -> 0.10. ~20 tok/s generation, ~19 GB of memory.
|
| 11 |
+
- LFM2.5's prompts are read in chunks: the matrix products on the NPU straight from the int4 weights (every program on one NPU configuration), the attention, the short convolutions and the routing on the CPU (a new library, lfx_glue). ~2.4 s for 1,000 tokens, ~8.5 s for 4,000 (~20x reading them one by one).
|
| 12 |
+
|
| 13 |
## v0.9.0 (2026-10-16)
|
| 14 |
- **Qwen3.6 35B-A3B in less memory (experts read from the SSD)**: the MoE's experts stay on the SSD; only the ones the router picks are read, and the recently used ones stay in memory (as in onw: unbuffered reads that bypass the OS file cache, the experts the next layer will likely use read ahead). The setting "Memory for MoE experts" picks how much (auto, all in memory, 4-12 GB); auto keeps them all when the whole model fits in 70% of the memory. On a Ryzen AI Max+ 395 with 4 GB: memory ~20 GB -> ~7 GB, generation ~20 -> ~15 tok/s, a 1,000-token prompt 3.1 -> 3.5 s.
|
| 15 |
- **Qwen3.6 35B-A3B reads images**: the image encoder (ViT) runs on the CPU as for Qwen3.5 (the model needs an update).
|
README.md
CHANGED
|
@@ -118,8 +118,9 @@ onwx chat ryugyosoft/Ternary-Bonsai-2-27B-onwx # 端末でチャット
|
|
| 118 |
| [Qwen3.5 9B](https://huggingface.co/ryugyosoft/Qwen3.5-9B-onwx)(int4、画像も読める) | 7.6 GB | 約 5.8 GB + 文脈 1k トークンあたり約 15 MB | 約 2.5 秒 / 約 7.7 秒(CPU も使う設定で約 2.1 秒 / 約 6.6 秒) | 約 12 tok/s、温度 0 なら MTP で約 15 tok/s |
|
| 119 |
| [Qwen3.5 4B](https://huggingface.co/ryugyosoft/Qwen3.5-4B-onwx)(int4、画像も読める) | 4.8 GB | 約 3.4 GB + 文脈 1k トークンあたり約 13 MB | 約 2.1 秒 / 約 6.4 秒(CPU も使う設定で約 1.8 秒 / 約 5.7 秒) | 約 18 tok/s |
|
| 120 |
| [llm-jp-4.1 8B thinking](https://huggingface.co/ryugyosoft/llm-jp-4.1-8b-onwx)(int4、日本語に強い) | 5.8 GB | 約 4.8 GB + 文脈 1k トークンあたり約 65 MB | 約 6.3 秒 / 約 13 秒 | 約 11 tok/s |
|
|
|
|
| 121 |
|
| 122 |
-
(Ryzen AI Max+ 395・Windows で測定。Qwen3.6 35B-A3B は
|
| 123 |
|
| 124 |
- **Qwen3.6 35B-A3B を少ないメモリで(0.9.0)**:エキスパートを SSD に置いたまま、ルーターが選んだものだけを読み込み、最近使ったものをメモリに残せるようになりました(onw と同じ方式。OS のファイルキャッシュを通さない直接読み込みで、次の層で使いそうなものは先読みします)。設定の「MoE のエキスパートのメモリ」で量を選べます(既定の「自動」は、モデル全体がメモリの 7 割に収まればすべてメモリに置きます)。4 GB にすると使用メモリは約 20 GB → 約 7 GB、生成は約 20 → 約 15 tok/s、プロンプトの読み込みは 1,000 トークンで 3.1 → 3.5 秒、4,000 トークンで 9.3 → 10.2 秒です(Ryzen AI Max+ 395 の内蔵 NVMe)。0.9.0 では画像も読めるようになり(画像のエンコーダーは CPU。画像を含むプロンプトは今のところ 1 トークンずつ読みます)、MTP も速くなりました(約 22 → 約 23 tok/s)。モデルの更新が必要です(変わったファイルだけをダウンロードします)。
|
| 125 |
- **Qwen3.6 35B-A3B(0.8.0 で追加)**:256 個のエキスパート(1 トークンに 8 個+共有の 1 個)を持つ MoE で、全体は 35B、1 トークンの計算は 3B 分です。重みは int4 で、エキスパートを選ぶルーターも含めて全部 NPU で動かします(ルーターは NPU が int4 で 16 候補まで絞り、CPU が上位 8 個を正確に選び直します)。生成は約 20 tok/s、温度 0 なら MTP で約 22 tok/s です。プロンプトは 1,024 / 2,048 トークンずつ NPU でまとめて読み、選ばれたエキスパートごとにトークンを集めて int4 の重みのまま行列計算します(1,000 トークンで約 3.1 秒、8,000 トークンで約 19 秒、32,000 トークンで約 93 秒)。同じ PC の FastFlowLM 1.0.8 と比べると、プロンプトの読み込み(TTFT)は 1K で 10.2 → 3.1 秒、4K で 18.4 → 9.3 秒、8K で 27.7 → 19.2 秒、32K で 97.9 → 92.7 秒、生成は 15.6 → 20.7 tok/s です。すべてメモリに置くと約 20 GB 使います(Ryzen AI Max+ 395 / 128 GB で確認。HX 370 / 32 GB では未確認)。読み込みの合間の処理はいつも CPU で行います(AVX-512 が必要です)。
|
|
@@ -128,6 +129,7 @@ onwx chat ryugyosoft/Ternary-Bonsai-2-27B-onwx # 端末でチャット
|
|
| 128 |
|
| 129 |
- プロンプトは NPU でまとめて読み込みます。256 トークン以上(llm-jp は 32 トークン以上)は 1,024 / 2,048 トークンずつ、NPU の 32 個の AI エンジン全部で 1 層ずつ計算します(行列計算は bfp16 の行列ユニット。重みはその層の直前に NPU 上で展開するので、使用メモリの増加は作業領域の約 0.5 GB だけです)。短いプロンプトは従来どおり 128 / 32 トークンずつの経路で読みます。
|
| 130 |
- 設定の「CPU も使ってプロンプトの読み込みを速くする(消費電力が増えます)」をオンにすると、NPU の計算の合間の小さな処理を CPU(省電力側のコア)で行い、1K トークン以上のプロンプトの読み込みが 1〜2 割速くなります(Qwen3.5 9B、4K トークンで 7.7 → 6.6 秒)。読み込み中の消費電力は数 W 増えます。長いプロンプトほど効きます。既定はオフです。
|
|
|
|
| 131 |
- アテンションのキャッシュ(キーと値)は int8 で持つので、文脈 1k トークンあたりのメモリは 0.3.0 の半分です。
|
| 132 |
- Qwen3.5 9B と Qwen3.6 35B-A3B は温度 0(API の既定)のとき、モデルの MTP 層で次のトークンを先読みして速く生成します。出力は 1 トークンずつ生成したときと同じです(Qwen3.6 はエキスパートの足し算の順序が違うため、ほぼ同点の候補のところで分かれることがあります。チャット画面の既定のサンプリングでは使いません。`ONWX_MTP=0` で止められます)。文脈が 24K トークン(Qwen3.6 は 8K)を超えると、先読みの検証のほうが重くなるので自動で使わなくなります(`ONWX_MTP_MAX` で変更)。Qwen3.6 は連続する 2 トークンのエキスパートがほとんど重ならないため検証が重く、効果は平均で約 1 割(コードの生成で約 15%)です。先読みが当たりにくい応答では、途中から自動で 1 トークンずつの生成に戻ります。
|
| 133 |
- 文脈は最大 262144 トークン(モデルの上限。llm-jp は 65536)です。Qwen3.5 9B / 4B は 32K を超える長いプロンプトも NPU の新しい経路で読みます(9B、64K トークンで約 340 tok/s)。既定は 3840 トークンで、設定で変えられます。文脈が長いほど遅くなります。
|
|
|
|
| 118 |
| [Qwen3.5 9B](https://huggingface.co/ryugyosoft/Qwen3.5-9B-onwx)(int4、画像も読める) | 7.6 GB | 約 5.8 GB + 文脈 1k トークンあたり約 15 MB | 約 2.5 秒 / 約 7.7 秒(CPU も使う設定で約 2.1 秒 / 約 6.6 秒) | 約 12 tok/s、温度 0 なら MTP で約 15 tok/s |
|
| 119 |
| [Qwen3.5 4B](https://huggingface.co/ryugyosoft/Qwen3.5-4B-onwx)(int4、画像も読める) | 4.8 GB | 約 3.4 GB + 文脈 1k トークンあたり約 13 MB | 約 2.1 秒 / 約 6.4 秒(CPU も使う設定で約 1.8 秒 / 約 5.7 秒) | 約 18 tok/s |
|
| 120 |
| [llm-jp-4.1 8B thinking](https://huggingface.co/ryugyosoft/llm-jp-4.1-8b-onwx)(int4、日本語に強い) | 5.8 GB | 約 4.8 GB + 文脈 1k トークンあたり約 65 MB | 約 6.3 秒 / 約 13 秒 | 約 11 tok/s |
|
| 121 |
+
| [LFM2.5 24B-A2B](https://huggingface.co/ryugyosoft/LFM2.5-24B-A2B-onwx-kit)(Liquid AI の MoE、この PC で変換) | 元のモデル 48 GB(変換後 17 GB) | 約 19 GB + 文脈 1k トークンあたり約 22 MB | 約 2.4 秒 / 約 8.5 秒 | 約 20 tok/s |
|
| 122 |
|
| 123 |
+
(Ryzen AI Max+ 395・Windows で測定。LFM2.5 24B-A2B は onwx 1.0.0、Qwen3.6 35B-A3B は 0.9.0、Ternary Bonsai 2 9B / 4B は 0.7.0、Qwen3.5 9B / 4B は 0.6.0、ほかは 0.5.0。0.6.0 ではプロンプトの読み込み中に NPU の完了を待つ方法を変え、1K トークン以上の読み込みが 0.5.0 より約 1 割速くなりました)
|
| 124 |
|
| 125 |
- **Qwen3.6 35B-A3B を少ないメモリで(0.9.0)**:エキスパートを SSD に置いたまま、ルーターが選んだものだけを読み込み、最近使ったものをメモリに残せるようになりました(onw と同じ方式。OS のファイルキャッシュを通さない直接読み込みで、次の層で使いそうなものは先読みします)。設定の「MoE のエキスパートのメモリ」で量を選べます(既定の「自動」は、モデル全体がメモリの 7 割に収まればすべてメモリに置きます)。4 GB にすると使用メモリは約 20 GB → 約 7 GB、生成は約 20 → 約 15 tok/s、プロンプトの読み込みは 1,000 トークンで 3.1 → 3.5 秒、4,000 トークンで 9.3 → 10.2 秒です(Ryzen AI Max+ 395 の内蔵 NVMe)。0.9.0 では画像も読めるようになり(画像のエンコーダーは CPU。画像を含むプロンプトは今のところ 1 トークンずつ読みます)、MTP も速くなりました(約 22 → 約 23 tok/s)。モデルの更新が必要です(変わったファイルだけをダウンロードします)。
|
| 126 |
- **Qwen3.6 35B-A3B(0.8.0 で追加)**:256 個のエキスパート(1 トークンに 8 個+共有の 1 個)を持つ MoE で、全体は 35B、1 トークンの計算は 3B 分です。重みは int4 で、エキスパートを選ぶルーターも含めて全部 NPU で動かします(ルーターは NPU が int4 で 16 候補まで絞り、CPU が上位 8 個を正確に選び直します)。生成は約 20 tok/s、温度 0 なら MTP で約 22 tok/s です。プロンプトは 1,024 / 2,048 トークンずつ NPU でまとめて読み、選ばれたエキスパートごとにトークンを集めて int4 の重みのまま行列計算します(1,000 トークンで約 3.1 秒、8,000 トークンで約 19 秒、32,000 トークンで約 93 秒)。同じ PC の FastFlowLM 1.0.8 と比べると、プロンプトの読み込み(TTFT)は 1K で 10.2 → 3.1 秒、4K で 18.4 → 9.3 秒、8K で 27.7 → 19.2 秒、32K で 97.9 → 92.7 秒、生成は 15.6 → 20.7 tok/s です。すべてメモリに置くと約 20 GB 使います(Ryzen AI Max+ 395 / 128 GB で確認。HX 370 / 32 GB では未確認)。読み込みの合間の処理はいつも CPU で行います(AVX-512 が必要です)。
|
|
|
|
| 129 |
|
| 130 |
- プロンプトは NPU でまとめて読み込みます。256 トークン以上(llm-jp は 32 トークン以上)は 1,024 / 2,048 トークンずつ、NPU の 32 個の AI エンジン全部で 1 層ずつ計算します(行列計算は bfp16 の行列ユニット。重みはその層の直前に NPU 上で展開するので、使用メモリの増加は作業領域の約 0.5 GB だけです)。短いプロンプトは従来どおり 128 / 32 トークンずつの経路で読みます。
|
| 131 |
- 設定の「CPU も使ってプロンプトの読み込みを速くする(消費電力が増えます)」をオンにすると、NPU の計算の合間の小さな処理を CPU(省電力側のコア)で行い、1K トークン以上のプロンプトの読み込みが 1〜2 割速くなります(Qwen3.5 9B、4K トークンで 7.7 → 6.6 秒)。読み込み中の消費電力は数 W 増えます。長いプロンプトほど効きます。既定はオフです。
|
| 132 |
+
- **LFM2.5 24B-A2B は onwx が重みを配布しません**:モデル一覧から選ぶと、変換用のキット(約 9 MB)と Liquid AI の元のモデル(約 48 GB)を Hugging Face から取ってきて、この PC で int4 に変換します(ダウンロードと変換で約 20〜30 分。途中で約 65 GB のディスクを使い、終わると元のファイルは消え、約 17 GB が残ります。ボタンは「ダウンロードして変換」で、始める前にこれらを確認します)。利用には LFM Open License v1.0 が適用されます。プロンプトは行列の計算を NPU、注意機構などを CPU でまとめて読みます。
|
| 133 |
- アテンションのキャッシュ(キーと値)は int8 で持つので、文脈 1k トークンあたりのメモリは 0.3.0 の半分です。
|
| 134 |
- Qwen3.5 9B と Qwen3.6 35B-A3B は温度 0(API の既定)のとき、モデルの MTP 層で次のトークンを先読みして速く生成します。出力は 1 トークンずつ生成したときと同じです(Qwen3.6 はエキスパートの足し算の順序が違うため、ほぼ同点の候補のところで分かれることがあります。チャット画面の既定のサンプリングでは使いません。`ONWX_MTP=0` で止められます)。文脈が 24K トークン(Qwen3.6 は 8K)を超えると、先読みの検証のほうが重くなるので自動で使わなくなります(`ONWX_MTP_MAX` で変更)。Qwen3.6 は連続する 2 トークンのエキスパートがほとんど重ならないため検証が重く、効果は平均で約 1 割(コードの生成で約 15%)です。先読みが当たりにくい応答では、途中から自動で 1 トークンずつの生成に戻ります。
|
| 135 |
- 文脈は最大 262144 トークン(モデルの上限。llm-jp は 65536)です。Qwen3.5 9B / 4B は 32K を超える長いプロンプトも NPU の新しい経路で読みます(9B、64K トークンで約 340 tok/s)。既定は 3840 トークンで、設定で変えられます。文脈が長いほど遅くなります。
|
README_en.md
CHANGED
|
@@ -101,8 +101,9 @@ onwx chat ryugyosoft/Ternary-Bonsai-2-27B-onwx # chat in the terminal
|
|
| 101 |
| [Qwen3.5 9B](https://huggingface.co/ryugyosoft/Qwen3.5-9B-onwx) (int4, reads images) | 7.6 GB | ~5.8 GB + ~15 MB per 1k tokens of context | ~2.5 s / ~7.7 s (~2.1 s / ~6.6 s with the CPU setting) | ~12 tok/s, ~15 tok/s with MTP at temperature 0 |
|
| 102 |
| [Qwen3.5 4B](https://huggingface.co/ryugyosoft/Qwen3.5-4B-onwx) (int4, reads images) | 4.8 GB | ~3.4 GB + ~13 MB per 1k tokens of context | ~2.1 s / ~6.4 s (~1.8 s / ~5.7 s with the CPU setting) | ~18 tok/s |
|
| 103 |
| [llm-jp-4.1 8B thinking](https://huggingface.co/ryugyosoft/llm-jp-4.1-8b-onwx) (int4, strong in Japanese) | 5.8 GB | ~4.8 GB + ~65 MB per 1k tokens of context | ~6.3 s / ~13 s | ~11 tok/s |
|
|
|
|
| 104 |
|
| 105 |
-
(Measured on a Ryzen AI Max+ 395, Windows: Qwen3.6 35B-A3B with
|
| 106 |
|
| 107 |
- **Qwen3.6 35B-A3B in less memory (0.9.0)**: the experts can stay on the SSD - only the ones the router picks are read, the recently used ones kept in memory (as in onw: unbuffered reads that bypass the OS file cache, the experts the next layer will likely use read ahead). The setting "Memory for MoE experts" picks how much (the default "auto" keeps them all when the whole model fits in 70% of the memory). With 4 GB: memory ~20 GB -> ~7 GB, generation ~20 -> ~15 tok/s, a 1,000-token prompt 3.1 -> 3.5 s, 4,000 tokens 9.3 -> 10.2 s (the Ryzen AI Max+ 395's built-in NVMe). 0.9.0 also reads images (the image encoder on the CPU; prompts with an image are read token by token for now) and has a faster MTP (~22 -> ~23 tok/s). The model needs an update (only the changed files are downloaded).
|
| 108 |
- **Qwen3.6 35B-A3B (new in 0.8.0)**: a mixture of experts with 256 experts (8 a token + a shared one): 35B in all, 3B of compute a token. int4 weights, all on the NPU including the router that picks the experts (the NPU narrows the 256 down to 16 candidates in int4, the CPU picks the top 8 of those exactly). About 20 tok/s generation, about 22 with MTP at temperature 0. Prompts are read on the NPU 1,024 / 2,048 tokens at a time, the tokens gathered per chosen expert and multiplied with its int4 weights as they are (1,000 tokens in ~3.1 s, 8,000 in ~19 s, 32,000 in ~93 s). Against FastFlowLM 1.0.8 on the same PC: prompt reading (TTFT) 1K 10.2 -> 3.1 s, 4K 18.4 -> 9.3 s, 8K 27.7 -> 19.2 s, 32K 97.9 -> 92.7 s; generation 15.6 -> 20.7 tok/s. All in memory it takes about 20 GB (checked on a Ryzen AI Max+ 395 with 128 GB; not tried on an HX 370 with 32 GB). The small steps between the NPU programs always run on the CPU while reading prompts (needs AVX-512).
|
|
@@ -111,6 +112,7 @@ onwx chat ryugyosoft/Ternary-Bonsai-2-27B-onwx # chat in the terminal
|
|
| 111 |
|
| 112 |
- Prompts are read on the NPU in batches. From 256 tokens on (llm-jp: 32) 1,024 / 2,048 tokens at a time, each layer on all 32 AI Engines of the NPU (bfp16 matrix products; the weights are expanded on the NPU right before their layer, so only a ~0.5 GB work area is added to the memory). Short prompts take the 128 / 32-token paths as before.
|
| 113 |
- The setting "Use the CPU too, to read prompts faster (uses more power)" moves the small steps between the NPU's computations to the CPU (its efficiency cores): prompts of 1K tokens and more read 10-20% faster (Qwen3.5 9B, 4K tokens: 7.7 -> 6.6 s), for a few watts more while reading. The longer the prompt, the more it helps. Off by default.
|
|
|
|
| 114 |
- The attention cache (keys and values) is kept as int8: half the memory per 1k tokens of context of 0.3.0.
|
| 115 |
- At temperature 0 (the API default) Qwen3.5 9B and Qwen3.6 35B-A3B draft the next token with the model's MTP layer and generate faster; the output is the same as one token at a time (Qwen3.6 adds the experts up in another order, so it can part ways where two candidates are all but tied; not used with the chat window's default sampling; `ONWX_MTP=0` turns it off). Past 24K tokens of context (Qwen3.6: 8K) the drafts' check costs more than it saves, so it turns itself off (`ONWX_MTP_MAX` changes that). For Qwen3.6 the check is heavy - two tokens in a row hardly share experts - so it gains about 10% on average (about 15% for code); when the drafts are rarely right, the rest of the answer goes back to one token at a time by itself.
|
| 116 |
- Context: up to 262144 tokens (the model's limit; llm-jp 65536). Qwen3.5 9B / 4B read prompts past 32K on the new NPU path too (9B: ~340 tok/s at 64K tokens). The default is 3840 tokens, changeable in the settings. Longer contexts are slower.
|
|
|
|
| 101 |
| [Qwen3.5 9B](https://huggingface.co/ryugyosoft/Qwen3.5-9B-onwx) (int4, reads images) | 7.6 GB | ~5.8 GB + ~15 MB per 1k tokens of context | ~2.5 s / ~7.7 s (~2.1 s / ~6.6 s with the CPU setting) | ~12 tok/s, ~15 tok/s with MTP at temperature 0 |
|
| 102 |
| [Qwen3.5 4B](https://huggingface.co/ryugyosoft/Qwen3.5-4B-onwx) (int4, reads images) | 4.8 GB | ~3.4 GB + ~13 MB per 1k tokens of context | ~2.1 s / ~6.4 s (~1.8 s / ~5.7 s with the CPU setting) | ~18 tok/s |
|
| 103 |
| [llm-jp-4.1 8B thinking](https://huggingface.co/ryugyosoft/llm-jp-4.1-8b-onwx) (int4, strong in Japanese) | 5.8 GB | ~4.8 GB + ~65 MB per 1k tokens of context | ~6.3 s / ~13 s | ~11 tok/s |
|
| 104 |
+
| [LFM2.5 24B-A2B](https://huggingface.co/ryugyosoft/LFM2.5-24B-A2B-onwx-kit) (Liquid AI's MoE, converted on your PC) | the original 48 GB (17 GB converted) | ~19 GB + ~22 MB per 1k tokens of context | ~2.4 s / ~8.5 s | ~20 tok/s |
|
| 105 |
|
| 106 |
+
(Measured on a Ryzen AI Max+ 395, Windows: LFM2.5 24B-A2B with onwx 1.0.0, Qwen3.6 35B-A3B with 0.9.0, Ternary Bonsai 2 9B / 4B with 0.7.0, Qwen3.5 9B / 4B with 0.6.0, the others with 0.5.0. 0.6.0 waits for the NPU differently while reading prompts: prompts of 1K tokens and more read ~10% faster than with 0.5.0.)
|
| 107 |
|
| 108 |
- **Qwen3.6 35B-A3B in less memory (0.9.0)**: the experts can stay on the SSD - only the ones the router picks are read, the recently used ones kept in memory (as in onw: unbuffered reads that bypass the OS file cache, the experts the next layer will likely use read ahead). The setting "Memory for MoE experts" picks how much (the default "auto" keeps them all when the whole model fits in 70% of the memory). With 4 GB: memory ~20 GB -> ~7 GB, generation ~20 -> ~15 tok/s, a 1,000-token prompt 3.1 -> 3.5 s, 4,000 tokens 9.3 -> 10.2 s (the Ryzen AI Max+ 395's built-in NVMe). 0.9.0 also reads images (the image encoder on the CPU; prompts with an image are read token by token for now) and has a faster MTP (~22 -> ~23 tok/s). The model needs an update (only the changed files are downloaded).
|
| 109 |
- **Qwen3.6 35B-A3B (new in 0.8.0)**: a mixture of experts with 256 experts (8 a token + a shared one): 35B in all, 3B of compute a token. int4 weights, all on the NPU including the router that picks the experts (the NPU narrows the 256 down to 16 candidates in int4, the CPU picks the top 8 of those exactly). About 20 tok/s generation, about 22 with MTP at temperature 0. Prompts are read on the NPU 1,024 / 2,048 tokens at a time, the tokens gathered per chosen expert and multiplied with its int4 weights as they are (1,000 tokens in ~3.1 s, 8,000 in ~19 s, 32,000 in ~93 s). Against FastFlowLM 1.0.8 on the same PC: prompt reading (TTFT) 1K 10.2 -> 3.1 s, 4K 18.4 -> 9.3 s, 8K 27.7 -> 19.2 s, 32K 97.9 -> 92.7 s; generation 15.6 -> 20.7 tok/s. All in memory it takes about 20 GB (checked on a Ryzen AI Max+ 395 with 128 GB; not tried on an HX 370 with 32 GB). The small steps between the NPU programs always run on the CPU while reading prompts (needs AVX-512).
|
|
|
|
| 112 |
|
| 113 |
- Prompts are read on the NPU in batches. From 256 tokens on (llm-jp: 32) 1,024 / 2,048 tokens at a time, each layer on all 32 AI Engines of the NPU (bfp16 matrix products; the weights are expanded on the NPU right before their layer, so only a ~0.5 GB work area is added to the memory). Short prompts take the 128 / 32-token paths as before.
|
| 114 |
- The setting "Use the CPU too, to read prompts faster (uses more power)" moves the small steps between the NPU's computations to the CPU (its efficiency cores): prompts of 1K tokens and more read 10-20% faster (Qwen3.5 9B, 4K tokens: 7.7 -> 6.6 s), for a few watts more while reading. The longer the prompt, the more it helps. Off by default.
|
| 115 |
+
- **onwx does not distribute LFM2.5 24B-A2B's weights**: picked from the model list, onwx fetches the conversion kit (~9 MB) and Liquid AI's original model (~48 GB) from Hugging Face and converts it to int4 on your PC (download and conversion ~20-30 minutes; ~65 GB of disk on the way, the original files deleted at the end, ~17 GB stays; the button reads "Download and convert" and asks first). The LFM Open License v1.0 applies. Its prompts are read in chunks: the matrix products on the NPU, the attention and the rest on the CPU.
|
| 116 |
- The attention cache (keys and values) is kept as int8: half the memory per 1k tokens of context of 0.3.0.
|
| 117 |
- At temperature 0 (the API default) Qwen3.5 9B and Qwen3.6 35B-A3B draft the next token with the model's MTP layer and generate faster; the output is the same as one token at a time (Qwen3.6 adds the experts up in another order, so it can part ways where two candidates are all but tied; not used with the chat window's default sampling; `ONWX_MTP=0` turns it off). Past 24K tokens of context (Qwen3.6: 8K) the drafts' check costs more than it saves, so it turns itself off (`ONWX_MTP_MAX` changes that). For Qwen3.6 the check is heavy - two tokens in a row hardly share experts - so it gains about 10% on average (about 15% for code); when the drafts are rarely right, the rest of the answer goes back to one token at a time by itself.
|
| 118 |
- Context: up to 262144 tokens (the model's limit; llm-jp 65536). Qwen3.5 9B / 4B read prompts past 32K on the new NPU path too (9B: ~340 tok/s at 64K tokens). The default is 3840 tokens, changeable in the settings. Longer contexts are slower.
|
onwx/__init__.py
CHANGED
|
@@ -2,6 +2,6 @@
|
|
| 2 |
|
| 3 |
(A working name: the product's name is set here and in the install scripts.)
|
| 4 |
"""
|
| 5 |
-
__version__ = "0.
|
| 6 |
NAME = "onwx"
|
| 7 |
TITLE = "onwx (AMD Ryzen AI NPU)"
|
|
|
|
| 2 |
|
| 3 |
(A working name: the product's name is set here and in the install scripts.)
|
| 4 |
"""
|
| 5 |
+
__version__ = "1.0.0"
|
| 6 |
NAME = "onwx"
|
| 7 |
TITLE = "onwx (AMD Ryzen AI NPU)"
|
onwx/app.py
CHANGED
|
@@ -21,10 +21,16 @@ CATALOG = [ # repo, title, takes images, download GB, decode speed (Ryzen AI 9
|
|
| 21 |
"三値(2 ビット)の 9B を、全部 NPU で動かします(Qwen3.5-9B ベースの Ternary Bonsai 2。画像も読めます)。Ryzen AI Max+ 395 で生成は約 20 tok/s、プロンプトの読み込みは 2 万トークンで約 40 秒(約 500 tok/s)。文脈は最大 262144 トークン"),
|
| 22 |
("ryugyosoft/Ternary-Bonsai-2-4B-onwx", "Ternary Bonsai 2 4B (2-bit, images)", True, 1.6, "30 tok/s",
|
| 23 |
"三値(2 ビット)の軽い 4B を、全部 NPU で動かします(Qwen3.5-4B ベースの Ternary Bonsai 2。画像も読めます)。Ryzen AI Max+ 395 で生成は約 30 tok/s、プロンプトの読み込みは 2 万トークンで約 34 秒(約 600 tok/s)。文脈は最大 262144 トークン"),
|
|
|
|
|
|
|
| 24 |
("ryugyosoft/llm-jp-4.1-8b-onwx", "llm-jp-4.1 8B thinking (int4)", False, 5.8, "11 tok/s",
|
| 25 |
"日本語に強い 8B を int4 の重みで、全部 NPU で動かします(国立情報学研究所の llm-jp-4.1-8b-thinking。考えてから答えるモデルです。生成は Ryzen AI 9 HX 370 で約 11 tok/s、1,000 トークン以上のプロンプトの読み込みは Ryzen AI Max+ 395 で約 300〜430 tok/s)。文脈は最大 65536 トークン"),
|
| 26 |
]
|
| 27 |
CATALOG_REPOS = {c[0] for c in CATALOG}
|
|
|
|
|
|
|
|
|
|
|
|
|
| 28 |
|
| 29 |
|
| 30 |
def catalog():
|
|
@@ -44,6 +50,9 @@ MEMORY = {
|
|
| 44 |
"ryugyosoft/llm-jp-4.1-8b-onwx": (4.8, 0.0, 65), # (4.19 GB at 1024, 5.13 at 8192: 32 layers x 8 kv heads)
|
| 45 |
"ryugyosoft/Ternary-Bonsai-2-9B-onwx": (2.9, 0.0, 15), # [0.7] (ternary weights 2.4 GB; the vision tower +1.7 GB)
|
| 46 |
"ryugyosoft/Ternary-Bonsai-2-4B-onwx": (2.0, 0.0, 13), # [0.7] (1.6 GB; vision +1.3 GB)
|
|
|
|
|
|
|
|
|
|
| 47 |
"ryugyosoft/Qwen3.6-35B-A3B-onwx": (20.2, 0.0, 24), # [0.8] (19.1 GB measured at 4096 with MTP, + the prompt
|
| 48 |
# programs' buffers ~1 GB; 10 attention layers + MTP's;
|
| 49 |
# [0.9] the vision tower +1.8 GB when an image comes; with
|
|
@@ -462,6 +471,7 @@ PKG_REV_OF.update({r: 4 for r in ("ryugyosoft/Ternary-Bonsai-2-27B-onwx", "ryugy
|
|
| 462 |
PKG_REV_OF.update({r: 1 for r in ("ryugyosoft/Ternary-Bonsai-2-9B-onwx", "ryugyosoft/Ternary-Bonsai-2-4B-onwx")})
|
| 463 |
# [0.7] Ternary Bonsai 2 9B / 4B: new (rev 1)
|
| 464 |
PKG_REV_OF["ryugyosoft/Qwen3.6-35B-A3B-onwx"] = 2
|
|
|
|
| 465 |
# [0.8] Qwen3.6-35B-A3B: new (rev 1); [0.9] rev 2: the image encoder (vision.npz), the faster MTP check (ver.xclbin)
|
| 466 |
|
| 467 |
|
|
|
|
| 21 |
"三値(2 ビット)の 9B を、全部 NPU で動かします(Qwen3.5-9B ベースの Ternary Bonsai 2。画像も読めます)。Ryzen AI Max+ 395 で生成は約 20 tok/s、プロンプトの読み込みは 2 万トークンで約 40 秒(約 500 tok/s)。文脈は最大 262144 トークン"),
|
| 22 |
("ryugyosoft/Ternary-Bonsai-2-4B-onwx", "Ternary Bonsai 2 4B (2-bit, images)", True, 1.6, "30 tok/s",
|
| 23 |
"三値(2 ビット)の軽い 4B を、全部 NPU で動かします(Qwen3.5-4B ベースの Ternary Bonsai 2。画像も読めます)。Ryzen AI Max+ 395 で生成は約 30 tok/s、プロンプトの読み込みは 2 万トークンで約 34 秒(約 600 tok/s)。文脈は最大 262144 トークン"),
|
| 24 |
+
("ryugyosoft/LFM2.5-24B-A2B-onwx-kit", "LFM2.5 24B-A2B (MoE, converted on this PC)", False, 65.0, "20 tok/s",
|
| 25 |
+
"Liquid AI の MoE(24B、1 トークンあたり 2.4B。64 エキスパートから 4 つ)を、この PC で変換して全部 NPU で動かします(LFM2.5-24B-A2B)。onwx は Liquid AI の重みを配布しません:元のモデル(約 48 GB)を Hugging Face から取ってきて、ここで int4 の重みに変換します(ダウンロードと変換で約 20〜30 分。変換中は約 65 GB の空きが要り、済むと元のファイルは消えて約 17 GB が残ります。使うには LFM Open License v1.0 に従ってください)。精度を保つため、エキスパート以外と、エキスパートの down は int4 を 2 枚重ねにしています(fp32 と比べた KL 0.10)。Ryzen AI Max+ 395 で生成は約 20 tok/s、プロンプトの読み込みは 1,000 トークンで約 2.4 秒、4,000 トークンで約 8.5 秒(行列は NPU、注意機構などは CPU)。メモリは約 19 GB"),
|
| 26 |
("ryugyosoft/llm-jp-4.1-8b-onwx", "llm-jp-4.1 8B thinking (int4)", False, 5.8, "11 tok/s",
|
| 27 |
"日本語に強い 8B を int4 の重みで、全部 NPU で動かします(国立情報学研究所の llm-jp-4.1-8b-thinking。考えてから答えるモデルです。生成は Ryzen AI 9 HX 370 で約 11 tok/s、1,000 トークン以上のプロンプトの読み込みは Ryzen AI Max+ 395 で約 300〜430 tok/s)。文脈は最大 65536 トークン"),
|
| 28 |
]
|
| 29 |
CATALOG_REPOS = {c[0] for c in CATALOG}
|
| 30 |
+
# [1.0] models converted on this PC (kit.py): the original model on Hugging Face, and the disk the build takes (the
|
| 31 |
+
# original's download + the converted weights; the original goes at the end)
|
| 32 |
+
KITS = {"ryugyosoft/LFM2.5-24B-A2B-onwx-kit": {"source": "LiquidAI/LFM2.5-24B-A2B", "disk_gb": 65.0, "source_gb": 48.0,
|
| 33 |
+
"final_gb": 17.0, "minutes": "20-30"}}
|
| 34 |
|
| 35 |
|
| 36 |
def catalog():
|
|
|
|
| 50 |
"ryugyosoft/llm-jp-4.1-8b-onwx": (4.8, 0.0, 65), # (4.19 GB at 1024, 5.13 at 8192: 32 layers x 8 kv heads)
|
| 51 |
"ryugyosoft/Ternary-Bonsai-2-9B-onwx": (2.9, 0.0, 15), # [0.7] (ternary weights 2.4 GB; the vision tower +1.7 GB)
|
| 52 |
"ryugyosoft/Ternary-Bonsai-2-4B-onwx": (2.0, 0.0, 13), # [0.7] (1.6 GB; vision +1.3 GB)
|
| 53 |
+
"ryugyosoft/LFM2.5-24B-A2B-onwx-kit": (18.6, 0.0, 22), # [1.0] (17 GB of blobs, the balanced variant, + the
|
| 54 |
+
# prompt's buffers ~1 GB (18.1-18.5 GB measured); 10
|
| 55 |
+
# attention layers x 4 cores x 544 B a position)
|
| 56 |
"ryugyosoft/Qwen3.6-35B-A3B-onwx": (20.2, 0.0, 24), # [0.8] (19.1 GB measured at 4096 with MTP, + the prompt
|
| 57 |
# programs' buffers ~1 GB; 10 attention layers + MTP's;
|
| 58 |
# [0.9] the vision tower +1.8 GB when an image comes; with
|
|
|
|
| 471 |
PKG_REV_OF.update({r: 1 for r in ("ryugyosoft/Ternary-Bonsai-2-9B-onwx", "ryugyosoft/Ternary-Bonsai-2-4B-onwx")})
|
| 472 |
# [0.7] Ternary Bonsai 2 9B / 4B: new (rev 1)
|
| 473 |
PKG_REV_OF["ryugyosoft/Qwen3.6-35B-A3B-onwx"] = 2
|
| 474 |
+
PKG_REV_OF["ryugyosoft/LFM2.5-24B-A2B-onwx-kit"] = 1 # [1.0] (a kit: an update would fetch and convert the 48 GB again)
|
| 475 |
# [0.8] Qwen3.6-35B-A3B: new (rev 1); [0.9] rev 2: the image encoder (vision.npz), the faster MTP check (ver.xclbin)
|
| 476 |
|
| 477 |
|
onwx/bin/lfx_glue.dll
ADDED
|
Binary file (60.9 kB). View file
|
|
|
onwx/bin/liblfx_glue.so
ADDED
|
Binary file (90 kB). View file
|
|
|
onwx/dl.py
CHANGED
|
@@ -21,6 +21,10 @@ def main():
|
|
| 21 |
from .hub import lfs_pointers
|
| 22 |
if lfs_pointers(dest):
|
| 23 |
sys.exit("download incomplete (pointer files left)")
|
|
|
|
|
|
|
|
|
|
|
|
|
| 24 |
open(os.path.join(dest, ".complete"), "w").write("ok\n")
|
| 25 |
os.remove(os.path.join(dest, ".downloading"))
|
| 26 |
|
|
|
|
| 21 |
from .hub import lfs_pointers
|
| 22 |
if lfs_pointers(dest):
|
| 23 |
sys.exit("download incomplete (pointer files left)")
|
| 24 |
+
if os.path.exists(os.path.join(dest, "kit.json")): # [1.0] a kit: the original model, converted here
|
| 25 |
+
import json
|
| 26 |
+
from .kit import build
|
| 27 |
+
build(dest, json.load(open(os.path.join(dest, "kit.json"))), lambda s: print(s, flush=True))
|
| 28 |
open(os.path.join(dest, ".complete"), "w").write("ok\n")
|
| 29 |
os.remove(os.path.join(dest, ".downloading"))
|
| 30 |
|
onwx/i18n.json
CHANGED
|
@@ -370,7 +370,11 @@
|
|
| 370 |
"MoE のモデル(Qwen3.6 35B-A3B)で、エキスパートの重みをどれだけメモリに置くか。足りない分は SSD から読みます。自動:モデル全体がメモリの 7 割に収まればすべてメモリに、収まらなければ残りの分だけ。少なくすると使用メモリが減り、生成が少し遅くなります(4 GB で約 7 GB・約 15 tok/s、すべてなら約 20 GB・約 20 tok/s)。": "For mixture-of-experts models (Qwen3.6 35B-A3B): how much of the experts' weights stays in memory; the rest is read from the SSD. Auto: all of them when the whole model fits in 70% of the memory, else what is left. Less memory, a little slower generation (4 GB: ~7 GB in all, ~15 tok/s; all: ~20 GB, ~20 tok/s).",
|
| 371 |
"自動": "Auto",
|
| 372 |
"すべてメモリに": "All in memory",
|
| 373 |
-
"256 エキスパートの MoE(35B、1 トークンあたり 3B)を int4 の重みで、全部 NPU で動かします(Qwen3.6-35B-A3B。画像も読めます)。Ryzen AI Max+ 395 で生成は約 20 tok/s(温度 0 なら MTP で約 23 tok/s)、プロンプトの読み込みは 1,000 トークンで約 3 秒、8,000 トークンで約 20 秒。すべてメモリに置くと約 20 GB、メモリが足りなければエキスパートを SSD から読みます(4 GB で計約 7 GB、生成は約 15 tok/s)。文脈は最大 262144 トークン": "A 35B mixture of experts (256 experts, 3B active per token) with int4 weights, all on the NPU (Qwen3.6-35B-A3B; reads images too). On a Ryzen AI Max+ 395: about 20 tok/s generation (about 23 with MTP at temperature 0), a 1,000-token prompt read in about 3 s, 8,000 tokens in about 20 s. All in memory it takes about 20 GB; with too little memory the experts are read from the SSD (4 GB of them: about 7 GB in all, about 15 tok/s). Context up to 262144 tokens"
|
|
|
|
|
|
|
|
|
|
|
|
|
| 374 |
},
|
| 375 |
"zh": {
|
| 376 |
"三値(2 ビット)の 9B を、全部 NPU で動かします(Qwen3.5-9B ベースの Ternary Bonsai 2。画像も読めます)。Ryzen AI Max+ 395 で生成は約 20 tok/s、プロンプトの読み込みは 2 万トークンで約 40 秒(約 500 tok/s)。文脈は最大 262144 トークン": "三值(2 位)9B,全部在 NPU 上运行(基于 Qwen3.5-9B 的 Ternary Bonsai 2,也能读图)。Ryzen AI Max+ 395 上生成约 20 tok/s,提示词读取约 500 tok/s(2 万词元约 40 秒)。上下文最多 262144 词元",
|
|
@@ -743,6 +747,10 @@
|
|
| 743 |
"MoE のモデル(Qwen3.6 35B-A3B)で、エキスパートの重みをどれだけメモリに置くか。足りない分は SSD から読みます。自動:モデル全体がメモリの 7 割に収まればすべてメモリに、収まらなければ残りの分だけ。少なくすると使用メモリが減り、生成が少し遅くなります(4 GB で約 7 GB・約 15 tok/s、すべてなら約 20 GB・約 20 tok/s)。": "用于混合专家模型(Qwen3.6 35B-A3B):专家权重在内存中保留多少,其余从 SSD 读取。自动:整个模型能放进内存的 70% 时全部放入内存,否则只放剩余的部分。内存越少,生成稍慢(4 GB:总共约 7 GB、约 15 tok/s;全部:约 20 GB、约 20 tok/s)。",
|
| 744 |
"自動": "自动",
|
| 745 |
"すべてメモリに": "全部放入内存",
|
| 746 |
-
"256 エキスパートの MoE(35B、1 トークンあたり 3B)を int4 の重みで、全部 NPU で動かします(Qwen3.6-35B-A3B。画像も読めます)。Ryzen AI Max+ 395 で生成は約 20 tok/s(温度 0 なら MTP で約 23 tok/s)、プロンプトの読み込みは 1,000 トークンで約 3 秒、8,000 トークンで約 20 秒。すべてメモリに置くと約 20 GB、メモリが足りなければエキスパートを SSD から読みます(4 GB で計約 7 GB、生成は約 15 tok/s)。文脈は最大 262144 トークン": "256 个专家的混合专家模型(35B,每个 token 激活 3B),int4 权重,全部在 NPU 上运行(Qwen3.6-35B-A3B,也能读图像)。在 Ryzen AI Max+ 395 上生成约 20 tok/s(温度为 0 时用 MTP 约 23 tok/s),1,000 个 token 的提示约 3 秒读完,8,000 个约 20 秒。全部放入内存约占 20 GB;内存不足时从 SSD 读取专家(专家 4 GB:总共约 7 GB,生成约 15 tok/s)。上下文最长 262144 个 token"
|
|
|
|
|
|
|
|
|
|
|
|
|
| 747 |
}
|
| 748 |
}
|
|
|
|
| 370 |
"MoE のモデル(Qwen3.6 35B-A3B)で、エキスパートの重みをどれだけメモリに置くか。足りない分は SSD から読みます。自動:モデル全体がメモリの 7 割に収まればすべてメモリに、収まらなければ残りの分だけ。少なくすると使用メモリが減り、生成が少し遅くなります(4 GB で約 7 GB・約 15 tok/s、すべてなら約 20 GB・約 20 tok/s)。": "For mixture-of-experts models (Qwen3.6 35B-A3B): how much of the experts' weights stays in memory; the rest is read from the SSD. Auto: all of them when the whole model fits in 70% of the memory, else what is left. Less memory, a little slower generation (4 GB: ~7 GB in all, ~15 tok/s; all: ~20 GB, ~20 tok/s).",
|
| 371 |
"自動": "Auto",
|
| 372 |
"すべてメモリに": "All in memory",
|
| 373 |
+
"256 エキスパートの MoE(35B、1 トークンあたり 3B)を int4 の重みで、全部 NPU で動かします(Qwen3.6-35B-A3B。画像も読めます)。Ryzen AI Max+ 395 で生成は約 20 tok/s(温度 0 なら MTP で約 23 tok/s)、プロンプトの読み込みは 1,000 トークンで約 3 秒、8,000 トークンで約 20 秒。すべてメモリに置くと約 20 GB、メモリが足りなければエキスパートを SSD から読みます(4 GB で計約 7 GB、生成は約 15 tok/s)。文脈は最大 262144 トークン": "A 35B mixture of experts (256 experts, 3B active per token) with int4 weights, all on the NPU (Qwen3.6-35B-A3B; reads images too). On a Ryzen AI Max+ 395: about 20 tok/s generation (about 23 with MTP at temperature 0), a 1,000-token prompt read in about 3 s, 8,000 tokens in about 20 s. All in memory it takes about 20 GB; with too little memory the experts are read from the SSD (4 GB of them: about 7 GB in all, about 15 tok/s). Context up to 262144 tokens",
|
| 374 |
+
"{0} GB(変換中は {1} GB)": "{0} GB ({1} GB while converting)",
|
| 375 |
+
"ダウンロードして変換": "Download and convert",
|
| 376 |
+
"このモデルはこの PC で変換して使います。\n・元のモデル(約 {0} GB)を Hugging Face からダウンロードします\n・変換中は約 {1} GB の空きが必要です(終わると元のファイルは消え、約 {2} GB が残ります)\n・かかる時間は約 {3} 分です(回線と PC による)\n・元のモデルのライセンスに従ってお使いください\n始めますか?": "This model is converted on this PC.\n- The original model (~{0} GB) is downloaded from Hugging Face\n- Converting needs ~{1} GB of free disk (the original files are deleted at the end, ~{2} GB stays)\n- It takes ~{3} minutes (depends on the line and the PC)\n- Use it under the original model's license\nStart?",
|
| 377 |
+
"Liquid AI の MoE(24B、1 トークンあたり 2.4B。64 エキスパートから 4 つ)を、この PC で変換して全部 NPU で動かします(LFM2.5-24B-A2B)。onwx は Liquid AI の重みを配布しません:元のモデル(約 48 GB)を Hugging Face から取ってきて、ここで int4 の重みに変換します(ダウンロードと変換で約 20〜30 分。変換中は約 65 GB の空きが要り、済むと元のファイルは消えて約 17 GB が残ります。使うには LFM Open License v1.0 に従ってください)。精度を保つため、エキスパート以外と、エキスパートの down は int4 を 2 枚重ねにしています(fp32 と比べた KL 0.10)。Ryzen AI Max+ 395 で生成は約 20 tok/s、プロンプトの読み込みは 1,000 トークンで約 2.4 秒、4,000 トークンで約 8.5 秒(行列は NPU、注意機構などは CPU)。メモリは約 19 GB": "Liquid AI's MoE (24B, 2.4B a token: 4 of 64 experts), converted on this PC and run all on the NPU (LFM2.5-24B-A2B). onwx does not distribute Liquid AI's weights: it downloads the original model (~48 GB) from Hugging Face and converts it here to int4 weights (download and conversion ~20-30 minutes; ~65 GB of free disk while converting, the original files deleted after, ~17 GB stays; use it under the LFM Open License v1.0). For accuracy, everything but the experts, and the experts' down projections, are two stacked int4 blocks (KL 0.10 against fp32). Ryzen AI Max+ 395: ~20 tok/s generation, a 1,000-token prompt read in ~2.4 s, 4,000 tokens in ~8.5 s (the matrices on the NPU, attention and the rest on the CPU). Memory ~19 GB"
|
| 378 |
},
|
| 379 |
"zh": {
|
| 380 |
"三値(2 ビット)の 9B を、全部 NPU で動かします(Qwen3.5-9B ベースの Ternary Bonsai 2。画像も読めます)。Ryzen AI Max+ 395 で生成は約 20 tok/s、プロンプトの読み込みは 2 万トークンで約 40 秒(約 500 tok/s)。文脈は最大 262144 トークン": "三值(2 位)9B,全部在 NPU 上运行(基于 Qwen3.5-9B 的 Ternary Bonsai 2,也能读图)。Ryzen AI Max+ 395 上生成约 20 tok/s,提示词读取约 500 tok/s(2 万词元约 40 秒)。上下文最多 262144 词元",
|
|
|
|
| 747 |
"MoE のモデル(Qwen3.6 35B-A3B)で、エキスパートの重みをどれだけメモリに置くか。足りない分は SSD から読みます。自動:モデル全体がメモリの 7 割に収まればすべてメモリに、収まらなければ残りの分だけ。少なくすると使用メモリが減り、生成が少し遅くなります(4 GB で約 7 GB・約 15 tok/s、すべてなら約 20 GB・約 20 tok/s)。": "用于混合专家模型(Qwen3.6 35B-A3B):专家权重在内存中保留多少,其余从 SSD 读取。自动:整个模型能放进内存的 70% 时全部放入内存,否则只放剩余的部分。内存越少,生成稍慢(4 GB:总共约 7 GB、约 15 tok/s;全部:约 20 GB、约 20 tok/s)。",
|
| 748 |
"自動": "自动",
|
| 749 |
"すべてメモリに": "全部放入内存",
|
| 750 |
+
"256 エキスパートの MoE(35B、1 トークンあたり 3B)を int4 の重みで、全部 NPU で動かします(Qwen3.6-35B-A3B。画像も読めます)。Ryzen AI Max+ 395 で生成は約 20 tok/s(温度 0 なら MTP で約 23 tok/s)、プロンプトの読み込みは 1,000 トークンで約 3 秒、8,000 トークンで約 20 秒。すべてメモリに置くと約 20 GB、メモリが足りなければエキスパートを SSD から読みます(4 GB で計約 7 GB、生成は約 15 tok/s)。文脈は最大 262144 トークン": "256 个专家的混合专家模型(35B,每个 token 激活 3B),int4 权重,全部在 NPU 上运行(Qwen3.6-35B-A3B,也能读图像)。在 Ryzen AI Max+ 395 上生成约 20 tok/s(温度为 0 时用 MTP 约 23 tok/s),1,000 个 token 的提示约 3 秒读完,8,000 个约 20 秒。全部放入内存约占 20 GB;内存不足时从 SSD 读取专家(专家 4 GB:总共约 7 GB,生成约 15 tok/s)。上下文最长 262144 个 token",
|
| 751 |
+
"{0} GB(変換中は {1} GB)": "{0} GB(转换时 {1} GB)",
|
| 752 |
+
"ダウンロードして変換": "下载并转换",
|
| 753 |
+
"このモデルはこの PC で変換して使います。\n・元のモデル(約 {0} GB)を Hugging Face からダウンロードします\n・変換中は約 {1} GB の空きが必要です(終わると元のファイルは消え、約 {2} GB が残ります)\n・かかる時間は約 {3} 分です(回線と PC による)\n・元のモデルのライセンスに従ってお使いください\n始めますか?": "此模型在本机转换后使用。\n・从 Hugging Face 下载原始模型(约 {0} GB)\n・转换时需要约 {1} GB 可用空间(结束后删除原始文件,保留约 {2} GB)\n・约需 {3} 分钟(取决于网络和电脑)\n・请遵守原始模型的许可证\n开始吗?",
|
| 754 |
+
"Liquid AI の MoE(24B、1 トークンあたり 2.4B。64 エキスパートから 4 つ)を、この PC で変換して全部 NPU で動かします(LFM2.5-24B-A2B)。onwx は Liquid AI の重みを配布しません:元のモデル(約 48 GB)を Hugging Face から取ってきて、ここで int4 の重みに変換します(ダウンロードと変換で約 20〜30 分。変換中は約 65 GB の空きが要り、済むと元のファイルは消えて約 17 GB が残ります。使うには LFM Open License v1.0 に従ってください)。精度を保つため、エキスパート以外と、エキスパートの down は int4 を 2 枚重ねにしています(fp32 と比べた KL 0.10)。Ryzen AI Max+ 395 で生成は約 20 tok/s、プロンプトの読み込みは 1,000 トークンで約 2.4 秒、4,000 トークンで約 8.5 秒(行列は NPU、注意機構などは CPU)。メモリは約 19 GB": "Liquid AI 的 MoE(24B,每个 token 2.4B:64 个专家中选 4 个),在本机转换后全部在 NPU 上运行(LFM2.5-24B-A2B)。onwx 不分发 Liquid AI 的权重:从 Hugging Face 下载原始模型(约 48 GB),在本机转换为 int4 权重(下载和转换约需 20~30 分钟;转换时需要约 65 GB 可用空间,转换后删除原始文件,保留约 17 GB;请遵守 LFM Open License v1.0)。为保持精度,专家以外的部分和专家的 down 使用两层 int4(与 fp32 相比 KL 0.10)。Ryzen AI Max+ 395 上生成约 20 tok/s,1,000 token 的提示词约 2.4 秒读完,4,000 token 约 8.5 秒(矩阵在 NPU 上,注意力等在 CPU 上)。内存约 19 GB"
|
| 755 |
}
|
| 756 |
}
|
onwx/kit.py
ADDED
|
@@ -0,0 +1,79 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""[1.0] Models onwx converts on this PC instead of downloading converted weights (a "kit": kit.json, the converter
|
| 2 |
+
and the NPU programs - none of the publisher's files). dl.py runs build() after downloading the kit into the model
|
| 3 |
+
folder: the publisher's original model into .source (Hugging Face), the converter into blobs/, then the folder made
|
| 4 |
+
complete (plans/, xdna.json with the model's config, its tokenizer files, LICENSE and a NOTICE), .source deleted.
|
| 5 |
+
Progress lines go to stdout (the manager reads the bytes on disk; the log keeps the lines)."""
|
| 6 |
+
import json, os, shutil, subprocess, sys, time
|
| 7 |
+
|
| 8 |
+
COPY = ("config.json", "generation_config.json", "tokenizer.json", "tokenizer_config.json", "special_tokens_map.json",
|
| 9 |
+
"chat_template.jinja", "LICENSE")
|
| 10 |
+
|
| 11 |
+
|
| 12 |
+
def jobs_for(per_job_gb=4.0, rest_gb=4.0):
|
| 13 |
+
"""converter processes: 3 cores each, and what the memory allows (each takes ~4 GB at its peak)"""
|
| 14 |
+
from .xdna_experts import total_ram
|
| 15 |
+
cpu = os.cpu_count() or 4
|
| 16 |
+
ram = total_ram() / 2**30 or 16
|
| 17 |
+
return max(1, min(cpu // 3, int((ram - rest_gb) // per_job_gb), 8))
|
| 18 |
+
|
| 19 |
+
|
| 20 |
+
def build(dest, kit, log=print):
|
| 21 |
+
from huggingface_hub import snapshot_download
|
| 22 |
+
from tqdm.auto import tqdm
|
| 23 |
+
|
| 24 |
+
class Quiet(tqdm):
|
| 25 |
+
def __init__(self, *a, **k):
|
| 26 |
+
k["disable"] = True
|
| 27 |
+
super().__init__(*a, **k)
|
| 28 |
+
variant = os.environ.get("ONWX_KIT_VARIANT", kit["default"])
|
| 29 |
+
v = kit["variants"][variant]
|
| 30 |
+
src = os.path.join(dest, ".source")
|
| 31 |
+
log(f"kit {kit['name']} variant {variant}: downloading {kit['source']}")
|
| 32 |
+
snapshot_download(kit["source"], local_dir=src, tqdm_class=Quiet,
|
| 33 |
+
allow_patterns=["*.safetensors", "*.json", "*.jinja", "LICENSE*"])
|
| 34 |
+
blobs = os.path.join(dest, "blobs")
|
| 35 |
+
t = time.time()
|
| 36 |
+
# (its own process, run as a script: its worker processes import it by its path - a module loaded from a file
|
| 37 |
+
# here could not be pickled to them)
|
| 38 |
+
cmd = [sys.executable, os.path.join(dest, kit["converter"]), src, blobs, "--r2", ",".join(v["r2"]),
|
| 39 |
+
"--jobs", str(int(os.environ.get("ONWX_KIT_JOBS", jobs_for())))]
|
| 40 |
+
p = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True, encoding="utf8",
|
| 41 |
+
errors="replace", env={**os.environ, "PYTHONUTF8": "1", "PYTHONDONTWRITEBYTECODE": "1"},
|
| 42 |
+
**({"creationflags": subprocess.CREATE_NO_WINDOW} if os.name == "nt" else {}))
|
| 43 |
+
for line in p.stdout:
|
| 44 |
+
log(line.rstrip())
|
| 45 |
+
if p.wait() != 0:
|
| 46 |
+
raise RuntimeError(f"the conversion failed (exit {p.returncode})")
|
| 47 |
+
log(f"converted in {time.time() - t:.0f}s")
|
| 48 |
+
vd = os.path.join(dest, v["dir"])
|
| 49 |
+
plans = os.path.join(dest, "plans")
|
| 50 |
+
if os.path.isdir(plans):
|
| 51 |
+
shutil.rmtree(plans)
|
| 52 |
+
shutil.copytree(os.path.join(vd, "plans"), plans)
|
| 53 |
+
shutil.copy(os.path.join(vd, "act.init"), os.path.join(blobs, "act.init"))
|
| 54 |
+
cfg = json.load(open(os.path.join(src, "config.json")))
|
| 55 |
+
man = json.load(open(os.path.join(vd, "xdna.json")))
|
| 56 |
+
man["config"] = cfg
|
| 57 |
+
json.dump(man, open(os.path.join(dest, "xdna.json"), "w"), indent=1)
|
| 58 |
+
eng = {"arch": man["arch"].replace("-w4", "-xdna"), "format": man["format"],
|
| 59 |
+
"config": {**cfg, "max_position_embeddings": man["tmax"]}, "sizes": [1]}
|
| 60 |
+
json.dump(eng, open(os.path.join(dest, "engine.json"), "w"), indent=1)
|
| 61 |
+
os.replace(os.path.join(blobs, "embed.bf16"), os.path.join(dest, "embed.bf16"))
|
| 62 |
+
for f in COPY:
|
| 63 |
+
if os.path.exists(os.path.join(src, f)):
|
| 64 |
+
shutil.copy(os.path.join(src, f), os.path.join(dest, f))
|
| 65 |
+
open(os.path.join(dest, "NOTICE"), "w", encoding="utf8").write(
|
| 66 |
+
f"Converted on this PC by onwx from {kit['source']} ({kit['license']}): its weights rewritten as int4 blocks "
|
| 67 |
+
f"for the NPU (variant {variant}). The original files' terms (LICENSE) apply to this folder.\n")
|
| 68 |
+
if os.environ.get("ONWX_KIT_KEEP_SOURCE") != "1":
|
| 69 |
+
shutil.rmtree(src, ignore_errors=True)
|
| 70 |
+
log("kit done")
|
| 71 |
+
|
| 72 |
+
|
| 73 |
+
def main():
|
| 74 |
+
dest = sys.argv[1]
|
| 75 |
+
build(dest, json.load(open(os.path.join(dest, "kit.json"))), lambda s: print(s, flush=True))
|
| 76 |
+
|
| 77 |
+
|
| 78 |
+
if __name__ == "__main__":
|
| 79 |
+
main()
|
onwx/manager.py
CHANGED
|
@@ -130,6 +130,8 @@ class Manager:
|
|
| 130 |
"deletable": bool(path) and app.downloaded_by_onwx(path), "cache_gb": self._cache_gb(path) if path else 0}
|
| 131 |
if app.mem_saver_on(repo, cfg):
|
| 132 |
m["note"] += T("・メモリ節約モード(回答の出だしは遅め)")
|
|
|
|
|
|
|
| 133 |
m["think"] = app.think_kind(repo, path)
|
| 134 |
m["state"] = self._state_of(repo, path, info, status, dl)
|
| 135 |
if m["state"] == "ready" and app.update_interrupted(path):
|
|
@@ -241,6 +243,8 @@ class Manager:
|
|
| 241 |
try: # (the network call outside the lock: cancel stays instant)
|
| 242 |
from huggingface_hub import HfApi
|
| 243 |
total = _to_fetch(HfApi().model_info(repo, files_metadata=True).siblings, dest) or 1
|
|
|
|
|
|
|
| 244 |
except Exception as e:
|
| 245 |
return {"error": T("Hugging Face に接続できません: {0}", e)}
|
| 246 |
with self.lock:
|
|
|
|
| 130 |
"deletable": bool(path) and app.downloaded_by_onwx(path), "cache_gb": self._cache_gb(path) if path else 0}
|
| 131 |
if app.mem_saver_on(repo, cfg):
|
| 132 |
m["note"] += T("・メモリ節約モード(回答の出だしは遅め)")
|
| 133 |
+
if repo in app.KITS: # [1.0] converted on this PC: the UI shows what it takes
|
| 134 |
+
m["kit"] = dict(app.KITS[repo])
|
| 135 |
m["think"] = app.think_kind(repo, path)
|
| 136 |
m["state"] = self._state_of(repo, path, info, status, dl)
|
| 137 |
if m["state"] == "ready" and app.update_interrupted(path):
|
|
|
|
| 243 |
try: # (the network call outside the lock: cancel stays instant)
|
| 244 |
from huggingface_hub import HfApi
|
| 245 |
total = _to_fetch(HfApi().model_info(repo, files_metadata=True).siblings, dest) or 1
|
| 246 |
+
if repo in app.KITS: # [1.0] a kit: the original model + what the converter writes
|
| 247 |
+
total += int(app.KITS[repo]["disk_gb"] * 2**30)
|
| 248 |
except Exception as e:
|
| 249 |
return {"error": T("Hugging Face に接続できません: {0}", e)}
|
| 250 |
with self.lock:
|
onwx/npu_lfx.py
ADDED
|
@@ -0,0 +1,477 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""[1.0] LFM2.5-24B-A2B's chunked prefill (xdna.json "lfx", plans/lfx/): every matrix product on the NPU, straight from
|
| 2 |
+
the decode int4 blobs (lfx_gemm: 64 tokens x 1024 rows a job, pfx_mx.cc's fused expand-and-multiply), the rest on the
|
| 3 |
+
CPU - the norms, the rotation and the NPU's A tiles (bin/lfx_glue: FWHT, bfp16ebs8), the short conv (its state into
|
| 4 |
+
the decode state image), the attention (int8 records into the decode KV cache, as the decode kernel writes them), the
|
| 5 |
+
router and the experts' sums. A layer, M tokens:
|
| 6 |
+
conv: rms -> in_proj (sc_in) -> B * x, conv over the chunk + state, * C -> out_proj (sc_out) -> + h
|
| 7 |
+
attn: rms -> qkv (at_qkv) -> q / k norms, rope -> cache records; causal attention -> out (at_out) -> + h
|
| 8 |
+
ffn: rms -> dense MLP (mlp_gu0 / gu1, act, mlp_dn (+ dn2)) | router -> per expert its tokens in segments of 64 ->
|
| 9 |
+
e_up -> silu(gate) * up, H_512 -> e_dn (+ e_dn2) -> + weight x y
|
| 10 |
+
Programs: dense ones read 4 segments a dispatch (the X / Y sub-buffers move), the experts' 8 (their W fills pointed
|
| 11 |
+
at the segments' experts by patching the instruction words: <name>_patch.json)."""
|
| 12 |
+
import ctypes as C
|
| 13 |
+
import json, os, time
|
| 14 |
+
import numpy as np
|
| 15 |
+
|
| 16 |
+
BLK, AW, CW = 544, 2304, 1024
|
| 17 |
+
HERE = os.path.dirname(os.path.abspath(__file__))
|
| 18 |
+
|
| 19 |
+
|
| 20 |
+
def _signs(k):
|
| 21 |
+
return np.where(np.random.default_rng(k).random(k) < 0.5, -1.0, 1.0).astype(np.float32)
|
| 22 |
+
|
| 23 |
+
|
| 24 |
+
class Glue:
|
| 25 |
+
def __init__(self):
|
| 26 |
+
lib = os.path.join(HERE, "bin", "lfx_glue.dll" if os.name == "nt" else "liblfx_glue.so")
|
| 27 |
+
x = self.x = C.CDLL(lib)
|
| 28 |
+
vp = C.c_void_p
|
| 29 |
+
x.lx_rot.argtypes = [C.c_int, C.c_int, vp, vp, C.c_int, C.c_float, vp]
|
| 30 |
+
x.lx_pack.argtypes = [C.c_int, C.c_int, vp, vp, C.c_int, C.c_int, vp]
|
| 31 |
+
x.lx_pack2.argtypes = [C.c_int, C.c_int, vp, vp, C.c_int, C.c_int, vp, C.c_int]
|
| 32 |
+
x.lx_threads.argtypes = [C.c_int]
|
| 33 |
+
x.lx_rows.argtypes = [vp, C.c_int, C.c_int, C.c_int, C.c_int, vp]
|
| 34 |
+
x.lx_rows2.argtypes = [vp, C.c_int, C.c_int, C.c_int, C.c_int, vp, vp, C.c_int, C.c_int]
|
| 35 |
+
x.lx_eact.argtypes = [vp, C.c_int, vp]
|
| 36 |
+
x.lx_attn.argtypes = [C.c_int, C.c_int, vp, vp, vp, vp]
|
| 37 |
+
x.lx_scatter.argtypes = [C.c_int, C.c_int, vp, vp, vp, vp]
|
| 38 |
+
x.lx_conv.argtypes = [C.c_int, vp, C.c_int, vp, vp, vp]
|
| 39 |
+
x.lx_rmsrot.argtypes = [C.c_int, C.c_int, vp, vp, C.c_float, vp, C.c_int, C.c_float, vp, vp]
|
| 40 |
+
x.lx_qkrope.argtypes = [C.c_int, vp, vp, vp, C.c_float, C.c_int, vp, vp, vp, vp]
|
| 41 |
+
x.lx_kvput.argtypes = [C.c_int, C.c_int, vp, vp, vp]
|
| 42 |
+
x.lx_kvget.argtypes = [C.c_int, vp, vp, vp]
|
| 43 |
+
x.lx_silumul.argtypes = [C.c_int, C.c_int, vp, C.c_int, vp, C.c_int, vp, C.c_int]
|
| 44 |
+
t = int(os.environ.get("ONWX_LFX_THREADS", "0"))
|
| 45 |
+
if t:
|
| 46 |
+
x.lx_threads(t)
|
| 47 |
+
|
| 48 |
+
def rot(self, x, s, blk=1024, scale=1 / 32, out=None):
|
| 49 |
+
x = np.ascontiguousarray(x, np.float32)
|
| 50 |
+
o = np.empty_like(x) if out is None else out
|
| 51 |
+
assert self.x.lx_rot(x.shape[0], x.shape[1], x.ctypes.data, s.ctypes.data, blk, scale, o.ctypes.data) == 0
|
| 52 |
+
return o
|
| 53 |
+
|
| 54 |
+
def pack(self, x, out, idx=None, n=None, dup=1, hdr=0):
|
| 55 |
+
"""rows of x (or x[idx]) into out (uint8 [segments][(hdr) K dup / 128][9216] flat; hdr: a header tile first,
|
| 56 |
+
word 0 the K-steps - the v2 programs); returns the segments"""
|
| 57 |
+
x = np.ascontiguousarray(x, np.float32)
|
| 58 |
+
n = x.shape[0] if n is None else n
|
| 59 |
+
ip = None
|
| 60 |
+
if idx is not None:
|
| 61 |
+
idx = np.ascontiguousarray(idx, np.int32)
|
| 62 |
+
ip = idx.ctypes.data
|
| 63 |
+
nseg = (n + 63) // 64
|
| 64 |
+
need = nseg * (x.shape[1] * dup // 128 + hdr) * AW * 4
|
| 65 |
+
assert out.nbytes >= need, (out.nbytes, need)
|
| 66 |
+
assert self.x.lx_pack2(x.shape[0], x.shape[1], x.ctypes.data, ip, n, dup, out.ctypes.data, hdr) == 0
|
| 67 |
+
return nseg
|
| 68 |
+
|
| 69 |
+
|
| 70 |
+
def _addr(a, off_elems=0):
|
| 71 |
+
return a.ctypes.data + off_elems * a.itemsize
|
| 72 |
+
|
| 73 |
+
|
| 74 |
+
def bf16(y):
|
| 75 |
+
return (np.asarray(y).view(np.uint16).astype(np.uint32) << 16).view(np.float32)
|
| 76 |
+
|
| 77 |
+
|
| 78 |
+
def hadamard(n):
|
| 79 |
+
h = np.array([[1.0]], np.float32)
|
| 80 |
+
while h.shape[0] < n:
|
| 81 |
+
h = np.block([[h, h], [h, -h]])
|
| 82 |
+
return h
|
| 83 |
+
|
| 84 |
+
|
| 85 |
+
class _Prog:
|
| 86 |
+
def __init__(self, lp, name, ent):
|
| 87 |
+
self.lp, self.name, self.e = lp, name, ent
|
| 88 |
+
d = lp.dir
|
| 89 |
+
self.ctx = lp.ctx(os.path.join(d, ent["xclbin"]))
|
| 90 |
+
a = np.fromfile(os.path.join(d, ent["insts"]), np.uint8)
|
| 91 |
+
self.ibo, m = self.ctx.buf(a.nbytes, 1, cacheable=True)
|
| 92 |
+
self.im = m.view(np.uint8)
|
| 93 |
+
self.im[:a.nbytes] = a
|
| 94 |
+
self.n = a.nbytes
|
| 95 |
+
self.ctx.sync(self.ibo, True)
|
| 96 |
+
self.nseg, self.nk, self.rbn = ent["nseg"], ent["nk"], ent["rbn"]
|
| 97 |
+
self.rmap = ent.get("rmap", "sr")
|
| 98 |
+
self.nq = self.rbn // 2
|
| 99 |
+
self.jobs = [(s, q) for s in range(self.nseg) for q in range(self.nq)]
|
| 100 |
+
self.patch = None
|
| 101 |
+
if "patch" in ent:
|
| 102 |
+
pt = json.load(open(os.path.join(d, ent["patch"])))
|
| 103 |
+
self.step = pt["step"]
|
| 104 |
+
segs = sorted(int(s) for s in pt["segs"])
|
| 105 |
+
self.pos = np.concatenate([np.asarray(pt["segs"][str(s)], np.int64) for s in segs])
|
| 106 |
+
self.seg_of = np.concatenate([np.full(len(pt["segs"][str(s)]), s, np.int64) for s in segs])
|
| 107 |
+
iw = self.im[:self.n].view(np.uint32)
|
| 108 |
+
self.base = iw.astype(np.int64)[self.pos]
|
| 109 |
+
self.patch = True
|
| 110 |
+
self.hdr = int(ent.get("hdr", 0)) # (v2: a header tile before each segment's tiles)
|
| 111 |
+
self.sw = (self.nk + self.hdr) * AW # (a segment's X words)
|
| 112 |
+
self.xw = self.nseg * self.sw # (an X dispatch's words)
|
| 113 |
+
self.yw = len(self.jobs) * 8 * 4 * CW
|
| 114 |
+
|
| 115 |
+
def point(self, experts):
|
| 116 |
+
e = np.zeros(self.nseg, np.int64)
|
| 117 |
+
e[:len(experts)] = experts
|
| 118 |
+
iw = self.im[:self.n].view(np.uint32)
|
| 119 |
+
iw[self.pos] = ((self.base + e[self.seg_of] * self.step) & 0xFFFFFFFF).astype(np.uint32)
|
| 120 |
+
self.ctx.sync(self.ibo, True)
|
| 121 |
+
|
| 122 |
+
def rows(self, Y, nseg, Mc):
|
| 123 |
+
"""Y (the dispatch's words, bf16) -> [nseg * 64, 16 Mc] fp32 in the op's row order"""
|
| 124 |
+
Y = bf16(Y[:len(self.jobs) * 8 * 4 * CW].view(np.uint16)).reshape(self.nseg, self.nq, 8, 4, 64, 32)
|
| 125 |
+
if self.rmap == "sr": # (core g = 2c + side, rb = 2q + r % 2; r = 2 side + rb%2)
|
| 126 |
+
Y = Y.reshape(self.nseg, self.nq, 8, 2, 2, 64, 32).transpose(0, 5, 2, 3, 1, 4, 6) # s, t, c, side, q, rr, 32
|
| 127 |
+
else: # (r = 2 rr + side)
|
| 128 |
+
Y = Y.reshape(self.nseg, self.nq, 8, 2, 2, 64, 32).transpose(0, 5, 2, 4, 1, 3, 6)
|
| 129 |
+
return Y.reshape(self.nseg * 64, 16 * Mc)[:nseg * 64]
|
| 130 |
+
|
| 131 |
+
|
| 132 |
+
class LfxPrefill:
|
| 133 |
+
MIN, ALIGN, tcost, RA = 16, 1, 0, 0
|
| 134 |
+
MS = (2048,)
|
| 135 |
+
CH = int(os.environ.get("ONWX_LFX_CHUNK", "2048")) # (tokens a call: the host arrays' size)
|
| 136 |
+
|
| 137 |
+
def __init__(self, b):
|
| 138 |
+
self.b = b
|
| 139 |
+
self.man = b.man["lfx"]
|
| 140 |
+
self.dir = os.path.join(b.d, "plans", self.man.get("dir", "lfx"))
|
| 141 |
+
self.CAP = b.tmax
|
| 142 |
+
self.g = Glue()
|
| 143 |
+
self._ctx = {}
|
| 144 |
+
ents = json.load(open(os.path.join(self.dir, "lfx.json")))
|
| 145 |
+
self.p = {k: _Prog(self, k, v) for k, v in ents.items()}
|
| 146 |
+
c = b.cfg
|
| 147 |
+
self.L, self.types, self.nd = c["num_hidden_layers"], c["layer_types"], c["num_dense_layers"]
|
| 148 |
+
self.eps = c["norm_eps"]
|
| 149 |
+
self.r2 = set(b.man.get("r2", []))
|
| 150 |
+
self.k2 = {p: 2 if p in self.r2 else 1 for p in ("conv", "attn", "mlp")}
|
| 151 |
+
self.eup = "eup" in self.r2
|
| 152 |
+
self.sm = np.load(os.path.join(b.d, "blobs", "small.npz"))
|
| 153 |
+
self.s2048, self.s512, self.s_i = _signs(2048), _signs(512), _signs(12288)
|
| 154 |
+
self.s512t = np.tile(self.s512, 3)
|
| 155 |
+
self.H64 = hadamard(64)
|
| 156 |
+
hd = 64
|
| 157 |
+
self.inv = 1.0 / (c["rope_parameters"]["rope_theta"] ** (np.arange(0, hd, 2, dtype=np.float64) / hd))
|
| 158 |
+
self.host = b.moe.host # (router, bias, gpost) per MoE layer
|
| 159 |
+
self.taps = {l: np.ascontiguousarray(self.sm[f"conv{l}"], np.float32) for l in range(self.L)
|
| 160 |
+
if self.types[l] == "conv"}
|
| 161 |
+
f32 = lambda a: np.ascontiguousarray(a, np.float32)
|
| 162 |
+
self.qn = {l: f32(self.sm[f"qn{l}"]) for l in range(self.L) if self.types[l] != "conv"}
|
| 163 |
+
self.kn = {l: f32(self.sm[f"kn{l}"]) for l in range(self.L) if self.types[l] != "conv"}
|
| 164 |
+
self.w_in = [f32(self.sm[f"in{l}"]) for l in range(self.L)]
|
| 165 |
+
self.w_post = [f32(self.sm[f"post{l}"]) for l in range(self.L)]
|
| 166 |
+
self.inv32 = f32(self.inv)
|
| 167 |
+
# buffers: X (the A tiles of a call), Y (a dispatch's outputs, the biggest program's)
|
| 168 |
+
segs_e = (self.CH * 4) // 64 + 64 # (the experts' segments of a call: every expert's last padded)
|
| 169 |
+
nk_e = max(p.nk + p.hdr for p in self.p.values() if p.patch)
|
| 170 |
+
nk_d = max(p.nk + p.hdr for p in self.p.values() if not p.patch)
|
| 171 |
+
xmax = max(segs_e * nk_e, -(-self.CH // 64) * nk_d) * AW + max(p.xw for p in self.p.values())
|
| 172 |
+
self.xbo, xm = b.dev.buf(xmax * 4, 3) # (not cacheable: a cacheable one's syncs cost ~3 ms)
|
| 173 |
+
self.xm = xm.view(np.uint8)
|
| 174 |
+
ymax = max(p.yw for p in self.p.values())
|
| 175 |
+
self.ybos = [b.dev.buf(ymax * 4, 5) for _ in range(2)] # (two: a dispatch's rows are read while the next
|
| 176 |
+
self.ybo, ym = self.ybos[0] # one runs - the C calls and the run release the GIL)
|
| 177 |
+
self.ym = ym.view(np.uint32)
|
| 178 |
+
from concurrent.futures import ThreadPoolExecutor
|
| 179 |
+
self.rowpool = ThreadPoolExecutor(1)
|
| 180 |
+
self.t = {}
|
| 181 |
+
perm = np.concatenate([np.concatenate([np.arange(g * 128, g * 128 + 128), 2048 + np.arange(g * 128, g * 128 + 128),
|
| 182 |
+
4096 + np.arange(g * 128, g * 128 + 128)]) for g in range(16)])
|
| 183 |
+
self._idn = {}
|
| 184 |
+
self._csum = None
|
| 185 |
+
self._ws = {} # (work arrays kept between calls: a fresh 50 MB array
|
| 186 |
+
# costs ~0.1 s of page faults on Windows)
|
| 187 |
+
self.map_sc = perm.astype(np.int32) # (sc_in's op column j -> B | C | x column perm[j])
|
| 188 |
+
rows = []
|
| 189 |
+
for a in range(4): # (at_qkv's op rows: core a = q heads 8a.., kv heads 2a, 2a + 1)
|
| 190 |
+
rows += [np.arange(h * 64, h * 64 + 64) for h in range(8 * a, 8 * a + 8)]
|
| 191 |
+
kv = np.arange(a * 128, a * 128 + 128)
|
| 192 |
+
rows += [2048 + kv, 2560 + kv]
|
| 193 |
+
self.map_at = np.concatenate(rows).astype(np.int32) # (at_qkv's op column -> q | k | v)
|
| 194 |
+
half, per = 6144, 384
|
| 195 |
+
gi, ui = [], [] # (mlp_gu pass p's op rows: per core cc [gate 384 | up 384])
|
| 196 |
+
for cc in range(16):
|
| 197 |
+
gi.append(cc * 768 + np.arange(per))
|
| 198 |
+
ui.append(cc * 768 + per + np.arange(per))
|
| 199 |
+
cm = np.empty(12288, np.int32) # (mlp_gu's op column -> [gate 6144 | up 6144] of the pass)
|
| 200 |
+
for cc in range(16):
|
| 201 |
+
cm[cc * 768:cc * 768 + per] = cc * per + np.arange(per)
|
| 202 |
+
cm[cc * 768 + per:(cc + 1) * 768] = 6144 + cc * per + np.arange(per)
|
| 203 |
+
self.map_gu = cm
|
| 204 |
+
self.eg = np.concatenate([g * 192 + np.arange(96) for g in range(16)]) # (e_up's: per core g [gate 96 | up 96])
|
| 205 |
+
self.eu = self.eg + 96
|
| 206 |
+
|
| 207 |
+
def ctx(self, xclbin):
|
| 208 |
+
import hashlib
|
| 209 |
+
h = hashlib.sha1(open(xclbin, "rb").read()).hexdigest()
|
| 210 |
+
if h not in self._ctx:
|
| 211 |
+
self._ctx[h] = self.b.dev.more(xclbin)
|
| 212 |
+
return self._ctx[h]
|
| 213 |
+
|
| 214 |
+
# ---------------------------------------------------------------- the prefill interface (Bonsai.prefill)
|
| 215 |
+
def plan(self, n, T=0):
|
| 216 |
+
return min(n, self.CH)
|
| 217 |
+
|
| 218 |
+
def chunk_for(self, m):
|
| 219 |
+
return m
|
| 220 |
+
|
| 221 |
+
# ---------------------------------------------------------------- NPU calls
|
| 222 |
+
def _gemm(self, name, wbo, nseg_total, Mc, experts=None, act=False, cmap=None, out=None, nrows=None):
|
| 223 |
+
"""the program over nseg_total segments of the packed X (in self.xm) -> rows fp32: [nseg_total * 64, 16 Mc] in
|
| 224 |
+
the op's order, or (cmap: op column -> out column) into out's first nrows rows; act: the experts' up -
|
| 225 |
+
silu(gate) * up, [.., 1536]"""
|
| 226 |
+
p = self.p[name]
|
| 227 |
+
b = self.b
|
| 228 |
+
if out is None:
|
| 229 |
+
out = self.ws("g_" + name, (nseg_total * 64 if nrows is None else nrows, 1536 if act else 16 * Mc))
|
| 230 |
+
nrows = out.shape[0] if nrows is None else nrows
|
| 231 |
+
if cmap is None and not act:
|
| 232 |
+
cmap = self._ident(16 * Mc)
|
| 233 |
+
t = time.perf_counter()
|
| 234 |
+
nseg_all = -(-nseg_total // p.nseg) * p.nseg # (the last dispatch's unused segments: their headers too -
|
| 235 |
+
if p.hdr: # a core reading garbage K-steps would never finish)
|
| 236 |
+
hw = self.xm.view(np.int32)
|
| 237 |
+
for s_ in range(nseg_total, nseg_all):
|
| 238 |
+
hw[s_ * p.sw] = p.nk
|
| 239 |
+
xa = b.dev.sub(self.xbo, nseg_all * p.sw * 4, 0) # (only the packed bytes to the device)
|
| 240 |
+
b.dev.sync(xa, True)
|
| 241 |
+
b.dev.free(xa)
|
| 242 |
+
ys = [b.dev.sub(bo, p.yw * 4, 0) for bo, _ in self.ybos]
|
| 243 |
+
yms = [m.view(np.uint32) for _, m in self.ybos]
|
| 244 |
+
th = [0.0]
|
| 245 |
+
|
| 246 |
+
def rows(k, s0, ns): # (on the row thread)
|
| 247 |
+
t1 = time.perf_counter()
|
| 248 |
+
b.dev.sync(ys[k], False)
|
| 249 |
+
ym = yms[k]
|
| 250 |
+
if act:
|
| 251 |
+
self.g.x.lx_eact(ym.ctypes.data, ns, _addr(out, s0 * 64 * 1536))
|
| 252 |
+
else:
|
| 253 |
+
self.g.x.lx_rows2(ym.ctypes.data, ns, p.nq, p.rmap == "rs", Mc, cmap.ctypes.data,
|
| 254 |
+
_addr(out, s0 * 64 * out.shape[1]), out.shape[1], nrows - s0 * 64)
|
| 255 |
+
th[0] += time.perf_counter() - t1
|
| 256 |
+
fut = [None, None]
|
| 257 |
+
try:
|
| 258 |
+
for i, s0 in enumerate(range(0, nseg_total, p.nseg)):
|
| 259 |
+
k = i % 2
|
| 260 |
+
if fut[k] is not None: # (its Y buffer's rows read first)
|
| 261 |
+
fut[k].result()
|
| 262 |
+
ns = min(p.nseg, nseg_total - s0)
|
| 263 |
+
if experts is not None:
|
| 264 |
+
p.point(experts[s0:s0 + ns])
|
| 265 |
+
xs = b.dev.sub(self.xbo, p.xw * 4, s0 * p.sw * 4) if s0 else self.xbo
|
| 266 |
+
try:
|
| 267 |
+
p.ctx.run(p.ibo, p.n, xs, wbo, self.ybos[k][0], None)
|
| 268 |
+
finally:
|
| 269 |
+
if s0:
|
| 270 |
+
b.dev.free(xs)
|
| 271 |
+
fut[k] = self.rowpool.submit(rows, k, s0, ns)
|
| 272 |
+
finally:
|
| 273 |
+
for f in fut:
|
| 274 |
+
if f is not None:
|
| 275 |
+
f.result()
|
| 276 |
+
for y_ in ys:
|
| 277 |
+
b.dev.free(y_)
|
| 278 |
+
th = th[0]
|
| 279 |
+
self.t[name] = self.t.get(name, 0.0) + time.perf_counter() - t - th
|
| 280 |
+
self.t["rows"] = self.t.get("rows", 0.0) + th
|
| 281 |
+
return out
|
| 282 |
+
|
| 283 |
+
def ws(self, key, shape, dtype=np.float32):
|
| 284 |
+
"""a work array (the first prod(shape) elements of a kept buffer)"""
|
| 285 |
+
n = int(np.prod(shape))
|
| 286 |
+
a = self._ws.get(key)
|
| 287 |
+
if a is None or a.size < n or a.dtype != dtype:
|
| 288 |
+
a = self._ws[key] = np.empty(max(n, 1), dtype)
|
| 289 |
+
return a[:n].reshape(shape)
|
| 290 |
+
|
| 291 |
+
def _rmsrot(self, h, w, norm=False):
|
| 292 |
+
"""(n = h / rms(h) * w or None, its rotation) - one C pass"""
|
| 293 |
+
M = h.shape[0]
|
| 294 |
+
xr = self.ws("rot0", (M, 2048))
|
| 295 |
+
n = self.ws("rms", (M, 2048)) if norm else None
|
| 296 |
+
t = time.perf_counter()
|
| 297 |
+
self.g.x.lx_rmsrot(M, 2048, h.ctypes.data, w.ctypes.data, self.eps, self.s2048.ctypes.data, 1024, 1 / 32,
|
| 298 |
+
None if n is None else n.ctypes.data, xr.ctypes.data)
|
| 299 |
+
self.t["glue"] = self.t.get("glue", 0.0) + time.perf_counter() - t
|
| 300 |
+
return n, xr
|
| 301 |
+
|
| 302 |
+
def _ident(self, n):
|
| 303 |
+
if n not in self._idn:
|
| 304 |
+
self._idn[n] = np.arange(n, dtype=np.int32)
|
| 305 |
+
return self._idn[n]
|
| 306 |
+
|
| 307 |
+
def _dense(self, name, wbo, x, Mc, dup=1, rot=True, s=None, blk=1024, cmap=None, width=None):
|
| 308 |
+
"""x [M, K] (normed) -> rows [M, 16 Mc] (op order, or cmap: op column -> column of a [M, width] result)"""
|
| 309 |
+
M = x.shape[0]
|
| 310 |
+
t = time.perf_counter()
|
| 311 |
+
xr = self.g.rot(x, self.s2048 if s is None else s, blk, out=self.ws("rot", x.shape)) if rot else x
|
| 312 |
+
nseg = self.g.pack(xr, self.xm, n=M, dup=dup, hdr=self.p[name].hdr)
|
| 313 |
+
self.t["glue"] = self.t.get("glue", 0.0) + time.perf_counter() - t
|
| 314 |
+
out = self.ws("d_" + name, (M, width or 16 * Mc))
|
| 315 |
+
return self._gemm(name, wbo, nseg, Mc, cmap=cmap, out=out, nrows=M)
|
| 316 |
+
|
| 317 |
+
def _rms(self, x, w, key=None):
|
| 318 |
+
ss = np.einsum("...i,...i->...", x, x)[..., None] / x.shape[-1]
|
| 319 |
+
r = (1.0 / np.sqrt(ss + self.eps)).astype(np.float32)
|
| 320 |
+
if key is None:
|
| 321 |
+
return x * r * w
|
| 322 |
+
o = self.ws(key, x.shape)
|
| 323 |
+
np.multiply(x, r, out=o)
|
| 324 |
+
o *= w
|
| 325 |
+
return o
|
| 326 |
+
|
| 327 |
+
# ---------------------------------------------------------------- the short conv's state (the decode image)
|
| 328 |
+
def _conv_state(self, l):
|
| 329 |
+
"""the state image's conv blocks of layer l: (view [16 groups][2 blocks][544] of the map, the map)"""
|
| 330 |
+
bo, m, _ = self.b.st[l]
|
| 331 |
+
w = m.view(np.float32)
|
| 332 |
+
# stream s (cores 2s, 2s + 1 interleaved), a core's image: op o's blocks [hdr, in, conv0, conv1]
|
| 333 |
+
st = w[:8 * 16 * BLK].reshape(4, 8, 2, BLK) # [stream][block i][core side][words]
|
| 334 |
+
return st, bo
|
| 335 |
+
|
| 336 |
+
def _conv(self, l, Y):
|
| 337 |
+
"""y = C * conv(B * x) over the chunk (Y: [M, 6144] rows B | C | x), from / into the state image"""
|
| 338 |
+
b = self.b
|
| 339 |
+
bo, m, _ = b.st[l]
|
| 340 |
+
b.dev.sync(bo, False)
|
| 341 |
+
st = m.view(np.float32)[:4 * 8 * 2 * BLK].reshape(4, 8, 2, BLK) # [stream][block][core side][words]
|
| 342 |
+
S = np.empty((3, 2048), np.float32) # (bx at t - 3, t - 2, t - 1 before the chunk)
|
| 343 |
+
blocks = []
|
| 344 |
+
for g in range(16): # (group g = 2 core + op o; core = 2 stream + side)
|
| 345 |
+
core, o = divmod(g, 2)
|
| 346 |
+
sm, side = divmod(core, 2)
|
| 347 |
+
for e in range(2):
|
| 348 |
+
blk = st[sm, o * 4 + 2 + e, side]
|
| 349 |
+
ch = slice(g * 128 + e * 64, g * 128 + e * 64 + 64)
|
| 350 |
+
S[:, ch] = blk[256:448].reshape(3, 64)
|
| 351 |
+
blocks.append((blk, ch))
|
| 352 |
+
y = self.ws("conv_y", (Y.shape[0], 2048))
|
| 353 |
+
self.g.x.lx_conv(Y.shape[0], Y.ctypes.data, Y.shape[1], self.taps[l].ctypes.data, S.ctypes.data, y.ctypes.data)
|
| 354 |
+
for blk, ch in blocks:
|
| 355 |
+
blk[256:448] = S[:, ch].reshape(-1)
|
| 356 |
+
b.dev.sync(bo, True)
|
| 357 |
+
return y
|
| 358 |
+
|
| 359 |
+
# ---------------------------------------------------------------- attention (q / k norms, rope, the int8 cache records: lfx_glue)
|
| 360 |
+
def _attn(self, l, qkv, T0):
|
| 361 |
+
b = self.b
|
| 362 |
+
M = qkv.shape[0]
|
| 363 |
+
T = T0 + M
|
| 364 |
+
K = self.ws("att_k", (T, 8, 64))
|
| 365 |
+
V = self.ws("att_v", (T, 8, 64))
|
| 366 |
+
q = self.ws("att_q", (M, 32, 64))
|
| 367 |
+
self.g.x.lx_qkrope(M, qkv.ctypes.data, self.qn[l].ctypes.data, self.kn[l].ctypes.data, self.eps, T0,
|
| 368 |
+
self.inv32.ctypes.data, q.ctypes.data, K[T0:].ctypes.data, V[T0:].ctypes.data)
|
| 369 |
+
m = b.kvm[l].view(np.uint8)
|
| 370 |
+
if T0: # (the earlier positions: from the cache, dequantized)
|
| 371 |
+
nb = (T0 + 3) // 4 * 4 * 2176
|
| 372 |
+
sub = b.dev.sub(b.kv[l], nb, 0)
|
| 373 |
+
try:
|
| 374 |
+
b.dev.sync(sub, False)
|
| 375 |
+
finally:
|
| 376 |
+
b.dev.free(sub)
|
| 377 |
+
self.g.x.lx_kvget(T0, m.ctypes.data, K.ctypes.data, V.ctypes.data)
|
| 378 |
+
Q0, Q1 = T0 // 4, (T - 1) // 4 + 1 # (the chunk's records: the first quad's earlier ones kept)
|
| 379 |
+
sub = b.dev.sub(b.kv[l], (Q1 - Q0) * 4 * 2176, Q0 * 4 * 2176)
|
| 380 |
+
try:
|
| 381 |
+
if T0 % 4:
|
| 382 |
+
b.dev.sync(sub, False)
|
| 383 |
+
self.g.x.lx_kvput(M, T0, K[T0:].ctypes.data, V[T0:].ctypes.data, m[Q0 * 4 * 2176:].ctypes.data)
|
| 384 |
+
b.dev.sync(sub, True)
|
| 385 |
+
finally:
|
| 386 |
+
b.dev.free(sub)
|
| 387 |
+
out = self.ws("att_o", (M, 32, 64))
|
| 388 |
+
self.g.x.lx_attn(M, T0, q.ctypes.data, K.ctypes.data, V.ctypes.data, out.ctypes.data)
|
| 389 |
+
return out.reshape(M, 2048)
|
| 390 |
+
|
| 391 |
+
# ---------------------------------------------------------------- the MoE
|
| 392 |
+
def _moe(self, l, x, xr, h):
|
| 393 |
+
"""h += the MoE of layer l on x (the normed rows; xr: rotated)"""
|
| 394 |
+
R, bias, _ = self.host[l]
|
| 395 |
+
M = x.shape[0]
|
| 396 |
+
sc = 1.0 / (1.0 + np.exp(-(x @ R.T)))
|
| 397 |
+
top = np.argsort(-(sc + bias), 1)[:, :4]
|
| 398 |
+
w = np.take_along_axis(sc, top, 1)
|
| 399 |
+
w = w / (w.sum(1, keepdims=True) + 1e-6)
|
| 400 |
+
tok = np.repeat(np.arange(M), 4)
|
| 401 |
+
ex, wt = top.reshape(-1), w.reshape(-1)
|
| 402 |
+
order = np.argsort(ex, kind="stable")
|
| 403 |
+
ex, tok, wt = ex[order], tok[order], wt[order]
|
| 404 |
+
seg_e, seg_rows = [], [] # (per segment its expert and 64 rows: pair indices, -1 pad)
|
| 405 |
+
for e in np.unique(ex):
|
| 406 |
+
idx = np.nonzero(ex == e)[0]
|
| 407 |
+
for s0 in range(0, len(idx), 64):
|
| 408 |
+
r = np.full(64, -1, np.int64)
|
| 409 |
+
r[:min(64, len(idx) - s0)] = idx[s0:s0 + 64]
|
| 410 |
+
seg_rows.append(r)
|
| 411 |
+
seg_e.append(int(e))
|
| 412 |
+
seg_rows = np.stack(seg_rows)
|
| 413 |
+
nseg = len(seg_e)
|
| 414 |
+
src = np.where(seg_rows >= 0, tok[np.maximum(seg_rows, 0)], -1).reshape(-1)
|
| 415 |
+
t = time.perf_counter()
|
| 416 |
+
self.g.pack(xr, self.xm, idx=src, n=nseg * 64, dup=2 if self.eup else 1, hdr=self.p["e_up"].hdr)
|
| 417 |
+
self.t["glue"] = self.t.get("glue", 0.0) + time.perf_counter() - t
|
| 418 |
+
wbo = self.b.wbo[l]
|
| 419 |
+
a = self._gemm("e_up", wbo, nseg, 192, seg_e, act=True) # [nseg * 64, 1536]: silu(gate) * up
|
| 420 |
+
t = time.perf_counter()
|
| 421 |
+
ar = self.g.rot(a, self.s512t, 512, out=self.ws("rot", a.shape))
|
| 422 |
+
self.g.pack(ar, self.xm, n=nseg * 64, dup=self.p["e_dn"].e.get("dup", 1), # (2: down + the residual's)
|
| 423 |
+
hdr=self.p["e_dn"].hdr)
|
| 424 |
+
self.t["glue"] = self.t.get("glue", 0.0) + time.perf_counter() - t
|
| 425 |
+
y = self._gemm("e_dn", wbo, nseg, 128, seg_e)
|
| 426 |
+
if "e_dn2" in self.p:
|
| 427 |
+
y += self._gemm("e_dn2", wbo, nseg, 128, seg_e)
|
| 428 |
+
t = time.perf_counter()
|
| 429 |
+
r = seg_rows.reshape(-1)
|
| 430 |
+
dst = np.ascontiguousarray(np.where(r >= 0, tok[np.maximum(r, 0)], -1), np.int32)
|
| 431 |
+
wr = np.ascontiguousarray(np.where(r >= 0, wt[np.maximum(r, 0)], 0.0), np.float32)
|
| 432 |
+
y = np.ascontiguousarray(y, np.float32)
|
| 433 |
+
self.g.x.lx_scatter(len(dst), 2048, y.ctypes.data, dst.ctypes.data, wr.ctypes.data, h.ctypes.data)
|
| 434 |
+
self.t["sum"] = self.t.get("sum", 0.0) + time.perf_counter() - t
|
| 435 |
+
|
| 436 |
+
def _mlp(self, l, xr):
|
| 437 |
+
wbo = self.b.wbo[l]
|
| 438 |
+
dup = self.k2["mlp"]
|
| 439 |
+
nseg = self.g.pack(xr, self.xm, n=xr.shape[0], dup=dup, hdr=self.p["mlp_gu0"].hdr)
|
| 440 |
+
M = xr.shape[0]
|
| 441 |
+
a = self.ws("mlp_a", (M, 12288))
|
| 442 |
+
Y = self.ws("mlp_y", (M, 12288))
|
| 443 |
+
for p in range(2):
|
| 444 |
+
self._gemm(f"mlp_gu{p}", wbo, nseg, 768, cmap=self.map_gu, out=Y, nrows=M) # ([gate | up] of the pass)
|
| 445 |
+
self.g.x.lx_silumul(M, 6144, Y.ctypes.data, 12288, Y[:, 6144:].ctypes.data, 12288,
|
| 446 |
+
a[:, p * 6144:].ctypes.data, 12288)
|
| 447 |
+
y = self._dense("mlp_dn", wbo, a, 128, s=self.s_i, dup=self.p["mlp_dn"].e.get("dup", 1))
|
| 448 |
+
if "mlp_dn2" in self.p:
|
| 449 |
+
y = y + self._dense("mlp_dn2", wbo, a, 128, s=self.s_i)
|
| 450 |
+
return y
|
| 451 |
+
|
| 452 |
+
# ---------------------------------------------------------------- a call
|
| 453 |
+
def run(self, toks, rows=False):
|
| 454 |
+
b = self.b
|
| 455 |
+
T0, M = b.pos, len(toks)
|
| 456 |
+
t0 = time.perf_counter()
|
| 457 |
+
h = np.stack([b.embed(int(t)) for t in toks]).astype(np.float32)
|
| 458 |
+
for l in range(self.L):
|
| 459 |
+
wbo = b.wbo[l]
|
| 460 |
+
_, xr = self._rmsrot(h, self.w_in[l])
|
| 461 |
+
if self.types[l] == "conv":
|
| 462 |
+
d = "_d" if l < self.nd else ""
|
| 463 |
+
Y = self._dense("sc_in" + d, wbo, xr, 384, dup=self.k2["conv"], cmap=self.map_sc, rot=False)
|
| 464 |
+
y = self._conv(l, Y)
|
| 465 |
+
h += self._dense("sc_out" + d, wbo, y, 128, dup=self.k2["conv"])
|
| 466 |
+
else:
|
| 467 |
+
qkv = self._dense("at_qkv", wbo, xr, 192, dup=self.k2["attn"], cmap=self.map_at, rot=False)
|
| 468 |
+
o = self._attn(l, qkv, T0)
|
| 469 |
+
h += self._dense("at_out", wbo, o, 128, dup=self.k2["attn"])
|
| 470 |
+
x, xr = self._rmsrot(h, self.w_post[l], norm=l >= self.nd)
|
| 471 |
+
if l < self.nd:
|
| 472 |
+
h += self._mlp(l, xr)
|
| 473 |
+
else:
|
| 474 |
+
self._moe(l, x, xr, h)
|
| 475 |
+
b.pos += M
|
| 476 |
+
self.t["all"] = self.t.get("all", 0.0) + time.perf_counter() - t0
|
| 477 |
+
return h[-1].copy(), None
|
onwx/web/manage.html
CHANGED
|
@@ -407,8 +407,10 @@
|
|
| 407 |
else if (dl) pill = `<span class="pill accent dlpct"></span>`;
|
| 408 |
else if (m.state === "missing") pill = `<span class="muted small">${T("未取得")}</span>`;
|
| 409 |
else if (PILL[m.state]) pill = `<span class="pill ${PILL[m.state][0]}">${PILL[m.state][1]}</span>`;
|
|
|
|
|
|
|
| 410 |
const cols = m.catalog
|
| 411 |
-
? `<td class="c">${
|
| 412 |
: `<td colspan="3" class="c"></td>`;
|
| 413 |
return `<tr class="${cls}"${style}><td class="name">${esc(m.title)}<span class="sub" title="${esc(note)}">${esc(note)}</span><span class="acts2">${acts}</span></td>${cols}<td class="c">${pill}</td><td class="acts">${acts}</td></tr>`;
|
| 414 |
}).join("");
|
|
@@ -489,7 +491,7 @@
|
|
| 489 |
case "live": return `<button disabled>${T("ロード済み")}</button>${more}`;
|
| 490 |
case "loading": return `<button disabled>${T("ロード中…")}</button>${more}`;
|
| 491 |
case "downloading": return btn("cancel", m.key, T("キャンセル"));
|
| 492 |
-
case "missing": return btn("download", m.key, T("ダウンロード"), "primary");
|
| 493 |
case "partial": return btn("delete", m.key, T("削除"), "danger");
|
| 494 |
case "outdated": return btn("download", m.key, T("更新"), "primary") + more;
|
| 495 |
default: return btn("load", m.key, S.server.model ? T("切り替え") : T("ロード"), "primary", loadBusy()) + more;
|
|
@@ -502,13 +504,20 @@
|
|
| 502 |
switch (b.dataset.act) {
|
| 503 |
case "menu": return menu(ev, b, key);
|
| 504 |
case "load": b.disabled = true; return post("load", {key});
|
| 505 |
-
case "download": return
|
| 506 |
case "cancel": return post("cancel_download");
|
| 507 |
case "delete": return deleteModel(key);
|
| 508 |
case "cache": return clearCache(key);
|
| 509 |
case "folder": return post("open", {what: "folder", key});
|
| 510 |
}
|
| 511 |
}
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 512 |
$("rows").addEventListener("click", rowAction);
|
| 513 |
$("menu").addEventListener("click", rowAction);
|
| 514 |
function menu(ev, el, key) {
|
|
|
|
| 407 |
else if (dl) pill = `<span class="pill accent dlpct"></span>`;
|
| 408 |
else if (m.state === "missing") pill = `<span class="muted small">${T("未取得")}</span>`;
|
| 409 |
else if (PILL[m.state]) pill = `<span class="pill ${PILL[m.state][0]}">${PILL[m.state][1]}</span>`;
|
| 410 |
+
const size = m.kit && !m.path ? T("{0} GB(変換中は {1} GB)", m.kit.final_gb.toFixed(0), m.kit.disk_gb.toFixed(0))
|
| 411 |
+
: `${(m.kit ? m.kit.final_gb : m.size_gb).toFixed(1)} GB`;
|
| 412 |
const cols = m.catalog
|
| 413 |
+
? `<td class="c">${size}</td><td class="c${m.fits ? "" : " bad"}">${T("約{0} GB", Math.round(m.mem_gb))}</td><td class="c">${esc(m.speed)}</td>`
|
| 414 |
: `<td colspan="3" class="c"></td>`;
|
| 415 |
return `<tr class="${cls}"${style}><td class="name">${esc(m.title)}<span class="sub" title="${esc(note)}">${esc(note)}</span><span class="acts2">${acts}</span></td>${cols}<td class="c">${pill}</td><td class="acts">${acts}</td></tr>`;
|
| 416 |
}).join("");
|
|
|
|
| 491 |
case "live": return `<button disabled>${T("ロード済み")}</button>${more}`;
|
| 492 |
case "loading": return `<button disabled>${T("ロード中…")}</button>${more}`;
|
| 493 |
case "downloading": return btn("cancel", m.key, T("キャンセル"));
|
| 494 |
+
case "missing": return btn("download", m.key, m.kit ? T("ダウンロードして変換") : T("ダウンロード"), "primary");
|
| 495 |
case "partial": return btn("delete", m.key, T("削除"), "danger");
|
| 496 |
case "outdated": return btn("download", m.key, T("更新"), "primary") + more;
|
| 497 |
default: return btn("load", m.key, S.server.model ? T("切り替え") : T("ロード"), "primary", loadBusy()) + more;
|
|
|
|
| 504 |
switch (b.dataset.act) {
|
| 505 |
case "menu": return menu(ev, b, key);
|
| 506 |
case "load": b.disabled = true; return post("load", {key});
|
| 507 |
+
case "download": return download(key);
|
| 508 |
case "cancel": return post("cancel_download");
|
| 509 |
case "delete": return deleteModel(key);
|
| 510 |
case "cache": return clearCache(key);
|
| 511 |
case "folder": return post("open", {what: "folder", key});
|
| 512 |
}
|
| 513 |
}
|
| 514 |
+
// [1.0] a kit model: the original model is downloaded and converted on this PC - say what it takes first
|
| 515 |
+
async function download(key) {
|
| 516 |
+
const m = S && S.models.find(x => x.key === key), k = m && m.kit;
|
| 517 |
+
if (k && !m.path && !await ask(T("このモデルはこの PC で変換して使います。\n・元のモデル(約 {0} GB)を Hugging Face からダウンロードします\n・変換中は約 {1} GB の空きが必要です(終わると元のファイルは消え、約 {2} GB が残ります)\n・かかる時間は約 {3} 分です(回線と PC による)\n・元のモデルのライセンスに従ってお使いください\n始めますか?", k.source_gb.toFixed(0), k.disk_gb.toFixed(0), k.final_gb.toFixed(0), k.minutes)))
|
| 518 |
+
return;
|
| 519 |
+
return post("download", {repo: key});
|
| 520 |
+
}
|
| 521 |
$("rows").addEventListener("click", rowAction);
|
| 522 |
$("menu").addEventListener("click", rowAction);
|
| 523 |
function menu(ev, el, key) {
|
onwx/xdna.py
CHANGED
|
@@ -349,6 +349,9 @@ while H32.shape[0] < 32:
|
|
| 349 |
H32 = np.block([[H32, H32], [H32, -H32]])
|
| 350 |
|
| 351 |
|
|
|
|
|
|
|
|
|
|
| 352 |
class Bonsai:
|
| 353 |
"""d: the model folder (xdna.json, plans/, blobs/, embed.npz). One token per step(). context: the positions the
|
| 354 |
attention caches are allocated for (pair-major programs: up to xdna.json "tmax"; older ones: their fixed 3840)."""
|
|
@@ -384,7 +387,7 @@ class Bonsai:
|
|
| 384 |
B = os.path.join(d, "blobs")
|
| 385 |
mt = man.get("mtp") if os.environ.get("ONWX_MTP", "1") != "0" else None
|
| 386 |
self.experts = None # [0.9] the MoE's experts: pools, streamed from the SSD or all
|
| 387 |
-
if "moe" in man:
|
| 388 |
from .xdna_experts import ExpertStore, budget_bytes
|
| 389 |
files = [os.path.join(B, f"L{l}.wgt") for l in range(self.L)]
|
| 390 |
if mt and mt.get("moe"):
|
|
@@ -395,7 +398,7 @@ class Bonsai:
|
|
| 395 |
else:
|
| 396 |
self.wbo = [self.load(os.path.join(B, f"L{l}.wgt"), 3)[0] for l in range(self.L)]
|
| 397 |
self.hbo = self.load(os.path.join(B, "head.wgt"), 3)[0]
|
| 398 |
-
self.st, self.kv = {}, {}
|
| 399 |
# [MTP] the DeltaNet states: 2 VT slots a layer (a verify block reads one, writes the state after each of its
|
| 400 |
# VT tokens into the VT slots of the other half; the host continues from the accepted token's); kv: + pair slots
|
| 401 |
self.nslot, self.slot, self.stsub = (2 * mt["vt"] if mt else 1), 0, {}
|
|
@@ -403,7 +406,7 @@ class Bonsai:
|
|
| 403 |
pfx = man.get("pfx") # [0.5] room past tmax for the last prompt chunk's padding and the
|
| 404 |
self.kv_slack = (max(pfx.get("ms", [2048])) + 256) if pfx and self.pair else 0 # attention's read-ahead
|
| 405 |
for l in range(self.L):
|
| 406 |
-
if self.types[l]
|
| 407 |
self.st[l] = self.load(os.path.join(B, f"L{l}.st"), 5, self.nslot) # (bo, map, path)
|
| 408 |
n = os.path.getsize(self.st[l][2])
|
| 409 |
self.stsz = n
|
|
@@ -411,7 +414,7 @@ class Bonsai:
|
|
| 411 |
self.stsub[l] = [self.dev.sub(self.st[l][0], n, k * n) for k in range(self.nslot)]
|
| 412 |
else:
|
| 413 |
n = ((self.tmax + self.kv_slack + self.kq - 1) // self.kq + self.kv_margin) * PAIR if self.pair else 4 * self.hstr
|
| 414 |
-
self.kv[l] = self.dev.buf(n * 4, 5)[0]
|
| 415 |
self.actbo, am = self.dev.buf(man["act_words"] * 4, 4)
|
| 416 |
self.am = am.view(np.int32)
|
| 417 |
init = np.fromfile(os.path.join(B, "act.init"), dtype=np.int32)
|
|
@@ -441,7 +444,7 @@ class Bonsai:
|
|
| 441 |
self.mtp = Mtp(self) if mt and not mt.get("moe") else None
|
| 442 |
# [0.4] the chunked all-NPU prefill (xdna.json "pfx": 1024 / 2048-token chunks; made on first use)
|
| 443 |
self.bigpf = None
|
| 444 |
-
self._big_ok = "pfx" in man and os.environ.get("ONWX_PFX", "1") != "0"
|
| 445 |
self._big_off = False
|
| 446 |
self.moe = None # [0.8] mixture of experts (Qwen3.6-35B-A3B): xdna_moe
|
| 447 |
if "moe" in man:
|
|
@@ -641,11 +644,14 @@ class Bonsai:
|
|
| 641 |
"""[0.4] the chunked NPU prefill (None: not in this package / disabled / failed to start)"""
|
| 642 |
if self.bigpf is None and self._big_ok:
|
| 643 |
try:
|
| 644 |
-
|
| 645 |
-
|
|
|
|
|
|
|
|
|
|
| 646 |
from .npu_pfx_moe import MoePrefill as BigPrefill
|
| 647 |
self.bigpf = BigPrefill(self)
|
| 648 |
-
self.bigpf.MIN = int(self.man
|
| 649 |
except Exception as e: # (the old prompt programs still work)
|
| 650 |
self._big_ok = False
|
| 651 |
print(f"[onwx] the chunked NPU prefill is off: {e!r}", flush=True)
|
|
@@ -1193,7 +1199,7 @@ class XdnaModel:
|
|
| 1193 |
self.S = (self.rt.pf128.NT if getattr(self.rt, "pf128", None) is not None else
|
| 1194 |
self.rt.pf.NT if self.rt.pf is not None else 1)
|
| 1195 |
if self.rt._big_ok: # [0.4] prompt blocks of the chunked prefill's biggest chunk
|
| 1196 |
-
self.S = max(self.S, max(self.rt.man
|
| 1197 |
self.delta = 0
|
| 1198 |
self.banned = []
|
| 1199 |
self._mutex = threading.RLock()
|
|
|
|
| 349 |
H32 = np.block([[H32, H32], [H32, -H32]])
|
| 350 |
|
| 351 |
|
| 352 |
+
STATEFUL = ("linear_attention", "conv") # [1.0] layers with a state image (DeltaNet; LFM2's short conv)
|
| 353 |
+
|
| 354 |
+
|
| 355 |
class Bonsai:
|
| 356 |
"""d: the model folder (xdna.json, plans/, blobs/, embed.npz). One token per step(). context: the positions the
|
| 357 |
attention caches are allocated for (pair-major programs: up to xdna.json "tmax"; older ones: their fixed 3840)."""
|
|
|
|
| 387 |
B = os.path.join(d, "blobs")
|
| 388 |
mt = man.get("mtp") if os.environ.get("ONWX_MTP", "1") != "0" else None
|
| 389 |
self.experts = None # [0.9] the MoE's experts: pools, streamed from the SSD or all
|
| 390 |
+
if "moe" in man and man["moe"].get("shared") is not None: # ([1.0] LFM2: the layer blobs as they are)
|
| 391 |
from .xdna_experts import ExpertStore, budget_bytes
|
| 392 |
files = [os.path.join(B, f"L{l}.wgt") for l in range(self.L)]
|
| 393 |
if mt and mt.get("moe"):
|
|
|
|
| 398 |
else:
|
| 399 |
self.wbo = [self.load(os.path.join(B, f"L{l}.wgt"), 3)[0] for l in range(self.L)]
|
| 400 |
self.hbo = self.load(os.path.join(B, "head.wgt"), 3)[0]
|
| 401 |
+
self.st, self.kv, self.kvm = {}, {}, {}
|
| 402 |
# [MTP] the DeltaNet states: 2 VT slots a layer (a verify block reads one, writes the state after each of its
|
| 403 |
# VT tokens into the VT slots of the other half; the host continues from the accepted token's); kv: + pair slots
|
| 404 |
self.nslot, self.slot, self.stsub = (2 * mt["vt"] if mt else 1), 0, {}
|
|
|
|
| 406 |
pfx = man.get("pfx") # [0.5] room past tmax for the last prompt chunk's padding and the
|
| 407 |
self.kv_slack = (max(pfx.get("ms", [2048])) + 256) if pfx and self.pair else 0 # attention's read-ahead
|
| 408 |
for l in range(self.L):
|
| 409 |
+
if self.types[l] in STATEFUL:
|
| 410 |
self.st[l] = self.load(os.path.join(B, f"L{l}.st"), 5, self.nslot) # (bo, map, path)
|
| 411 |
n = os.path.getsize(self.st[l][2])
|
| 412 |
self.stsz = n
|
|
|
|
| 414 |
self.stsub[l] = [self.dev.sub(self.st[l][0], n, k * n) for k in range(self.nslot)]
|
| 415 |
else:
|
| 416 |
n = ((self.tmax + self.kv_slack + self.kq - 1) // self.kq + self.kv_margin) * PAIR if self.pair else 4 * self.hstr
|
| 417 |
+
self.kv[l], self.kvm[l] = self.dev.buf(n * 4, 5) # ([1.0] the map: LFM2.5's prefill writes it)
|
| 418 |
self.actbo, am = self.dev.buf(man["act_words"] * 4, 4)
|
| 419 |
self.am = am.view(np.int32)
|
| 420 |
init = np.fromfile(os.path.join(B, "act.init"), dtype=np.int32)
|
|
|
|
| 444 |
self.mtp = Mtp(self) if mt and not mt.get("moe") else None
|
| 445 |
# [0.4] the chunked all-NPU prefill (xdna.json "pfx": 1024 / 2048-token chunks; made on first use)
|
| 446 |
self.bigpf = None
|
| 447 |
+
self._big_ok = ("pfx" in man or "lfx" in man) and os.environ.get("ONWX_PFX", "1") != "0"
|
| 448 |
self._big_off = False
|
| 449 |
self.moe = None # [0.8] mixture of experts (Qwen3.6-35B-A3B): xdna_moe
|
| 450 |
if "moe" in man:
|
|
|
|
| 644 |
"""[0.4] the chunked NPU prefill (None: not in this package / disabled / failed to start)"""
|
| 645 |
if self.bigpf is None and self._big_ok:
|
| 646 |
try:
|
| 647 |
+
if "lfx" in self.man: # [1.0] LFM2.5: GEMMs on the int4 blobs, the rest on the CPU
|
| 648 |
+
from .npu_lfx import LfxPrefill as BigPrefill
|
| 649 |
+
else:
|
| 650 |
+
from .npu_pfx import BigPrefill
|
| 651 |
+
if self.man.get("pfx", {}).get("moe"): # [0.8] the MoE's own chunked prefill
|
| 652 |
from .npu_pfx_moe import MoePrefill as BigPrefill
|
| 653 |
self.bigpf = BigPrefill(self)
|
| 654 |
+
self.bigpf.MIN = int(self.man.get("pfx", self.man.get("lfx", {})).get("min", min(self.bigpf.MS)))
|
| 655 |
except Exception as e: # (the old prompt programs still work)
|
| 656 |
self._big_ok = False
|
| 657 |
print(f"[onwx] the chunked NPU prefill is off: {e!r}", flush=True)
|
|
|
|
| 1199 |
self.S = (self.rt.pf128.NT if getattr(self.rt, "pf128", None) is not None else
|
| 1200 |
self.rt.pf.NT if self.rt.pf is not None else 1)
|
| 1201 |
if self.rt._big_ok: # [0.4] prompt blocks of the chunked prefill's biggest chunk
|
| 1202 |
+
self.S = max(self.S, max(self.rt.man.get("pfx", self.rt.man.get("lfx", {})).get("ms", [2048])))
|
| 1203 |
self.delta = 0
|
| 1204 |
self.banned = []
|
| 1205 |
self._mutex = threading.RLock()
|
onwx/xdna_moe.py
CHANGED
|
@@ -99,6 +99,19 @@ def route(host, h, approx, topk, shared, norm_topk=True):
|
|
| 99 |
return experts, weights
|
| 100 |
|
| 101 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 102 |
class Moe:
|
| 103 |
def __init__(self, b):
|
| 104 |
self.b = b
|
|
@@ -108,21 +121,32 @@ class Moe:
|
|
| 108 |
pts = m["patch"] if isinstance(m["patch"], dict) else {"all": m["patch"]}
|
| 109 |
self.prog = {k: _Patched(b, os.path.join(P, f), os.path.join(P, b.man["plans"]["moe" if k == "all" else f"moe_{k}"]))
|
| 110 |
for k, f in pts.items()}
|
| 111 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 112 |
self.topk, self.shared, self.nslot = m["topk"], m["shared"], m["slots"]
|
| 113 |
self.norm_topk = m.get("norm_topk", True)
|
| 114 |
B = os.path.join(b.d, "blobs")
|
| 115 |
self.host = []
|
| 116 |
for l in range(b.L):
|
|
|
|
|
|
|
|
|
|
| 117 |
z = np.load(os.path.join(B, f"L{l}.host.npz"))
|
| 118 |
-
self.host.append((z["router"], z["sgate"], z["gpost"]))
|
| 119 |
act = b.act
|
| 120 |
# (only H and MOEH move between the host and the NPU between the layers: sub-buffers of act)
|
| 121 |
self.h_off, self.mh_off = act["H"], act["MOEH"]
|
| 122 |
self.h_sub = b.dev.sub(b.actbo, 16 * BLK_W * 4, self.h_off * 4)
|
| 123 |
# [NPU router] the mixer plans end with the router (int4): its logits pick 16 candidates, the host re-scores
|
| 124 |
# them exactly (fp32) - the top-8 as the fp32 router's, at 16 of its 256 rows
|
| 125 |
-
self.npu_router = bool(m.get("router")) and os.environ.get("ONWX_MOE_HOSTROUTER") != "1"
|
| 126 |
if self.npu_router:
|
| 127 |
self.rl_off = act["RL"]
|
| 128 |
self.rl_sub = b.dev.sub(b.actbo, 16 * BLK_W * 4, self.rl_off * 4)
|
|
@@ -144,6 +168,8 @@ class Moe:
|
|
| 144 |
|
| 145 |
def route(self, l, h):
|
| 146 |
"""the experts of layer l for residual h: (10 slot experts, 10 weights)"""
|
|
|
|
|
|
|
| 147 |
approx = None
|
| 148 |
if self.npu_router:
|
| 149 |
b = self.b
|
|
@@ -161,10 +187,10 @@ class Moe:
|
|
| 161 |
ex[self.topk] = self.shared
|
| 162 |
experts = st.need(l, ex)
|
| 163 |
self.prog[self.kind_of(l)].point(experts)
|
| 164 |
-
f = np.asarray(weights, np.float32).view(np.int32)
|
| 165 |
-
s0 = 0
|
| 166 |
for g, n in enumerate(self.groups):
|
| 167 |
-
self.mh[g, 0:2, H_XS:H_XS + n] = f[s0:s0 + n]
|
| 168 |
s0 += n
|
| 169 |
self.b.dev.sync(self.mh_sub, True)
|
| 170 |
|
|
@@ -215,7 +241,7 @@ class Moe:
|
|
| 215 |
ins = it[0]
|
| 216 |
if ins is self.prog[self.kind_of(prev)].bo if prev is not None else False:
|
| 217 |
key = ("moe", prev)
|
| 218 |
-
elif l < b.L and
|
| 219 |
key = ("at", l, tcap)
|
| 220 |
elif l < b.L:
|
| 221 |
key = ("dn", l)
|
|
@@ -254,7 +280,10 @@ class Moe:
|
|
| 254 |
pg = self.prog[self.kind_of(prev)]
|
| 255 |
items.append((pg.bo, pg.n, b.moe_bo(prev), b.actbo, b.dummy, b.dummy))
|
| 256 |
if l < b.L:
|
| 257 |
-
if b.types[l] == "
|
|
|
|
|
|
|
|
|
|
| 258 |
items.append((dn[0], dn[1], b.wbo[l], b.actbo, b.st_cur(l), b.dummy))
|
| 259 |
else:
|
| 260 |
subs.append(b.kv_at(l, T))
|
|
@@ -272,6 +301,11 @@ class Moe:
|
|
| 272 |
b.dev.free(s)
|
| 273 |
if b.experts is not None:
|
| 274 |
b.experts.done()
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 275 |
if l < b.L:
|
| 276 |
t1 = time.perf_counter()
|
| 277 |
t2 = time.perf_counter()
|
|
|
|
| 99 |
return experts, weights
|
| 100 |
|
| 101 |
|
| 102 |
+
def route_lfm(host, h, topk, norm_topk=True):
|
| 103 |
+
"""[1.0] LFM2's router: sigmoid(R rms(h) gamma); the top-k of the scores + expert_bias; their scores (normalized)"""
|
| 104 |
+
R, bias, g = host
|
| 105 |
+
x = h.astype(np.float32)
|
| 106 |
+
xn = x / np.sqrt(np.mean(x * x) + 1e-5) * g
|
| 107 |
+
sc = 1.0 / (1.0 + np.exp(-(R @ xn)))
|
| 108 |
+
top = np.argsort(-(sc + bias))[:topk]
|
| 109 |
+
w = sc[top]
|
| 110 |
+
if norm_topk:
|
| 111 |
+
w = w / (w.sum() + 1e-6)
|
| 112 |
+
return list(map(int, top)), list(map(float, w))
|
| 113 |
+
|
| 114 |
+
|
| 115 |
class Moe:
|
| 116 |
def __init__(self, b):
|
| 117 |
self.b = b
|
|
|
|
| 121 |
pts = m["patch"] if isinstance(m["patch"], dict) else {"all": m["patch"]}
|
| 122 |
self.prog = {k: _Patched(b, os.path.join(P, f), os.path.join(P, b.man["plans"]["moe" if k == "all" else f"moe_{k}"]))
|
| 123 |
for k, f in pts.items()}
|
| 124 |
+
# [1.0] LFM2 (router "sigmoid_bias"): mixers "sc" (short conv) / "at", the first dense_layers without a MoE
|
| 125 |
+
# (their mixer plan "scm" ends with the dense MLP), sigmoid scores + expert_bias, no shared expert
|
| 126 |
+
self.lfm = m.get("router") == "sigmoid_bias"
|
| 127 |
+
self.nd = m.get("dense_layers", 0)
|
| 128 |
+
self.slices = m.get("slices", 1) # (F_XS floats a slot: its 512-slices)
|
| 129 |
+
if self.lfm:
|
| 130 |
+
self.kind_of = lambda l: "sc" if b.types[l] == "conv" else "at"
|
| 131 |
+
else:
|
| 132 |
+
self.kind_of = lambda l: "all" if "all" in self.prog else ("dn" if b.types[l] == "linear_attention" else "at")
|
| 133 |
self.topk, self.shared, self.nslot = m["topk"], m["shared"], m["slots"]
|
| 134 |
self.norm_topk = m.get("norm_topk", True)
|
| 135 |
B = os.path.join(b.d, "blobs")
|
| 136 |
self.host = []
|
| 137 |
for l in range(b.L):
|
| 138 |
+
if l < self.nd:
|
| 139 |
+
self.host.append(None)
|
| 140 |
+
continue
|
| 141 |
z = np.load(os.path.join(B, f"L{l}.host.npz"))
|
| 142 |
+
self.host.append((z["router"], z["bias"] if self.lfm else z["sgate"], z["gpost"]))
|
| 143 |
act = b.act
|
| 144 |
# (only H and MOEH move between the host and the NPU between the layers: sub-buffers of act)
|
| 145 |
self.h_off, self.mh_off = act["H"], act["MOEH"]
|
| 146 |
self.h_sub = b.dev.sub(b.actbo, 16 * BLK_W * 4, self.h_off * 4)
|
| 147 |
# [NPU router] the mixer plans end with the router (int4): its logits pick 16 candidates, the host re-scores
|
| 148 |
# them exactly (fp32) - the top-8 as the fp32 router's, at 16 of its 256 rows
|
| 149 |
+
self.npu_router = bool(m.get("router")) and not self.lfm and os.environ.get("ONWX_MOE_HOSTROUTER") != "1"
|
| 150 |
if self.npu_router:
|
| 151 |
self.rl_off = act["RL"]
|
| 152 |
self.rl_sub = b.dev.sub(b.actbo, 16 * BLK_W * 4, self.rl_off * 4)
|
|
|
|
| 168 |
|
| 169 |
def route(self, l, h):
|
| 170 |
"""the experts of layer l for residual h: (10 slot experts, 10 weights)"""
|
| 171 |
+
if self.lfm:
|
| 172 |
+
return route_lfm(self.host[l], h, self.topk, self.norm_topk)
|
| 173 |
approx = None
|
| 174 |
if self.npu_router:
|
| 175 |
b = self.b
|
|
|
|
| 187 |
ex[self.topk] = self.shared
|
| 188 |
experts = st.need(l, ex)
|
| 189 |
self.prog[self.kind_of(l)].point(experts)
|
| 190 |
+
f = np.repeat(np.asarray(weights, np.float32), self.slices).view(np.int32) # (per 512-slice)
|
| 191 |
+
s0, k = 0, self.slices
|
| 192 |
for g, n in enumerate(self.groups):
|
| 193 |
+
self.mh[g, 0:2, H_XS:H_XS + n * k] = f[s0 * k:(s0 + n) * k]
|
| 194 |
s0 += n
|
| 195 |
self.b.dev.sync(self.mh_sub, True)
|
| 196 |
|
|
|
|
| 241 |
ins = it[0]
|
| 242 |
if ins is self.prog[self.kind_of(prev)].bo if prev is not None else False:
|
| 243 |
key = ("moe", prev)
|
| 244 |
+
elif l < b.L and it[4] is b.kv.get(l):
|
| 245 |
key = ("at", l, tcap)
|
| 246 |
elif l < b.L:
|
| 247 |
key = ("dn", l)
|
|
|
|
| 280 |
pg = self.prog[self.kind_of(prev)]
|
| 281 |
items.append((pg.bo, pg.n, b.moe_bo(prev), b.actbo, b.dummy, b.dummy))
|
| 282 |
if l < b.L:
|
| 283 |
+
if self.lfm and b.types[l] == "conv":
|
| 284 |
+
sc = b.ins["scm" if l < self.nd else "sc"]
|
| 285 |
+
items.append((sc[0], sc[1], b.wbo[l], b.actbo, b.st_cur(l), b.dummy))
|
| 286 |
+
elif b.types[l] == "linear_attention":
|
| 287 |
items.append((dn[0], dn[1], b.wbo[l], b.actbo, b.st_cur(l), b.dummy))
|
| 288 |
else:
|
| 289 |
subs.append(b.kv_at(l, T))
|
|
|
|
| 301 |
b.dev.free(s)
|
| 302 |
if b.experts is not None:
|
| 303 |
b.experts.done()
|
| 304 |
+
if l < self.nd: # ([1.0] a dense layer: its MLP was in the mixer plan)
|
| 305 |
+
if self.trace is not None:
|
| 306 |
+
self.trace.append((T, l, self.read_h(), [], []))
|
| 307 |
+
prev = None
|
| 308 |
+
continue
|
| 309 |
if l < b.L:
|
| 310 |
t1 = time.perf_counter()
|
| 311 |
t2 = time.perf_counter()
|
pyproject.toml
CHANGED
|
@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
|
|
| 4 |
|
| 5 |
[project]
|
| 6 |
name = "onwx"
|
| 7 |
-
version = "0.
|
| 8 |
description = "LLMs entirely on the AMD Ryzen AI NPU (XDNA2)"
|
| 9 |
readme = "README.md"
|
| 10 |
license = {text = "Apache-2.0"}
|
|
|
|
| 4 |
|
| 5 |
[project]
|
| 6 |
name = "onwx"
|
| 7 |
+
version = "1.0.0"
|
| 8 |
description = "LLMs entirely on the AMD Ryzen AI NPU (XDNA2)"
|
| 9 |
readme = "README.md"
|
| 10 |
license = {text = "Apache-2.0"}
|
tools/lfx_glue/build.bat
ADDED
|
@@ -0,0 +1,4 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
@echo off
|
| 2 |
+
call "C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Auxiliary\Build\vcvars64.bat" >nul
|
| 3 |
+
cd /d "%~dp0"
|
| 4 |
+
cl /nologo /O2 /arch:AVX2 /fp:precise /EHsc /MD /std:c++17 /LD lfx_glue.cpp /Fe:lfx_glue.dll
|
tools/lfx_glue/build.sh
ADDED
|
@@ -0,0 +1,4 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
#!/bin/bash
|
| 2 |
+
# (WSL Ubuntu-22.04 gcc 11: glibc 2.35 - runs on newer distros too)
|
| 3 |
+
cd "$(dirname "$0")"
|
| 4 |
+
g++ -O3 -mavx2 -mfma -std=c++17 -shared -fPIC -fvisibility=hidden -pthread lfx_glue.cpp -o liblfx_glue.so
|
tools/lfx_glue/lfx_glue.cpp
ADDED
|
@@ -0,0 +1,621 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
// lfx_glue: the CPU side of onwx's LFM2.5 prefill (npu_lfx.py) - the rotation of the activations (x' = H (s * x) *
|
| 2 |
+
// scale per block, H the unnormalized Hadamard) and the A tiles the NPU GEMM (pfx_mx.cc) reads: bfp16ebs8, per token
|
| 3 |
+
// and 8 inputs an exponent byte + 8 int8, laid out [segment of 64 rows][K-step of 128][zp 4][i 16][h 2][8 rows x 9 B]
|
| 4 |
+
// (row r of a segment = z * 8 + a, z = 2 zp + h; i: the step's group of 8 inputs). Plain C++ (the compiler
|
| 5 |
+
// vectorizes), threads over rows.
|
| 6 |
+
// lx_rot(M, K, x, s, blk, scale, out) rows x [M][K] -> out (may be x)
|
| 7 |
+
// lx_pack(M, K, x, idx, n, dup, X) n rows (row j = x[idx[j]], idx < 0: zeros; idx null: row j) x dup
|
| 8 |
+
// copies along K -> X [ceil(n / 64)][K dup / 128][9216 B]
|
| 9 |
+
// lx_rows(Y, nseg, nq, rmap, Mc, out) a dispatch's job outputs (bf16 [seg][q][c 8][r 4][64][32]) -> out
|
| 10 |
+
// fp32 [nseg * 64][16 Mc] in the op's row order (core g, rb: rmap 0
|
| 11 |
+
// r = 2 side + rb % 2, 1: r = 2 (rb % 2) + side; g = 2c + side)
|
| 12 |
+
// lx_rows2(Y, nseg, nq, rmap, Mc, cmap, out, ostride, nrows) the same into out's columns cmap[op column]
|
| 13 |
+
// lx_eact(Y, nseg, out) the experts' up dispatch (nq 3, Mc 192: per core [gate 96 | up 96])
|
| 14 |
+
// -> out fp32 [nseg * 64][1536] = silu(gate) * up
|
| 15 |
+
// lx_threads(n) the threads (0: the hardware's)
|
| 16 |
+
#include <algorithm>
|
| 17 |
+
#include <immintrin.h>
|
| 18 |
+
#include <atomic>
|
| 19 |
+
#include <chrono>
|
| 20 |
+
#include <condition_variable>
|
| 21 |
+
#include <functional>
|
| 22 |
+
#include <mutex>
|
| 23 |
+
#include <cmath>
|
| 24 |
+
#include <cstdint>
|
| 25 |
+
#include <cstring>
|
| 26 |
+
#include <thread>
|
| 27 |
+
#include <vector>
|
| 28 |
+
#ifdef _WIN32
|
| 29 |
+
#define API extern "C" __declspec(dllexport)
|
| 30 |
+
#else
|
| 31 |
+
#define API extern "C" __attribute__((visibility("default")))
|
| 32 |
+
#endif
|
| 33 |
+
|
| 34 |
+
static int g_threads = 0;
|
| 35 |
+
|
| 36 |
+
// a persistent pool: par(n, f) runs f(0 .. n - 1) on the workers and the caller (spawning threads every call cost
|
| 37 |
+
// ~1 ms - a prefill makes thousands of calls)
|
| 38 |
+
struct Pool {
|
| 39 |
+
std::vector<std::thread> th;
|
| 40 |
+
std::mutex mu;
|
| 41 |
+
std::condition_variable cv, done;
|
| 42 |
+
std::function<void(int)> f;
|
| 43 |
+
std::atomic<int> next{0}, left{0}, gen_a{0};
|
| 44 |
+
int n = 0, gen = 0, active = 0;
|
| 45 |
+
bool stop = false;
|
| 46 |
+
void start(int t) {
|
| 47 |
+
for (int k = 0; k < t; k++)
|
| 48 |
+
th.emplace_back([this] {
|
| 49 |
+
int seen = 0;
|
| 50 |
+
for (;;) {
|
| 51 |
+
// (spin ~3 ms before sleeping: a prefill calls every few ms, and waking 30 sleeping threads took ~1-2 ms)
|
| 52 |
+
auto t0 = std::chrono::steady_clock::now();
|
| 53 |
+
while (gen_a.load(std::memory_order_acquire) == seen && !stop &&
|
| 54 |
+
std::chrono::steady_clock::now() - t0 < std::chrono::microseconds(3000))
|
| 55 |
+
std::this_thread::yield();
|
| 56 |
+
{
|
| 57 |
+
std::unique_lock<std::mutex> lk(mu);
|
| 58 |
+
cv.wait(lk, [&] { return stop || gen != seen; });
|
| 59 |
+
if (stop)
|
| 60 |
+
return;
|
| 61 |
+
seen = gen;
|
| 62 |
+
active++; // (checked in under the lock: run() waits for it to leave)
|
| 63 |
+
}
|
| 64 |
+
work();
|
| 65 |
+
{
|
| 66 |
+
std::lock_guard<std::mutex> lk(mu);
|
| 67 |
+
if (--active == 0)
|
| 68 |
+
done.notify_all();
|
| 69 |
+
}
|
| 70 |
+
}
|
| 71 |
+
});
|
| 72 |
+
}
|
| 73 |
+
void work() {
|
| 74 |
+
for (int i; (i = next.fetch_add(1)) < n;) {
|
| 75 |
+
f(i);
|
| 76 |
+
if (left.fetch_sub(1) == 1) {
|
| 77 |
+
std::lock_guard<std::mutex> lk(mu);
|
| 78 |
+
done.notify_all();
|
| 79 |
+
}
|
| 80 |
+
}
|
| 81 |
+
}
|
| 82 |
+
void run(int cnt, std::function<void(int)> fn) {
|
| 83 |
+
{
|
| 84 |
+
std::lock_guard<std::mutex> lk(mu);
|
| 85 |
+
f = std::move(fn);
|
| 86 |
+
n = cnt;
|
| 87 |
+
next = 0;
|
| 88 |
+
left = cnt;
|
| 89 |
+
gen++;
|
| 90 |
+
gen_a.store(gen, std::memory_order_release);
|
| 91 |
+
}
|
| 92 |
+
cv.notify_all();
|
| 93 |
+
work();
|
| 94 |
+
std::unique_lock<std::mutex> lk(mu); // (every item done and no worker still inside work(): the
|
| 95 |
+
done.wait(lk, [&] { return left.load() == 0 && active == 0; }); // next run may replace f / n / next)
|
| 96 |
+
}
|
| 97 |
+
};
|
| 98 |
+
static Pool *g_pool = nullptr;
|
| 99 |
+
static std::mutex g_pool_mu;
|
| 100 |
+
|
| 101 |
+
template <class F>
|
| 102 |
+
static void par(int n, F f) {
|
| 103 |
+
if (n <= 1) {
|
| 104 |
+
for (int i = 0; i < n; i++)
|
| 105 |
+
f(i);
|
| 106 |
+
return;
|
| 107 |
+
}
|
| 108 |
+
std::lock_guard<std::mutex> lk(g_pool_mu); // (one call at a time)
|
| 109 |
+
if (!g_pool) {
|
| 110 |
+
int t = g_threads > 0 ? g_threads : (int)std::max(1u, std::thread::hardware_concurrency());
|
| 111 |
+
g_pool = new Pool();
|
| 112 |
+
g_pool->start(std::max(0, t - 1));
|
| 113 |
+
}
|
| 114 |
+
g_pool->run(n, std::function<void(int)>(f));
|
| 115 |
+
}
|
| 116 |
+
|
| 117 |
+
static void fwht(float *x, int n) {
|
| 118 |
+
for (int h = 1; h < n; h <<= 1)
|
| 119 |
+
for (int i = 0; i < n; i += 2 * h)
|
| 120 |
+
for (int j = i; j < i + h; j++) {
|
| 121 |
+
float a = x[j], b = x[j + h];
|
| 122 |
+
x[j] = a + b;
|
| 123 |
+
x[j + h] = a - b;
|
| 124 |
+
}
|
| 125 |
+
}
|
| 126 |
+
|
| 127 |
+
API void lx_threads(int n) { g_threads = n; }
|
| 128 |
+
|
| 129 |
+
API int lx_rot(int M, int K, const float *x, const float *s, int blk, float scale, float *out) {
|
| 130 |
+
if (blk <= 0 || K % blk)
|
| 131 |
+
return -1;
|
| 132 |
+
par(M, [&](int m) {
|
| 133 |
+
const float *xr = x + (size_t)m * K;
|
| 134 |
+
float *o = out + (size_t)m * K;
|
| 135 |
+
for (int k = 0; k < K; k++)
|
| 136 |
+
o[k] = xr[k] * s[k];
|
| 137 |
+
for (int c = 0; c < K; c += blk) {
|
| 138 |
+
fwht(o + c, blk);
|
| 139 |
+
for (int k = c; k < c + blk; k++)
|
| 140 |
+
o[k] *= scale;
|
| 141 |
+
}
|
| 142 |
+
});
|
| 143 |
+
return 0;
|
| 144 |
+
}
|
| 145 |
+
|
| 146 |
+
static inline void bfp8(const float *v, uint8_t *o) { // (AVX2, as glue_cpu.cpp's)
|
| 147 |
+
__m256 x = _mm256_loadu_ps(v);
|
| 148 |
+
__m256 ax = _mm256_andnot_ps(_mm256_set1_ps(-0.0f), x);
|
| 149 |
+
__m256 m1 = _mm256_max_ps(ax, _mm256_permute2f128_ps(ax, ax, 1));
|
| 150 |
+
m1 = _mm256_max_ps(m1, _mm256_shuffle_ps(m1, m1, 0x4E));
|
| 151 |
+
m1 = _mm256_max_ps(m1, _mm256_shuffle_ps(m1, m1, 0xB1));
|
| 152 |
+
uint32_t mb = (uint32_t)_mm_cvtsi128_si32(_mm_castps_si128(_mm256_castps256_ps128(m1)));
|
| 153 |
+
int E = (int)(mb >> 23);
|
| 154 |
+
if (E == 0) {
|
| 155 |
+
std::memset(o, 0, 9);
|
| 156 |
+
return;
|
| 157 |
+
}
|
| 158 |
+
int e2 = 133 - E + 127;
|
| 159 |
+
__m256 sc = (e2 >= 1 && e2 <= 254) ? _mm256_castsi256_ps(_mm256_set1_epi32(e2 << 23))
|
| 160 |
+
: _mm256_set1_ps(std::ldexp(1.0f, 133 - E));
|
| 161 |
+
__m256 y = _mm256_mul_ps(x, sc);
|
| 162 |
+
__m256 r = _mm256_round_ps(y, _MM_FROUND_TO_NEAREST_INT | _MM_FROUND_NO_EXC);
|
| 163 |
+
if (_mm256_movemask_ps(_mm256_cmp_ps(r, _mm256_set1_ps(127.0f), _CMP_GT_OQ))) {
|
| 164 |
+
E++;
|
| 165 |
+
y = _mm256_mul_ps(y, _mm256_set1_ps(0.5f));
|
| 166 |
+
r = _mm256_round_ps(y, _MM_FROUND_TO_NEAREST_INT | _MM_FROUND_NO_EXC);
|
| 167 |
+
}
|
| 168 |
+
r = _mm256_min_ps(_mm256_max_ps(r, _mm256_set1_ps(-128.0f)), _mm256_set1_ps(127.0f));
|
| 169 |
+
__m256i iv = _mm256_cvtps_epi32(r);
|
| 170 |
+
__m128i lo = _mm256_castsi256_si128(iv), hi = _mm256_extracti128_si256(iv, 1);
|
| 171 |
+
__m128i p16 = _mm_packs_epi32(lo, hi);
|
| 172 |
+
__m128i p8 = _mm_packs_epi16(p16, p16);
|
| 173 |
+
o[0] = (uint8_t)E;
|
| 174 |
+
_mm_storel_epi64((__m128i *)(o + 1), p8);
|
| 175 |
+
}
|
| 176 |
+
|
| 177 |
+
API int lx_pack2(int M, int K, const float *x, const int32_t *idx, int n, int dup, uint8_t *X, int hdr);
|
| 178 |
+
API int lx_pack(int M, int K, const float *x, const int32_t *idx, int n, int dup, uint8_t *X) {
|
| 179 |
+
return lx_pack2(M, K, x, idx, n, dup, X, 0);
|
| 180 |
+
}
|
| 181 |
+
// lx_pack with hdr: every segment's tiles after a header tile whose word 0 is the K-steps (lfx2's cores read it)
|
| 182 |
+
API int lx_pack2(int M, int K, const float *x, const int32_t *idx, int n, int dup, uint8_t *X, int hdr) {
|
| 183 |
+
if (K % 128)
|
| 184 |
+
return -1;
|
| 185 |
+
int KS = K * dup / 128, nseg = (n + 63) / 64;
|
| 186 |
+
size_t SS = (size_t)(KS + (hdr ? 1 : 0)) * 2304, OFF = hdr ? 2304 : 0; // (words a segment, its tiles' offset)
|
| 187 |
+
if (hdr)
|
| 188 |
+
for (int s = 0; s < nseg; s++) {
|
| 189 |
+
int32_t *h = (int32_t *)(X + s * SS * 4);
|
| 190 |
+
std::memset(h, 0, 64);
|
| 191 |
+
h[0] = KS;
|
| 192 |
+
}
|
| 193 |
+
std::vector<float> zero(K, 0.0f);
|
| 194 |
+
par(nseg * 8, [&](int u) { // (a unit: 8 rows of a segment)
|
| 195 |
+
int seg = u / 8, z = u % 8;
|
| 196 |
+
for (int a = 0; a < 8; a++) {
|
| 197 |
+
int j = seg * 64 + z * 8 + a;
|
| 198 |
+
const float *row = zero.data();
|
| 199 |
+
if (j < n) {
|
| 200 |
+
int src = idx ? idx[j] : j;
|
| 201 |
+
if (src >= 0 && src < M)
|
| 202 |
+
row = x + (size_t)src * K;
|
| 203 |
+
}
|
| 204 |
+
for (int ks = 0; ks < KS; ks++)
|
| 205 |
+
for (int i = 0; i < 16; i++) {
|
| 206 |
+
size_t w = seg * SS + OFF + (size_t)ks * 2304 + (z >> 1) * 576 + i * 36 + (z & 1) * 18;
|
| 207 |
+
bfp8(row + ((ks * 128) % K) + i * 8, X + w * 4 + a * 9);
|
| 208 |
+
}
|
| 209 |
+
}
|
| 210 |
+
});
|
| 211 |
+
return 0;
|
| 212 |
+
}
|
| 213 |
+
|
| 214 |
+
static inline float bf(uint16_t v) {
|
| 215 |
+
uint32_t u = (uint32_t)v << 16;
|
| 216 |
+
float f;
|
| 217 |
+
std::memcpy(&f, &u, 4);
|
| 218 |
+
return f;
|
| 219 |
+
}
|
| 220 |
+
|
| 221 |
+
API int lx_rows2(const uint16_t *Y, int nseg, int nq, int rmap, int Mc, const int32_t *cmap, float *out, int ostride,
|
| 222 |
+
int nrows) {
|
| 223 |
+
par(nseg * 8, [&](int u) { // (lx_rows into out [rows][ostride]: op column j -> cmap[j]; rows
|
| 224 |
+
int s = u / 8, c = u % 8; // past nrows dropped)
|
| 225 |
+
for (int q = 0; q < nq; q++)
|
| 226 |
+
for (int r = 0; r < 4; r++) {
|
| 227 |
+
int side = rmap ? r % 2 : r / 2, rr = rmap ? r / 2 : r % 2;
|
| 228 |
+
int g = 2 * c + side, rb = 2 * q + rr;
|
| 229 |
+
const uint16_t *y = Y + ((((size_t)s * nq + q) * 8 + c) * 4 + r) * 2048;
|
| 230 |
+
const int32_t *cm = cmap + g * Mc + rb * 32;
|
| 231 |
+
for (int t = 0; t < 64; t++) {
|
| 232 |
+
int row = s * 64 + t;
|
| 233 |
+
if (row >= nrows)
|
| 234 |
+
break;
|
| 235 |
+
float *o = out + (size_t)row * ostride;
|
| 236 |
+
for (int i = 0; i < 32; i++)
|
| 237 |
+
o[cm[i]] = bf(y[t * 32 + i]);
|
| 238 |
+
}
|
| 239 |
+
}
|
| 240 |
+
});
|
| 241 |
+
return 0;
|
| 242 |
+
}
|
| 243 |
+
|
| 244 |
+
API int lx_rows(const uint16_t *Y, int nseg, int nq, int rmap, int Mc, float *out) {
|
| 245 |
+
int W = 16 * Mc;
|
| 246 |
+
par(nseg * 8, [&](int u) { // (a unit: a segment's column c)
|
| 247 |
+
int s = u / 8, c = u % 8;
|
| 248 |
+
for (int q = 0; q < nq; q++)
|
| 249 |
+
for (int r = 0; r < 4; r++) {
|
| 250 |
+
int side = rmap ? r % 2 : r / 2, rr = rmap ? r / 2 : r % 2;
|
| 251 |
+
int g = 2 * c + side, rb = 2 * q + rr;
|
| 252 |
+
const uint16_t *y = Y + ((((size_t)s * nq + q) * 8 + c) * 4 + r) * 2048;
|
| 253 |
+
for (int t = 0; t < 64; t++) {
|
| 254 |
+
float *o = out + ((size_t)s * 64 + t) * W + g * Mc + rb * 32;
|
| 255 |
+
for (int i = 0; i < 32; i++)
|
| 256 |
+
o[i] = bf(y[t * 32 + i]);
|
| 257 |
+
}
|
| 258 |
+
}
|
| 259 |
+
});
|
| 260 |
+
return 0;
|
| 261 |
+
}
|
| 262 |
+
|
| 263 |
+
API int lx_eact(const uint16_t *Y, int nseg, float *out) {
|
| 264 |
+
par(nseg * 8, [&](int u) {
|
| 265 |
+
int s = u / 8, c = u % 8;
|
| 266 |
+
for (int side = 0; side < 2; side++) {
|
| 267 |
+
int g = 2 * c + side;
|
| 268 |
+
for (int k = 0; k < 3; k++) { // (gate rb k with up rb 3 + k: rows 96 g + 32 k ..)
|
| 269 |
+
int qg = k / 2, rg = 2 * side + k % 2, qu = (k + 3) / 2, ru = 2 * side + (k + 3) % 2;
|
| 270 |
+
const uint16_t *yg = Y + ((((size_t)s * 3 + qg) * 8 + c) * 4 + rg) * 2048;
|
| 271 |
+
const uint16_t *yu = Y + ((((size_t)s * 3 + qu) * 8 + c) * 4 + ru) * 2048;
|
| 272 |
+
for (int t = 0; t < 64; t++) {
|
| 273 |
+
float *o = out + ((size_t)s * 64 + t) * 1536 + g * 96 + k * 32;
|
| 274 |
+
for (int i = 0; i < 32; i++) {
|
| 275 |
+
float a = bf(yg[t * 32 + i]), b = bf(yu[t * 32 + i]);
|
| 276 |
+
o[i] = a / (1.0f + std::exp(-a)) * b;
|
| 277 |
+
}
|
| 278 |
+
}
|
| 279 |
+
}
|
| 280 |
+
}
|
| 281 |
+
});
|
| 282 |
+
return 0;
|
| 283 |
+
}
|
| 284 |
+
|
| 285 |
+
// causal attention of M queries at positions T0 .. T0 + M - 1 over keys 0 .. T0 + M - 1 (GQA: 32 q heads, 8 kv heads
|
| 286 |
+
// of 64): q [M][32][64] (normed, roped), K / V [T][8][64], out [M][32][64]; scale 1 / 8. A unit: one head's QB
|
| 287 |
+
// queries; keys in blocks of KB: the scores, one rescale of the accumulators a block (flash attention), AVX2.
|
| 288 |
+
static constexpr int QB = 16, KB = 64;
|
| 289 |
+
static inline float dot64(const float *a, const float *b) {
|
| 290 |
+
__m256 s0 = _mm256_setzero_ps(), s1 = _mm256_setzero_ps();
|
| 291 |
+
for (int d = 0; d < 64; d += 16) {
|
| 292 |
+
s0 = _mm256_fmadd_ps(_mm256_loadu_ps(a + d), _mm256_loadu_ps(b + d), s0);
|
| 293 |
+
s1 = _mm256_fmadd_ps(_mm256_loadu_ps(a + d + 8), _mm256_loadu_ps(b + d + 8), s1);
|
| 294 |
+
}
|
| 295 |
+
s0 = _mm256_add_ps(s0, s1);
|
| 296 |
+
__m128 x = _mm_add_ps(_mm256_castps256_ps128(s0), _mm256_extractf128_ps(s0, 1));
|
| 297 |
+
x = _mm_add_ps(x, _mm_movehl_ps(x, x));
|
| 298 |
+
x = _mm_add_ss(x, _mm_shuffle_ps(x, x, 1));
|
| 299 |
+
return _mm_cvtss_f32(x);
|
| 300 |
+
}
|
| 301 |
+
static inline __m256 exp8(__m256 x) { // e^x, x <= 0 (Cephes-style: 2^n * poly, ~1e-7 relative)
|
| 302 |
+
x = _mm256_max_ps(x, _mm256_set1_ps(-87.0f));
|
| 303 |
+
__m256 fx = _mm256_round_ps(_mm256_mul_ps(x, _mm256_set1_ps(1.44269504088896341f)), _MM_FROUND_TO_NEAREST_INT);
|
| 304 |
+
__m256 r = _mm256_fnmadd_ps(fx, _mm256_set1_ps(0.693359375f), x);
|
| 305 |
+
r = _mm256_fnmadd_ps(fx, _mm256_set1_ps(-2.12194440e-4f), r);
|
| 306 |
+
__m256 y = _mm256_set1_ps(1.9875691500e-4f);
|
| 307 |
+
y = _mm256_fmadd_ps(y, r, _mm256_set1_ps(1.3981999507e-3f));
|
| 308 |
+
y = _mm256_fmadd_ps(y, r, _mm256_set1_ps(8.3334519073e-3f));
|
| 309 |
+
y = _mm256_fmadd_ps(y, r, _mm256_set1_ps(4.1665795894e-2f));
|
| 310 |
+
y = _mm256_fmadd_ps(y, r, _mm256_set1_ps(1.6666665459e-1f));
|
| 311 |
+
y = _mm256_fmadd_ps(y, r, _mm256_set1_ps(5.0000001201e-1f));
|
| 312 |
+
y = _mm256_fmadd_ps(y, _mm256_mul_ps(r, r), _mm256_add_ps(r, _mm256_set1_ps(1.0f)));
|
| 313 |
+
__m256i e = _mm256_slli_epi32(_mm256_add_epi32(_mm256_cvtps_epi32(fx), _mm256_set1_epi32(127)), 23);
|
| 314 |
+
return _mm256_mul_ps(y, _mm256_castsi256_ps(e));
|
| 315 |
+
}
|
| 316 |
+
|
| 317 |
+
API int lx_attn(int M, int T0, const float *q, const float *K, const float *V, float *out) {
|
| 318 |
+
// a unit: one kv head's 4 q heads x QB / 4 queries (GQA: its keys / values read once for the 4 heads); a block of
|
| 319 |
+
// KB keys transposed ([d][key]): the scores 8 keys a vector, no horizontal sums; e^x vectorized
|
| 320 |
+
constexpr int NQ = QB / 4;
|
| 321 |
+
int nqb = (M + NQ - 1) / NQ;
|
| 322 |
+
par(8 * nqb, [&](int u) {
|
| 323 |
+
int kv = u % 8, qb = nqb - 1 - u / 8;
|
| 324 |
+
int i0 = qb * NQ, i1 = std::min(M, i0 + NQ), nq = (i1 - i0) * 4;
|
| 325 |
+
alignas(32) float acc[QB][64], Kt[64][KB], S[QB][KB];
|
| 326 |
+
float m[QB], l[QB];
|
| 327 |
+
const float *qr[QB];
|
| 328 |
+
for (int a = 0; a < nq; a++) {
|
| 329 |
+
m[a] = -1e30f;
|
| 330 |
+
l[a] = 0.0f;
|
| 331 |
+
qr[a] = q + ((size_t)(i0 + a / 4) * 32 + 4 * kv + a % 4) * 64;
|
| 332 |
+
for (int d = 0; d < 64; d++)
|
| 333 |
+
acc[a][d] = 0.0f;
|
| 334 |
+
}
|
| 335 |
+
int T = T0 + i1;
|
| 336 |
+
for (int t0 = 0; t0 < T; t0 += KB) {
|
| 337 |
+
int nk = std::min(KB, T - t0);
|
| 338 |
+
for (int j = 0; j < KB; j++) { // (the block transposed; past nk: zeros)
|
| 339 |
+
const float *k = K + ((size_t)(t0 + j) * 8 + kv) * 64;
|
| 340 |
+
if (j < nk)
|
| 341 |
+
for (int d = 0; d < 64; d++)
|
| 342 |
+
Kt[d][j] = k[d];
|
| 343 |
+
else
|
| 344 |
+
for (int d = 0; d < 64; d++)
|
| 345 |
+
Kt[d][j] = 0.0f;
|
| 346 |
+
}
|
| 347 |
+
for (int a = 0; a < nq; a++) {
|
| 348 |
+
__m256 s0 = _mm256_setzero_ps(), s1 = s0, s2 = s0, s3 = s0, s4 = s0, s5 = s0, s6 = s0, s7 = s0;
|
| 349 |
+
const float *qq = qr[a];
|
| 350 |
+
for (int d = 0; d < 64; d++) {
|
| 351 |
+
__m256 qd = _mm256_set1_ps(qq[d] * 0.125f);
|
| 352 |
+
const float *kr = Kt[d];
|
| 353 |
+
s0 = _mm256_fmadd_ps(qd, _mm256_load_ps(kr), s0);
|
| 354 |
+
s1 = _mm256_fmadd_ps(qd, _mm256_load_ps(kr + 8), s1);
|
| 355 |
+
s2 = _mm256_fmadd_ps(qd, _mm256_load_ps(kr + 16), s2);
|
| 356 |
+
s3 = _mm256_fmadd_ps(qd, _mm256_load_ps(kr + 24), s3);
|
| 357 |
+
s4 = _mm256_fmadd_ps(qd, _mm256_load_ps(kr + 32), s4);
|
| 358 |
+
s5 = _mm256_fmadd_ps(qd, _mm256_load_ps(kr + 40), s5);
|
| 359 |
+
s6 = _mm256_fmadd_ps(qd, _mm256_load_ps(kr + 48), s6);
|
| 360 |
+
s7 = _mm256_fmadd_ps(qd, _mm256_load_ps(kr + 56), s7);
|
| 361 |
+
}
|
| 362 |
+
float *sr = S[a];
|
| 363 |
+
_mm256_store_ps(sr, s0);
|
| 364 |
+
_mm256_store_ps(sr + 8, s1);
|
| 365 |
+
_mm256_store_ps(sr + 16, s2);
|
| 366 |
+
_mm256_store_ps(sr + 24, s3);
|
| 367 |
+
_mm256_store_ps(sr + 32, s4);
|
| 368 |
+
_mm256_store_ps(sr + 40, s5);
|
| 369 |
+
_mm256_store_ps(sr + 48, s6);
|
| 370 |
+
_mm256_store_ps(sr + 56, s7);
|
| 371 |
+
int lim = std::min(nk - 1, T0 + i0 + a / 4 - t0); // (the last key the query sees)
|
| 372 |
+
float mx = m[a];
|
| 373 |
+
for (int j = 0; j <= lim; j++)
|
| 374 |
+
mx = std::max(mx, sr[j]);
|
| 375 |
+
for (int j = std::max(lim + 1, 0); j < KB; j++) // (lim < 0: a block past the query - all masked)
|
| 376 |
+
sr[j] = -1e30f;
|
| 377 |
+
float c = std::exp(m[a] - mx);
|
| 378 |
+
__m256 mv = _mm256_set1_ps(mx), sum = _mm256_setzero_ps();
|
| 379 |
+
for (int j = 0; j < KB; j += 8) {
|
| 380 |
+
__m256 pv = exp8(_mm256_sub_ps(_mm256_load_ps(sr + j), mv));
|
| 381 |
+
_mm256_store_ps(sr + j, pv);
|
| 382 |
+
sum = _mm256_add_ps(sum, pv);
|
| 383 |
+
}
|
| 384 |
+
alignas(32) float sv[8];
|
| 385 |
+
_mm256_store_ps(sv, sum);
|
| 386 |
+
l[a] = l[a] * c + sv[0] + sv[1] + sv[2] + sv[3] + sv[4] + sv[5] + sv[6] + sv[7];
|
| 387 |
+
m[a] = mx;
|
| 388 |
+
__m256 cv = _mm256_set1_ps(c);
|
| 389 |
+
__m256 a0 = _mm256_mul_ps(_mm256_load_ps(acc[a]), cv), a1 = _mm256_mul_ps(_mm256_load_ps(acc[a] + 8), cv);
|
| 390 |
+
__m256 a2 = _mm256_mul_ps(_mm256_load_ps(acc[a] + 16), cv), a3 = _mm256_mul_ps(_mm256_load_ps(acc[a] + 24), cv);
|
| 391 |
+
__m256 a4 = _mm256_mul_ps(_mm256_load_ps(acc[a] + 32), cv), a5 = _mm256_mul_ps(_mm256_load_ps(acc[a] + 40), cv);
|
| 392 |
+
__m256 a6 = _mm256_mul_ps(_mm256_load_ps(acc[a] + 48), cv), a7 = _mm256_mul_ps(_mm256_load_ps(acc[a] + 56), cv);
|
| 393 |
+
for (int j = 0; j <= lim; j++) {
|
| 394 |
+
const float *v = V + ((size_t)(t0 + j) * 8 + kv) * 64;
|
| 395 |
+
__m256 pv = _mm256_set1_ps(sr[j]);
|
| 396 |
+
a0 = _mm256_fmadd_ps(pv, _mm256_loadu_ps(v), a0);
|
| 397 |
+
a1 = _mm256_fmadd_ps(pv, _mm256_loadu_ps(v + 8), a1);
|
| 398 |
+
a2 = _mm256_fmadd_ps(pv, _mm256_loadu_ps(v + 16), a2);
|
| 399 |
+
a3 = _mm256_fmadd_ps(pv, _mm256_loadu_ps(v + 24), a3);
|
| 400 |
+
a4 = _mm256_fmadd_ps(pv, _mm256_loadu_ps(v + 32), a4);
|
| 401 |
+
a5 = _mm256_fmadd_ps(pv, _mm256_loadu_ps(v + 40), a5);
|
| 402 |
+
a6 = _mm256_fmadd_ps(pv, _mm256_loadu_ps(v + 48), a6);
|
| 403 |
+
a7 = _mm256_fmadd_ps(pv, _mm256_loadu_ps(v + 56), a7);
|
| 404 |
+
}
|
| 405 |
+
_mm256_store_ps(acc[a], a0);
|
| 406 |
+
_mm256_store_ps(acc[a] + 8, a1);
|
| 407 |
+
_mm256_store_ps(acc[a] + 16, a2);
|
| 408 |
+
_mm256_store_ps(acc[a] + 24, a3);
|
| 409 |
+
_mm256_store_ps(acc[a] + 32, a4);
|
| 410 |
+
_mm256_store_ps(acc[a] + 40, a5);
|
| 411 |
+
_mm256_store_ps(acc[a] + 48, a6);
|
| 412 |
+
_mm256_store_ps(acc[a] + 56, a7);
|
| 413 |
+
}
|
| 414 |
+
}
|
| 415 |
+
for (int a = 0; a < nq; a++) {
|
| 416 |
+
float *o = out + ((size_t)(i0 + a / 4) * 32 + 4 * kv + a % 4) * 64;
|
| 417 |
+
float il = 1.0f / l[a];
|
| 418 |
+
for (int d = 0; d < 64; d++)
|
| 419 |
+
o[d] = acc[a][d] * il;
|
| 420 |
+
}
|
| 421 |
+
});
|
| 422 |
+
return 0;
|
| 423 |
+
}
|
| 424 |
+
|
| 425 |
+
// h[dst[j]] += w[j] * y[j] for the n rows of y [n][W] (dst < 0: skipped); threads over column blocks (no races)
|
| 426 |
+
API int lx_scatter(int n, int W, const float *y, const int32_t *dst, const float *w, float *h) {
|
| 427 |
+
int nb = (W + 255) / 256;
|
| 428 |
+
par(nb, [&](int b) {
|
| 429 |
+
int c0 = b * 256, c1 = std::min(W, c0 + 256);
|
| 430 |
+
for (int j = 0; j < n; j++) {
|
| 431 |
+
int t = dst[j];
|
| 432 |
+
if (t < 0)
|
| 433 |
+
continue;
|
| 434 |
+
const float *yr = y + (size_t)j * W;
|
| 435 |
+
float *hr = h + (size_t)t * W, ww = w[j];
|
| 436 |
+
for (int c = c0; c < c1; c++)
|
| 437 |
+
hr[c] += ww * yr[c];
|
| 438 |
+
}
|
| 439 |
+
});
|
| 440 |
+
return 0;
|
| 441 |
+
}
|
| 442 |
+
|
| 443 |
+
// LFM2's short conv over M tokens: rows Y [M][ys] hold B | C | x at columns 0 / 2048 / 4096; taps [2048][3]
|
| 444 |
+
// (out_t = w0 bx_t-2 + w1 bx_t-1 + w2 bx_t); st [3][2048]: in rows 1, 2 = bx at the chunk's t - 2, t - 1; out: the
|
| 445 |
+
// last three bx; y [M][2048] = C * conv
|
| 446 |
+
API int lx_conv(int M, const float *Y, int ys, const float *taps, float *st, float *y) {
|
| 447 |
+
par(128, [&](int g) { // (16 channels a unit; the tokens outer: each row read once)
|
| 448 |
+
int c0 = g * 16;
|
| 449 |
+
float w0[16], w1[16], w2[16], p0[16], p1[16], p2[16];
|
| 450 |
+
for (int i = 0; i < 16; i++) {
|
| 451 |
+
int c = c0 + i;
|
| 452 |
+
w0[i] = taps[c * 3];
|
| 453 |
+
w1[i] = taps[c * 3 + 1];
|
| 454 |
+
w2[i] = taps[c * 3 + 2];
|
| 455 |
+
p0[i] = st[c]; // (bx at t - 3, t - 2, t - 1)
|
| 456 |
+
p2[i] = st[2048 + c];
|
| 457 |
+
p1[i] = st[2 * 2048 + c];
|
| 458 |
+
}
|
| 459 |
+
for (int t = 0; t < M; t++) {
|
| 460 |
+
const float *r = Y + (size_t)t * ys;
|
| 461 |
+
float *o = y + (size_t)t * 2048 + c0;
|
| 462 |
+
for (int i = 0; i < 16; i++) {
|
| 463 |
+
float bx = r[c0 + i] * r[4096 + c0 + i];
|
| 464 |
+
o[i] = r[2048 + c0 + i] * (w0[i] * p2[i] + w1[i] * p1[i] + w2[i] * bx);
|
| 465 |
+
p0[i] = p2[i];
|
| 466 |
+
p2[i] = p1[i];
|
| 467 |
+
p1[i] = bx;
|
| 468 |
+
}
|
| 469 |
+
}
|
| 470 |
+
for (int i = 0; i < 16; i++) {
|
| 471 |
+
int c = c0 + i;
|
| 472 |
+
st[c] = p0[i];
|
| 473 |
+
st[2048 + c] = p2[i];
|
| 474 |
+
st[2 * 2048 + c] = p1[i];
|
| 475 |
+
}
|
| 476 |
+
});
|
| 477 |
+
return 0;
|
| 478 |
+
}
|
| 479 |
+
|
| 480 |
+
// rows x [M][K] -> n = x / rms(x) * w (eps) into nout (may be null), and its rotation H (s * n) * scale per blk into rout
|
| 481 |
+
API int lx_rmsrot(int M, int K, const float *x, const float *w, float eps, const float *s, int blk, float scale,
|
| 482 |
+
float *nout, float *rout) {
|
| 483 |
+
par(M, [&](int m) {
|
| 484 |
+
const float *xr = x + (size_t)m * K;
|
| 485 |
+
double ss = 0.0;
|
| 486 |
+
for (int k = 0; k < K; k++)
|
| 487 |
+
ss += (double)xr[k] * xr[k];
|
| 488 |
+
float r = (float)(1.0 / std::sqrt(ss / K + eps));
|
| 489 |
+
float *n = nout ? nout + (size_t)m * K : nullptr, *o = rout + (size_t)m * K;
|
| 490 |
+
for (int k = 0; k < K; k++) {
|
| 491 |
+
float v = xr[k] * r * w[k];
|
| 492 |
+
if (n)
|
| 493 |
+
n[k] = v;
|
| 494 |
+
o[k] = v * s[k];
|
| 495 |
+
}
|
| 496 |
+
for (int c = 0; c < K; c += blk) {
|
| 497 |
+
fwht(o + c, blk);
|
| 498 |
+
for (int k = c; k < c + blk; k++)
|
| 499 |
+
o[k] *= scale;
|
| 500 |
+
}
|
| 501 |
+
});
|
| 502 |
+
return 0;
|
| 503 |
+
}
|
| 504 |
+
|
| 505 |
+
// qkv rows [M][3072] (q 32 x 64 | k 8 x 64 | v 8 x 64) -> q, k: rms per head (eps) * qn / kn, rope (rotate-half,
|
| 506 |
+
// positions pos0 .., inv_freq [32]); v copied. q [M][32][64], k / v [M][8][64]
|
| 507 |
+
API int lx_qkrope(int M, const float *qkv, const float *qn, const float *kn, float eps, int pos0, const float *inv,
|
| 508 |
+
float *q, float *k, float *v) {
|
| 509 |
+
par(M, [&](int m) {
|
| 510 |
+
const float *r = qkv + (size_t)m * 3072;
|
| 511 |
+
float cs[32], sn[32];
|
| 512 |
+
for (int i = 0; i < 32; i++) {
|
| 513 |
+
double a = (double)(pos0 + m) * inv[i];
|
| 514 |
+
cs[i] = (float)std::cos(a);
|
| 515 |
+
sn[i] = (float)std::sin(a);
|
| 516 |
+
}
|
| 517 |
+
for (int h = 0; h < 40; h++) {
|
| 518 |
+
const float *x = r + h * 64, *w = h < 32 ? qn : kn;
|
| 519 |
+
float *o = h < 32 ? q + ((size_t)m * 32 + h) * 64 : k + ((size_t)m * 8 + h - 32) * 64;
|
| 520 |
+
float ss = 0.0f;
|
| 521 |
+
for (int d = 0; d < 64; d++)
|
| 522 |
+
ss += x[d] * x[d];
|
| 523 |
+
float rr = 1.0f / std::sqrt(ss / 64 + eps), t[64];
|
| 524 |
+
for (int d = 0; d < 64; d++)
|
| 525 |
+
t[d] = x[d] * rr * w[d];
|
| 526 |
+
for (int d = 0; d < 32; d++) {
|
| 527 |
+
o[d] = t[d] * cs[d] - t[d + 32] * sn[d];
|
| 528 |
+
o[d + 32] = t[d + 32] * cs[d] + t[d] * sn[d];
|
| 529 |
+
}
|
| 530 |
+
}
|
| 531 |
+
std::memcpy(v + (size_t)m * 512, r + 2560, 512 * 4);
|
| 532 |
+
});
|
| 533 |
+
return 0;
|
| 534 |
+
}
|
| 535 |
+
|
| 536 |
+
// the decode kernel's int8 cache records (onw_npu.cc at_tok / at_put, AT_KVC 2): positions T0 .. T0 + M - 1 of k / v
|
| 537 |
+
// [M][8][64] into the quad blocks at C ([quad Q][core 4][2176 B], C = the block of quad T0 / 4): per kv head the
|
| 538 |
+
// unnormalized H_64 of the row, int8 at the step s = bf16(max * g / 127 * (1 + 2^-7)) (truncated), g = 1/8
|
| 539 |
+
static inline void q8row(const float *x, int8_t *q, float *sc) {
|
| 540 |
+
float t[64];
|
| 541 |
+
std::memcpy(t, x, 256);
|
| 542 |
+
fwht(t, 64);
|
| 543 |
+
float mx = 0.0f;
|
| 544 |
+
for (int d = 0; d < 64; d++)
|
| 545 |
+
mx = std::max(mx, std::fabs(t[d]));
|
| 546 |
+
float sv = (mx > 0.0f ? mx : 1.0f) * (0.125f / 127.0f * 1.0078125f);
|
| 547 |
+
uint32_t b;
|
| 548 |
+
std::memcpy(&b, &sv, 4);
|
| 549 |
+
b &= 0xFFFF0000u;
|
| 550 |
+
std::memcpy(&sv, &b, 4);
|
| 551 |
+
float f = 0.125f / sv;
|
| 552 |
+
for (int d = 0; d < 64; d++)
|
| 553 |
+
q[d] = (int8_t)std::min(std::max(std::nearbyint(t[d] * f), -128.0f), 127.0f);
|
| 554 |
+
*sc = sv;
|
| 555 |
+
}
|
| 556 |
+
API int lx_kvput(int M, int T0, const float *k, const float *v, uint8_t *C) {
|
| 557 |
+
int Q0 = T0 / 4;
|
| 558 |
+
par(M, [&](int m) {
|
| 559 |
+
int p = T0 + m, Q = p / 4 - Q0, j = p % 4;
|
| 560 |
+
for (int a = 0; a < 4; a++) {
|
| 561 |
+
uint8_t *blk = C + ((size_t)Q * 4 + a) * 2176;
|
| 562 |
+
int8_t kq[128], vq[128];
|
| 563 |
+
float ks[2], vs[2];
|
| 564 |
+
for (int c = 0; c < 2; c++) {
|
| 565 |
+
q8row(k + ((size_t)m * 8 + 2 * a + c) * 64, kq + c * 64, ks + c);
|
| 566 |
+
q8row(v + ((size_t)m * 8 + 2 * a + c) * 64, vq + c * 64, vs + c);
|
| 567 |
+
}
|
| 568 |
+
for (int dt = 0; dt < 16; dt++) {
|
| 569 |
+
std::memcpy(blk + (dt >> 1) * 64 + j * 16 + (dt & 1) * 8, kq + dt * 8, 8);
|
| 570 |
+
std::memcpy(blk + 1024 + dt * 32 + j * 8, vq + dt * 8, 8);
|
| 571 |
+
}
|
| 572 |
+
float *sc = (float *)(blk + 2048);
|
| 573 |
+
for (int c = 0; c < 2; c++) {
|
| 574 |
+
sc[(j * 2 + c) * 2] = ks[c];
|
| 575 |
+
sc[(j * 2 + c) * 2 + 1] = vs[c];
|
| 576 |
+
}
|
| 577 |
+
}
|
| 578 |
+
});
|
| 579 |
+
return 0;
|
| 580 |
+
}
|
| 581 |
+
// positions 0 .. T - 1 of the cache (C at quad 0) -> k / v [T][8][64] in the natural basis (H (q8 s) / 8)
|
| 582 |
+
API int lx_kvget(int T, const uint8_t *C, float *k, float *v) {
|
| 583 |
+
par(T, [&](int p) {
|
| 584 |
+
int Q = p / 4, j = p % 4;
|
| 585 |
+
for (int a = 0; a < 4; a++) {
|
| 586 |
+
const uint8_t *blk = C + ((size_t)Q * 4 + a) * 2176;
|
| 587 |
+
const float *sc = (const float *)(blk + 2048);
|
| 588 |
+
int8_t kq[128], vq[128];
|
| 589 |
+
for (int dt = 0; dt < 16; dt++) {
|
| 590 |
+
std::memcpy(kq + dt * 8, blk + (dt >> 1) * 64 + j * 16 + (dt & 1) * 8, 8);
|
| 591 |
+
std::memcpy(vq + dt * 8, blk + 1024 + dt * 32 + j * 8, 8);
|
| 592 |
+
}
|
| 593 |
+
for (int c = 0; c < 2; c++) {
|
| 594 |
+
float tk[64], tv[64], s1 = sc[(j * 2 + c) * 2], s2 = sc[(j * 2 + c) * 2 + 1];
|
| 595 |
+
for (int d = 0; d < 64; d++) {
|
| 596 |
+
tk[d] = kq[c * 64 + d] * s1;
|
| 597 |
+
tv[d] = vq[c * 64 + d] * s2;
|
| 598 |
+
}
|
| 599 |
+
fwht(tk, 64);
|
| 600 |
+
fwht(tv, 64);
|
| 601 |
+
float *ko = k + ((size_t)p * 8 + 2 * a + c) * 64, *vo = v + ((size_t)p * 8 + 2 * a + c) * 64;
|
| 602 |
+
for (int d = 0; d < 64; d++) {
|
| 603 |
+
ko[d] = tk[d] * 0.125f;
|
| 604 |
+
vo[d] = tv[d] * 0.125f;
|
| 605 |
+
}
|
| 606 |
+
}
|
| 607 |
+
}
|
| 608 |
+
});
|
| 609 |
+
return 0;
|
| 610 |
+
}
|
| 611 |
+
|
| 612 |
+
// out[i][j] = silu(g[i][j]) * u[i][j] (rows of n values; row strides gs / us / os)
|
| 613 |
+
API int lx_silumul(int M, int n, const float *g, int gs, const float *u, int us, float *out, int os) {
|
| 614 |
+
par(M, [&](int m) {
|
| 615 |
+
const float *gr = g + (size_t)m * gs, *ur = u + (size_t)m * us;
|
| 616 |
+
float *o = out + (size_t)m * os;
|
| 617 |
+
for (int j = 0; j < n; j++)
|
| 618 |
+
o[j] = gr[j] / (1.0f + std::exp(-gr[j])) * ur[j];
|
| 619 |
+
});
|
| 620 |
+
return 0;
|
| 621 |
+
}
|