GRepoWatch
Daily at 07:00 UTC+8Subscribe中文
← Back to today’s ranking

ashhart /

TensorFold

LLM Inference Engine for Metal, CUDA and Vulkan.

Visit GitHub repository ↗
Last recorded · 2026-11-15#9

Rank in that snapshot

+53Daily Stars2,149Total Stars

Project overview

LLM Inference Engine for Metal, CUDA and Vulkan.

Recorded Trending signals

ashhart/TensorFold was recorded 2 times between 2026-11-13 and 2026-11-15. Its best recorded rank is #7. The latest saved snapshot reports 53 daily Stars and 2,149 total Stars. These are archive observations, not quality ratings or continuous activity measurements.

Official project documentation

The following is an excerpt from the project’s official README, fetched from GitHub. It may change independently of this archive. RepoWatch has not installed or functionally tested this project.

TensorFold

TensorFold serves language models from a Zig binary on Apple Silicon and NVIDIA GPUs. The engine reads checkpoints, tokenizes requests and runs Metal or CUDA kernels directly. Serving needs no Python or MLX installation.

Building with a coding agent? Have it load the contributor skill before it writes any code.

Contributing with agents

Please use Agent Traffic Control before editing TensorFold so contributors and agents can avoid duplicate work and conflicting changes. Check the live agent radar before choosing an issue and follow CONTRIBUTING.md and the ATC contributor instructions.

Install or upgrade ATC to 0.9.4 or later, then run from an updated TensorFold clone:

npm install -g github:ashhart/ATC
atc check ISSUE --json
atc claim ISSUE --json

Replace ISSUE with the issue number and claim only after a complete check says the work is free. TensorFold enables automatic public check-ins: a successful new claim starts an anonymous session on the radar and returns its session number. Keep that number instead of starting a duplicate session. For work already yours, start a session with atc start ISSUE --json. Add --github-name to claim or start only if you want your verified GitHub name shown; --local keeps a new session on your machine. The board shows the issue, activity state and last check-in; GitHub still associates the verification comment with its author. Check that publication is public; unavailable means the public update failed even if the local claim or session succeeded.

Pulse only while the agent is working, before its five-minute check-in expires:

atc pulse ISSUE SESSION working --json
atc who path/to/file --json
atc overlap --json
atc land ISSUE SESSION --json
atc release ISSUE --json

Use reviewing or blocked when appropriate, land the session when it stops, and resolve overlaps before opening a pull request. Before opening a pull request, have the agent run the pull request skill; agents that support skills can install it as tensorfold-pull-request.

Quick start

Start with Qwen3.5-2B, the small 4-bit checkpoint checked on Metal and a GB10. The commands below download its pinned revision, then serve it as local-model. Model weights have their own licenses. Reviewing or benchmarking TensorFold? Evaluating TensorFold covers pinning, signature checks, a container and the tests that need no GPU.

Mac with Apple Silicon

Install with Homebrew, then start the server:

brew install ashhart/tensorfold/tensorfold
tensorfold pull mlx-community/Qwen3.5-2B-MLX-4bit@93760be4f1f69842a46bc13dbdc0f19e291392a3
tensorfold serve mlx-community/Qwen3.5-2B-MLX-4bit --name local-model --context 8192 --no-thinking

Linux with an NVIDIA GPU

Use the release archive for your CPU: linux-aarch64 on a GB10, linux-x86_64 on an x86_64 host. It needs glibc 2.28 or newer and NVIDIA driver R580 or newer (CUDA 13.0), providing libcuda.so.1. Keep the archive's directories together. Its kernels are built with CUDA 13, so TensorFold refuses an older driver at startup; nvidia-smi shows the installed one. See the CUDA runbook for runtime libraries and supported GPUs.

V=1.2.9
case "$(uname -m)" in
  aarch64|arm64) PLATFORM=linux-aarch64 ;;
  x86_64) PLATFORM=linux-x86_64 ;;
  *) echo 'No release archive for this CPU' >&2; exit 1 ;;
esac
curl -fLO "https://github.com/ashhart/TensorFold/releases/download/v$V/tensorfold-$V-$PLATFORM.tar.gz"
curl -fLO "https://github.com/ashhart/TensorFold/releases/download/v$V/tensorfold-$V-$PLATFORM.tar.gz.sha256"
sha256sum -c "tensorfold-$V-$PLATFORM.tar.gz.sha256"
tar -xzf "tensorfold-$V-$PLATFORM.tar.gz"
export PATH="$PWD/tensorfold-$V-$PLATFORM/bin:$PATH"
tensorfold() { command tensorfold-native "$@"; }
tensorfold pull mlx-community/Qwen3.5-2B-MLX-4bit@93760be4f1f69842a46bc13dbdc0f19e291392a3
tensorfold serve mlx-community/Qwen3.5-2B-MLX-4bit --name local-model --context 8192 --no-thinking

Send a request

Leave the server running. In another terminal, on either platform:

curl -fsS http://127.0.0.1:8080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"local-model","messages":[{"role":"user","content":"Say hello in one sentence."}],"max_tokens":128,"temperature":0}'

The API also serves completions, Responses and Anthropic Messages. /v1/models lists the served name. On Windows with an NVIDIA GPU, follow the native Windows build guide. Use tensorfold --help, tensorfold capabilities --json and tensorfold serve REPO --help to inspect your binary. You can also serve a complete local checkpoint directory or an existing Hugging Face cache.

tensorfold web [ARGS...] opens TensorFold Web, a small local web app for picking, running and chatting with models. If it needs installing, an interactive release binary offers to install its matching version with uv after you say yes. See the web launcher for manual installation and unattended use.

--parallel N caps active requests at N; extra requests wait for a free slot. Omitting the flag uses --parallel auto, which asks for up to eight streams as memory and the engine allow. An explicit number is refused if the engine cannot fit that many streams. Read the engine's startup capacity line, such as 8 streams at once on CUDA, for the count it accepted. With API authentication off, /health reports live.connections as active plus waiting requests and live.waiting as the queued count. For example, 8 connections with 4 waiting means 4 active requests. Completing or cancelling an active request frees its slot; cancelling a waiting request removes it from the queue.

Mac app

The TensorFold desktop workspace bundles the local engine, model downloads, hardware recommendations, live generation metrics, coding workers and document editing. Download the Apple-silicon beta 9 DMG, or build from app/. The desktop app launches the bundled tensorfold-native executable directly. Mobile clients in app/mobile/ remain preview software.

TensorFold Starlight (enterprise)

TensorFold Starlight gives an organisation one governed OpenAI- and Anthropic-compatible endpoint for a fleet of TensorFold engines on Macs, Linux GPU servers and Windows machines. It signs people in through the organisation's identity provider (OIDC or SAML, with SCIM provisioning), decides who may use which model and where it runs, places and supervises models on nodes, and accounts for every token. It runs fully offline, or brokers approved frontier providers under the same identity, policy and budgets; the install guide covers Docker, Kubernetes and Linux services. Starlight lives in its own repository, ashhart/tf-starlight: it pins the TensorFold release it is tested with and checks every flag its nodes pass to tensorfold serve against that release in CI. Starlight is new: the gates that still need real GPU clusters or identity-provider tenants are listed in its qualification notes.

Living Weights (experimental)

Living Weights teaches a running model new facts in its own weights. Start a Nemotron 3.5 Lightning server with --slide and tell it something: each fact that passes the lesson's checks is written into the checkpoint folder's files. It is experimental: related facts can disturb each other, recall drops past a few dozen facts and with thinking on, and learned facts cost some decode speed on a Mac. Known issues has the details.

tensorfold pull TensorFold/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-MLX-4bit
hub="${HF_HUB_CACHE:-${HF_HOME:-$HOME/.cache/huggingface}/hub}/models--TensorFold--NVIDIA-Nemotron-3.5-Lightning-30B-A3B-MLX-4bit"
src="$hub/snapshots/$(cat "$hub/refs/main")"
mkdir -p ~/models && cp -RL "$src" ~/models/nemotron-living   # always teach a copy
(umask 077; openssl rand -hex 24 > ~/models/slide-key)        # --slide starts only with a key
tensorfold serve ~/models/nemotron-living --slide --api-key-file ~/models/slide-key

Leave that server running and send a lesson from another terminal:

curl -N http://127.0.0.1:8080/v1/slide/learn -H "Authorization: Bearer $(cat ~/models/slide-key)" \
  -H 'Content-Type: application/json' -d '{"text": "I like blue."}'

Send it a document instead and it finds the facts in it and learns them one by one. Open http://127.0.0.1:8080/slide to watch in 3D: the facts, the neurons each one took in the model's layers, the shard files that hold them, and the lesson running now.

Learning runs on Apple-silicon Macs and on an NVIDIA GB10 (DGX Spark), and a folder that learned on one serves on the other. On an APFS volume, cp -cRL makes a clone that shares the original's data until lessons change it. It rewrites the folder's files: before the first lesson the server keeps the originals in a hidden snapshot, and tensorfold slide rollback ~/models/nemotron-living puts them back, undoing every lesson. The Living Weights guide covers the key, teaching files and web pages, omp's /learn command, the 3D view, undoing lessons, and what to expect.

Pick a model

These are the native release's serving paths. A recipe may also describe the older Python 0.6.6 engine; use its native section and the current changes together. Hardware below records the documented scope, not an unmeasured minimum. Context, parallel requests, prompt caches and drafters need memory beyond the weights. tensorfold info REPO reports the checkpoint's memory floor. Where no RAM capacity is published, inspect that floor before serving a large checkpoint. CUDA release images include sm_80, sm_86, sm_120 and sm_121. The A100 (sm_80) has kernel-check evidence (#846), but no served checkpoint on record, so it is not model-qualified. Nemotron and dense Qwen have NVFP4/FP8 fallback code for sm_80 to sm_89 (#901, #1470); no such model run on sm_80 to sm_89 is recorded.

Run a table command after installation. On a Linux archive installation, use the tensorfold shell function from Quick start. REPO@revision pins a pull; serve the repository name without the revision after pulling it. Conversions are explicitly marked: their upstream weights need the linked conversion before native serving.

Model / formatDownloadMachine and memory scopeNative recipe / limits
dots.mocr, BF16 OCRUse the local dots-studio/dots.mocr checkpoint.Native HIP; AMD R9700 (gfx1201) checked, one GPU.Native OCR recipe; full model-card demo matches HF tokens. Native CUDA serves text prompts only (CUDA recipe). No video, MTP head, or Metal qualification.
Qwen3.5-2B, MLX 4-bittensorfold pull mlx-community/Qwen3.5-2B-MLX-4bit@93760be4f1f69842a46bc13dbdc0f19e291392a3One Mac (M5 Max checked), or one GB10; no RAM minimum published.2B; CUDA release checks. Text only.
Qwen3.8-27B, MLX 4-bittensorfold pull TensorFold/Qwen3.8-27B-MLX-4bit
tensorfold pull z-lab/Qwen3.8-27B-DFlash2
One Mac (M5 Max/M3 Ultra), or one GB10; a 32 GB Mac needs iogpu.wired_limit_mb raised (the startup refusal names the value).27B. Add --drafter z-lab/Qwen3.8-27B-DFlash2; without it decoding is plain.
Nemotron 3.5 Lightning 30B-A3B, MLX 4-bittensorfold pull TensorFold/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-MLX-4bitOne Mac (M1-M5), or one GB10; also checked on RTX 3090 / RTX PRO 6000.Lightning. Includes MTP.
Lightning, mixed oQ4 / oQ6 / oQ8tensorfold pull TensorFold/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-oQ4
tensorfold pull TensorFold/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-oQ6
tensorfold pull TensorFold/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-oQ8
One M3 Ultra checked.Width and MTP instructions. Metal only; copy the 4-bit checkpoint's MTP head to draft.
Lightning, NVIDIA NVFP4tensorfold pull nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4One GB10 checked. Non-cluster fallback code exists for sm_80 to sm_89; no NVFP4/FP8 serving run is recorded on sm_80.Lightning; CUDA changes.
Nemotron 3.5 Super VL 120B-A12B, NVFP4tensorfold pull nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-NVFP4@82d93699acd3be9e4617c53360fe5b3f5dbeb1b3One or two GB10s; weights 70.8 GiB, 73.2 GiB with vision on one.One/two-GB10 commands. Add --vision for images.
Nemotron 3.5 Bolt 74B-A9BNVIDIA's checkpointNative CUDA; 24 GB checkpoint, checked on RTX PRO 6000.Bolt/Omni video; release qualification.
Qwen3.8 Flash Next, MLX 6-bittensorfold pull TensorFold/Qwen3.8-Flash-Next-MLX-6bit-MTPOne M5 Ultra checked; RAM minimum not published.Metal, runbook.
Qwen3.8 Flash Next, MLX 4-bittensorfold pull TensorFold/Qwen3.8-Flash-Next-MLX-4bit-MTPOne GB10; 73.3 GiB uploaded, plus mapped tables and caches. Metal has a native 4-bit path too (M3 Ultra checked).Native CUDA. NVFP4/two-local-rank and vision paths are experimental. Metal's shared-session limits are in known issues.
GLM-5.3-Flash, MLX 4-bittensorfold pull TensorFold/GLM-5.3-Flash-MLX-4bit-MTPTwo M5 Ultras; single-Mac native serving is not qualified.Paired Metal runbook. Do not use the Python recipe's single-Mac command.
GLM-5.3-Flash, EXL3/TR3tensorfold pull Mia-AiLab/GLM-5.3-Flash-EXL3-TR3-4bpwTwo CUDA ranks; the release checks the two-GB10 setup.Release scope, checkpoint. Experimental checkpoint; use native rank options, not the Python installation.
GLM-5.3-Flash, NVFP4tensorfold pull nvidia/GLM-5.3-Flash-NVFP4@423acf37583782c51c142d145aef733d72943d93Two GB10s at TP=2: @Bizuayeu loaded the whole checkpoint on 1.1.0 in 302.1 s and served it.Native CUDA; contributed run: one token hash per prompt across three runs and a passphrase found in 199,307 tokens; scored prompts and more than two ranks were not covered.
GLM-5.3-Flash, FP4-Localtensorfold pull nvidia/GLM-5.3-Flash-FP4-Local@76348ddb2be2e35e39ecfc00e75dc07703f4f3d5Two GB10s at TP=2, or three at TP=3 over a ring of direct links: @Bizuayeu loaded it as stored and served it at both sizes. 165.3 GiB of weights, 13% less than NVFP4's.Native CUDA, in 1.1.5 (#1050). NVIDIA's mixed-precision checkpoint: NVFP4 routed experts, FP8 for most of the rest. As close to BF16 as NVFP4 at its smaller size (receipt).
GLM-5.3-Flash, XSFP4tensorfold pull Bizuayeu/GLM-5.3-Flash-XSFP4@88736f916f8939607fde8a4b881dcae749f7551dTwo GB10s at TP=2, or three at TP=3 over a ring of direct links: @Bizuayeu served it at both sizes. 165.7 GiB of weights.Native CUDA, in 1.1.5 through FP4-Local's path. FP4-Local with its routed experts refitted from the BF16 weights and a smoothing folded in, on the same kernels. Closer to BF16 than FP4-Local and NVFP4 by all three measures in its card.
GLM-5.3-Flash, Jammed XSFP4tensorfold pull Bizuayeu/GLM-5.3-Flash-Jammed-XSFP4@0a22bb46ecef3d0f5f96e41dc6011be12974fbf1Two GB10s at TP=2, or three at TP=3 over a ring of direct links: @Bizuayeu served it at both sizes. 160.2 GiB of weights; at TP=2 each GB10 holds 6.6 GiB less than at block 16.Native CUDA, in 1.2.3 (#2041). Refitted from BF16 at block 32 with XSFP4's smoothing, each row's scales packed (csf0_v1, read only by this path), at XSFP4's speed. Its KL and top-1 are nearer BF16 than NVIDIA's two checkpoints', short of XSFP4's (its card).
GLM-5.3-Flash, Jammed XSFP4 NZ5tensorfold pull Bizuayeu/GLM-5.3-Flash-Jammed-XSFP4-NZ5@1c900ea0294e28a8e9ab56c1156c0338eed60aefTwo GB10s at TP=2: @Bizuayeu served it. 160.2 GiB of weights.Native CUDA, on main after 1.2.7 (#2894). Jammed XSFP4 refitted with E2M1's -0 code read as +5 (NVFP4_NZ5, read only by this path). Nearer BF16 than Jammed XSFP4 (its card).
GLM-5.3, EXLR8 hometensorfold pull kindlingai/glm-5.3-exlr8-k2-k3-k4@f459f5d0272a529403dd9b3fd32f771f4250a0d4 --exlr8 home --node 0/2 --dir ~/models/glm-5.3-exlr8-homeTwo M5 Ultra Macs, 256 GB each; about 160 GB resident per Mac at 8k context / four streams.Two-Mac recipe. Use --node 1/2 on the other Mac; Thunderbolt 5 and MCDMA required. No CUDA backend.
LFM2.5-24B-A2B, MLX 4 / 6 / 8-bittensorfold pull LiquidAI/LFM2.5-24B-A2B-MLX-4bit@667408d494bde191c2cb9dd4f4549827816b9275
tensorfold pull LiquidAI/LFM2.5-24B-A2B-MLX-6bit@1deec53eb49571e21cdcbf82fd5839248f07ed88
tensorfold pull LiquidAI/LFM2.5-24B-A2B-MLX-8bit@67ab417dd02d0c9545802a726df668620eae049d
One Mac (M4 Pro checked), or one NVIDIA GPU with the 4-bit export (GB10 checked).Native recipe (CUDA), release history. Weights: LFM Open License v1.0.
Qwen3.6-35B-A3B, MLX with MTPtensorfold pull TensorFold/Qwen3.6-35B-A3B-MLX-4bit-MTP@81169a9bc511a27c1b4eedb77a2cd98ced431847One GB10; 20.9 GB checkpoint.Qwen3.6. Native CUDA receipt here. Metal also registers this model type; the recipe's Metal commands describe Python 0.6.6.
Qwen3.6-35B-A3B, NVIDIA NVFP4 with MTPtensorfold pull nvidia/Qwen3.6-35B-A3B-NVFP4@1355db6a052410cfd62085d94b58866fd0f2c3c5One GB10; 20.3 GiB on the device, 20.9 GiB with the drafter.Native CUDA. Serve with --drafter naming the same checkpoint for MTP drafts.
Gemma 4 31B, MLX 4-bittensorfold pull mlx-community/gemma-4-31B-it-4bit@696d436c404745a59f30e4939a658162b0a9e57f
tensorfold pull z-lab/gemma-4-31B-it-DFlash@eabd648301ce28583cc14757912e5e0f84e152e1
One Mac (M5 Ultra checked), or one GB10 on CUDA. Pre-M5 Metal has a row path; its assistant and DFlash drafters require M5 tensor units.Native 31B on Metal, CUDA qualification. Name the drafter with --drafter; images with --vision.
Gemma 4 12B, converted MLX 4-bithf download google/gemma-4-12B-it --revision 707f0a3b8a3c7ad586ed01e27eafbad8a27dd0f7 --local-dir ~/models/gemma-4-12b-hf
hf download google/gemma-4-12B-it-assistant --revision 46d4c6f13f0ac0ad827b915669b8df9b81c64c51 --local-dir ~/models/gemma-4-12b-assistant
One Mac (M5 Ultra checked); converted text weights 6.3 GB, image weights 100 MB, plus assistant/caches. MLX 4-bit also served on one GB10. Pre-M5 Metal has a row path; the MTP assistant requires M5 tensor units.Convert and serve. Upstream download needs hf; conversion needs Python, mlx and mlx-lm. Serving is native; CUDA qualification includes the MTP assistant.
Step-5-Preview, converted affinehf download stepfun-ai/Step-5-Preview --local-dir ~/models/step5-preview-fp8One 256 GB Mac; text 166.9 GiB, MTP adds 2.5 GiB and vision 3.7 GiB.Convert and serve. The documented 256 GB setup serves converted affine weights; converting the upstream FP8 download needs Python and mlx.
MiniMax-M3.1-Flash, community affine 3/4-bithf download BobClawblaw/MiniMax-M3.1-Flash-MLX-mix34 --revision 9524999057e0d277d26438606e033e2191fb465c --local-dir ~/models/MiniMax-M3.1-Flash-MLX-mix34
hf download BobClawblaw/MiniMax-M3.1-Flash-MLX-mix256k --revision 916e7ca1044f1c3e7566163096243c940327224d --local-dir ~/models/MiniMax-M3.1-Flash-MLX-mix256k
One 256 GB M5 Ultra; 203.7 GiB (32,768-token window) or 189.2 GiB (147,456), plus caches.MiniMax-M3.1-Flash. Set MINIMAX_LOAD_LIMIT_GB; images with --vision. Community checkpoints: lossy conversions of MiniMax's NVFP4 release (KL 0.244 / 0.295). MiniMax Community License.
Recursive Flint 40B-A2.5B, MLX INT4hf download RecursiveSI/flint-int4 --revision 3b93f9053b530488405ae9fca51d4900d6ce21f0 --local-dir ~/models/flint-int4One M5 Mac; M5 Ultra checked; 21.4 GiB resident plus caches. M1-M4 untested.Flint. Text and tools; always reasons; no draft head.
Kolibri-1, block FP8 (research)tensorfold pull Aleph-Alpha/Kolibri-1@e52eb4627d11516b0c01de49210ab5a4e4061444One NVIDIA GPU; GB10 measured; 74 GB checkpoint.Kolibri. Text only; copy drafts, no EAGLE-3/neural drafter.
Liquid AI d1-3B, decisionsNot documented yet.Native CUDA; RAM minimum not published.API section below. /v1/systemone and /v1/decisions.
Fish Audio Drama 3, speechNot documented yet.Native Metal or CUDA; GB10 measured.Release audio instructions, reference voices and dialogue speakers. /v1/audio/speech with sampling controls; checkpoint and codec.pth required.
Drama 3 pruned NVFP4, speechhf download plotarmordev/drama-3-NVFP4-Pruned --revision 494db08481db952f834b2241bce63fb3dfa72554 --local-dir drama-3-NVFP4-PrunedNative Metal; roughly 21 GiB checkpoint plus runtime state. Not a minimum-RAM or cross-backend quality claim.Packed experts, BF16 dense/tower weights and FP32 accumulation; no conversion. Use serve drama-3-NVFP4-Pruned --cfg-gamma 1.5 --cfg-batched with an attested catalog voice.

GLM-5.3: one machine or two?

The native release does not qualify GLM-5.3 on one machine. GLM-5.3 EXLR8 home needs two 256 GB M5 Ultra Macs; its recipe gives both pull and serve commands. GLM-5.3-Flash is a different model: its native Metal setup needs two M5 Ultras. Native CUDA has recorded TP=2 and TP=3 setups, including FP4-Local and XSFP4 on two and three GB10s. The single-256-GB-Mac GLM-5.3-Flash command in the older recipe is for Python 0.6.6, not this release. For the NVFP4 path, see GLM-5.3-Flash on CUDA and what the contributed run on two GB10s covered.

DeepSeek checkpoints are not served yet by any release binary: whole-model gate #827 is still open. MiMo's native backend is not in a release yet. Bonsai's ternary pack and Gemma 4 26B-A4B remain Python-only; they are not native download recommendations. Known issues lists current limits, including Flash Next shared-session cache/performance behavior and the 27B's drafter limits.

Exact decoding

Every accepted draft must equal the token the same native engine would produce with "draft": false. A resumed request must equal fresh execution, and each concurrent stream must equal its solo run. The comparison fixes the checkpoint, backend, settings and runtime. Different quantizations and different backends can produce different outputs. Flash Next on Metal uses shared target rounds with an explicit --parallel N above one, when that many sessions fit. A lone greedy request there uses GPU-side rounds and hands over when a second request arrives. --parallel auto (the default) and --parallel 1, learning and two-Mac serving use the single-reply driver. FZ_BATCH_MTP_SERIAL=1 restores serial MTP drafting on the shared host. The earlier MTP chain receipt records its own pinned commits. Nemotron, GLM, the 2B model and Qwen3.8-27B use shared lane rounds. On Metal, each of up to 128 streams in the 27B's shared rounds drafts its own DFlash2 tree.

Models on disk

tensorfold pull TensorFold/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-MLX-4bit
tensorfold models
tensorfold models --json
tensorfold info TensorFold/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-MLX-4bit

pull downloads a checkpoint into the Hugging Face cache and checks what it downloads against the Hub: large files by sha256, small files by their git blob sha1. Large files come down in pieces over up to 16 connections, and an interrupted pull resumes with the pieces it lacks. It refuses a checkpoint whose model_type has no native family, and fetches the 27B's DFlash2 drafter for --drafter. models lists the cached checkpoints with a model type it knows, and info shows one checkpoint's family, format, context and memory floor. models --json prints the same list as one document for programs: schema 1, and each checkpoint's repo, family and model_type; family is a display title and model_type is the key to match on. tensorfold serve REPO serves a cached checkpoint by its repository name.

Context compaction

Compaction is off by default, and with it off every reply is unchanged. With --compact-at, a conversation that would overflow is compacted instead of refused. The server keeps the system prompt and the most recent turns word for word, and never separates a tool call from its result. The older turns become a memory note: goal, constraints, progress, decisions, next steps, exact names and numbers, and the files the tool calls read or changed. Each later compaction updates the previous note instead of starting over. A client that sends back the compacted conversation keeps its note. The reply carries the note and the compaction counts under tensorfold.compaction, and every reply reports context_window and context_used. The same request gives the same compaction and the same reply. With --compact-memory, the stored note carries over between requests.

Cortheon mode

Off by default. --cortheon gives every chat request Cortheon's reasoning program and web research run by the server, so any OpenAI-compatible client gets it with no setup. The model works through Cortheon's stages for its task type and can call cortheon_search and cortheon_fetch, which the server runs without the client seeing them. A question about current facts, or a request to build with an outside library, is researched before the model starts. A coding task that changed files is asked to run its tests before it finishes. Web text is filtered for instructions aimed at the model, and reads reach public addresses only. With the flag, search queries written from the conversation go to DuckDuckGo, or to Brave with BRAVE_SEARCH_API_KEY, and pages are fetched from this machine; without it, the server makes no searches and reads no pages. See Cortheon mode.

Model discovery

GET /v1/models (also /models) advertises each served name and alias. When the engine reports a finite context window, each entry includes equal context_length and max_model_len integer extensions. These are the loaded engine's effective prompt-plus-reply token limit, including any startup memory fitting; they are not the default reply limit. An unspecified engine window omits both fields. Clients should still handle per-request memory refusals: an advertised context is not a guarantee that every workload fits.

A model that thinks also carries an OpenRouter-style reasoning: default_enabled, default_effort (only with --reasoning-effort), and supported_efforts, the levels its template names, highest first, then none when the template has a thinking switch. A template that names no level leaves supported_efforts out. These describe chat and Responses requests.

Serve flags

The read-only /memory diagnostics report process footprint and lifetime peak, available backend allocation counters, and the applied retained-prefix plan without changing memory policy.

The binary's capabilities --json response lists its supported flags and platform-specific values. serve MODEL --help prints usage without loading a model.

FlagMeaning
--host HOST, --port PORTListen address and port, default 127.0.0.1:8080.
--unix-socket PATHListen on a new owner-only Unix socket at PATH instead of TCP, refused with --host or --port; macOS and Linux. An existing PATH, or a directory other users can write to (unless sticky, like /tmp), is refused, and connections from other users are closed. API keys, --metrics-open and the Host check apply as on TCP (a client sends a loopback Host or one in --allowed-host): the socket's permissions are a second layer, not a substitute for keys.
--profile stableExplicitly preserve current defaults; profiles describes the initial switch registry and startup report.
--config FILE.tomlRead serve flags from a TOML file: a key is a flag without its --, and model names the model. The command line wins over the file, and a relative path in the file starts at the file's folder. See serve config files.
--name NAMEModel ID advertised to clients.
--alias NAMEAnother accepted model ID; repeat for multiple aliases.
--api-key KEYRequire a bearer key; repeat for multiple keys.
--api-key-file FILERead keys from a restricted-permission file; a key can be limited to some routes.
--metrics-openAllow /metrics, /v1/metrics and /v1/load without a key when API authentication is enabled.
--dashboardEnable the local /dashboard page and /stats endpoint.
--max-inflight N, --max-inflight-per-key NRefuse model requests past N in progress at once, overall or per key, with 429; 0, the default, is off. See admission.
--max-connections N, --max-connections-per-address N, --trusted-proxy ADDRESSOpen connections at most (default 128); from three quarters full, one address's share (default a quarter, 0 off); addresses never counted for that share, as loopback is not, such as a reverse proxy or Docker's bridge gateway. See admission.
--requests-per-second N, --request-burst NRefuse a key's model requests past N a second, after a burst; 0, the default, is off.
--context NBound prompt plus reply tokens; the model's window and memory checks still apply. Qwen3.8-27B defaults to 32,768 and takes up to 262,144.
--rope-yarn FYaRN context extension with this factor over the checkpoint's trained window, for Qwen3.5 and 3.6 on a CUDA build; other builds refuse the flag.
--loops NPasses over a looped model's recurrent block, from 1; without it the checkpoint's own count runs. Other builds refuse the flag.
--speed-up FILERank and link settings for two-Mac Flash Next or GLM serving.
--max-tokens NDefault reply limit, 4096; requests can override it.
--temperature TSampling temperature; zero requests greedy decoding.
--prompt-seedUse native prompt-derived keys instead of the fresh OS fallback; request and configured default seeds still win.
--top-p P, --top-k K, --min-p PSampling defaults.
--instruct-sampling SETSampling defaults for sampled chat requests with thinking off: card (the Qwen 3.5, 3.6 and 3.8 cards' temperature 0.7, top_p 0.8, top_k 20, presence_penalty 1.5), name=value pairs over it, or none; the serve flags still win. Greedy requests, raw completions and the server's own requests never take them.
--thinking, --no-thinkingChoose whether the chat template opens a reasoning block.
--reasoning-effort LEVELDefault template effort: minimal, low, medium, high, xhigh or max.
--thinking-budget NLimit reasoning tokens where the engine supports closing the reasoning block.
--thinking-reserve NWith thinking on, let reasoning use up to N tokens beyond a request's max_tokens, which then limits the answer (lane engines).
--loop-guardClose reasoning that repeats itself verbatim (a short cycle, or a long passage) where the engine supports it.
--loop-guard-stopAs --loop-guard, but end the reply where the guard fires: the reasoning so far, no answer or tool calls, finish_reason content_filter and a reasoning_loop error (API).
--no-draftsProduce the plain reference through the same engine.
--drafter DRAFTERAn external drafter, as a directory or a repo id in the Hugging Face cache, for models that take one: Qwen3.8-27B uses DFlash2 and Gemma 4 31B the drafter named in its recipe. Without one they decode plain.
--drafter-bits 0|4Drafter weight precision where the model offers a choice: 4 bits (the default), or 0 to keep bf16. Qwen3.8-27B on CUDA takes 4 only.
--record DIRWrite each stream's committed tokens, positions and tapped rows under DIR, for drafter training. CUDA Nemotron only; a single stream runs through the lane rounds, and recording stops below 25 GiB free disk.
--keep-warm SECONDSKeep the Metal GPU active while idle for this long after a request, default 900; zero disables it.
--cfg-gamma GAMMAMetal Drama speech: semantic guidance strength, finite 1–5; 0/default disables it. Only reference-conditioned requests are guided. Enabled replies report X-Cfg-Gamma; codec heads stay conditional.
--cfg-allow-request-gammaMetal Drama speech: allow a finite 1–5 X-Cfg-Gamma request header; invalid values return 400 before engine submission. Without this opt-in, the server's gamma is fixed.
--cfg-force-guidedWith an enabled Metal Drama guidance policy, exercise the unconditional state even at gamma 1. The semantic mix at 1 remains exact conditional-logit identity.
--cfg-batchedWith enabled Metal Drama guidance, submit both independent backbone states in one command buffer rather than separate submissions. Weights are shared; extra state is included in admission. Not vectorized batching.
--silence-floor-dbfs DBFSDrama speech only: explicit frame-power floor for the silence guard, with no default floor. Calibrate below quiet intended speech using public or built-in speech.
--silence-seconds SECONDSDrama speech only: stop after this continuous duration below the explicit floor, retaining audio and reporting silence. Unset by default: off; calibrate above intentional pauses.
--silence-onset-s SECONDSDrama speech only: with the silence guard on, stop once this many seconds run below its floor while the synthesis has voiced less than --silence-onset-voiced-s, retaining audio and reporting silence. Off unless given; must be below --silence-seconds.
--silence-onset-voiced-s SECONDSDrama speech only: the voiced seconds below which --silence-onset-s applies; required with it, with no default. Calibrate on public or built-in speech.
--silence-mid-s SECONDSDrama speech only: with the silence guard on, finish as silence when end of speech falls in a run this long below the floor, after the explicit voiced minimum. Whole replies carry the decimal X-Silence-Mid-Trips count, including zero. Must be below --silence-seconds and paired with --silence-mid-min-voiced-s. Unset by default: off.
--silence-mid-min-voiced-s SECONDSRequired with --silence-mid-s: positive finite cumulative voiced duration before the mid rule applies, including equality. No default; calibrate with the other silence thresholds.
--silence-redraws NDrama speech only: draw a whole reply the silence guard ended again, up to N times, as seed salt + k or explicit request seed + k; N of 1 or more needs the guard. Unset by default: off, as is 0; streams are never redrawn. See speech redraws.
--silence-redraw-budget-s SECONDSStart a redraw only while elapsed time plus the last take's duration fits; a request's X-Deadline-S can lower it. No default budget.
--silence-redraw-controls LISTRedraws speak the input filtered to the listed controls, as --allowed-controls filters; an empty list removes every control.
--short-reply-ratio RAlso redraw a whole reply that finished stop with less audio than R (above 0, below 1) of its text's expected length; needs --silence-redraws and --short-reply-chars-per-s. No default.
--short-reply-chars-per-s CPSInput characters per second of audio in complete replies of the served voice, setting the expected length; needs --short-reply-ratio. No default; calibrate on your own voice.
--register-redraw-st STDrama speech only: draw a whole reply once more when a phrase after the first opens ST semitones above the other phrases' median opening; needs the two flags below. Unset by default: off, with no default for any of the three. See speech redraws.
--register-redraw-high PERCENTThe share of that phrase's voiced frames that must lie above --register-high-hz.
--register-high-hz HZThe pitch line, 75 to 600 Hz, that marks a high register; calibrate it on your own voice.
--parallel N|autoCap active requests at N and queue the rest. Default auto asks for up to 8 as memory and the engine allow; an explicit count that cannot fit is refused. Read the accepted count at startup.
--prompt-cache-gib GIBRetained-prefix budget; by default half the room left after streams on Metal and unified-memory CUDA (16 GiB floor where room permits), 8 GiB host snapshots on discrete CUDA, 16 GiB for Flash Next's batch host. An explicit value overrides it; 0 disables it.
--prompt-cache-over-capPermit an explicit prefix budget past what 70% of RAM leaves once the server loads.
--learnKeep shared prompt prefixes on disk for new conversations to resume; the models are below the table.
--learn-dir DIRWhere --learn keeps them, ~/.cache/tensorfold/learned by default; implies --learn.
--learn-gib GIBDisk for learned prefixes on each Mac, 32 by default; the least recently used go first. Implies --learn.
--learn-min-free-gib GIBFree disk --learn leaves on each Mac, 4 by default; below it learning pauses and serving goes on.
--learn-conversationsKeep every prompt state on disk too, each conversation's and the shared prefixes, in one store in place of --learn's, so a next turn after a restart, a crash or an eviction reads it back. Implies --learn; see Conversations on disk.
--learn-daily-gib GIBWhat --learn-conversations may write to disk a day, every model together, 32 by default; past it states stay in RAM only.
--recallDraft from a record of the token ids earlier requests read and wrote, so repeated text is verified in a few rounds; replies are unchanged. Off by default; see Recall for what stays on disk.
--recall-dir DIRThe record's directory, one a tokenizer under ~/.cache/tensorfold/recall by default; implies --recall.
--recall-gib GIBDisk for the record, 1/16 GiB by default; the oldest ids go first. Implies --recall.
--recall-read-onlyDraft from the record without adding to it, as gates and benchmarks run; implies --recall.
--snapshot-dir noneKeep prefix state in memory; none is the supported value.
--max-snapshots 0Disable disk snapshots at startup; 0 is the supported value.
--compact-at auto|FRACTIONTurn on context compaction (below). auto compacts when the prompt and reply would pass the window minus a reserve.
--compact-keep TOKENSRecent tokens kept word for word when compacting; the default is 20,000 or a quarter of the window, whichever is smaller.
--compact-memory DIRAlso keep each conversation's memory note as a Markdown file in DIR.
--cortheonCortheon's reasoning program and server-run web research for chat requests; see Cortheon mode.
--no-update-checkAccepted compatibility switch; update the installed binary through its package manager.
--backend autoUse the backend compiled for the platform; mlx selects native Metal on macOS and cuda selects CUDA on Linux.

--prompt-cache-gib applies to Nemotron, Flash Next, GLM, Qwen3.8-27B and LFM2.5-24B-A2B, and --prompt-cache-over-cap to the same models but GLM. --learn keeps shared prompt prefixes, such as a system prompt and its tools, on disk, so new conversations resume them after a restart or an upgrade that computes the same bits. It covers GLM-5.3-Flash, Nemotron, and Flash Next on one Mac; Qwen3.8-27B, Qwen3.5-2B, LFM2.5-24B-A2B, GLM-5.3 (EXLR8) and a CUDA server refuse it at startup. Qwen3.8-27B on Metal also resumes a follow-up turn from the state its last reply left in the server's first slot, when the prompt cache holds nothing as far: the turn reads only its new tokens, even on a Mac whose cache cannot keep a long conversation. Where the cache cannot keep a whole context, the server also keeps that slot's DeltaNet states at each prompt's history end, so a turn whose template drops the last reply's reasoning resumes there. Replies are the same tokens as a fresh prompt pass; TF_QWEN27_SLOT_RESUME=0 turns both off.

Sampling defaults come from the checkpoint's generation_config.json, then serve flags and request fields override them. --context 0 is family-specific; use a positive limit for GLM and inspect the capacity reported at startup. The supported environment variables are listed in capabilities --json, including API keys, request logging and Hugging Face cache paths. TF_Q27_PROFILE=N makes a native Metal Qwen3.8-27B server log a per-class GPU-time breakdown every N decode rounds (1 means 64) and add gpu_seconds to its replies; it is diagnostic only, and a profiled round runs slower.

Community benchmarks

tensorfold benchmark MODEL runs public code and chat fixtures on a cached model and saves a portable receipt; --publish optionally submits it for review at tensorfold.dev. See benchmark commands and measurement definitions.

Build and contribute

Use the Zig version pinned in .zig-version, currently 0.17.0. The .metal sources need Xcode's Metal compiler. Install full Xcode and its Metal Toolchain component (xcodebuild -downloadComponent MetalToolchain); the Command Line Tools alone fail every kernel compile with unable to find utility "metal".

zig build native -Dcpu=apple_m1
zig build test test-golden -Dcpu=apple_m1

Structured output and required tool calls need the release archive's lib/ or zig build grammar -Dxgrammar=DIR; prepare DIR with git clone --branch v0.2.8 --recurse-submodules https://github.com/mlc-ai/xgrammar (use the native build's same --prefix for grammar).

To build one stripped server for one GPU (zig build -Dgpu=spark -Dnvcc=..., about 69% smaller than the four-target server), see Build a server for one GPU.

On macOS, test also runs available Metal device checks. See CONTRIBUTING.md for the host-test subset.

The server is zig-out/native/bin/tensorfold-native. Release archives include the native executable, runtime assets and license notices; packaging describes the qualified CUDA inputs. For cache-aware performance receipts, see observed cache benchmark evidence. The clients retain server usage and output fingerprints, and verify cold/reused states rather than inferring them.

Read CONTRIBUTING.md for exactness, precision and performance gates. Coding agents should also load docs/recipes/SKILL.md; agents that support skills can install it as tensorfold-contributor. TensorFold is Apache-2.0; see LICENSE, NOTICE and THIRD_PARTY_NOTICES.md.

Contributors

Thank you to everyone who has sent TensorFold a pull request, a measurement or a bug report:

@0mao0, @321sssrt-bit, @AbrahamCete, @aditya1503, @AdrianBinDC, @ajensenwaud, @akol1, @alenperic, @Alexbob0, @alexderz, @anvilsong, @Arminova, @b-ostrov, @BadAd84, @barelyworkingcode, @Benjamin-Wegener, @benthecarman, @benwilson, @BHCC2025, @Bizuayeu, @BlivionIaG, @BobClawblaw, @borodach23, @Boscoeuk, @boxabirds, @brandonmmusic-max, @bunnyfu, @CanadaApollo6, @CerebralCoding, @cesarswong, @chadhurley25075-png, @chaog992, @Charlie-Louis, @Chedrian07, @chris247474, @christrade215, @crescit, @cshintov, @cwschroeder, @DakotaTexas, @danyo1399, @Deesha08, @Defilan, @DevRico003, @di37, @diwakergupta, @drowzeys, @easthambuilds, @ecohash-co, @edurdias, @eleqtrizit, @ericlsimplifi, @EugeneClaw, @fche, @feni6, @florianbrede-ayet, @gbgbgbg, @GDACONSULT, @gecobattya, @gilby, @Gogo6969, @gprot42, @GraithSecurity, @grantoverton, @grearjake-star, @greatyingzi, @GustavBlack, @harrisonfriia, @haxudev, @heitke, @hichaiuse, @ivanfioravanti, @jasontitus, @javierbits, @jayleaton, @jeffpeng3, @jeidbugs404, @Jenpo, @jerryfane, @jetnet, @jishnuvenugopal, @jkuepker, @joeynyc, @johnymoo, @JordiPosthumus, @joshuaswarren, @JRaxworthy, @jrb-ai, @jschmied, @juliankang4, @justincasey, @kelchm, @kieranscotney, @kingjamez, @kky42, @laudney, @lcgutierrez, @LECYWZA, @lijian1999, @liumorrisclaw, @LXD-8, @m-naoki-m, @mapamalu, @mcclanahanaman, @MESevenJourney, @mgoldwasser, @MiaAI-Lab, @mikolaj92, @millaguie, @Mirrdhyn, @Moutonc, @MovieMaker93, @mrpmorris, @mrv777, @MV10, @NeoAiLabs, @Nipale-ai, @nmcbride, @nood-co1, @nullburn, @olexale, @omar16100, @optimisme, @otarkhan, @outcastofmusic, @paragontasx, @peacockesq, @philip-pentatonic, @plotarmordev, @pmeenan, @ppardus, @pulseandthread, @quigles1977, @rafafortes, @raymondkpwong, @robertpitt, @rootsbymenda, @RoscoeTT, @RS91Dev, @russellmm, @salmanarshad321, @samwang0041-star, @sanjaibalajee, @satindergrewal, @scottleimroth, @sethforprivacy, @sfxnz, @shantanugoel, @simon-lin88, @simonmd, @spenchey, @squarrier, @ss-cong, @Stephane-Eden, @styles01, @SvangenStudios, @swbiggart, @SxMShaDoW, @sxuff, @taussoe, @tfolkman, @ThinkOffApp, @ThomasHalwax, @Thotheris, @tinyapps, @tolewis, @tomByrer, @tonydehnke, @tournierjc, @tovereast, @tpischke, @urtho, @valacroix, @vcruz305, @vinicius-symetrix, @wojo, @xjqx2z, @yguanrong, @Yuepixel, @YvesLaRose, @ZackSample, @zweck.

docs/credits.md names whose Python engine work each part of the native engine carries.

Usable memory profiles explains how to record applied budgets, context, cache state and process peaks.