onnx-genai

ONNX Runtime GenAI Inference Engine

Usage:  onnx-genai -s -m chat_completion_model -e embedding_model -p port 

 -m path     : chat completion model
 -e path     : embedding model (pooling=mean)
 -r path     : rerank model
 -T path     : text to speach model
 -D path     : System One decision ONNX model (Kev, d1, LiquidAI d1, English Laya or Strands Decider v1, text only)
 -t path     : chat template
 -j          : chat template from stdin
 -d          : pooling=e2e 
 -b          : pooling=multi-vector 
 -l          : pooling=last-token
 -c          : pooling=cls
 -s          : server
 -p          : server listening port (default=8080)
 -h host     : server host (default=127.0.0.1)
 -i          : input
 -o          : output (default=stdout)
 -           : use stdin for input

The CLI is built for 4 platforms:

• macOS Apple Silicon
• macOS Intel
• Windows AMD
• Windows ARM

Dependencies

• onnxruntime-genai-0.12.0
• onnxruntime-extensions-0.14.0
• onnxruntime-1.29.0 (CPU; macOS universal, Windows x64 and ARM64)

The bundled ONNX Runtime macOS library targets macOS 13.3 or later. Other dependencies may impose higher deployment requirements. GenAI and Extensions are separate libraries and retain the versions listed above.

OpenAI Compatible Endpoints

• /v1/models
• /v1/chat/completions
• /v1/embeddings
• /v1/audio/speech

Cohere Compatible Endpoints

• /v1/rerank

MongoDB Compatible Endpoints

• /v1/contextualizedembeddings
• /v1/contextualized/embeddings (alias)

[!TIP] The input_type property has no special effect. You must specifically add the prefixes “Represent the query for retrieving supporting documents: “ and “Represent the document for retrieval: “ if required by the model. https://www.mongodb.com/docs/

TypeSafe Compatible Endpoint

• POST /v1/systemone

Load one decision model independently of chat, embeddings, reranking and speech:

onnx-genai -s -D /models/kev-0.6b-ONNX/onnx/model_q4.onnx
# Same adapter for kev-4b-ONNX:
onnx-genai -s -D /models/kev-4b-ONNX/onnx/model_q4.onnx
# d1 q8:
onnx-genai -s -D /models/d1-3B-ONNX/onnx/decoder_model_merged_quantized.onnx
# LiquidAI d1-2B or d1-Omni-2B q8 (text only):
onnx-genai -s -D /models/d1-2B-ONNX/onnx/decoder_q8.onnx
# Laya English encoder:
onnx-genai -s -D /models/laya-onnx/laya.onnx
# Strands Decider v1 q8 (fused readout, not a causal-LM export):
onnx-genai -s -D /models/strands-decider-2B-qwen3.5-v1-2610-onnx/onnx/model_q8.onnx

Supported exports: Kev-0.6B, Kev-4B, d1-3B, d1-2B, d1-Omni-2B, Laya English, and Strands Decider v1. These are decision models, not chat or embedding models; do not load them with -m, -e or -r.

For Kev/d1, keep config.json and tokenizer.json in the model directory, with the selected graph and all its external data files in onnx/. They may alternatively all be placed in one directory. Kev, d1 and Strands are identified by exactly one of kev, d1 or decider metadata in config.json, respectively (LiquidAI d1 by d1_attention), not the directory name or model_type. Kev uses the selected graph directly. d1 additionally loads the matching embed_tokens_quantized.onnx or embed_tokens_q4.onnx beside its selected decoder. The text-only adapter does not load the vision encoder.

LiquidAI’s own d1 exports, d1-2B and d1-Omni-2B (text only), are identified by d1_attention in config.json and are served as backend d1 too (d1-latest). Keep config.json, tokenizer.json, tokenizer_config.json and calibration.json at the repository root, with the selected onnx/decoder[_q4|_q8].onnx and all of its .onnx_data* shards beside it; the vision and audio encoders are not needed. A state and up to d1_set_size (16) questions are packed into one decoder row and read in a single pass without a cache (“set” attention: each question attends to the state and itself, the state reads every question). Answers therefore differ slightly from asking each question alone (at most about 0.02 in probability on the d1-2B q8 checks, with no change of decision). Prompts, option codes (d1_codes), readout tokens and the calibration.json temperature (per question bucket, else default) follow the checkpoint’s published reference code. Typed questions (text, number, date, box, boxes, multiple choice), the calibration.json tables and picture, video or audio input are not supported. d1-Omni-2B uses the same adapter with an empty audio input. decoder_q8 is verified on CPU; decoder_q4 also targets CPU, while the f16 decoders are meant for GPUs.

Strands uses the selected fused graph directly: onnx/model_q8.onnx needs onnx/model_q8.onnx_data (~1.7 GB); model.onnx and model_q4f16.onnx likewise need their corresponding external data. Keep the export’s tokenizer assets and configuration, not those from a differently named checkpoint. The official v1 checkpoint is a rank-48 LoRA plus dedicated option-scoring head on Qwen3.5-2B-Base; the community export fuses both. Official adapter-only weights and older hobson-v19/v21 exports are not interchangeable with this fused v1 layout.

The Laya adapter runs the 421M-parameter ModernBERT-large-based encoder plus decision head on CPU, English only. Use the community ONNX bundle receptron/laya-onnx pinned to revision d42786bdff8a7550316339c5fbf8f46caa83ea1b; preserve its layout:

laya-onnx/
  laya.onnx
  laya.onnx.data
  laya_config.json
  tokenizer/
    tokenizer.json
    tokenizer_config.json

laya_config.json alongside the selected graph identifies the native Laya backend. Both tokenizer files are required; external laya.onnx.data must stay beside the graph. Backend selection distinguishes Kev, D1, Laya and Strands; ambiguous Kev/d1/Strands config.json metadata mixed with Laya metadata is rejected, not resolved by preferring one backend. No backend flag is needed. This supports the shipped English ONNX export, not the multilingual checkpoint. An independent CPU comparison against official convaiinnovations/laya revision 7b928d828b7b0e022f929d9bd2e44165aa270148 measured a maximum raw-logit difference of 2.58e-5 in a four-row mixed-question batch using PyTorch 2.15.0 and ONNX Runtime 1.29.0. This is a bounded smoke comparison, not universal 1e-5 parity or model-quality validation. Published temperatures in laya_config.json are consumed as-is and must be finite and positive. In particular, choice:11+ = 0.945 is the community export’s calibration, not the official checkpoint’s 0.10058280825614929; probabilities for those questions deliberately follow the published ONNX bundle rather than the original checkpoint.

Runtime requirement: the published d1 export uses ONNX IR 13. The bundled ONNX Runtime has been upgraded to 1.29.0 in both the C++ build dependencies and the macOS/Windows deployment directories, so a runtime override is no longer needed. Older installations using 1.23.2 cannot load this graph. Laya and Strands use the same bundled CPU runtime 1.29.0; neither requires a runtime version bump.

For deployment, keep the 1.29.0 runtime alongside the rebuilt executable (on Windows, deploy its corresponding provider DLLs too). System One still requires an executable rebuilt from the updated source; replacing a runtime library alone does not add the endpoint to an older executable. Prefer Kev q4, Strands q8 and d1 q8 for initial CPU evaluation; other export precisions depend on runtime/kernel support. Probabilities may differ across quantizations and runtime versions. Quantized Kev probabilities can also vary slightly between a packed request and the same questions evaluated separately, despite the graph’s branch isolation.

curl http://127.0.0.1:8080/v1/systemone \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "kev-latest",
    "state": {"message": "I was charged twice. Please refund the extra charge."},
    "questions": {
      "refund": {"type": "noul", "instructions": "Is a refund requested?"},
      "team": {
        "type": "choice",
        "instructions": "Which team should handle this?",
        "criteria": {"billing": "Payments and refunds", "technical": "Software faults"}
      },
      "urgency": {
        "type": "score",
        "instructions": "How urgent is this?",
        "criteria": ["Routine", "Urgent", "Emergency"]
      }
    }
  }'

Use kev-latest for Kev, d1-latest for d1, laya-latest for Laya, strands-decider-latest for Strands, or the model ID returned by GET /v1/models (the model directory’s basename). Family aliases only resolve to the loaded backend (kev, d1, laya or strands-decider); an unknown model is rejected. The response contains model, answers keyed by question name, and usage with input_tokens and output_tokens: 0. Noul returns noul (probability of true); Choice returns choice, probabilities and confidence; Score returns a zero-based expected score, legend, probabilities and confidence. Laya uses this same response schema; no act fields are added.

Confidence follows TypeSafe’s documented formulas: Choice measures the largest probability above a uniform distribution, and Score measures probability-weighted distance from the modal level. A single option/level has confidence 1. These are distribution statistics, not guarantees of accuracy or Jev-equivalent calibration. Strands divides each option logit by decider.temperature_by_kind[type] (falling back to decider.temperature) before softmax; missing, nonpositive or nonfinite effective temperatures fail model loading. It does not generate answer letters or apply training-time ordinal smoothing at inference.

The endpoint accepts string/object/array state and structured instructions/descriptions. It rejects images, assets, box, unsupported fields, duplicate JSON keys and inputs exceeding limits:

Over-limit inputs are rejected, not truncated. Kev shares the state within a packed graph run. d1 and Strands evaluate each question independently, with fresh caches (Strands’ empty caches are inside the graph); their input usage counts repeated state tokens. Mixed question kinds and option counts do not share padding or logits. Strands follows the upstream engine’s strict-window path: the export’s max_question_fraction is a truncation policy, not an additional strict-mode limit, and is not applied. The exported models’ long-context behavior is not necessarily validated up to these serving limits.

Laya constructs each row in the reference order [CLS] type question: instructions [SEP] [MASK] options [SEP] state [SEP]. Structured state uses insertion-ordered JSON with Python-style spacing; structured instructions additionally use ASCII JSON escapes, as in the Python reference. Structured option descriptions use ordered JSON as an API extension to the reference’s text descriptions. Literal [MASK] in caller content is replaced with a space so it cannot inject option markers. It follows the reference equal-budget rule for options, but rejects the request if any option would need shrinking (including to meet the 48-token option cap). The remaining question/instruction head capacity is max(8, head_max_len - sum(option tokens including markers)); a head that would need truncation is rejected too. State plus the complete head and separators must fit max_len = 512. These are reject-not-truncate limits, not permission to silently shorten instructions, options or state. Question rows are batched with right padding; the marker-position tensor has width at least 2. The padded batch must fit 32,768 tokens, while usage.input_tokens sums the unpadded row lengths (including repeated state). The existing 1–64 question, 1–255 Choice option and 1–10 Score level limits remain unchanged for Laya.

Invalid requests return HTTP 422 with a detail array, an unloaded decision model returns 503, and inference failures return 500. Errors are logged. Decision inference has its own mutex and does not repurpose the other model slots. Without -s, -D selects decision evaluation from -i/stdin, with -o for a JSON response file; failures return a nonzero exit status. If both -m and -D are supplied in CLI mode, decision evaluation takes precedence.

Kev, the English Laya weights and Strands Decider (including the community fused v1 export) are Apache-2.0 licensed. d1 uses the LFM Open License v1.0, including commercial-use restrictions; review it before deploying. LiquidAI’s d1-2B and d1-Omni-2B exports use the same license.

Strands calibration is fitted per primitive on held-out short classification, not on arbitrary traffic or every quantization. The export author’s limited 11-case CPU comparison reports maximum probability differences against the upstream torch engine of 0.0021 (FP32, no argmax changes), 0.0504 (q8, no argmax changes), and 0.2074 (q4f16, one argmax change). Classification agreement is not numerical parity or evidence that confidence thresholds remain calibrated. Local q8 native/Python ORT 1.29.0 parity passed the basic three-question and structured five-question requests with exact token IDs/readout positions and a largest displayed probability difference of 1e-8 on the pinned export above. This is adapter parity, not a fresh official torch comparison. Windows and native 4D Strands execution remain unverified here. See the focused checks for pinned revisions, local results and unverified paths.

Converted ONNX Models

Quantisation

The fp32 format is accurate but consumes 4 bytes per weight and pretty slow on a CPU. It is generally not suitable for production. You should only use it as a reference.

The fp16 format consumes 2 bytes per weight. The CPU backend may be forced to perform calculations in float32 except on a CPU like Apple Silicon that has native 16-bit maths which is not as fast as an NVIDIA GPU. It is usually best to avoid this format on a CPU.

The int8 format takes advantage of NEON instructions on Apple Silicon and AVX2 AVX-512 VNNI instructions on Intel or AMD to accelerate maths. For encoders, the accuracy drop is said to be negligible ( less than 1%). You should always use the int8 format on a PC or Mac with no GPU.

The int4 format is designed to compress large language models. Just as a reference, a 7B parameter in native float32 format would requires 28GB of memory, and on a CPU the data must go through the processor for every single token generation. int4 reduces the bandwidth by 8. The format internally groups multiple weights (e.g. 32) to share a scale factor to maintain accuracy. However the quantisation is less precise compared to a GGUF (llama.cpp) model of a similar size.

Rerank

Embedding

Chat Completion