Documentation Overview
ReleasesGitHub

Local inference · 4D integration

Any model.
Any architecture.

Bring local AI to Mac and Windows. One CPU-first inference server for chat, embeddings, reranking, speech, and decisions — with a 4D component to connect it all.

Make your first request Use with 4D
The local stackOn your machine
Your application4D component or any HTTP client
HTTP / 127.0.0.1:8080
onnx-genai serverONNX Runtime · CPU-firstInference
loads
A model you chooseONNX GenAI model directory
ChatEmbeddingsRerankingSpeechDecisions
Mac & WindowsIntel & ARMOpen source · MIT4D 20+

More than a conversation

One server. A whole AI toolkit.

Load the models your application needs. Call familiar HTTP endpoints from 4D, your terminal, or another client.

01 / GENERATE

Chat & completions

Build conversational features with local language models and an OpenAI-compatible chat endpoint.

/v1/chat/completions
02 / RETRIEVE

Embeddings & reranking

Turn text into vectors, add contextualized embeddings, and rank documents for retrieval.

/v1/embeddings · /v1/rerank
03 / SPEAK

Text to speech

Give your application a voice with a compatible speech model, served through the local API.

/v1/audio/speech
04 / DECIDE

Structured decisions

Evaluate choices with dedicated Kev, d1, Laya, or Strands decision models — separately from chat.

/v1/systemone

Each capability requires its own compatible model. API compatibility is endpoint-specific, not a complete implementation of the upstream services. Read the API & model reference →

Two ways in

Start standalone. Or start in 4D.

01 / NATIVE EXECUTABLE

Your tools. Your workflow.

Start the server from the command line and send HTTP requests. No 4D installation needed.

Explore the CLI → 02 / 4D COMPONENT

At home in your app.

Let the component download configured models, launch the server in the background, and shut it down when needed.

Integrate with 4D →

Compatibility before complexity

Local AI. Without a dedicated GPU.

This distribution primarily runs inference on CPU cores for broad compatibility. Choose a model and quantization that fit your machine's memory and performance budget.

Deployment at a glance
PlatformArchitectureWhat to know
macOSApple Silicon & IntelUniversal runtime ONNX Runtime targets macOS 13.3+. Other bundled dependencies may require a newer system.
Windowsx64 & ARM64Deploy the matching executable and its runtime libraries together.
4D integration4D 20+Optional for direct HTTP use. Use the component for application-managed inference.
ModelsCompatible ONNX exportsDownload separately. Keep required tokenizer, configuration, and external data files with the model.
ONNX Runtime 1.29.0, bundled.

GenAI 0.12.0 and Extensions 0.14.0 are separate dependencies. Not every ONNX model is interchangeable: chat uses GenAI-compatible exports; other tasks use their own model layouts. Runtime provenance & build details →

A short path to inference

From a local model to a first reply.

Get a matching build from the releases or build from source. Keep its runtime libraries alongside the executable, then download a compatible chat model. Replace the example path with its complete GenAI model directory.

01 — Start the serverShell / macOS
./onnx-genai -s \
  -m /path/to/chat-model \
  -h 127.0.0.1 \
  -p 8080

On Windows, use onnx-genai.exe with the same flags and your local model path. Leave the server running. In another terminal, wait until model discovery succeeds; starting the process does not mean model loading is complete.

02 — Discover loaded modelsShell
curl http://127.0.0.1:8080/v1/models

Then send a message to the loaded chat model:

03 — Make a requestShell / macOS
curl http://127.0.0.1:8080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "messages": [
      {"role": "user", "content": "What can I build with local AI?"}
    ],
    "max_tokens": 128,
    "temperature": 0.7,
    "stream": false
  }'
Keep local inference local.

The examples bind to 127.0.0.1. Do not expose the listener to a network without deliberately securing access. Model downloads require a connection; review each model's license before deployment.

Build something that stays with you

Put local inference to work.

Connect through AI Kit, explore model options, or build the native engine for your target platform.

4D integration guide CLI & modelsBuild from source