orvlet engine
A model server
that lives in the app
A bundled native helper built in Swift on MLX and llama.cpp. It starts only when a request needs it, listens only on this Mac, and requires a token for every call.
Photograph — Tyler via Unsplash
Speed on Apple Silicon
Techniques that make a local model feel quick, each one checked against plain decoding before it is trusted.
- Speculative decodingDraft models, prompt lookup, self-speculation and EAGLE-3 draft heads propose tokens the target model verifies.
- Multi-token predictionFor models that ship prediction heads, with proofs that greedy and seeded output is token-identical to plain decoding and a depth autotuner that follows the acceptance rate.
- Parallel draftsSeveral draft candidates verified together in one vectorized step.
- Prompt cachingPrefix reuse across turns and between parallel subagents, and a per-session cache on SSD that survives a restart.
- Continuous batchingConcurrent agent tasks share the model with chunked prefill and a prefix tree for shared prompts.
- Fused kernelsCustom Metal kernels, ahead-of-time shader compilation and fused decode megakernels for supported models.
Memory, managed
Unified memory is the budget. The Engine shows how it is spent and keeps a model from taking the whole Mac.
- Quantized KV cache at 8 or 4 bits as a load option, per model or preset.
- Quantize as it loads, including 2- and 3-bit weights for large models, plus a GGUF-to-MLX converter.
- Long conversations keep going with context shifting and sliding windows instead of a full recompute.
- Weight paging streams very large models from fast storage, and memory-mapped loading avoids duplicate copies of a checkpoint.
- Memory limits and controls: an admission budget below physical memory, idle unload, pinning and cache compaction.
- A fit estimate before you load, with memory by context length for every library model.
One port, familiar APIs
Point your own scripts and tools at the Engine. Settings → Engine API lists every route with copyable examples.
Chat and agents
OpenAI Chat Completions and Responses (with WebSocket and durable background responses), Anthropic Messages with token counting, and Ollama-compatible chat, generate and model routes — side by side on one loopback port.
Structured output
Responses constrained to a JSON schema or grammar, tool-call parsers per model family, separated reasoning, and log-probabilities.
Beyond text
Embeddings and reranking (text and image), tokenization, image generation and editing, speech transcription and neural text-to-speech, and vision for multi-image requests.
Typed decisions
Ask a model to choose one answer from a closed set — to classify, route or pick the next step — with probabilities, from text or image context.
Adapters on demand
Attach, swap and stack LoRA adapters per request on a running base model without reloading it.
A provider for other apps
The Engine can act as a Foundation Models provider for other apps on this Mac, and an embeddable client lets isolated tools call it.
Never selected for you. Never started at launch. Never listening beyond this Mac unless you share it.
Photograph — Stephan Widua via Unsplash
More than one Mac
Turn on sharing and the Engine advertises itself on your network over Bonjour; nothing connects until you confirm it. Shared access uses TLS with a pinned identity and a key. For models larger than one machine, a GGUF model's layers can be split across two to eight Macs you pair, in proportion to each one's free memory, with recovery if a worker drops out.
Instruments, not guesses
The Engine page shows speed, memory, recent requests and a redacted local log. A floating overlay reports measured memory, GPU use and thermal state; a latency waterfall breaks down each request from queue to first token; a single-request flow view shows what happens inside one inference. Benchmarks compare settings on the same machine.