otelma icon

otelma

A local LLM inference runtime, built from scratch. Smaller and more didactic than Ollama, with a fixed unified memory budget as an explicit constraint.

GitHub Go Apple Silicon llama.cpp MLX v1.1

Install

brew tap albertobarrago/otelma
brew install otelma

Builds from the latest release source and pulls in llama.cpp automatically. Formula: homebrew-otelma.

Quick start

otelma list
otelma pull qwen2.5-0.5b        # by catalog name, resolves the source for you
otelma run qwen2.5-0.5b "What is the capital of Italy?"

otelma chat qwen2.5-0.5b        # interactive, multi-turn

No separate "start the server" step: otelma serve auto-starts in the background the first time it's needed. Full docs: Guide, Architecture, Configuration.

Architecture

01
CLI

pull, list, ps, rm, run, chat, serve, config, version — talks to the API over HTTP even for local calls, and auto-starts the server when needed.

02
Local runtime API

Model manager: registry + explicit state machine (NOT_PRESENT → DOWNLOADED → LOADING → READY → BUSY → UNLOADING), with a memory Budget that rejects any load exceeding the ceiling. Scheduler: serializes concurrent requests through the manager. Also exposes a minimal OpenAI-compatible /v1/chat/completions + /v1/models, so any tool with "custom OpenAI endpoint" support can use otelma as its backend.

03
Inference backend abstraction

A common interface behind which concrete engines live: llamacpp (spawns llama-server, GGUF) and mlx (spawns mlx_lm.server, native Apple Silicon), both talking OpenAI-compatible HTTP to their subprocess, plus an echo stand-in for testing.

04
Model storage

Checksum/size of a local model — a single GGUF file, or (for MLX) a whole directory — plus a Hugging Face downloader for both pull <name> hf:<user>/<repo> and pull <name> mlx:<user>/<repo>.

Why the memory budget matters

On 24GB of unified memory, two large models READY at once isn't something you can casually afford. otelma's Budget tracks reservations explicitly and rejects a load before it happens, instead of finding out via a kernel OOM kill.

Status: v1.1

Works end-to-end

  • pull → ps → run/chat pipeline, real inference
  • Interactive multi-turn chat with real context
  • Pull from local path or Hugging Face
  • Curated model catalog (otelma list)
  • Configurable via a single JSON file (otelma config)
  • Server auto-starts in the background when needed
  • Pulled models survive a server restart (registry persists to disk)
  • OpenAI-compatible /v1/chat/completions (no streaming)
  • otelma rm to unregister a model (unloads it first if READY)
  • MLX backend (Apple Silicon native) alongside llamacpp

Known limitations

  • Scheduler: single mutex, no priority queue
  • One backend per server process, no per-model selection