AbstractCore
AbstractCore is an offline-capable, open-source-first LLM layer for Python. One create_llm(...) API works across local runtimes, self-hosted servers, cloud APIs and OpenAI-compatible gateways. Use it in your Python code, or run it as an OpenAI-compatible /v1 server with a web console that picks, downloads and runs local models that fit your machine.
pip install abstractcore
abstractcore serve
# open the printed link: http://127.0.0.1:8000/console#claim=...One API, every provider
Your application code stays the same while the model underneath moves between local runtimes, your own inference server and hosted APIs. The light install covers every remote provider; [apple] and [gpu] add the local engines.
Ten providers behind create_llm()
Local and open-weight backends are first-class; cloud and gateway providers work with the light install.
- ✓ Local servers:
ollama,lmstudio,vllm,openai-compatible - ✓ In-process:
mlx(Apple silicon),huggingface(Transformers and GGUF) - ✓ Cloud APIs:
openai,anthropic - ✓ Gateways:
openrouter,portkey - ✓ No silent downloads: you fetch local weights on purpose (
abstractcore models download ...), then run without internet
from abstractcore import create_llm
# Local model on Apple silicon (MLX). Any provider works the same way, e.g.
# create_llm("openai", model="gpt-5-mini") or create_llm("ollama", model="qwen3:4b").
llm = create_llm("mlx", model="mlx-community/Qwen3.5-4B-4bit")
response = llm.generate("In two sentences, what is a durable workflow runtime?")
print(response.content)abstractcore command and its apps on your PATH (commands you get), so you already have them after an install. Install AbstractCore on its own, as below, when you only need the LLM layer in your own Python environment.Pick, download and run a model that fits
abstractcore serve starts the server on 127.0.0.1:8000, creates its own bearer token on first run and prints a one-time link to AbstractCore’s own web console. The console shows what this machine can run and installs engines with one confirmation. Every action shows the matching command. AbstractGateway has its own consoles; both pairs share the Models and Engines screens.
| Console tab | What you can do | Command |
|---|---|---|
| Overview | See this machine: OS, accelerator, memory and the usable model-memory ceiling, free disk, engines summary, server health. | abstractcore host profile --json |
| Models | Browse the catalog with fit badges, download a model in a click, see installed models and their sizes, delete weights, follow job progress. | abstractcore models download, models list, models delete |
| Engines | Install Ollama, LM Studio, MLX or llama.cpp with one confirmation that shows the exact command; check vLLM and Hugging Face too. | abstractcore engines status --probe, engines install <id> |
| Providers | Read the provider list and the configured capability defaults. | abstractcore config providers --probe |
The same from a terminal
Everything the console does is a command. For a keyboard-first screen, AbstractCore’s terminal console, the abstractcore-console crate (v0.4.1, Rust 1.87+, built on AbstractTUI), has the same Models and Engines screens and also edits your default models, providers, API keys and capability routes.
- ✓ Lost the link?
abstractcore serve --claim-url --port 8000prints a fresh one (valid 10 minutes, one use) - ✓ API keys and default models are set on the host with
abstractcore --config
abstractcore host profile # accelerator, memory ceiling, free disk
abstractcore models catalog --fits # models that fit this machine
abstractcore models download ollama qwen3:8b
abstractcore models list # installed models, with sizes
abstractcore models verify qwen3:8b --provider ollama
abstractcore engines status --probe
abstractcore engines install ollama --dry-run # the exact command
cargo install abstractcore-console # terminal console
abstractcore-consoleFit verdicts
Each catalog entry gets an advisory verdict that compares the memory the model needs (weights, context cache and overhead) with the usable ceiling of this machine.
| Verdict | Meaning |
|---|---|
fits | Runs with headroom. |
tight | Runs, with little headroom. |
partial_offload | CUDA only: too big for VRAM, fits VRAM plus part of RAM (slower). |
needs_gpu_limit | Apple silicon: does not fit the current GPU memory limit, but fits once macOS lets the GPU use more. The verdict carries the exact command, for example sudo sysctl iogpu.wired_limit_mb=114688 for Qwen3.8 Flash-Next 4-bit on a 128 GB Mac (the recommendations table); the setting lasts until the Mac restarts, and both consoles print it. |
too_large | Does not fit. |
unknown | No size or parameter count, or no measurable ceiling. |
engine_missing with the engine, the reason and an install command for this interpreter. That is different from route_unavailable (this host cannot run the engine at all) and from weights that are not downloaded yet.What you get
The parts that usually break when you move beyond one hosted API: tool syntax, structured output, media support, token budgets and caching.
Universal tool calling
Decorate functions with @tool. Native and prompted tool syntax are normalized. Core is pass-through by default (execute_tools=False): it returns response.tool_calls and your host or runtime decides how to run them.
Structured output
Pass a Pydantic model as response_model= and get a typed object back, with provider-aware strategies and retry.
Input: text, images, audio, video, documents
Attach files with media=[...] next to your prompt (part of the light install: Media input). Nothing is changed silently: images go to a vision-capable model, or a text-only model gets short observations from a configured vision fallback; audio follows audio_policy: speech is transcribed through AbstractVoice (the input.voice route) or sent natively to an audio-capable model, while sound and music understanding come in through audio-capable models such as Qwen2.5-Omni (the input.sound and input.music routes, no dedicated engine); a dedicated music input is planned (backlog 0986); video follows video_policy (native video, or sampled frames sent through the vision path).
Output: text, voice, music, images, video, 3D
The same generate(..., output=...) returns speech and voice clones, music, images with edits and upscales, text-to-video and image-to-video, and 3D scenes through capability plugins. Text-only generate() is unchanged.
Capability plugins
llm.voice and llm.audio (AbstractVoice), llm.music (AbstractMusic), llm.vision (AbstractVision), llm.scene3d (Abstract3D, text and image to 3D) and llm.camera (AbstractCamera, real-camera piloting).
Native MLX runtime
In-process Qwen3.8 inference with vision, selectable MTP depth, concurrent requests and prefix reuse. Stop any generation with cancel_event=.
Embeddings
EmbeddingManager for semantic search and RAG, with local models or Ollama, LM Studio, OpenAI, OpenRouter, Portkey and OpenAI-compatible endpoints.
Sessions and prompt caching
BasicSession keeps conversation state; prompt caching, events, interaction tracing and retry hooks help you run it in production.
MCP, glyphs, CLI apps
Use MCP (Model Context Protocol) servers, over HTTP or stdio, as tool sources, compress long documents into images for vision models (glyphs), and use the built-in summarizer, extractor, judge, intent and deepsearch apps.
One call shape for every model and modality
You create an LLM once, naming its provider (it is required: create_llm), then every request goes through generate: inputs (images, audio, video, documents) go in media=, and the output modality (text by default, or voice, music, image, video, 3D) in output=.
llm = create_llm(provider, model=...)
llm.generate(prompt, media=[...], output=...)Code examples
Set the capability routes once per machine, then each output is one call, and the same model reads the picture back through media=.
# Once per machine (the gateway console's "Use recommended defaults" writes the same store)
abstractcore config set-default output.image --provider mlx-gen --model AbstractFramework/flux.2-klein-4b-8bit
abstractcore config set-default output.voice --provider supertonic --model supertonic-3
abstractcore config set-default output.music --provider acestep --model ACE-Step/acestep-v15-xl-turbo-diffusers# Defaults set once: see defaults.sh (or the gateway console's "Use recommended defaults")
from abstractcore import create_llm
llm = create_llm("mlx", model="mlx-community/Qwen3.5-4B-4bit")
image = llm.generate("A watercolor illustration of a lighthouse on a rocky coast at dawn, soft light, calm sea", output="image")
speech = llm.generate("AbstractFramework: write once, generate everything, run anywhere.", output="voice")
music = llm.generate("Calm ambient piano with soft strings, instrumental", output="music")
open("lighthouse.png", "wb").write(image.outputs["image"][0].data)
open("tagline.wav", "wb").write(speech.outputs["voice"][0].data)
open("ambient.wav", "wb").write(music.outputs["music"][0].data)
# Ask the text model about the picture it just made (the image goes in through media=)
answer = llm.generate("What is in this picture?", media=["lighthouse.png"])
print(answer.content)What the run printed (examples/R6/with_defaults.run.log; RESULTS.md R6): “This picture depicts a serene, watercolor-style illustration of a lighthouse situated on a rocky outcrop. Here are the key details:”
This picture depicts a serene, watercolor-style illustration of a lighthouse situated on a rocky outcrop. Here are the key details:
- The Lighthouse: It is a white, cylindrical tower with a stone foundation. It features a few small rectangular windows and a reddish-brown arched door at the base. At the very top is a dark lantern room with a glass enclosure, from which a warm light appears to be glowing.
- The Setting: The lighthouse is perched on a cluster of large, grey rocks that are covered in patches of green vegetation. The water around the rocks is depicted in shades of blue with white foam, suggesting gentle waves.
- The Water: The foreground shows the water with soft, wavy patterns and distinct reflections of the lighthouse tower and the sky.
- The Sky: The sky is painted with soft, blended washes of pastel colors—yellows, pinks, and light blues—creating a peaceful atmosphere that suggests either a sunrise or sunset.
- Background: In the distance, there is a faint, hazy outline of land or hills on the horizon.
Overall, the image has a soft, artistic quality typical of watercolor painting, with visible texture on the "paper."
from abstractcore import create_llm
# generate() is a method on an LLM instance: the text model is the entry point; each output below names its own engine
llm = create_llm("mlx", model="mlx-community/Qwen3.5-4B-4bit")
image = llm.generate("A watercolor illustration of a lighthouse on a rocky coast at dawn, soft light, calm sea",
output={"modality": "image", "provider": "mlx-gen", "model": "AbstractFramework/flux.2-klein-4b-8bit"})
speech = llm.generate("AbstractFramework: write once, generate everything, run anywhere.",
output={"modality": "voice", "provider": "supertonic", "model": "supertonic-3"})
music = llm.generate("Calm ambient piano with soft strings, instrumental",
output={"modality": "music", "provider": "acestep", "model": "ACE-Step/acestep-v15-xl-turbo-diffusers"})
open("lighthouse.png", "wb").write(image.outputs["image"][0].data)
open("tagline.wav", "wb").write(speech.outputs["voice"][0].data)
open("ambient.wav", "wb").write(music.outputs["music"][0].data)
# Ask the text model about the picture it just made (the image goes in through media=)
answer = llm.generate("What is in this picture?", media=["lighthouse.png"])
print(answer.content)What the run printed (examples/R6/no_defaults.run.log; RESULTS.md R6): “This picture depicts a serene watercolor painting of a lighthouse. Here are the key elements:”
This picture depicts a serene watercolor painting of a lighthouse. Here are the key elements:
- The Lighthouse: A tall, white lighthouse stands centrally in the composition. It features a red lantern room at the very top and a small flag flying on a pole. It is situated on a raised, rocky platform with railings around the base and the lantern room.
- The Rocks: The lighthouse is perched atop a rugged, jagged cliff of brown and grey rocks. There is some green vegetation, likely moss or grass, growing on the rocks.
- The Ocean: The water is painted in varying shades of blue and white, suggesting gentle waves lapping against the rocks. The water reflects the lighthouse and the rocks below.
- The Sky: The background shows a soft, pastel sky with a gradient of colors ranging from pale yellow near the horizon to light blue and pink clouds above, suggesting a peaceful sunrise or sunset.
Overall, the image has a soft, dreamy quality typical of watercolor art.
# <token>: printed by abstractcore serve --print-token
# Start the server first (another terminal): abstractcore serve --port 19740
curl -sS http://127.0.0.1:19740/v1/audio/speech -H "Authorization: Bearer <token>" \
-H "Content-Type: application/json" \
-d '{"provider": "supertonic", "model": "supertonic-3",
"input": "AbstractFramework: write once, generate everything, run anywhere.", "response_format": "wav"}' \
--output tagline.wav
curl -sS http://127.0.0.1:19740/v1/audio/music -H "Authorization: Bearer <token>" \
-H "Content-Type: application/json" \
-d '{"provider": "acestep", "model": "ACE-Step/acestep-v15-xl-turbo-diffusers",
"prompt": "Calm ambient piano with soft strings, instrumental", "response_format": "wav"}' \
--output ambient.wav
curl -sS http://127.0.0.1:19740/v1/images/generations -H "Authorization: Bearer <token>" \
-H "Content-Type: application/json" \
-d '{"provider": "mlx-gen", "model": "AbstractFramework/flux.2-klein-4b-8bit",
"prompt": "A watercolor illustration of a lighthouse on a rocky coast at dawn, soft light, calm sea",
"width": 1024, "height": 1024, "response_format": "b64_json"}' > image.json
# Save the generated picture, then ask the chat model about it: the PNG travels as a base64 data URL
python3 -c 'import base64,json; open("lighthouse.png","wb").write(base64.b64decode(json.load(open("image.json"))["data"][0]["b64_json"]))'
IMG=$(base64 < lighthouse.png | tr -d '\n')
cat > question.json <<JSON
{"model": "mlx/mlx-community/Qwen3.5-4B-4bit", "reasoning_effort": "none",
"messages": [{"role": "user", "content": [
{"type": "text", "text": "What is in this picture?"},
{"type": "image_url", "image_url": {"url": "data:image/png;base64,$IMG"}}]}]}
JSON
curl -sS http://127.0.0.1:19740/v1/chat/completions -H "Authorization: Bearer <token>" \
-H "Content-Type: application/json" -d @question.jsonThe run used port 19740; abstractcore serve listens on 127.0.0.1:8000 by default.
What the run printed (examples/R6/http.run.log; RESULTS.md R6; its bullet glyphs omitted): “This is a watercolor painting of a lighthouse situated on a rocky coastal cliff, overlooking the sea.”
This is a watercolor painting of a lighthouse situated on a rocky coastal cliff, overlooking the sea.
Here’s a detailed breakdown of what’s in the picture:
Lighthouse:
- A tall, white cylindrical tower with a red band around its base and a black lantern room on top.
- It has small windows along its shaft and a gallery (railing) around the lantern room.
- A smaller, connected building (possibly a keeper’s house or utility room) sits beside the base of the lighthouse.
Rocks & Coastline:
- The lighthouse stands on a rugged, rocky outcrop that juts into the sea.
- Large, weathered boulders are scattered around the base and extend into the water, suggesting a tidal or shallow shoreline.
- Some greenery (grass or low shrubs) grows among the rocks.
Sea & Water:
- The water is depicted with soft, flowing brushstrokes in blues and whites, suggesting gentle waves.
- There’s a reflection of the lighthouse and sky on the water’s surface near the rocks.
Sky & Atmosphere:
- The sky is painted with warm pastel tones — pinks, yellows, and light blues — evoking a sunset or sunrise.
- Soft clouds are visible, adding to the serene, tranquil mood.
- In the distance, faint landforms (hills or mountains) appear on the horizon, blending into the sky.
Art Style:
- This is clearly a watercolor illustration, likely digital or AI-generated to mimic traditional watercolor effects.
- The edges are intentionally soft and slightly irregular, giving it an organic, hand-painted feel.
- The lighting is warm and atmospheric, enhancing the peaceful, romantic coastal scene.
Overall, the image captures a classic, idyllic maritime scene — peaceful, nostalgic, and beautifully rendered with impressionistic watercolor techniques. It evokes feelings of solitude, safety, and the timeless beauty of coastal landscapes.
An OpenAI-compatible /v1 endpoint
Apps that already speak the OpenAI API can use any provider through AbstractCore: route with model="provider/model". Optional /v1/audio/*, /v1/images/* and /v1/videos/* routes serve generated media, and Swagger UI is at /docs.
Authenticated by default
On your own machine the server creates its bearer token on first run and keeps it readable by you only. On a network, set ABSTRACTCORE_AUTH_TOKEN yourself and pass --host 0.0.0.0.
- ✓
abstractcore serve --print-tokenprints the token for API clients - ✓
abstractcore-endpointserves one provider and model per worker - ✓ Docker:
ghcr.io/lpalbou/abstractcore-server:2.19.0, built from PyPI with the server and the remote providers (no local engines)
The curl requests for chat, speech and images are in the OpenAI-compatible endpoint tab above. Any OpenAI SDK works the same way, against abstractcore serve on its default 127.0.0.1:8000.
# <token>: printed by abstractcore serve --print-token
from openai import OpenAI
client = OpenAI(base_url="http://127.0.0.1:8000/v1", api_key="<token>")
resp = client.chat.completions.create(
model="mlx/mlx-community/Qwen3.5-4B-4bit", # route to any provider/model, e.g. "ollama/qwen3:4b"
messages=[{"role": "user", "content": "In one sentence, what is an append-only ledger?"}],
reasoning_effort="none", # no thinking phase for a one-line answer
)
print(resp.choices[0].message.content)Three installs
Pick the one that matches where your models run. Apple and GPU contain everything in Light and add local engines.
Light
Every remote provider: OpenAI, Anthropic, OpenRouter, Portkey, LM Studio, Ollama, vLLM and any OpenAI-compatible endpoint, plus the built-in tools, media input, the HTTP server and the voice, music, vision and 3D plugins with their remote backends.
pip install abstractcoreApple Light +
Everything in Light, plus every local engine Apple Silicon runs: MLX for text and image input, Hugging Face (Transformers and GGUF), local embeddings, and the local voice, image and video, and music engines.
pip install "abstractcore[apple]"GPU Light +
Everything in Light, plus every local engine NVIDIA and AMD machines run: vLLM in-process, Hugging Face (Transformers and GGUF), local embeddings, and the local voice, image and video, and music engines.
pip install "abstractcore[gpu]"The light install already carries the OpenAI-compatible server and its web console (Server) and the AbstractVoice, AbstractMusic, AbstractVision and Abstract3D plugins (what each adds); AbstractCamera installs on its own: pip install abstractcamera.
Music defaults to the lightweight remote ACE Music backend: set ACEMUSIC_API_KEY before use. Local music engines come with [apple] and [gpu]. To upgrade a library install, run pip install -U abstractcore (or the [apple] / [gpu] form you installed); an installer install upgrades with the framework (Upgrade).