AbstractCore — Documentation

Overview

What AbstractCore Is

A unified Python interface over multiple LLM backends with consistent support for streaming, tool calling, structured output, media handling, embeddings, and an optional OpenAI-compatible HTTP server.

One API, Every Provider

Write your code once and run it across cloud providers, local servers, and in-process engines. AbstractCore normalizes provider differences so you can switch backends without rewriting your application.

✓ Cloud: OpenAI, Anthropic
✓ Gateways: OpenRouter, Portkey
✓ Local HTTP: Ollama, LM Studio, vLLM, any OpenAI-compatible endpoint
✓ In-process: MLX (Apple Silicon), HuggingFace (GPU/CPU)

from abstractcore import create_llm

# Cloud provider
llm = create_llm("openai", model="gpt-4o-mini")

# Local server — same API
llm = create_llm("ollama", model="qwen3:4b")

# In-process on Apple Silicon
llm = create_llm("mlx", model="mlx-community/Qwen3-4B")

resp = llm.generate("Say hello in French.")
print(resp.content)

Design Principles

AbstractCore is designed to be lightweight by default. The core install is small and fast. Heavy dependencies stay behind optional extras and are imported lazily.

✓ Lightweight default install — no torch, no transformers, no server deps unless you ask
✓ Pass-through tool execution — Core returns tool calls; your host/runtime executes them
✓ Policy-driven media — audio/video/image input is explicit, never silently altered
✓ Composable extras — install only the provider SDKs and features you need

# Lightweight core (HTTP-only providers)
pip install abstractcore

# Add only what you need
pip install "abstractcore[openai]"
pip install "abstractcore[anthropic]"
pip install "abstractcore[media]"
pip install "abstractcore[server]"

# Turnkey Apple Silicon install
pip install "abstractcore[all-apple]"

# Turnkey NVIDIA GPU install
pip install "abstractcore[all-gpu]"

Capabilities

Features

Everything you need to build production AI applications, from basic text generation to multimodal pipelines.

Providers

9+ LLM Backends

OpenAI, Anthropic, OpenRouter, Portkey, Ollama, LM Studio, vLLM, MLX, HuggingFace — and any generic OpenAI-compatible endpoint. Same create_llm() interface across all.

Streaming

Unified Streaming

Pass stream=True to any provider and iterate over chunks. Works identically across cloud, local HTTP, and in-process backends.

Agentic

Tool Calling

Decorate functions with @tool for a universal tool representation. Pass-through by default — your host or runtime executes; Core normalizes across providers.

Structured

Structured Output

Pass a Pydantic model as response_model=... and get typed objects back. Uses native provider structured output when available, prompted strategies + validation otherwise.

Multimodal

Media Handling

Policy-driven image, audio, video, and document input. Configurable vision/audio/video fallback pipelines. Capability plugins for TTS, STT, image gen, video gen, and music.

Embeddings

Embedding Manager

Install abstractcore[embeddings] and use EmbeddingManager for text embeddings with local models. Clean API for RAG pipelines and semantic search.

Server

OpenAI-Compatible Server

Turn AbstractCore into an API gateway with /v1/chat/completions, optional image/audio/video endpoints, Swagger UI, and configurable server auth.

Config

CLI & Centralized Config

Persistent configuration at ~/.abstractcore/config/. Set API keys, default models, vision/audio/video strategies, logging, and server settings from the command line.

Sessions

Session & Prompt Caching

BasicSession for multi-turn state. CachedSession adds provider prompt caching with stable prefix reuse. Attach files as context — cache them across turns.

Quickstart

Getting Started

From zero to generating in under a minute.

1. Install

pip install abstractcore

# Add provider SDK extras as needed
pip install "abstractcore[openai]"      # OpenAI SDK
pip install "abstractcore[anthropic]"   # Anthropic SDK

2. Minimal Generation

from abstractcore import create_llm

llm = create_llm("openai", model="gpt-4o-mini")
resp = llm.generate("Say hello in French.")
print(resp.content)

3. Streaming

for chunk in llm.generate("Write a short poem.", stream=True):
    print(chunk.content or "", end="", flush=True)

4. Tool Calling

from abstractcore import create_llm, tool

@tool
def get_weather(city: str) -> str:
    """Return weather for a city."""
    return f"{city}: 22C and sunny"

llm = create_llm("openai", model="gpt-4o-mini")
resp = llm.generate(
    "What's the weather in Paris?",
    tools=[get_weather]
)
print(resp.tool_calls)  # structured calls for your host to execute

5. Structured Output

from pydantic import BaseModel
from abstractcore import create_llm

class Answer(BaseModel):
    title: str
    bullets: list[str]

llm = create_llm("openai", model="gpt-4o-mini")
result = llm.generate(
    "Summarize HTTP/3 in 3 bullets.",
    response_model=Answer
)
print(result.title)
print(result.bullets)

6. Sessions & Multi-Turn

from abstractcore import BasicSession, create_llm

session = BasicSession(
    create_llm("anthropic", model="claude-haiku-4-5"),
    temperature=0.3
)
print(session.generate("Give me 3 startup ideas.").content)
print(session.generate("Pick the best one and explain why.").content)

Reference

API Surface

Key patterns and provider configuration.

Providers & Environment Variables

Cloud

OpenAI
OPENAI_API_KEY

Anthropic
ANTHROPIC_API_KEY

Gateways

OpenRouter
OPENROUTER_API_KEY

Portkey
PORTKEY_API_KEY

Local HTTP

Ollama
OLLAMA_BASE_URL

LM Studio
LMSTUDIO_BASE_URL

vLLM
VLLM_BASE_URL

OpenAI-compat
OPENAI_BASE_URL

In-Process

MLX
Apple Silicon

HuggingFace
GPU / CPU

Media Handling & Policies

Media input is policy-driven. Audio and video require explicit policy configuration to avoid silent semantic changes.

from abstractcore import create_llm

llm = create_llm("anthropic", model="claude-haiku-4-5")
resp = llm.generate("Describe this image.", media=["./image.png"])
print(resp.content)

# Audio/video policies: native_only | speech_to_text | auto | caption
# Configure via CLI:
#   abstractcore --set-audio-strategy auto
#   abstractcore --set-video-strategy auto

Embeddings

pip install "abstractcore[embeddings]"

from abstractcore.embeddings import EmbeddingManager

em = EmbeddingManager()
vec = em.embed_text("hello world")
print(len(vec))

OpenAI-Compatible Server

pip install "abstractcore[server]"
python -m abstractcore.server.app

# Health check
curl http://localhost:8000/health

# Chat completions (provider/model format)
curl -X POST http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "openai/gpt-4o-mini", "messages": [{"role": "user", "content": "Hello!"}]}'

# Swagger UI: http://localhost:8000/docs

Capability Plugins

AbstractCore discovers optional modality backends via Python entry points:

AbstractVoice

TTS & STT via llm.voice and llm.audio. Local and server-backed speech pipelines.

AbstractVision

Image & video generation via llm.vision. Text-to-image, image-to-image, text-to-video, image-to-video.

AbstractMusic

Text-to-music via llm.music. Local ACE-Step backend for in-process generation.

CLI Configuration

# Interactive config wizard
abstractcore --config

# Set API keys
abstractcore --set-api-key openai sk-...
abstractcore --set-api-key anthropic sk-ant-...

# Set default models
abstractcore --set-chat-model openai/gpt-4o-mini
abstractcore --set-code-model anthropic/claude-haiku-4-5

# Media strategies
abstractcore --set-video-strategy auto
abstractcore --set-audio-strategy auto

# Check status
abstractcore --status

Install Extras Reference

Extra	What it adds
`openai`	OpenAI Python SDK
`anthropic`	Anthropic Python SDK
`remote`	OpenAI + Anthropic SDKs
`huggingface`	Transformers + Torch (heavy)
`mlx` / `apple`	Apple Silicon local LLM (heavy)
`vllm` / `gpu`	NVIDIA GPU inference (heavy)
`tools`	Built-in web search & filesystem tools
`media`	Image + PDF/Office document extraction
`embeddings`	EmbeddingManager + local models
`server`	OpenAI-compatible /v1 HTTP server (FastAPI)
`tokens`	Precise token counting (tiktoken)
`all-apple` / `all-gpu`	Full local dev environment (turnkey)