ai_client

Talk to LLM providers over HTTP from GreyCat. Three wire formats behind one transport: the OpenAI-compatible surface that nearly every server speaks, plus Anthropic’s and Ollama’s native APIs for the parts that surface cannot reach.

Pure GCL on top of the http library. This is the counterpart to the ai library, which runs a model in-process; this one calls one over the network.

@library("ai_client", "0.0.0");

http comes along with it and does not need declaring.

Which module

Module	Wire format	Use it for
openai	OpenAI-compatible /v1	Anything portable. Point base_url at any of the servers below.
anthropic	Anthropic native /v1/messages	Prompt caching, thinking, citations, PDF input.
ollama	Ollama native /api/*	Model management: pull, tags, show, ps, copy, delete.

Every type is private, so it is reached qualified: openai::Client, anthropic::Client, ollama::Client. The three formats name the same concepts differently often enough that bare names would collide.

Portability

/chat/completions is the only endpoint every OpenAI-compatible server implements. openai::Client reaches all of these by base_url alone:

Server	chat	embeddings	models	Notes
OpenAI	yes	yes	yes
Anthropic compat	yes	no	no	/v1/models takes only x-api-key
Ollama	yes	yes	yes
vLLM	yes	yes	yes
llama.cpp	yes	yes	yes
LM Studio	yes	yes	yes
Groq	yes	no	yes
Gemini compat	yes	yes	yes
OpenRouter	yes	no	yes

A provider that does not support an optional request field ignores it silently rather than failing, so a request relying on one can appear to succeed while doing something else. The worst offenders are documented on the fields themselves; the short version:

• Anthropic compat ignores seed, logprobs, top_logprobs, presence_penalty, frequency_penalty, logit_bias, service_tier, store, metadata, modalities and user, and caps temperature at 1. It rejects rather than ignores n above 1, and response_format unless it is a json_schema carrying strict. It drops prompt caching and thinking entirely, which is what anthropic::Client exists for. Its /v1/models authenticates only with x-api-key, so openai::Client.models() gets a 401 there unless extra_headers supplies one; the completion endpoints do take the bearer token.
• Ollama ignores tool_choice, logit_bias, user and n.
• Groq rejects logprobs, logit_bias and top_logprobs.

Chat

var client = openai::Client {
    base_url: "http://localhost:11434/v1",   // or openai::Client::openai(key)
    api_key: System::getEnv("OPENAI_API_KEY"),
    timeout: 60s,
};

var res = client.chat(openai::CreateChatCompletionRequest {
    model: "llama3.2:3b",
    messages: Array<openai::Message> {
        openai::Message::system("You are terse."),
        openai::Message::user("Capital of France?"),
    },
    temperature: 0.0,
});
println(res.choices!![0].message?.content);

Set both max_completion_tokens and max_tokens: OpenAI deprecated the second in favour of the first, and several compatible servers still read only the second.

Streaming

One read per Server-Sent Event. The reader ends on the [DONE] sentinel every compatible endpoint closes with, so the loop never meets an event it cannot deserialize.

var reader = client.chat_stream(request);
while (reader.can_read()) {
    var chunk = openai::Client::next_chunk(reader);
    for (_, choice in chunk.choices?) {
        print(choice.delta?.content ?? "");
    }
}
reader.close();

Ask for stream_options: openai::StreamOptions { include_usage: true } to get a final chunk carrying usage for the whole request; its choices is empty.

The reader holds the connection open, so it can neither be persisted nor resumed: drain it, or close it, from the task that opened it.

Tools

var tool = openai::Tool::of("get_weather", "Weather for a city", Map<String, any?> {
    "type": "object",
    "properties": Map<String, any?> { "city": Map<String, any?> { "type": "string" } },
    "required": ["city"],
});

var first = client.chat(openai::CreateChatCompletionRequest {
    model: model,
    messages: Array<openai::Message> { openai::Message::user("Weather in Paris?") },
    tools: Array<openai::Tool> { tool },
});

var call = first.choices!![0].message?.tool_calls!![0];
var args = Json<any> {}.parse(call.function!!.arguments!!);   // a JSON *string* on the wire

// Feed the assistant turn back unchanged, then answer the call.
var second = client.chat(openai::CreateChatCompletionRequest {
    model: model,
    messages: Array<openai::Message> {
        openai::Message::user("Weather in Paris?"),
        first.choices[0].message!!,
        openai::Message::tool(call.id!!, "{\"temp_c\":18}"),
    },
});

The strict OpenAI endpoint restricts function names to [A-Za-z0-9_-]+, so a name carrying a dot is rejected there even though local servers accept it.

Images

Message.content takes a String or an Array<ContentPart>:

openai::Message::user(Array<openai::ContentPart> {
    openai::ContentPart::text("What is in this image?"),
    openai::ContentPart::image("https://example.com/cat.png"),
});

Embeddings and models

var vectors = client.embeddings(openai::CreateEmbeddingRequest {
    model: "nomic-embed-text",
    input: ["hello", "world"],          // a String, or an Array<String>
});
println(vectors.data!![0].embedding?.size());

for (_, model in client.models().data?) {
    println(model.id);
}

Anthropic native

Reach for this when the compatibility layer is not enough. The shape differs in three ways: the system prompt is a top-level field rather than a message, content is a list of typed blocks rather than a string, and max_tokens is required.

var claude = anthropic::Client::anthropic(System::getEnv("ANTHROPIC_API_KEY"));

var message = claude.messages(anthropic::CreateMessageParams {
    model: "claude-opus-4-5",
    max_tokens: 1024,
    // Everything up to a cache_control breakpoint is cached, and read back at a
    // fraction of the input rate by a later request sharing that prefix.
    system: Array<anthropic::ContentBlock> {
        anthropic::ContentBlock {
            type: "text",
            text: long_document,
            cache_control: anthropic::CacheControl::ephemeral_1h(),
        },
    },
    thinking: anthropic::ThinkingConfig::enabled(2000),
    messages: Array<anthropic::InputMessage> { anthropic::InputMessage::user("Summarize it.") },
});

println(message.text());                                  // text blocks, joined
println(message.usage?.cache_read_input_tokens);          // billed at a discount
for (_, use in message.tool_uses()) {
    println("${use.name} ${Json::to_string(use.input)}");  // input is decoded, not a string
}

count_tokens prices a request without running it. Streaming works the same way as above but yields MessageStreamEvent, which carries every event type; switch on event.type. There is no [DONE]: the sequence ends with message_stop.

Ollama native

Inference is better served by the compatible surface. This module is for the things /v1 cannot do at all:

var ollama = ollama::Client::local();

for (_, model in ollama.tags().models?) {
    println("${model.name} ${model.details?.parameter_size}");
}

// Check what a model can do before sending it tools or an image.
println(ollama.show(ollama::ShowRequest::of("llama3.2:3b")).capabilities);

// Pull with progress.
var pull = ollama.pull(ollama::PullRequest::of("llama3.2:3b"));
while (pull.can_read()) {
    var event = ollama::Client::next_status(pull);
    println("${event.status} ${event.completed}/${event.total}");
}
pull.close();

println(ollama.ps().models);        // loaded right now, and when each expires
ollama.delete(ollama::DeleteRequest::of("old-model"));

keep_alive on a request decides how long the model stays resident, and options.num_ctx sets the context window. Neither has an OpenAI equivalent.

Demo chat app

webroot/index.html is a single-file chat page in GreyCat’s own colours, backed by src/chat.gcl:

CHAT_PASSWORD=secret greycat serve      # then open http://localhost:8080

Sign in with root and that password. It lists the models the daemon can chat with, streams answers token by token, renders them as markdown, and shows the model’s reasoning in a collapsible block when thinking is on.

Conversations live in the graph, so they survive a restart and can be reopened and continued from the sidebar. They are held in a nodeIndex<String, node<Conversation>> keyed by Uuid::v7(): v7 leads with its timestamp, so the index walks in creation order. The title is inferred by asking the model to name the first exchange, and can be replaced with chat_rename.

The browser holds no history at all. It sends the next prompt and a conversation id, and the server replays the stored turns, so resuming an old conversation is just chat_open.

The browser also never reaches Ollama: it calls chat_setup and chat_send, and those use ollama::Client. So the daemon needs no CORS opening and no exposure beyond this server, and the page authenticates through runtime::Identity::login, which sets the session cookie, like any other GreyCat endpoint.

Streaming works around path-RPC, which cannot hold a response open: chat_send runs as a background task (task: true) and appends one JSON object per token to files/<user>/chat_<conversation_id>_<task_id>.ndjson, flushing each, which the page polls. Those file writes are visible while the task still runs, where graph writes would not be, since a task commits its graph changes only at the end. The name carries the conversation id because task ids restart from one whenever the store is wiped, and a task id alone collides with a file left by an earlier run. Nothing prunes those files, so a long-lived server wants a sweep.

The page pulls marked and DOMPurify from a CDN, both pinned with an integrity hash. Answers are markdown produced by a model, and marked does not sanitize, so its output is purified before it reaches the DOM.

Errors

A non-2xx throws, carrying the provider’s own message:

try {
    client.chat(request);
} catch (e) {
    println(e.message);   // openai: HTTP 429 rate_limit_exceeded: Rate limit reached
}

The three providers disagree on the shape of an error body; all three are read, and anything unrecognized falls back to the body as it came, which is what a proxy in front of the provider tends to send.

To branch on the status instead of catching, call AiHttp::send yourself: it returns the response whatever the status and throws nothing.

Configuration

Every client takes timeout, max_response_size and extra_headers. extra_headers is merged last, so it overrides anything the library sets, which is where a provider-specific header goes: HTTP-Referer and X-Title for OpenRouter, api-key for Azure.

Not covered

• Anything needing a multipart or binary body: audio transcription, file upload, image edits. http::HttpRequest.body is a String.
• The OpenAI Responses API. 311 schemas, and neither Anthropic, llama.cpp nor Gemini serve it. /chat/completions reaches every provider instead.
• Server-side conversation state (previous_response_id), which only OpenAI and LM Studio implement.

Tests

greycat test runs the offline suite, which reaches no network. Live tests are skipped unless the matching variable is set to a non-empty value:

Variable	Enables
OPENAI_BASE_URL	the OpenAI-compatible tests against that root
OPENAI_API_KEY	optional, for a server that wants a credential
OPENAI_MODEL	chat model, default llama3.2:3b
ANTHROPIC_API_KEY	the native Anthropic tests
ANTHROPIC_MODEL	default claude-sonnet-4-5
OLLAMA_BASE_URL	the native Ollama tests