Standalone inference

Use standalone inference when you want Cosine CLI to send model requests directly to an OpenAI-compatible endpoint without using Cosine cloud services.

Standalone inference is supported only in the local interactive TUI started with cos. It is not supported by one-shot/headless operation, remote runtime, daemon, IPC, or other non-TUI entry points.

While standalone mode is active, the CLI sends inference directly to your configured endpoint. It does not authenticate to Cosine cloud, synchronize sessions to Cosine cloud, call Cosine-hosted inference, or send Cosine telemetry. Conversations, compaction state, and resume data remain on the local machine in the platform-specific Cosine home directory.

Configure an endpoint

Run the interactive setup:

cos configure

The wizard first asks for an optional profile name, then asks for the endpoint, model, protocol, context window, authentication method, and whether to verify the connection. Leave the profile blank to save the unnamed default configuration.

After saving, the wizard prints the exact command to run. The unnamed configuration uses:

cos

For scripts or terminals without an interactive TTY, provide every required value:

cos configure \
  --mode standalone \
  --url http://127.0.0.1:11434/v1 \
  --model llama3.2 \
  --protocol chat-completions \
  --context-tokens 131072 \
  --no-auth \
  --verify

Use --api-key instead of --no-auth when your endpoint requires a bearer token. The CLI saves the key to the selected profile’s credential file rather than its TOML config. Avoid putting a key directly in shell history; the interactive wizard reads it without echoing it.

Standalone setup requires a URL, model, and positive context-token count. It also requires either an API key or --no-auth.

Choose protocol and context

• --protocol responses calls the endpoint’s /responses route. This is the default.
• --protocol chat-completions calls /chat/completions.
• --context-tokens sets the model’s positive maximum context-window size. Match this to the model and server you operate.

The selected protocol remains fixed for the session, including after context compaction and when a locally persisted session is resumed. A session configured for Chat Completions never switches to the Responses protocol. Your endpoint must therefore support the selected protocol for the entire session.

With --verify, Cosine sends a minimal request to the selected protocol route before saving. Verification checks that the endpoint is reachable and accepts the configured authentication, model, and protocol. A failed verification does not save the new setup.

Standalone binary distribution

Standalone-enabled binaries are built only by the repository’s custom .github/workflows/build-cli2-standalone.yml workflow. Ordinary CLI builds, CI builds, and standard release artifacts do not enable standalone inference. Maintainers must use that custom workflow and its dedicated standalone Make targets when producing a standalone distribution.

Inspect or change your setup

Show the selected profile’s configuration:

cos configure --show

The output reports whether an API key is configured, but redacts the credential.

Return to managed inference and remove the saved standalone credential:

cos configure --reset

You can also switch to managed mode without deleting the saved standalone fields and credential:

cos configure --mode managed

Use profiles

The configure flow selects its target profile before asking configuration questions. An explicit global --profile flag wins, followed by the target profile value and then the product default. Each profile keeps separate inference settings and credentials:

cos --profile local configure \
  --mode standalone \
  --url http://127.0.0.1:11434/v1 \
  --model llama3.2 \
  --protocol chat-completions \
  --context-tokens 131072 \
  --no-auth

cos --profile local configure --show
cos --profile local
cos --profile local configure --reset

Profiles live in the platform-specific Cosine config/home directory (typically ~/.cosine on macOS and Linux). The default profile uses:

• settings: config.toml
• inference credential: inference.json

A named profile uses:

• settings: config.<profile>.toml
• inference credential: inference.<profile>.json

Run cos to select the default and cos --profile local to select the named personal profile. Named repository files such as cosine.local.toml are unsupported. If an explicitly selected profile is missing, the CLI fails and shows how to create it with cos --profile local configure or return to the default with cos.

Cosine creates its config/home directory with permissions 0700 and writes inference credential files with permissions 0600. Treat these files as secrets and do not commit them.

Override saved settings for one run

Runtime flags take precedence over saved and binary-provided defaults. For standalone startup, endpoint precedence is --inference, saved inference.base_url in the selected profile, then the binary default. Model precedence is --model, the selected profile’s saved model, then the build/code default. Repository config does not set the model.

Inference credentials use this order:

1. --inference-api-key
2. COSINE_INFERENCE_API_KEY
3. the selected profile’s inference credential file

The CLI does not use your Cosine login token as a standalone endpoint credential.

You can force standalone mode and temporary values without changing the persisted setup:

COSINE_INFERENCE_API_KEY="$INFERENCE_KEY" \
  cos --standalone \
  --inference https://inference.example.com/v1 \
  --model example-model

For endpoints without authentication, configure --no-auth persistently before starting. See Configuration for the broader configuration precedence and file reference.