Browser and native app automation

By default, Cosine CLI opens browser tasks in a new window in your OS-default browser. The window uses your normal browser profile, including its existing sign-ins, and native accessibility control keeps supported actions in the background. You can explicitly choose the isolated managed browser instead.

On macOS and Windows, Cosine can also inspect and operate native applications. Use computer use for tasks such as testing a local web app, checking a desktop workflow, filling forms, and capturing screenshots.

Configure automation

Open Tools with Ctrl+P, move to the Settings tab, then use the Browser section.

• Browser control turns browser tools on or off.
• Desktop control turns native app and foreground coordinate control on or off.
• Browser mode switches the managed browser between Visible and Headless.
• Browser profile selects a completed managed-browser profile.
• Create new browser profile opens Chrome so you can sign in manually, including MFA, before the managed browser uses that profile.

The default user-browser window also needs Desktop control because Cosine drives it through the operating system’s accessibility APIs. Browser mode and Browser profile do not change your OS-default browser or its profile.

Cosine saves these settings to ~/.cosine/config.toml and applies the browser and desktop toggles to the current session.

You can also configure the same behavior directly:

[computer]
enabled = true
allow_desktop_automation = "enabled"
allow_existing_browser_attach = false
allow_browser_eval = false

[computer.browser]
enabled = true
launch_mode = "visible"
agent_profile_id = ""
allowed_domains = ["example.com"]

[computer.desktop]
foreground_ok = true

allowed_domains limits browser navigation to the listed domains and their subdomains.

Use the JavaScript computer-use runtime

The agent uses one cua_repl tool for browser and native application interaction. JavaScript state persists between calls, so an app or tab can be bound once and reused:

const finder = await cua.getApp("com.apple.finder");
const state = await finder.getAXState();
nodeRepl.write(state);

Browser and app observation methods display their state automatically unless you pass { emit: false }. Actions retain the successor observation internally, so later calls use current element indexes instead of stale UI references.

const tab = await cua.createBrowserTab("user", "https://example.com");
const state = await tab.getAXState({ emit: false });
await tab.click(2);
await tab.getAXState();

Passing an empty browser ID selects the user browser when native computer use is available. A configured CDP endpoint, Headless mode, or a selected managed-browser profile makes the managed browser the default instead. You can always pass "user" or "managed" explicitly.

The runtime exposes the cua and nodeRepl APIs only. It does not expose Node.js require, filesystem, shell, process, or network globals. Existing computer_*, browser_*, and desktop_* calls remain internal compatibility adapters but are not advertised to new agent turns.

Use your default browser

When the agent first needs a browser, Cosine opens a new background window in your OS-default browser. It does not create or copy a profile, so the window uses the same profile and sign-ins as a window you open yourself.

Cosine tries native accessibility actions first. Setting field values, selecting text, invoking exposed actions, and many clicks can run without bringing the browser forward. Input that the browser does not expose through accessibility can still bring the window to the foreground.

The window belongs to the agent session. Cosine closes it when the session ends when the browser exposes a window-close action. allowed_domains applies before Cosine opens or navigates the user-browser window, and Cosine resets a detected blocked navigation to a blank page.

Use the managed browser

Choose browser ID "managed" when you need browser isolation, Headless mode, a named Cosine browser profile, or CDP behavior. Cosine first checks a configured Chrome DevTools Protocol endpoint. If that endpoint is unavailable, it looks for Chrome, Chromium, or Brave on the machine, then checks its managed browser cache.

Run a task such as:

Open http://localhost:3000, submit the sign-up form, and confirm the success message appears.

The agent captures an accessibility observation before it acts. Each observation has an ID and owns its element references. After an action changes the page, the agent uses the successor observation instead of reusing old references.

Attach to an existing Chrome instance

Start Chrome with remote debugging enabled:

open -n -a "Google Chrome" --args \
  --remote-debugging-port=9222 \
  --user-data-dir=/tmp/cosine-chrome

Then start Cosine with the endpoint:

cos --cdp-url http://127.0.0.1:9222

You can save the endpoint instead:

[computer.browser]
cdp_url = "http://127.0.0.1:9222"

Do not expose the debugging port to another machine. A process that can reach it can control that Chrome instance.

Install Chrome for Testing

If no supported browser is available, inspect the diagnosis first:

cos browser doctor

Install the managed browser after reviewing the source, version, size, and destination:

cos browser install

For non-interactive setup:

cos browser install --yes

Chrome for Testing does not publish Linux ARM64 builds. On Linux ARM64, install Chromium yourself and set computer.browser.executable_path.

Control native applications on macOS

With Desktop control enabled, the agent can:

• list running applications;
• capture the focused application’s Accessibility tree;
• attach a screenshot of the focused app;
• click an Accessibility element or screenshot coordinate;
• set writable field values;
• select text or place the cursor around a match;
• invoke actions exposed by an element;
• scroll, drag, type, and press key combinations;
• paste plain text, Markdown, or HTML while restoring your previous clipboard contents.

Ask for a concrete app by name or bundle identifier:

Open TextEdit, replace the document text with the release notes, and save it.

The agent prefers Accessibility element references. It uses screenshot coordinates when the app does not expose a usable element.

Element references are observation-local. If the app changes, the agent must observe it again before using another reference. Cosine rejects stale observations instead of guessing which element an old reference now identifies.

Grant macOS permissions

Native app observation needs Accessibility permission. App screenshots need Screen Recording permission. These grants belong to the separately signed Cosine Computer Use helper.

Request both permissions from the CLI:

cos computer permissions --request

You can also open System Settings > Privacy & Security and grant Cosine Computer Use access under:

1. Device Control and Data Access on macOS 27, or Accessibility on macOS 26 and earlier
2. Screen & System Audio Recording, called Screen Recording on older macOS versions

macOS 27 changed the Settings label for Accessibility access, but the underlying permission and request API are unchanged.

After changing either permission manually, open Activity Monitor, select Cosine Computer Use, and choose Quit before checking again.

Native computer use requires a client and helper signed by the same trusted identity. The helper also checks the client’s signed application identifier. Release builds provide matching identities. Standard ad-hoc local builds fail closed, so use a release-signed build when testing native application control.

The helper keeps the same bundle identity across CLI updates, so replacing the cos binary does not move the grants to a new process.

Cosine blocks control of sensitive or self-hosting targets such as the terminal running Cosine and System Settings.

Approval and privacy

In Manual mode, Cosine asks before browser mutations, native app observation, screenshots, and desktop actions. Auto mode accepts ordinary tool confirmations automatically.

Accessibility observations and screenshots may contain private information. Check the target app before approving access. Password fields are redacted from native Accessibility values, but surrounding labels and visible content may still appear.

The default browser window can access the sites and data available to your normal browser profile. Review browser actions carefully, especially when you are signed in to personal or production accounts.

Managed-browser profiles remain separate from your normal Chrome profile. Choose the managed browser and a dedicated profile when a task needs isolation or saved sessions that should not share your everyday browser data.

Diagnose problems

Check browser and desktop readiness:

cos computer doctor

Use JSON output in scripts:

cos computer doctor --json
cos browser doctor --json

Browser connection refused

1. Confirm Chrome is running with --remote-debugging-port.
2. Open http://127.0.0.1:9222/json/version locally.
3. Check the configured computer.browser.cdp_url or --cdp-url value.
4. Remove the CDP URL if you want Cosine to launch a detected browser instead.

Native app is unavailable

1. Confirm the app is running.
2. Use the exact app name or bundle identifier if several names match.
3. Run cos computer permissions --request and enable Cosine Computer Use under Device Control and Data Access on macOS 27, or Accessibility on macOS 26 and earlier.
4. Observe the app again if an action reports a stale observation.

Screenshot capture fails

Run cos computer permissions --request and enable Cosine Computer Use under Screen & System Audio Recording. Re-run cos computer doctor before retrying the task.