Desktop automation CLI for AI agents

usecomputer

Desktop automation CLI for AI agents. Screenshot, click, type, scroll, and drag on macOS, Linux, and Windows.

123npm install -g usecomputer
usecomputer screenshot ./shot.png --json
usecomputer click -x 400 -y 220 --coord-map "0,0,1600,900,1568,882"

Native Zig backend. No Node.js runtime required.

Features

Screenshot with coord-map so clicks land on the real screen
Mouse move, click, drag, and scroll
Keyboard type and press
Window-scoped capture for focused agent loops
Observe global input as SSE on macOS

  screenshot ──────────────────────────────────────────► vision model
       ▲                                                      │
       │                                                      ▼
       │                                               click / type / scroll
       │                                                      │
       └──────────────────── next frame ◄─────────────────────┘

Install

1npm install -g usecomputer

Agent skill

If you use an AI coding agent (OpenCode, Claude Code, etc.), install the usecomputer skill so the agent knows how to use the CLI correctly:

1npx skills add remorses/usecomputer

The skill teaches the screenshot → act → screenshot feedback loop, coord-map usage, and window-scoped screenshot workflow.

Requirements

macOS. Accessibility permission enabled for your terminal app
Linux. X11 session with DISPLAY set (Wayland via XWayland works too)
Windows. Run in an interactive desktop session. Automation input is blocked on a locked desktop

Quick start

12345usecomputer mouse position --json
usecomputer mouse move -x 500 -y 500
usecomputer click -x 500 -y 500 --button left --count 1
usecomputer type "hello"
usecomputer press "cmd+s"

Library usage

123456789101112131415161718192021import * as usecomputer from 'usecomputer'

const screenshot = await usecomputer.screenshot({
  path: './tmp/shot.png',
  display: null,
  window: null,
  region: null,
  annotate: null,
})

const coordMap = usecomputer.parseCoordMapOrThrow(screenshot.coordMap)
const point = usecomputer.mapPointFromCoordMap({
  point: { x: 400, y: 220 },
  coordMap,
})

await usecomputer.click({
  point,
  button: 'left',
  count: 1,
})

These exported functions mirror the native command shapes used by the Zig N-API module. Optional native fields are passed as null when absent.

Screenshot scaling and coord-map

usecomputer screenshot always scales the output image so the longest edge is at most 1568 px. This keeps screenshots in a model-friendly size for computer-use agents.

Screenshot output includes:

desktopIndex (display index used for capture)
coordMap in the form captureX,captureY,captureWidth,captureHeight,imageWidth,imageHeight
hint with usage text for coordinate mapping

Always pass the exact --coord-map value emitted by usecomputer screenshot to pointer commands when you are clicking coordinates from that screenshot. This maps screenshot-space coordinates back to real screen coordinates.

  Desktop 1600x900                              Screenshot 1568x882
  ┌───────────────────────────┐                 ┌────────────────────────┐
  │                           │                 │                        │
  │         click ●           │                 │        model ●         │
  │                           │                 │                        │
  └────────────┬──────────────┘                 └───────────┬────────────┘
               ▲                                            │
               │              --coord-map                   │
               └──────── 0,0,1600,900,1568,882 ◄────────────┘

123usecomputer screenshot ./shot.png --json
usecomputer click -x 400 -y 220 --coord-map "0,0,1600,900,1568,882"
usecomputer mouse move -x 100 -y 80 --coord-map "0,0,1600,900,1568,882"

Validate with debug-point

To validate a target before clicking, use debug-point. It takes the same coordinates and --coord-map, captures a fresh full-desktop screenshot, and draws a red marker where the click would land. When --coord-map is present, it captures that same region so the overlay matches the screenshot you are targeting:

1usecomputer debug-point -x 400 -y 220 --coord-map "0,0,1600,900,1568,882"

Window-scoped screenshots

Capture only a specific application window for a smaller, more focused image. This improves model accuracy because the screenshot contains only the target app; no dock, menu bar, or background windows.

  Desktop                                         Window screenshot
  ┌──────────┬──────────────────────┐             ┌──────────────────┐
  │  dock    │                      │             │                  │
  │          │     target app       │────────────►│   target app     │
  │          │                      │             │                  │
  └──────────┴──────────────────────┘             └────────┬─────────┘
                                                           │
              click uses window coord-map                  │
              200,100,1200,800,1568,1045 ◄─────────────────┘

123456789# 1. find the window ID
usecomputer window list --json

# 2. screenshot that window
usecomputer screenshot ./tmp/app.png --window 12345 --json
# output: {"path":"./tmp/app.png","coordMap":"200,100,1200,800,1568,1045",...}

# 3. click using the coord-map (maps window screenshot pixels to desktop coords)
usecomputer click -x 400 -y 220 --coord-map "200,100,1200,800,1568,1045"

The coord-map from a window screenshot includes the window's position on screen, so pointer commands land on the correct desktop coordinates even though the screenshot only shows one window.

Keyboard commands

Type text

12345678# Short text
usecomputer type "hello from usecomputer"

# Type from stdin (good for multiline or very long text)
cat ./notes.txt | usecomputer type --stdin --chunk-size 4000 --chunk-delay 15

# Simulate slower typing for apps that drop fast input
usecomputer type "hello" --delay 20

--delay is the per-character delay in milliseconds.

For very long text, prefer --stdin + --chunk-size so shell argument limits and app input buffers are less likely to cause dropped characters.

Press keys and shortcuts

12345678910# Single key
usecomputer press "enter"

# Chords
usecomputer press "cmd+s"
usecomputer press "cmd+shift+p"
usecomputer press "ctrl+s"

# Repeats
usecomputer press "down" --count 10 --delay 30

Modifier aliases: cmd/command/meta, ctrl/control, alt/option, shift, fn.

Platform note:

macOS: cmd maps to Command.
Windows/Linux: cmd maps to Win/Super.
For app shortcuts that should work on Windows/Linux too, prefer ctrl+....

Drag commands

Drag moves the mouse while holding a button down. Coordinates are x,y pairs. The format is drag <from> <to> [cp] where cp is an optional quadratic bezier control point that curves the path.

12345678# Straight line drag (2 points)
usecomputer drag 100,200 500,600

# Curved drag (3 points; cp pulls the curve toward it)
usecomputer drag 100,200 500,600 300,50

# With coord-map from a screenshot
usecomputer drag 100,200 500,600 --coord-map "0,0,1600,900,1568,882"

Duration is computed automatically from arc length at about 500 px/s (average human drawing speed). Shorter drags are faster; longer drags take proportionally more time.

Bezier control point

The optional third argument [cp] is a quadratic bezier control point. It pulls the curve toward itself. The cursor does not pass through it:

  Straight (2 points)                         Curved (3 points)

                                                      * cp
  from ────────────────────────────────► to     from .´  `.
                                                    ´      `.
                                                   ´         to

Drawing circles and ellipses

A circle at center (cx, cy) with radius r uses 4 quadratic bezier arcs. Each arc goes between two cardinal points (top, right, bottom, left), with the control point at the bounding box corner between them:

                    top
                     ●
              cp TL / \ cp TR
                   /   \
             left ●     ● right
                   \   /
              cp BL \ / cp BR
                     ●
                   bottom

12345# Circle at center (400, 300) radius 50
usecomputer drag 400,250 450,300 450,250   # top → right,    cp = top-right corner
usecomputer drag 450,300 400,350 450,350   # right → bottom, cp = bottom-right corner
usecomputer drag 400,350 350,300 350,350   # bottom → left,  cp = bottom-left corner
usecomputer drag 350,300 400,250 350,250   # left → top,     cp = top-left corner

The pattern for any circle:

1234drag cx,cy-r   cx+r,cy   cx+r,cy-r    # top → right
drag cx+r,cy   cx,cy+r   cx+r,cy+r    # right → bottom
drag cx,cy+r   cx-r,cy   cx-r,cy+r    # bottom → left
drag cx-r,cy   cx,cy-r   cx-r,cy-r    # left → top

For an ellipse, use different rx and ry instead of r:

12345# Ellipse at center (400, 300) rx=30 ry=80
usecomputer drag 400,220 430,300 430,220   # top → right
usecomputer drag 430,300 400,380 430,380   # right → bottom
usecomputer drag 400,380 370,300 370,380   # bottom → left
usecomputer drag 370,300 400,220 370,220   # left → top

Observe; global input event stream

The observe command streams all mouse and keyboard events as Server-Sent Events (SSE) to stdout. It runs until interrupted with Ctrl+C.

1usecomputer observe

  mouse / keyboard ──► CGEventTap ──► SSE stdout
                                          │
                                          ▼
                                   event: mouseClick
                                   data: { "x": 540, "y": 320, ... }

Output:

1234567891011event: mouseClick
data: {"type":"mouseClick","button":"left","x":540,"y":320,"timestamp":1719500000123}

event: keyDown
data: {"type":"keyDown","key":"a","keyCode":0,"timestamp":1719500000456}

event: keyUp
data: {"type":"keyUp","key":"a","keyCode":0,"timestamp":1719500000489}

event: scroll
data: {"type":"scroll","x":200,"y":300,"deltaX":0,"deltaY":-3,"timestamp":1719500000600}

Every event has a type field that doubles as the SSE event name and as a TypeScript discriminated union tag.

Event types

Type	Fields
`mouseClick`	`button`, `x`, `y`, `timestamp`
`mouseRelease`	`button`, `x`, `y`, `timestamp`
`mouseMove`	`x`, `y`, `timestamp`
`keyDown`	`key`, `keyCode`, `timestamp`
`keyUp`	`key`, `keyCode`, `timestamp`
`flagsChanged`	`key`, `keyCode`, `timestamp`
`scroll`	`x`, `y`, `deltaX`, `deltaY`, `timestamp`

flagsChanged fires when modifier keys (Shift, Command, Option, Control, Fn) are pressed or released. The key field identifies which modifier, and keyCode is the macOS virtual keycode.

The button field on mouse events is "left", "right", "middle", or "other" (for side buttons).

TypeScript async generator

The npm package exports a typed observe() function that spawns the native binary and yields events as a typed async generator:

12345678910import { observe } from 'usecomputer'

for await (const event of observe()) {
  if (event.type === 'keyDown') {
    console.log(event.key, event.timestamp)
  }
  if (event.type === 'mouseClick') {
    console.log(event.x, event.y, event.button)
  }
}

The generator kills the child process automatically when you break out of the loop. The InputEvent union type provides full autocomplete for each event shape.

To stop observing from outside the loop, pass an AbortSignal:

1234567const controller = new AbortController()

setTimeout(() => controller.abort(), 5000) // stop after 5 seconds

for await (const event of observe({ signal: controller.signal })) {
  console.log(event.type)
}

Consecutive mouseMove events are automatically coalesced so the queue stays bounded even if the consumer is slow.

Platform support

Currently macOS only. Requires Input Monitoring permission (System Settings → Privacy & Security → Input Monitoring). This is the same permission needed for global event observation; click/type commands use Accessibility permission instead. Linux and Windows support is planned.

Kitty Graphics Protocol (agent-friendly screenshots)

When the AGENT_GRAPHICS environment variable contains kitty, the screenshot command emits the PNG image inline to stdout using the Kitty Graphics Protocol. This lets AI agents receive screenshots in a single tool call. No separate file read needed.

  usecomputer screenshot ──► Kitty Graphics escape ──► kitty-graphics-agent
                                                              │
                                                              ▼
                                                     image in model context

The protocol is supported by kitty-graphics-agent, an OpenCode plugin that intercepts Kitty Graphics escape sequences from CLI output and injects them as LLM-visible image attachments. To use it, add the plugin to your opencode.json:

123{
  "plugin": ["kitty-graphics-agent"]
}

The plugin sets AGENT_GRAPHICS=kitty in the shell environment automatically. When the agent runs usecomputer screenshot, the image appears directly in the model's context window.

The JSON output includes "agentGraphics": true when the image was emitted inline, so programmatic consumers know the screenshot is already in context.

OpenAI computer tool example

12345678910111213141516171819202122232425262728293031323334353637383940414243444546474849505152535455565758596061626364656667686970717273import fs from 'node:fs'
import * as usecomputer from 'usecomputer'

async function sendComputerScreenshot() {
  const screenshot = await usecomputer.screenshot({
    path: './tmp/computer-tool.png',
    display: null,
    window: null,
    region: null,
    annotate: null,
  })

  return {
    screenshot,
    imageBase64: await fs.promises.readFile(screenshot.path, 'base64'),
  }
}

async function runComputerAction(action, coordMap) {
  if (action.type === 'click') {
    await usecomputer.click({
      point: usecomputer.mapPointFromCoordMap({
        point: { x: action.x, y: action.y },
        coordMap: usecomputer.parseCoordMapOrThrow(coordMap),
      }),
      button: action.button ?? 'left',
      count: 1,
    })
    return
  }

  if (action.type === 'double_click') {
    await usecomputer.click({
      point: usecomputer.mapPointFromCoordMap({
        point: { x: action.x, y: action.y },
        coordMap: usecomputer.parseCoordMapOrThrow(coordMap),
      }),
      button: action.button ?? 'left',
      count: 2,
    })
    return
  }

  if (action.type === 'scroll') {
    await usecomputer.scroll({
      direction: action.scrollY && action.scrollY < 0 ? 'up' : 'down',
      amount: Math.abs(action.scrollY ?? 0),
      at: typeof action.x === 'number' && typeof action.y === 'number'
        ? usecomputer.mapPointFromCoordMap({
            point: { x: action.x, y: action.y },
            coordMap: usecomputer.parseCoordMapOrThrow(coordMap),
          })
        : null,
    })
    return
  }

  if (action.type === 'keypress') {
    await usecomputer.press({
      key: action.keys.join('+'),
      count: 1,
      delayMs: null,
    })
    return
  }

  if (action.type === 'type') {
    await usecomputer.typeText({
      text: action.text,
      delayMs: null,
    })
  }
}

Anthropic computer use example

Anthropic's computer tool uses action names like left_click, double_click, mouse_move, key, type, scroll, and screenshot. usecomputer provides the execution layer for those actions.

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119import fs from 'node:fs'
import Anthropic from '@anthropic-ai/sdk'
import type {
  BetaToolResultBlockParam,
  BetaToolUseBlock,
} from '@anthropic-ai/sdk/resources/beta/messages/messages'
import * as usecomputer from 'usecomputer'

const anthropic = new Anthropic({ apiKey: process.env.ANTHROPIC_API_KEY })

const message = await anthropic.beta.messages.create({
  model: 'claude-opus-4-6',
  max_tokens: 1024,
  tools: [
    {
      type: 'computer_20251124',
      name: 'computer',
      display_width_px: 1024,
      display_height_px: 768,
      display_number: 1,
    },
  ],
  messages: [{ role: 'user', content: 'Open Safari and search for usecomputer.' }],
  betas: ['computer-use-2025-11-24'],
})

for (const block of message.content) {
  if (block.type !== 'tool_use' || block.name !== 'computer') {
    continue
  }

  const toolUse = block as BetaToolUseBlock
  await usecomputer.screenshot({
    path: './tmp/claude-current-screen.png',
    display: null,
    window: null,
    region: null,
    annotate: null,
  })
  const coordinate = Array.isArray(toolUse.input.coordinate)
    ? toolUse.input.coordinate
    : null
  const point = coordinate
    ? { x: coordinate[0] ?? 0, y: coordinate[1] ?? 0 }
    : null

  switch (toolUse.input.action) {
    case 'screenshot': {
      break
    }
    case 'left_click': {
      if (point) {
        await usecomputer.click({ point, button: 'left', count: 1 })
      }
      break
    }
    case 'double_click': {
      if (point) {
        await usecomputer.click({ point, button: 'left', count: 2 })
      }
      break
    }
    case 'mouse_move': {
      if (point) {
        await usecomputer.mouseMove(point)
      }
      break
    }
    case 'type': {
      if (typeof toolUse.input.text === 'string') {
        await usecomputer.typeText({ text: toolUse.input.text, delayMs: null })
      }
      break
    }
    case 'key': {
      if (typeof toolUse.input.text === 'string') {
        await usecomputer.press({ key: toolUse.input.text, count: 1, delayMs: null })
      }
      break
    }
    case 'scroll': {
      await usecomputer.scroll({
        direction: toolUse.input.scroll_direction === 'up' || toolUse.input.scroll_direction === 'down' || toolUse.input.scroll_direction === 'left' || toolUse.input.scroll_direction === 'right'
          ? toolUse.input.scroll_direction
          : 'down',
        amount: typeof toolUse.input.scroll_amount === 'number' ? toolUse.input.scroll_amount : 3,
        at: point,
      })
      break
    }
    default: {
      throw new Error(`Unsupported Claude computer action: ${String(toolUse.input.action)}`)
    }
  }

  const afterActionScreenshot = await usecomputer.screenshot({
    path: './tmp/claude-computer-tool.png',
    display: null,
    window: null,
    region: null,
    annotate: null,
  })
  const imageBase64 = await fs.promises.readFile(afterActionScreenshot.path, 'base64')
  const toolResult: BetaToolResultBlockParam = {
    type: 'tool_result',
    tool_use_id: toolUse.id,
    content: [
      {
        type: 'image',
        source: {
          type: 'base64',
          media_type: 'image/png',
          data: imageBase64,
        },
      },
    ],
  }
  // Append toolResult to the next user message in your agent loop.
}

Coordinate options

Commands that target coordinates accept -x and -y flags:

usecomputer click -x <n> -y <n>
usecomputer hover -x <n> -y <n>
usecomputer mouse move -x <n> -y <n>

mouse move is optional before click when click coordinates are already provided.

Legacy coordinate forms are also accepted where available.

Display index options

For commands that accept --display, the index is 0-based:

0 = first display
1 = second display
2 = third display

Example:

1usecomputer screenshot ./shot.png --display 0 --json