Every tool of the built-in desktop and sessionboxer MCP servers

Every session's agent gets two MCP servers from Sessionboxer itself, next to the ones you add in Global settings: desktop, which sees and drives the box's screen, and sessionboxer, which tells the agent which session it is and lets it act on Sessionboxer. Both are always on (Advanced… → MCP & connectors shows them as switches you cannot turn off; only the sessionboxer policy changes). This page lists each tool as the agent sees it: the exact name, every parameter with its accepted values and defaults, what comes back, and what happens on the Sessionboxer side. The sources are packages/computer-use-mcp and packages/sessionboxer-mcp in the repository; the user guide says how the same things look from the UI.

Conventions: parameters marked required have no default. Sizes are in characters. A tool that fails (a parameter out of range, a policy that forbids it, a session that does not exist) returns an MCP tool error whose text says why, so the agent can tell you rather than guess.

Where the servers run

Both servers are stdio MCP processes inside the Linux box of the session (the Sandbox container). The agent's MCP configuration (.mcp.json for Claude Code, the equivalent for Codex, Cursor and Devin) lists them under the names desktop and sessionboxer.

What this means for a Windows or macOS session: screenshots are of the remote-desktop stream, full screen at the display's size (the guest runs at 1024×768 too), typing is paced at 40 ms per key (SESSIONBOXER_TYPE_DELAY_MS, 12 ms on Linux) because RDP and VNC drop keys at xdotool's default pace, and a few combinations may be kept by the viewer instead of reaching the guest (Win-key shortcuts over RDP; on macOS ⌘ is super and most combinations pass, Spotlight is the fallback). The agent's briefing says so and tells it to open programs from the Start menu or Spotlight. Recordings are of the same display, so they work unchanged; their paths are Linux paths under /workspace, which the daemon mirrors into the guest's Workspace so the file shows up for the agent and in the chat.

The desktop server

The desktop MCP mirrors Anthropic's computer-use tool set, implemented with xdotool and ffmpeg on the box's X display. Coordinates are integer pixels, [x, y], origin at the top-left of the screen; the screenshot tool's description tells the agent the screen size. Every tool that moves the pointer to a target glides there (an eased motion of 100–300 ms, about 120 positions a second) so hover effects and drag-and-drop see a moving pointer; SESSIONBOXER_MOUSE_GLIDE=0 in the server's environment makes it jump.

Tools that act (move, click, type, scroll…) return the text OK; screenshot, zoom and wait return a PNG image; the recording tools return JSON.

Seeing the screen

screenshot

Take a screenshot of the whole desktop. The agent is told to call it before acting and after any action whose result it needs to see, since the other tools only answer OK.

No parameters. Returns a PNG of the full screen.

zoom

Capture a rectangular region and scale it up to the full screen size, to read small text or inspect a detail. Coordinates the agent sees in the zoomed image are not screen coordinates; it maps them back through the region.

Returns a PNG.

cursor_position

No parameters. Returns { "x": …, "y": … }, the pointer's current position.

Mouse

mouse_move

Move the pointer to a coordinate without clicking.

left_click, right_click, middle_click, double_click, triple_click

Click the left, right or middle button; double-click; triple-click (selects a line or paragraph in most programs). All five take the same parameter.

left_click_drag

Press the left button at start_coordinate, move to coordinate, release. The move between the two is a glide, so drop targets that watch the pointer travel see it.

left_mouse_down, left_mouse_up

Press and hold the left button, and release it, as two calls, for drags that need something in between (a mouse_move through several points, a key while dragging). Both take the optional coordinate of the click tools: the pointer moves there first.

Keyboard

type

Type a string at the current focus, as a keyboard would. For shortcuts and special keys the agent uses key.

Typed in chunks of 50 characters with a pause between keys (12 ms on a Linux desktop, 40 ms on a Windows or macOS one).

key

Press a key or a combination, named the way xdotool names them: Return, Escape, Tab, BackSpace, Delete, Home, End, Page_Down, Up, F5, ctrl+s, alt+Tab, super, ctrl+shift+t. Several presses in a row are separated by spaces ("ctrl+a Delete").

hold_key

Hold a key or combination down for a number of seconds, then release it (for a program that reacts to a long press, or to keep a modifier down during another action).

Scrolling and waiting

scroll

Scroll the mouse wheel at a coordinate.

wait

Wait for a page to load or an animation to finish, then take a screenshot.

Returns a PNG of the screen after the wait.

Recording

A recording is ffmpeg grabbing the display into an H.264 .mp4 (yuv420p, faststart, so browsers play it). It runs detached from the MCP process and its state lives in a file on the box's tmpfs, so one recording runs at a time per session and a stop_recording from a later turn still finds it. The lifecycle the agent follows: start_recording → (annotate_recording, then the step, repeated) → stop_recording → mention the returned path in the reply, and the chat shows a player with the captions as clickable steps. Auto QA runs use the same tools and hand the path to e2e_finish.

start_recording

Start recording the desktop until stop_recording. Fails when a recording is already running.

Returns { path, startedAt, seconds: 0, captions: 0 }.

annotate_recording

Add a caption to the running recording at this moment: one short sentence saying what the agent is about to do or what the screen now shows ("Submitting the form with an empty email"). Each caption stays on screen until the next one. The agent calls it right before each step.

Returns the caption with its time from the start of the recording ({ at, text, path }). Fails when nothing is recording.

stop_recording

Stop the recording and finish the video. In order: the captions are burned into a band added under the desktop (nothing on screen is covered; the band fits about three lines and grows upwards for a longer caption); static stretches are condensed; the .vtt is written with the re-timed captions; narration is added when it applies. The tool result reports the recorded and the final length.

Returns JSON:

{
  path, startedAt,
  recordedSeconds,      // wall-clock length of the recording
  seconds,              // length of the finished video
  condensed,            // whether static stretches were cut
  bytes,
  captions: [{ at, text }],   // with their times in the finished video
  track,                // the .vtt path, when one was written
  narration,            // see below; absent when there were no captions
  warning               // what did not happen as asked; the video is still usable
}

narration is one of:

• { added: true, language, voice, speechSeconds, processingSeconds }: the captions were spoken (Kokoro, a text-to-speech model inside the box; nothing leaves the machine) and muxed in as an audio track. A step shorter than its sentence holds its last frame until the sentence ends, so the video may grow a little; captions and the .vtt carry the new times.
• { pending: true, estimatedSeconds, language, voice, nextStep }: the setting is Ask when it takes longer than N seconds (the default, N = 5) and this video is above it. The silent video is delivered and the agent is told to ask you first, then call narrate_recording if you want it.
• { skipped: "…" }: the setting is Never, narrate: false was passed, or the TTS model is not in the image.

Fails when nothing is recording.

narrate_recording

Add spoken narration to a finished recording: after a pending answer and your yes, or when you ask for narration later. Rewrites the .mp4 in place with the audio track (extending short steps as above) and returns the same shape as stop_recording minus the recording-time fields, with the re-timed captions and .vtt.

recording_status

No parameters. Returns { path, startedAt, seconds, captions } for the running recording, or { "recording": false }.

The sessionboxer server

How a call travels, and what the agent knows without asking

The box has no route to Sessionboxer's API and no token. The sessionboxer MCP posts each call to the box's own daemon (POST http://127.0.0.1:7000/sessionboxer; the Auto QA tools use /e2e), the daemon forwards it over the WebSocket the Control Plane already holds to the box, and the Control Plane, which knows from that socket which session is talking, applies the session's policy, performs the action and answers. A box can only ever speak for its own session; the agent cannot claim to be another one. When the box's daemon is unreachable the tool error says so.

Before any tool call the agent already knows where it is: its briefing opens with the session's name and environment, and .sessionboxer/session.json in the Workspace (/workspace, C:\workspace or /Users/agent/workspace) holds the id, title, URL, provider, model, environment, guest OS, creation time, what it was forked from, which session's agent created it, the conversation branch, the snapshot count, the Sessionboxer version and the policy in force, kept current by the box.

Results are JSON text. Every action that changes something (a PR attached, a snapshot, a rename, a queued prompt, a verification run, a notification, a pane opened, a session created, forked, messaged or stopped, a schedule) is also written into the transcript as a small marker at the point where it happened ("attached PR #12 (owner/repo)", "took Snapshot 3", "sent a message to Backend tests"), so the chat says what the agent did.

Policy, approval and limits

Two settings decide what the agent may do: the default in Global settings → MCP & connectors → sessionboxer and the session's own value in Advanced… / Session settings → MCP & connectors. A change applies to the running agent at its next turn.

When the Agent creates a Session (same place): Ask me (default) or Do not ask. With Ask me, session_create returns { pending: true, id, summary, expiresAt, hint } and a card appears in your chat ("Claude Code wants to create a Session 'Backend tests'") with Allow and Deny; the agent waits with approval_wait. A card nobody answers in 10 minutes expires and counts as denied. Settled cards stay in the transcript with a link to the session they created.

Limits that hold whatever the policy: at most 3 alive sessions created by one agent at a time (stopped ones do not count; session_stop frees a place) and a global cap on agent-created sessions (Global settings → MCP & connectors → sessionboxer, 10 by default); a chain of agents prompting agents stops after 4 hops; one message in flight per sender and target (wait for the reply first); a session cannot message itself; session_stop only stops sessions the calling agent created. Sessions an agent created show child of … under their name in the sidebar and in sessions_list.

The wait tools (session_wait, approval_wait) block for at most 20 s per call (default 15) because the bridge itself times out at 20 s; the agent calls them again while the answer is still pending.

Self-knowledge

whoami

No parameters. Returns the session as the Control Plane sees it, live:

{
  id, title, url, provider, model, environment,   // "docker-linux" | "qemu-windows" | "qemu-macos"
  guest,                // { os: "windows" | "macos", workspace } or null
  createdAt,
  forkedFrom,           // { sessionId, title, snapshotId } or null
  createdBy,            // { sessionId, title } or null: the agent that created this session
  branch, snapshotCount, sessionboxerVersion,
  agentTools,           // "off" | "session" | "all"
  status,               // running, idle, stopped…
  usage,                // the provider's usage windows, when it reports them
  context,              // context window use of the current conversation
  queueLength,
  panes,                // which panes you have open right now
  terminals: [{ id, createdAt, exitCode }],
  prs: [ summaries of the attached pull requests ],
  verification,         // { id, status, brief } of the current Auto QA run, or null
  repos: [{ name, path }]   // /workspace/<name> as the agent sees it
}

docs

Look a topic up in the Sessionboxer user guide, which ships in the box (/opt/sessionboxer/docs/GUIDE.md). The agent's briefing tells it to answer questions about Sessionboxer from this rather than from memory.

Returns { query, sections: [{ heading, path, text }] }, the three best-matching sections by word overlap with the heading and text, each cut at 6,000 characters, path being the chain of headings ("Using it › MCP servers"). With no match, sections is empty and headings lists every heading of the guide so the agent can retry with one.

settings_get

No parameters. Returns the public settings that apply to this session (the same object the UI reads: environments available, defaults, git identity, the MCP registry without its secrets, the sessionboxer policy…), with any secret-shaped value removed before it leaves the Control Plane.

This session

pr_attach

Attach a pull request to the session: you see it in the PRs pane with its checks and review comments from then on, and the session gets notified when reviewers comment or a check fails. The agent is told to call it right after creating a PR.

Returns the attached PR's summary. Marker in the transcript: "attached PR #n (owner/repo)".

pr_list

No parameters. Returns the pull requests attached to the session with their state, check counts and unseen review items.

pr_items

Returns the review comments, check results and other items of the PR, each with its id and whether it was marked addressed.

pr_mark_addressed

The items show as addressed in the PRs pane.

snapshot

No parameters. Takes a Snapshot of the Sandbox now (its disk, the Workspace and the conversation), the same as the Snapshot button; you can fork from it or revert to it. Returns { id, ordinal, sizeBytes, createdAt }. Marker: "took Snapshot n". Not available in Windows and macOS sessions (their VM disk is outside the Sandbox image).

queue_add

Queue a prompt for the agent itself: it is sent as the next user turn once the current one ends, after anything you already queued. Meant for a follow-up the agent wants a fresh turn for; you see it in the queue and can edit or drop it before it goes.

Returns { id, position }. Marker: "queued a message (…)".

queue_list

No parameters. Returns the prompts queued for the session, in order, with their ids.

title_set

Marker: "renamed the Session to …".

verify

Open an Auto QA (verification) run for the work of this turn, with a brief of what it checks that the Auto QA pane shows above the cases; then the agent follows the e2e-verification skill (e2e_plan unless the cases were given here, e2e_case_start / e2e_case_end, e2e_finish). The agent is told not to call it when Sessionboxer already asked it to verify the turn (the Auto QA setting does that after every user turn).

Returns { id, status, cases: [{ index, title, status }] }. Marker: "opened verification run …".

notify

Send you a short notification about the session: a browser push (when the device is enrolled under Devices) and the bell in Sessionboxer. For something that cannot wait for the reply, not for progress.

Returns { ok: true }. Marker: "notified you: …".

terminal_list

No parameters. Returns the Terminals of the session, yours and the ones ui_open opened for the agent, with id, createdAt and exitCode (null while the shell still runs).

terminal_read

Returns { id, text, exitCode }: the last lines of the Terminal's retained output (what you would see scrolling up in the pane) and the shell's exit code, null while it runs.

ui_open

Ask your browser to show a pane of this session. Honoured only when you are looking at this session and not typing; otherwise it is ignored, so an agent cannot pull you away from another session or interrupt a message. With pane: "terminal" and a terminal.command, a new Terminal opens and runs the command where you can watch it (a dev server, a test run).

Returns { pane, terminalId, shown }, shown being whether a browser of yours had the session open to receive it; terminalId can be read back with terminal_read. Marker: "opened the … pane" / "opened a Terminal running …".

Other sessions (policy All Sessions)

Session ids in these tools are the ids sessions_list returns; a prefix of 6 or more characters is enough when it is unambiguous.

sessions_list

No parameters. Returns every session on this Control Plane: id, title, url, status, provider, environment, repos (names), createdAt, createdBy, forkedFrom, queueLength, with the calling session marked self: true and the ones its agent created mine: true.

session_get

Returns that session's summary plus the last thing its agent said (capped in length). Never another session's whole conversation.

session_create

Start a new session whose first prompt is first_prompt; it is marked as created by this session (child of …). The new agent shares nothing with the caller but that text, so the prompt has to be the whole task, self-contained.

Returns { pending: false, sessionId, title, url }, or with approval on { pending: true, id, summary, expiresAt, hint } where id is the approval to pass to approval_wait. The new session takes the Global defaults (environment Docker · Linux, model, instructions, policy). Errors: 3 alive children already, the global cap reached, provider not connected. Marker: "created Session …", linking to it.

session_fork

Fork this session from a Snapshot taken now: the fork has the same files, repositories and tools. Works under the This Session only policy too, since it acts on the caller's own session.

Returns { pending: false, sessionId, title, url, snapshotOrdinal }. Counts as a child for the 3-alive limit. Not available in Windows and macOS sessions (no snapshots there).

session_message

Send a prompt to another session's agent. Its transcript shows the prompt as from Session X (its Agent), never as your words, and the sender's shows sent a message to Y.

Refused when: the target is the caller itself (queue_add is for that), a message from this sender to that target is still in flight (session_wait for the reply first), the chain of agents prompting agents would go past hop 4, the target is in error, or the target is stopped and when is now (queue leaves it for when you resume it). Returns { sessionId, title, delivery } with delivery prompted or queued.

session_wait

Wait until another session's agent finishes its turn and its queue is empty, or the timeout passes.

Returns { sessionId, title, still_running, status, lastReply }; the agent calls again while still_running is true. Waiting for itself is refused.

session_stop

Stops its Sandbox (you can resume it) and frees one of the 3 child places; returns { sessionId, title, status }. Any other session is yours to stop, and the tool refuses it; so is stopping itself (the agent ends its turn instead).

approval_wait

Wait for your answer to a pending approval.

Returns the approval: { id, kind, summary, status, expiresAt, result?, error? } with status one of pending, allowed, denied, expired; with allowed, result is what the approved action returned (the created session). The agent is told to call again while pending and to tell you what it is waiting for; its turn may also end, in which case the card in the chat still creates the session when you allow it.

Scheduled tasks

schedule_create

Create a scheduled task, the same as one made on the Scheduled tasks page: on a cron schedule, prompt a session or start a new one each time. Prompting the caller's own session works under This Session only; another session's needs All Sessions.

action:

• { type: "prompt", sessionId?, text }: send text (1–20,000 characters) to a session at each run; sessionId defaults to the caller.
• { type: "new_session", title?, provider?, repos, prompt, stopAfter }: start a session at each run with prompt (1–20,000) as its first prompt; repos as in session_create (default []); stopAfter (default true) stops the session once its first turn ends.

Returns { id, name, cron, timezone, nextRunAt }. The schedule is enabled at once and skips runs missed while Sessionboxer was down (the skip policy; you can change it on the Scheduled tasks page). Marker: "scheduled …"; the agent is told to say what it scheduled in its reply.

schedule_list

No parameters. Returns every scheduled task on the Control Plane with id, name, cron, timezone, enabled, action, nextRunAt and lastRunAt.

Auto QA (end-to-end verification) runs

When Auto QA is on, Sessionboxer opens a verification run after each of your turns and asks the agent to follow the e2e-verification skill; verify opens one on the agent's own initiative. These four tools fill the run in so the Auto QA pane follows along as it happens: the cases appear when planned, each turns running, passed, failed or skipped with its note and screenshot, and the video is attached at the end. They go through the daemon's /e2e endpoint rather than the general bridge and need an open run; without one they error.

e2e_plan

Register the test cases of the current run, or skip the run when the turn changed nothing testable (an answer-only turn, research). Called once, before start_recording. Cases are numbered from 1 in the order given.

e2e_case_start

Marks the case running. A failed case is started again after the fix; the run keeps every attempt.

e2e_case_end

e2e_finish

Close the run after stop_recording. Cases never started are marked skipped; the run's verdict is passed when no case's last attempt failed.

Returns the closed run. The agent then ends its reply with a short summary that mentions the video's path, so the chat shows the player.