Handy

io.github.miisodevPublished by GitHub account miisodevv1.4.0Updated Nov 21, 2026

Native mouse, keyboard, window, clipboard, and screen control for LLMs via stdio MCP.

Overview

AI-generated overview

Local stdio MCP server that lets an assistant see the desktop screen with mapped coordinates and drive native mouse, keyboard, clipboard and windows.

What you can do
  • Capture the screen, a single window or a zoomed region with coordinate rulers, cursor color, active-window info, frame diffs and optional numbered marks on interactive elements
  • Run single or chained native actions: move, click, drag, scroll, type, paste, key chords, mouse/key button holds, window focus and visual waits, with optional post-action verification and after-shot
  • List visible desktop windows and bring one to the foreground
  • Read or write plain-text system clipboard
  • Check permission state, active backend, displays, active window and cursor position
  • Turn computer control off as a one-way stop switch
When to use it
When you want an assistant to actually operate native desktop apps — fill forms, click through UI, verify that an action changed the screen — rather than only describe it. Useful for agent builders and developers testing real desktop workflows on Windows, macOS or Linux.
Requirements
Node.js 20+ and a local MCP client over stdio. macOS and Windows need OS accessibility/screen-recording permissions. Linux Wayland input needs ydotool/ydotoold; capture uses the freedesktop screenshot portal or grim. Optional environment variables HANDY_BACKEND and HANDY_ENABLED.
Before you install
This grants an assistant real control of your machine: it can click, type, paste and read the screen and clipboard, so anything visible on screen may reach the model. Keep the permission toggle and view-only mode in mind, and note that once turned off by the user a client cannot re-enable it. Screen captures go only to the requesting client; no telemetry or hosted account is claimed.

Installation

In SourceWeft

  1. Open Handy in the dashboard and add it to a workspace.
  2. Enable the server for the chats that should use its tools.

Desktop only via STDIO. STDIO servers start a local process, so they need the SourceWeft desktop host.

Other MCP clients

Follow the launch instructions in the repository.

README

[Handy logo]

handy

Give your agent eyes and hands — see and drive the real desktop.

Real-time native mouse, keyboard, window, clipboard and coordinate-mapped screen control for LLMs and agents, served over the Model Context Protocol as a stdio server and CLI — with numbered Set-of-Mark targets, chained multi-step actions, an instant on/off permission toggle and a physical (0, 0) corner-slam failsafe.

Site · Quick start · The tool surface · How it works · Privacy

[Version] [CI] [License: MIT] [Platforms] [MCP] [Sponsor]


What is handy?

Agents get screenshots, but few get honest control of the machine they are looking at. Coordinates are guessed from pixels, one-shot clicks land where nothing happened, and the human watching has no switch to stop it all.

Handy is an MCP server (with a full CLI alongside) that gives a model real-time native control of a desktop — mouse, keyboard, clipboard and windows — paired with a live coordinate-mapped view of the screen: dual-contrast (x, y) rulers, active-window telemetry, pixel hex under the cursor, sub-2ms frame-to-frame diffs, and Set-of-Mark detection that numbers every interactive element on screen so the agent clicks mark 13 instead of guessing (631, 447). Up to 100 steps — focus a window, click, paste, wait for the UI to settle — run in one atomic chained call, with an after-shot that shows numbered trail markers where every action landed.

It installs cleanly on any PC with zero C++ addon compilation (node-gyp free), speaks plain stdio to any MCP client, and treats permission as a product surface: a ~/.handy/permissions.json toggle checked before every capture and every chain step, a --view-only mode, --ttl timers, a live loopback Control Panel, and a physical failsafe — slam your mouse into the top-left (0, 0) corner and any in-flight chain aborts instantly.

Who it's for

Developers and agent builders who want their assistant to actually operate the desktop — fill forms, drive native apps, verify what happened — rather than describe it. Any MCP-capable client works (Claude Desktop, Claude Code, Cursor, Windsurf, and everything else speaking MCP); the CLI makes the same surface usable from shell scripts and CI.

From experiment to open source

Handy began as the input layer for the author's own agents, then grew through daily real-machine use — real installs, real Wayland compositors, real failsafes — into a tool that behaves the same on Windows, macOS and Linux. What you're reading is the same code that runs the author's desktop every day, not a demo.

The core principle

An agent that can see and click must be interruptible.

Handy checks its permission store synchronously before every capture and before every individual step of a chain, and the failsafe reads the real cursor position each step, so a human can always stop the machine — by toggle, by corner-slam, or by timer. When the toggle is OFF, no MCP client can turn it back on by itself; handy_permission({ "action": "off" }) remains a one-way stop switch.

Design highlights

  • Set-of-Mark vision (marks: true) — handy_see detects, classifies (control / text / icon / region), numbers and outlines the interactive elements on screen in reading order; every pointer step then accepts { "mark": N } and the chain DSL understands click mark 5 / click #5. Marks resolve to the element's own centre at execution time, so a chain survives layout shift and HiDPI scaling. Detection is DPI-independent (~18ms at 1280x800, ~22ms at 2560x1440, ~49ms at 4K) because the work buffer is area-capped rather than dimension-capped.
  • One atomic call for a whole interaction (handy_act) — up to 100 sequential steps (focus_window, move, click, drag, scroll, paste, type, key, mouse_down/up, key_down/up, wait, wait_for_change, wait_for_settled, wait_for_color, screenshot) in a single call, with screen/normalized (0..1000)/window coordinate spaces and numbered amber trail markers on screenshotAfter.
  • Human-like pointer motion (motion: "human") — Bézier arcs with Fitts's-law timing instead of teleporting straight lines, with the duration derived automatically from travel distance and target width when omitted.
  • Post-action verification (verifyChange: true) — a baseline capture before the chain and a frame diff after, so the receipt says whether the click actually did anything.
  • Captures that survive real screens — dual-contrast rulers legible on #000 and #fff apps alike, area-average box-filter downscaling that keeps 1px text alive, and per-row PNG filter selection for ~26% smaller encodes.
  • Fail-soft enhancements — mark detection degrades to a plain screenshot (marksError) instead of failing handy_see, and stale marks fail loudly with the available indices instead of clicking somewhere arbitrary.
  • Strict chain-DSL validation — malformed pointer arguments throw with the offending segment quoted; out-of-range marks list what exists. Nothing degrades to "act wherever the cursor happens to be".

How it works

Handy is one TypeScript process with zero runtime dependencies beyond the MCP SDK and Zod. Each platform speaks its own native bridge — no wrappers, no VMs:

PlatformBackendHow
Windows 10/11win32Persistent Win32 SendInput (KEYEVENTF_UNICODE, MOUSEEVENTF_VIRTUALDESK), EnumWindows/SetForegroundWindow, MTA-safe OpenClipboard, GDI CopyFromScreen, Per-Monitor V2 DPI awareness
macOSdarwinPersistent CoreGraphics CGEvent + NSWorkspace + NSPasteboard, in-memory CGWindowListCreateImage PNG capture (with screencapture fallback), Retina 2x logical-point mapping
Linux — X11linux-x11Pure native X11 wire-protocol client (XTEST, GetImage, BIG-REQUESTS, RANDR, EWMH _NET_CLIENT_LIST/_NET_ACTIVE_WINDOW) — zero external packages
Linux — Waylandlinux-cliAuto-selected on Wayland sessions: screen capture through the freedesktop screenshot portal (works out of the box on GNOME, whose Shell screenshot API rejects third-party callers) or grim on wlroots; input via ydotool (uinput daemon) or wtype/xdotool; clipboard via wl-clipboard/xclip/xsel
Headless / CIvirtualBuilt-in 1280x800 multi-window interactive desktop workbench for safe agent sandboxing — no display required

Everything above the bridge is shared: controller.ts enforces permissions and the failsafe between steps, marks.ts detects and numbers UI elements, overlay.ts renders rulers and reticles, png.ts handles decode/encode, and the MCP layer (mcp.ts) wraps it all as six tools.

A capture-and-act round trip looks like: the agent calls handy_see { marks: true } → the backend grabs a frame (portal on GNOME Wayland, native protocol on X11/Windows/macOS) → detection numbers the elements → the agent calls handy_act with a chain referencing mark indices → the controller re-checks permissions and the failsafe before each step → a receipt (and optionally a diff and after-shot) comes back.


Quick start

Requires Node.js 20+.

1. One-Command Auto-Setup (Claude Desktop, Cursor, Windsurf, Claude Code)

bash
npx -y @miisodev/handy install

Preview changes first with --dry-run, target a single host (handy install cursor), or remove cleanly (handy uninstall). Backups (*.handy.bak) are created automatically before modifying existing config files.

2. Prepackaged Plugin (Claude Code / Agent Marketplaces)

Installs the stdio MCP server, the handy operational skill (skills/handy/SKILL.md), and the /handy:on, /handy:off, /handy:toggle, /handy:status, /handy:doctor, and /handy:windows commands together:

bash
claude plugin marketplace add miisodev/handyclaude plugin install handy

3. Global CLI + Live Control Panel

bash
npm install -g @miisodev/handyhandy doctor                # verify input & capture readiness for your platformhandy ui                    # live Control Panel at http://127.0.0.1:7788

4. Manual Stdio MCP Configuration

json
{  "mcpServers": {    "handy": {      "command": "npx",      "args": ["-y", "@miisodev/handy", "mcp"]    }  }}

Linux (Wayland) setup

Capture works out of the box on GNOME and any compositor whose portal serves Screenshot. Input needs a running ydotoold (uinput), which most distros ship:

bash
sudo apt install ydotool            # or your distro's packagesudo systemctl enable --now ydotoold

If your distro only ships a user-level unit (some recent Ubuntu releases do), it cannot open /dev/uinput; run the daemon as a root system service with a user-owned socket instead:

bash
sudo systemctl edit --force --full ydotoold   # then set:# [Service]# ExecStart=/usr/bin/ydotoold -p /run/user/1000/.ydotool_socket -P 0660 -o 1000:1000# RuntimeDirectory=ydotoold

handy doctor tells you exactly which routes and tools it found — session type, capture routes, input tool.


The tool surface

All returns are structured JSON. Input mistakes come back as informational errors the model can self-correct from; every receipt names what was executed, where, and how long it took.

ToolDescription
handy_seeCapture the live screen, a single window, or a zoomed region with dual-contrast (x, y) rulers, cursor reticle, pixelUnderMouse, activeWindow, screenDiff, and highlightDiff. With marks: true it also detects, numbers, and returns every interactive UI element (elements).
handy_actExecute single or chained native mouse, keyboard, paste, window-focus, and smart visual-wait (wait_for_change, wait_for_settled, wait_for_color) controls with screen/normalized/window coordinates, mark element references, motion profiles, numbered trail markers, and optional verifyChange post-conditions.
handy_windowsList visible desktop windows (title, app, bounds, focused) or focus a window by title, app, or id.
handy_clipboardRead or write plain-text system clipboard directly.
handy_statusInspect live permission toggle state (ON/OFF), mode (full/view_only), OS backend, displays, active window, and cursor coordinates.
handy_permissionQuery permission status or immediately turn Handy off.

Example: See numbered elements, then act on them

json
// handy_see{ "marks": true, "maxMarks": 30 }
json
// handy_act — marks 5 and 13 come from the capture above{  "actions": [    { "type": "click", "mark": 5 },    { "type": "paste", "text": "Sam — ready for review 🚀" },    { "type": "click", "mark": 13 },    { "type": "wait_for_settled", "quietMs": 200 }  ],  "verifyChange": true,  "screenshotAfter": true}

handy_see returns each element's index, kind, bounds, centerX/centerY, and a score; marks stay valid in the server process until the next marked capture. The CLI re-detects on demand per invocation (detection is deterministic for an unchanged screen), and one chain can do both atomically:

bash
handy see --marks --out marked.pnghandy act 'click mark 5 -> paste "Sam" -> click mark 13 -> wait_for_settled' --screenshot-after after.pnghandy act '[{"type":"screenshot","marks":true},{"type":"click","mark":5}]'

Standalone CLI

bash
handy see --out screen.png                 # coordinate-mapped capturehandy see --window "Scratchpad" --highlight-diff --out shot.pnghandy see --marks --max-marks 20 --json    # numbered UI elementshandy windows && handy focus "Visual Studio Code"handy clipboard get | handy clipboard set "Copied from terminal"handy act 'focus "Scratchpad" -> click 800 420 -> paste "Hello 🌍" -> wait_for_settled'handy key ctrl+s                           # chords work on every platform bridge

Deployment

Permission controls & human failsafe

Handy stores its permission toggle at ~/.handy/permissions.json (0600) and checks it synchronously before every handy_see capture, handy_windows/handy_clipboard call, and before every individual step of a handy_act chain:

bash
handy off                 # immediately block all LLM screen viewing & input controlhandy on                  # full screen vision & mouse/keyboard controlhandy toggle              # flip between ON and OFFhandy on --view-only      # allow screen viewing, block mouse & keyboardhandy on --ttl 30m        # auto-disable after 30 minuteshandy status              # toggle state, backend, windows & displays
  • Physical failsafe: slamming your mouse into the top-left (0, 0) corner aborts any in-flight chain and releases held keys/buttons immediately — the controller reads the real cursor position each step, so it works even when the agent commands a different pointer path.
  • Prompt-injection guard: when turned OFF by the user, an MCP client cannot re-enable Handy on its own; handy_permission({ "action": "off" }) is always permitted as a one-way stop switch.

Backend selection & environment

The backend is picked automatically per platform; HANDY_BACKEND (auto, virtual, linux-x11, linux-cli/wayland, darwin, win32) overrides it. On a Wayland session the CLI backend is chosen even when DISPLAY is set (XWayland), because that is the only backend that can see the real desktop; on a pure X11 session the native X11 client takes over with full window listing and focus support. HANDY_ENABLED=false in the host environment is a second, independent kill switch.

Known platform limits, stated plainly: handy_windows/focus_window enumerate windows through the X11/Wayland-of-record APIs (EWMH, wmctrl) and see XWayland windows only on GNOME Wayland — native windows there are not publicly enumerable by any client-side API. Capture and input are unaffected.


Adapting to your environment

A handful of values reflect the author's machine rather than yours. None are secrets:

WhatWhereMake it yours
Linux input daemonydotool + ydotoold (see Linux (Wayland) setup)Distros differ in how the daemon ships; handy doctor reports what it found
Capture tools (non-GNOME Wayland)grim (wlroots) / maim / scrotOptional — the portal route needs nothing installed
Display overrideHANDY_BACKEND in the MCP host envOnly when auto-detection needs help
Permission store~/.handy/permissions.jsonCreated on first run; 0600
Author · repo · fundingpackage.json, the badges aboveYour own details if you fork; the funding links support the original author

Architecture

src/├── index.ts            public API barrel├── mcp.ts              six MCP tools over stdio + resources + prompts├── controller.ts       permission + failsafe enforcement between steps, chains, receipts├── cli.ts              the `handy` command surface├── installer.ts        host auto-setup (`handy install`, backups, uninstall)├── marks.ts            Set-of-Mark detection, DPI-independent, fail-soft├── overlay.ts          dual-contrast rulers, reticles, trail markers├── png.ts              decode/encode, box-filter resize, per-row filters├── diff.ts             frame diffing with early exit├── permissions.ts      ~/.handy/permissions.json store, TTLs, view-only├── keys.ts             per-platform key maps (X11 keysyms, Darwin, Win32 VK, Linux EV_KEY)├── ui.ts               loopback Control Panel (SSE)└── backends/    ├── win32.ts        SendInput, EnumWindows, GDI capture    ├── darwin.ts       CGEvent, NSWorkspace, CGWindowList    ├── linux-x11.ts    native X11 wire protocol client (XTEST, GetImage, RANDR, EWMH)    ├── linux-cli.ts    Wayland: portal capture + ydotool/wtype/xdotool/wl-clipboard    └── virtual.ts      headless 1280x800 workbench for CI

Key properties: no native build steps (pure wire-protocol clients), permission checks synchronously before every effect, the failsafe reads the real cursor each step, mark references fail loudly when stale, and every route that can fail falls through with its error named.

Development

bash
npm run typecheck    # src + tests, strictnpm run build        # lib/, bin/handy.js, public/handy.pngnpm test             # vitest across every backend bridge and the stdio MCP surfacenpm run validate     # version-site agreement, MCPB manifest, plugin commands, skill references

Real input is exercised against real desktop sessions — Windows SendInput, macOS CGEvent, Linux XTEST/uinput — with the virtual backend covering headless CI; a mock X11 server tests the binary protocol byte-for-byte.

Contributing

Handy is maintained by its author with contributions welcome: open an issue first for anything non-trivial, keep changes focused, match the existing code style, and update the docs a change touches — a tool change without its README/skill update is half a change. npm run typecheck, npm run build and npm test must pass clean; CI runs the same checks on every push and PR, on all three operating systems.

Security

Please do not report suspected vulnerabilities in a public issue — Handy is, by design, a tool that can click and type on someone's computer. Follow the private reporting process in SECURITY.md; never include screenshots or captured screen content in a report.

Privacy and support

Handy adds no analytics, no telemetry and no hosted account. Screen captures go only to the MCP client that asked for them; permission state lives in ~/.handy/; the Control Panel binds to loopback only. The full privacy policy says what is stored where. For help, open a GitHub issue or email [email protected].


Support the project

Handy is built and maintained by one person. If it's useful to you, sponsorship directly funds its continued development:

Starring the repo, reporting bugs, and spreading the word help too.

License

MIT © Kevin Miiso Novo

Credits

  • Model Context Protocol — the open standard that lets one server drive every MCP-capable client.
  • The GNOME, wlroots and freedesktop portal teams — Wayland's capture and input story works because of them.

Source: README.md at commit 803c0d9

Tools

6

handy_see

See Computer Screen (Mapped Coordinates + Window Zoom + Diff)

Capture the computer screen (or a single desktop window via `window`) in real-time with mapped (x, y) coordinate rulers, dual-contrast grid labels, live mouse reticle, hex color under cursor, active window info, and visual diff since the previous capture (`changedPercent`, `changedBounds`). Use `region` or `window` to zoom into dense UI areas at 1:1 resolution while preserving global screen coordinates, and set `highlightDiff: true` to outline changed pixels. Set `marks: true` to run Set-of-Mark UI element detection: Handy outlines and numbers every detected button, field, icon, text line, and panel, and returns their exact bounds and click centres in `elements` — then act on one with `{ "type": "click", "mark": 3 }` instead of estimating coordinates.

ParametersgridmarksregionwindowdisplaygridStepmaxMarksmaxWidthmarkKindsshowMouseminMarkScorehighlightDiff

handy_act

Control Mouse & Keyboard (Single or Chained)

Execute native mouse, keyboard, clipboard-paste, window-focus, and smart visual-wait controls in real-time. Supports chaining multiple steps sequentially in one call — e.g. [{ "type": "focus_window", "title": "Code" }, { "type": "click", "x": 420, "y": 260 }, { "type": "paste", "text": "Hello!" }, { "type": "wait_for_settled" }]. Steps support `coordinateSpace: "screen" | "normalized" | "window"`, and any pointer step accepts `mark: N` to target a detected UI element from the last `handy_see({ marks: true })` instead of raw x/y. Set `screenshotAfter: true` to receive an updated coordinate-mapped screenshot with numbered action-trail markers, or `verifyChange: true` for a cheap did-anything-happen diff.

ParametersgridmarksactionsrequireddisplayfailsafemaxMarksmaxWidthsettleMsshowMouseshowTrailstepDelayMsverifyChange+2 more

handy_windows

List or Focus Desktop Windows

List visible desktop windows with their titles, owning apps, screen bounds `{ x, y, width, height }`, and focused state (`action: 'list'`), or bring a specific window to the foreground (`action: 'focus'`).

Parametersidapptitleactionrequired

handy_clipboard

Read or Write System Clipboard

Read current plain-text clipboard contents (`action: 'read'`, works in `full` and `view_only` modes) or write Unicode text directly to the system clipboard (`action: 'write'`, requires `full` control mode).

Parameterstextactionrequired

handy_status

Check Handy Status & Permission Toggle

Check whether Handy permission is currently ON or OFF, which native OS backend is active, connected monitor dimensions, active window, and current mouse position (when enabled).

Takes no parameters.

handy_permission

Handy Permission Control (On/Off Toggle)

Inspect or turn OFF Handy computer control permissions. Turning Handy OFF (`action: 'off'`) immediately disables screen capture and mouse/keyboard control and releases any held inputs. When disabled by the user, re-enabling requires running `handy on` (or `/handy:on`) unless HANDY_ALLOW_MCP_ENABLE=true.

Parametersmodeactionrequired

Version history

1
  1. v1.4.0LatestNov 21, 2026

Score

90% of the applicable checks met

How complete this listing is, computed from the registry entry, the endpoint and public sources. Not a rating of the server and not a statement that it is safe or working today: reachability is a dated check, and no installs or reviews go into the score.

  • Reached in the last 30 days25 pts
  • Auth declared10 pts

    Declare the credential headers in the registry entry, or keep the endpoint answering the probe so its auth can be classified.

  • Tools listedRequired15 pts
  • DescriptionRequired15 pts
  • Logo10 pts
  • Documentation15 pts
  • License10 pts