Handy

io.github.miisodev由 GitHub 账号 miisodev 发布v1.4.0更新于 Nov 21, 2026

Native mouse, keyboard, window, clipboard, and screen control for LLMs via stdio MCP.

概览

AI 生成的概览

本地 stdio MCP 服务器,让助手看到带坐标映射的桌面画面,并操控原生鼠标、键盘、剪贴板和窗口。

可以做什么
  • 截取整个屏幕、单个窗口或放大的局部区域,附带坐标标尺、光标处颜色、活动窗口信息、帧间差异,以及可选的交互元素编号标记
  • 执行单步或链式的原生操作:移动、点击、拖拽、滚动、输入、粘贴、组合键、按住鼠标或按键、聚焦窗口和视觉等待,可选操作后校验与事后截图
  • 列出可见桌面窗口并将指定窗口置于前台
  • 读取或写入系统纯文本剪贴板
  • 查看权限状态、当前后端、显示器、活动窗口和光标位置
  • 关闭电脑控制权限,作为单向停止开关
适用场景
当你希望助手真正操作原生桌面应用——填写表单、点击界面、确认操作是否改变了画面——而不只是描述时使用。适合在 Windows、macOS 或 Linux 上测试真实桌面工作流的开发者和智能体构建者。
运行要求
需要 Node.js 20+ 和通过 stdio 连接的本地 MCP 客户端。macOS 与 Windows 需要系统的辅助功能/屏幕录制权限。Linux Wayland 的输入需要 ydotool/ydotoold;截图使用 freedesktop 截图门户或 grim。可选环境变量 HANDY_BACKEND 和 HANDY_ENABLED。
安装前请注意
这会赋予助手对你机器的真实控制权:它可以点击、输入、粘贴并读取屏幕和剪贴板,因此屏幕上可见的内容都可能传给模型。请留意权限开关和仅查看模式;用户关闭后客户端无法自行重新开启。截图只发送给发起请求的客户端;据称无遥测、无托管账户。

安装

在 SourceWeft 中

  1. 打开 控制台中的 Handy,将其添加到工作区。
  2. 为需要使用其工具的对话启用该服务。

Desktop only,通过 STDIO。 STDIO 服务会启动本地进程,因此需要 SourceWeft 桌面宿主。

其他 MCP 客户端

参照 仓库 中的启动说明。

README

[Handy logo]

handy

Give your agent eyes and hands — see and drive the real desktop.

Real-time native mouse, keyboard, window, clipboard and coordinate-mapped screen control for LLMs and agents, served over the Model Context Protocol as a stdio server and CLI — with numbered Set-of-Mark targets, chained multi-step actions, an instant on/off permission toggle and a physical (0, 0) corner-slam failsafe.

Site · Quick start · The tool surface · How it works · Privacy

[Version] [CI] [License: MIT] [Platforms] [MCP] [Sponsor]


What is handy?

Agents get screenshots, but few get honest control of the machine they are looking at. Coordinates are guessed from pixels, one-shot clicks land where nothing happened, and the human watching has no switch to stop it all.

Handy is an MCP server (with a full CLI alongside) that gives a model real-time native control of a desktop — mouse, keyboard, clipboard and windows — paired with a live coordinate-mapped view of the screen: dual-contrast (x, y) rulers, active-window telemetry, pixel hex under the cursor, sub-2ms frame-to-frame diffs, and Set-of-Mark detection that numbers every interactive element on screen so the agent clicks mark 13 instead of guessing (631, 447). Up to 100 steps — focus a window, click, paste, wait for the UI to settle — run in one atomic chained call, with an after-shot that shows numbered trail markers where every action landed.

It installs cleanly on any PC with zero C++ addon compilation (node-gyp free), speaks plain stdio to any MCP client, and treats permission as a product surface: a ~/.handy/permissions.json toggle checked before every capture and every chain step, a --view-only mode, --ttl timers, a live loopback Control Panel, and a physical failsafe — slam your mouse into the top-left (0, 0) corner and any in-flight chain aborts instantly.

Who it's for

Developers and agent builders who want their assistant to actually operate the desktop — fill forms, drive native apps, verify what happened — rather than describe it. Any MCP-capable client works (Claude Desktop, Claude Code, Cursor, Windsurf, and everything else speaking MCP); the CLI makes the same surface usable from shell scripts and CI.

From experiment to open source

Handy began as the input layer for the author's own agents, then grew through daily real-machine use — real installs, real Wayland compositors, real failsafes — into a tool that behaves the same on Windows, macOS and Linux. What you're reading is the same code that runs the author's desktop every day, not a demo.

The core principle

An agent that can see and click must be interruptible.

Handy checks its permission store synchronously before every capture and before every individual step of a chain, and the failsafe reads the real cursor position each step, so a human can always stop the machine — by toggle, by corner-slam, or by timer. When the toggle is OFF, no MCP client can turn it back on by itself; handy_permission({ "action": "off" }) remains a one-way stop switch.

Design highlights

  • Set-of-Mark vision (marks: true) — handy_see detects, classifies (control / text / icon / region), numbers and outlines the interactive elements on screen in reading order; every pointer step then accepts { "mark": N } and the chain DSL understands click mark 5 / click #5. Marks resolve to the element's own centre at execution time, so a chain survives layout shift and HiDPI scaling. Detection is DPI-independent (~18ms at 1280x800, ~22ms at 2560x1440, ~49ms at 4K) because the work buffer is area-capped rather than dimension-capped.
  • One atomic call for a whole interaction (handy_act) — up to 100 sequential steps (focus_window, move, click, drag, scroll, paste, type, key, mouse_down/up, key_down/up, wait, wait_for_change, wait_for_settled, wait_for_color, screenshot) in a single call, with screen/normalized (0..1000)/window coordinate spaces and numbered amber trail markers on screenshotAfter.
  • Human-like pointer motion (motion: "human") — Bézier arcs with Fitts's-law timing instead of teleporting straight lines, with the duration derived automatically from travel distance and target width when omitted.
  • Post-action verification (verifyChange: true) — a baseline capture before the chain and a frame diff after, so the receipt says whether the click actually did anything.
  • Captures that survive real screens — dual-contrast rulers legible on #000 and #fff apps alike, area-average box-filter downscaling that keeps 1px text alive, and per-row PNG filter selection for ~26% smaller encodes.
  • Fail-soft enhancements — mark detection degrades to a plain screenshot (marksError) instead of failing handy_see, and stale marks fail loudly with the available indices instead of clicking somewhere arbitrary.
  • Strict chain-DSL validation — malformed pointer arguments throw with the offending segment quoted; out-of-range marks list what exists. Nothing degrades to "act wherever the cursor happens to be".

How it works

Handy is one TypeScript process with zero runtime dependencies beyond the MCP SDK and Zod. Each platform speaks its own native bridge — no wrappers, no VMs:

PlatformBackendHow
Windows 10/11win32Persistent Win32 SendInput (KEYEVENTF_UNICODE, MOUSEEVENTF_VIRTUALDESK), EnumWindows/SetForegroundWindow, MTA-safe OpenClipboard, GDI CopyFromScreen, Per-Monitor V2 DPI awareness
macOSdarwinPersistent CoreGraphics CGEvent + NSWorkspace + NSPasteboard, in-memory CGWindowListCreateImage PNG capture (with screencapture fallback), Retina 2x logical-point mapping
Linux — X11linux-x11Pure native X11 wire-protocol client (XTEST, GetImage, BIG-REQUESTS, RANDR, EWMH _NET_CLIENT_LIST/_NET_ACTIVE_WINDOW) — zero external packages
Linux — Waylandlinux-cliAuto-selected on Wayland sessions: screen capture through the freedesktop screenshot portal (works out of the box on GNOME, whose Shell screenshot API rejects third-party callers) or grim on wlroots; input via ydotool (uinput daemon) or wtype/xdotool; clipboard via wl-clipboard/xclip/xsel
Headless / CIvirtualBuilt-in 1280x800 multi-window interactive desktop workbench for safe agent sandboxing — no display required

Everything above the bridge is shared: controller.ts enforces permissions and the failsafe between steps, marks.ts detects and numbers UI elements, overlay.ts renders rulers and reticles, png.ts handles decode/encode, and the MCP layer (mcp.ts) wraps it all as six tools.

A capture-and-act round trip looks like: the agent calls handy_see { marks: true } → the backend grabs a frame (portal on GNOME Wayland, native protocol on X11/Windows/macOS) → detection numbers the elements → the agent calls handy_act with a chain referencing mark indices → the controller re-checks permissions and the failsafe before each step → a receipt (and optionally a diff and after-shot) comes back.


Quick start

Requires Node.js 20+.

1. One-Command Auto-Setup (Claude Desktop, Cursor, Windsurf, Claude Code)

bash
npx -y @miisodev/handy install

Preview changes first with --dry-run, target a single host (handy install cursor), or remove cleanly (handy uninstall). Backups (*.handy.bak) are created automatically before modifying existing config files.

2. Prepackaged Plugin (Claude Code / Agent Marketplaces)

Installs the stdio MCP server, the handy operational skill (skills/handy/SKILL.md), and the /handy:on, /handy:off, /handy:toggle, /handy:status, /handy:doctor, and /handy:windows commands together:

bash
claude plugin marketplace add miisodev/handyclaude plugin install handy

3. Global CLI + Live Control Panel

bash
npm install -g @miisodev/handyhandy doctor                # verify input & capture readiness for your platformhandy ui                    # live Control Panel at http://127.0.0.1:7788

4. Manual Stdio MCP Configuration

json
{  "mcpServers": {    "handy": {      "command": "npx",      "args": ["-y", "@miisodev/handy", "mcp"]    }  }}

Linux (Wayland) setup

Capture works out of the box on GNOME and any compositor whose portal serves Screenshot. Input needs a running ydotoold (uinput), which most distros ship:

bash
sudo apt install ydotool            # or your distro's packagesudo systemctl enable --now ydotoold

If your distro only ships a user-level unit (some recent Ubuntu releases do), it cannot open /dev/uinput; run the daemon as a root system service with a user-owned socket instead:

bash
sudo systemctl edit --force --full ydotoold   # then set:# [Service]# ExecStart=/usr/bin/ydotoold -p /run/user/1000/.ydotool_socket -P 0660 -o 1000:1000# RuntimeDirectory=ydotoold

handy doctor tells you exactly which routes and tools it found — session type, capture routes, input tool.


The tool surface

All returns are structured JSON. Input mistakes come back as informational errors the model can self-correct from; every receipt names what was executed, where, and how long it took.

ToolDescription
handy_seeCapture the live screen, a single window, or a zoomed region with dual-contrast (x, y) rulers, cursor reticle, pixelUnderMouse, activeWindow, screenDiff, and highlightDiff. With marks: true it also detects, numbers, and returns every interactive UI element (elements).
handy_actExecute single or chained native mouse, keyboard, paste, window-focus, and smart visual-wait (wait_for_change, wait_for_settled, wait_for_color) controls with screen/normalized/window coordinates, mark element references, motion profiles, numbered trail markers, and optional verifyChange post-conditions.
handy_windowsList visible desktop windows (title, app, bounds, focused) or focus a window by title, app, or id.
handy_clipboardRead or write plain-text system clipboard directly.
handy_statusInspect live permission toggle state (ON/OFF), mode (full/view_only), OS backend, displays, active window, and cursor coordinates.
handy_permissionQuery permission status or immediately turn Handy off.

Example: See numbered elements, then act on them

json
// handy_see{ "marks": true, "maxMarks": 30 }
json
// handy_act — marks 5 and 13 come from the capture above{  "actions": [    { "type": "click", "mark": 5 },    { "type": "paste", "text": "Sam — ready for review 🚀" },    { "type": "click", "mark": 13 },    { "type": "wait_for_settled", "quietMs": 200 }  ],  "verifyChange": true,  "screenshotAfter": true}

handy_see returns each element's index, kind, bounds, centerX/centerY, and a score; marks stay valid in the server process until the next marked capture. The CLI re-detects on demand per invocation (detection is deterministic for an unchanged screen), and one chain can do both atomically:

bash
handy see --marks --out marked.pnghandy act 'click mark 5 -> paste "Sam" -> click mark 13 -> wait_for_settled' --screenshot-after after.pnghandy act '[{"type":"screenshot","marks":true},{"type":"click","mark":5}]'

Standalone CLI

bash
handy see --out screen.png                 # coordinate-mapped capturehandy see --window "Scratchpad" --highlight-diff --out shot.pnghandy see --marks --max-marks 20 --json    # numbered UI elementshandy windows && handy focus "Visual Studio Code"handy clipboard get | handy clipboard set "Copied from terminal"handy act 'focus "Scratchpad" -> click 800 420 -> paste "Hello 🌍" -> wait_for_settled'handy key ctrl+s                           # chords work on every platform bridge

Deployment

Permission controls & human failsafe

Handy stores its permission toggle at ~/.handy/permissions.json (0600) and checks it synchronously before every handy_see capture, handy_windows/handy_clipboard call, and before every individual step of a handy_act chain:

bash
handy off                 # immediately block all LLM screen viewing & input controlhandy on                  # full screen vision & mouse/keyboard controlhandy toggle              # flip between ON and OFFhandy on --view-only      # allow screen viewing, block mouse & keyboardhandy on --ttl 30m        # auto-disable after 30 minuteshandy status              # toggle state, backend, windows & displays
  • Physical failsafe: slamming your mouse into the top-left (0, 0) corner aborts any in-flight chain and releases held keys/buttons immediately — the controller reads the real cursor position each step, so it works even when the agent commands a different pointer path.
  • Prompt-injection guard: when turned OFF by the user, an MCP client cannot re-enable Handy on its own; handy_permission({ "action": "off" }) is always permitted as a one-way stop switch.

Backend selection & environment

The backend is picked automatically per platform; HANDY_BACKEND (auto, virtual, linux-x11, linux-cli/wayland, darwin, win32) overrides it. On a Wayland session the CLI backend is chosen even when DISPLAY is set (XWayland), because that is the only backend that can see the real desktop; on a pure X11 session the native X11 client takes over with full window listing and focus support. HANDY_ENABLED=false in the host environment is a second, independent kill switch.

Known platform limits, stated plainly: handy_windows/focus_window enumerate windows through the X11/Wayland-of-record APIs (EWMH, wmctrl) and see XWayland windows only on GNOME Wayland — native windows there are not publicly enumerable by any client-side API. Capture and input are unaffected.


Adapting to your environment

A handful of values reflect the author's machine rather than yours. None are secrets:

WhatWhereMake it yours
Linux input daemonydotool + ydotoold (see Linux (Wayland) setup)Distros differ in how the daemon ships; handy doctor reports what it found
Capture tools (non-GNOME Wayland)grim (wlroots) / maim / scrotOptional — the portal route needs nothing installed
Display overrideHANDY_BACKEND in the MCP host envOnly when auto-detection needs help
Permission store~/.handy/permissions.jsonCreated on first run; 0600
Author · repo · fundingpackage.json, the badges aboveYour own details if you fork; the funding links support the original author

Architecture

src/├── index.ts            public API barrel├── mcp.ts              six MCP tools over stdio + resources + prompts├── controller.ts       permission + failsafe enforcement between steps, chains, receipts├── cli.ts              the `handy` command surface├── installer.ts        host auto-setup (`handy install`, backups, uninstall)├── marks.ts            Set-of-Mark detection, DPI-independent, fail-soft├── overlay.ts          dual-contrast rulers, reticles, trail markers├── png.ts              decode/encode, box-filter resize, per-row filters├── diff.ts             frame diffing with early exit├── permissions.ts      ~/.handy/permissions.json store, TTLs, view-only├── keys.ts             per-platform key maps (X11 keysyms, Darwin, Win32 VK, Linux EV_KEY)├── ui.ts               loopback Control Panel (SSE)└── backends/    ├── win32.ts        SendInput, EnumWindows, GDI capture    ├── darwin.ts       CGEvent, NSWorkspace, CGWindowList    ├── linux-x11.ts    native X11 wire protocol client (XTEST, GetImage, RANDR, EWMH)    ├── linux-cli.ts    Wayland: portal capture + ydotool/wtype/xdotool/wl-clipboard    └── virtual.ts      headless 1280x800 workbench for CI

Key properties: no native build steps (pure wire-protocol clients), permission checks synchronously before every effect, the failsafe reads the real cursor each step, mark references fail loudly when stale, and every route that can fail falls through with its error named.

Development

bash
npm run typecheck    # src + tests, strictnpm run build        # lib/, bin/handy.js, public/handy.pngnpm test             # vitest across every backend bridge and the stdio MCP surfacenpm run validate     # version-site agreement, MCPB manifest, plugin commands, skill references

Real input is exercised against real desktop sessions — Windows SendInput, macOS CGEvent, Linux XTEST/uinput — with the virtual backend covering headless CI; a mock X11 server tests the binary protocol byte-for-byte.

Contributing

Handy is maintained by its author with contributions welcome: open an issue first for anything non-trivial, keep changes focused, match the existing code style, and update the docs a change touches — a tool change without its README/skill update is half a change. npm run typecheck, npm run build and npm test must pass clean; CI runs the same checks on every push and PR, on all three operating systems.

Security

Please do not report suspected vulnerabilities in a public issue — Handy is, by design, a tool that can click and type on someone's computer. Follow the private reporting process in SECURITY.md; never include screenshots or captured screen content in a report.

Privacy and support

Handy adds no analytics, no telemetry and no hosted account. Screen captures go only to the MCP client that asked for them; permission state lives in ~/.handy/; the Control Panel binds to loopback only. The full privacy policy says what is stored where. For help, open a GitHub issue or email [email protected].


Support the project

Handy is built and maintained by one person. If it's useful to you, sponsorship directly funds its continued development:

Starring the repo, reporting bugs, and spreading the word help too.

License

MIT © Kevin Miiso Novo

Credits

  • Model Context Protocol — the open standard that lets one server drive every MCP-capable client.
  • The GNOME, wlroots and freedesktop portal teams — Wayland's capture and input story works because of them.

来源:README.md,提交 803c0d9

工具

6

handy_see

See Computer Screen (Mapped Coordinates + Window Zoom + Diff)

Capture the computer screen (or a single desktop window via `window`) in real-time with mapped (x, y) coordinate rulers, dual-contrast grid labels, live mouse reticle, hex color under cursor, active window info, and visual diff since the previous capture (`changedPercent`, `changedBounds`). Use `region` or `window` to zoom into dense UI areas at 1:1 resolution while preserving global screen coordinates, and set `highlightDiff: true` to outline changed pixels. Set `marks: true` to run Set-of-Mark UI element detection: Handy outlines and numbers every detected button, field, icon, text line, and panel, and returns their exact bounds and click centres in `elements` — then act on one with `{ "type": "click", "mark": 3 }` instead of estimating coordinates.

输入参数gridmarksregionwindowdisplaygridStepmaxMarksmaxWidthmarkKindsshowMouseminMarkScorehighlightDiff

handy_act

Control Mouse & Keyboard (Single or Chained)

Execute native mouse, keyboard, clipboard-paste, window-focus, and smart visual-wait controls in real-time. Supports chaining multiple steps sequentially in one call — e.g. [{ "type": "focus_window", "title": "Code" }, { "type": "click", "x": 420, "y": 260 }, { "type": "paste", "text": "Hello!" }, { "type": "wait_for_settled" }]. Steps support `coordinateSpace: "screen" | "normalized" | "window"`, and any pointer step accepts `mark: N` to target a detected UI element from the last `handy_see({ marks: true })` instead of raw x/y. Set `screenshotAfter: true` to receive an updated coordinate-mapped screenshot with numbered action-trail markers, or `verifyChange: true` for a cheap did-anything-happen diff.

输入参数gridmarksactions必填displayfailsafemaxMarksmaxWidthsettleMsshowMouseshowTrailstepDelayMsverifyChange另有 2 个

handy_windows

List or Focus Desktop Windows

List visible desktop windows with their titles, owning apps, screen bounds `{ x, y, width, height }`, and focused state (`action: 'list'`), or bring a specific window to the foreground (`action: 'focus'`).

输入参数idapptitleaction必填

handy_clipboard

Read or Write System Clipboard

Read current plain-text clipboard contents (`action: 'read'`, works in `full` and `view_only` modes) or write Unicode text directly to the system clipboard (`action: 'write'`, requires `full` control mode).

输入参数textaction必填

handy_status

Check Handy Status & Permission Toggle

Check whether Handy permission is currently ON or OFF, which native OS backend is active, connected monitor dimensions, active window, and current mouse position (when enabled).

无需输入参数。

handy_permission

Handy Permission Control (On/Off Toggle)

Inspect or turn OFF Handy computer control permissions. Turning Handy OFF (`action: 'off'`) immediately disables screen capture and mouse/keyboard control and releases any held inputs. When disabled by the user, re-enabling requires running `handy on` (or `/handy:on`) unless HANDY_ALLOW_MCP_ENABLE=true.

输入参数modeaction必填

版本历史

1
  1. v1.4.0最新Nov 21, 2026

评分

已满足适用检查项的 90%

该评分衡量此收录条目的完整程度,依据注册中心条目、端点与公开来源计算得出。它不是对服务器本身的评价,也不表示服务器安全或当前可用:可连通性是带日期的检查结果,评分也不包含安装量或用户评价。

  • 近 30 天内可连通25 分
  • 已声明认证方式10 分

    请在注册中心条目中声明凭据请求头,或保持端点能够响应探测,以便判定其认证方式。

  • 已列出工具必需15 分
  • 描述必需15 分
  • 图标10 分
  • 文档15 分
  • 许可证10 分