Noracu

noracu

A local MCP server that lets Claude, Cursor, or any MCP client operate macOS apps and a Playwright browser by element_index — and, experimentally, a real iPhone through Apple's iPhone Mirroring (OCR on the Mac, normalized taps). No API key; no telemetry from the server.

[CI] [npm] [node] [license]

中文说明见 README.zh-CN.md。

One stdio MCP server. The agent snapshots a tree, gets an indexed element list, then clicks or types by element_index instead of raw pixels — the same observe/act split on the Mac and in the browser. The experimental iPhone layer runs the same observe → act loop over OCR text and normalized coordinates.

[browser element_index demo]

browser_snapshot returns an indexed ARIA tree; browser_click acts by element_index. The Mac layer runs the same loop (mac_ax_state → mac_click). Highlight boxes are overlays from demo/record-browser.mjs, not a product feature. The iPhone layer is not in this demo.

What it does

One MCP server, three control surfaces:

The iPhone layer is experimental and ships with its driver (imctl, in imctl/). It needs Apple's iPhone Mirroring with the iPhone locked and near the Mac, and a python3 with pyobjc; it observes with OCR and acts on normalized coordinates rather than element_index. Without pyobjc, phone_* tools return a one-line imctl unavailable error with the install command, and the Mac and browser layers are unaffected. The observe/act vocabulary follows the action/observation split popularized by ChatGPT's computer-use, re-implemented on open local tooling.

Why noracu

noracu is the local hands, not the model. Your MCP client (Claude Desktop, Claude Code, Cursor) decides what to do; this server only observes and acts on this machine.

• The server runs entirely on your machine. No cloud service, no API key, no telemetry, no postinstall scripts. Sole runtime dependency: Playwright. noracu makes no network calls of its own; the only automatic one is the one-time Chromium download on the first browser_launch (NORACU_DISABLE_AUTO_INSTALL=1 turns that off). The browser still fetches the pages you point it at, and your MCP client still sends tool results — screenshots, OCR text — to whatever model it uses.
• Acts on semantics, not pixels. Observation returns the accessibility tree (Mac) or ARIA tree (browser) as an indexed list — text, not screenshots. Actions reference element_index; indices expire after 30 s, and clicks re-verify the target (tag/name/href/position in the browser, frontmost app on the Mac) before firing. A stale reference returns element_no_longer_valid instead of clicking the wrong thing. Screenshots are there when the tree is not enough.
• One loop, three surfaces. Native Mac apps and a persistent-profile browser share the element_index protocol (observe a tree, act by index). The experimental iPhone layer runs the same observe → act loop on Apple's iPhone Mirroring, with OCR text and normalized coordinates, because the phone exposes no accessibility tree to the Mac.
• Uses your real sessions. Your logged-in browser profile, your Mac apps — not a hosted VM. Compared with cloud computer-use: neither is universally more reliable; they fail differently. noracu keeps the screen on your machine.
• Off switches that are actually off. NORACU_READ_ONLY=1 (observation-only, default-deny), NORACU_DISABLE_EVAL=1, and file:// blocked by default. See Security.
• Verifiable. 39 end-to-end checks (npm test), CI on macos-latest, MIT, and the whole server is one file: src/server.mjs.

Not a good fit if you need multiple browser tabs or sessions (single session for now), you're not on macOS (the Mac and iPhone layers are macOS-only; the browser layer is cross-platform), or you need a finished iPhone layer today (it ships, but experimental — and Apple does not offer iPhone Mirroring in the EU). If you only need browser automation, playwright-mcp has tab management and selector APIs that noracu v0.1 lacks; noracu is for when one agent also has to leave the browser and drive Mac apps.

Quickstart

First success is mac_ax_state (or a browser snapshot) — not a click, and not a phone_* tool. The biggest day-one failure is granting macOS permissions to node instead of to the host app; grants apply only after a full quit and reopen.

1. Install (Node ≥ 20; no postinstall script):

npm install noracu          # server at node_modules/noracu/src/server.mjsbrew install cliclick       # required for Mac input (mac_click/type/key/paste)

Or from source: git clone https://github.com/sgInnora/noracu.git && cd noracu && npm install && npm test (add NORACU_CI=1 to skip hardware-gated checks).

2. Add to your MCP client — use an absolute path to src/server.mjs. GUI apps launched from Finder/Dock do not inherit your shell PATH, so give an absolute path to the node binary as well (which node) — a bare node, npx, or noracu needs Node on the GUI PATH and otherwise fails with ENOENT.

Claude Desktop (~/Library/Application Support/Claude/claude_desktop_config.json) / Cursor (~/.cursor/mcp.json):

{  "mcpServers": {    "noracu": {      "command": "/opt/homebrew/bin/node",      "args": ["/ABSOLUTE/PATH/TO/noracu/src/server.mjs"]    }  }}

Claude Code: claude mcp add noracu -- /ABSOLUTE/PATH/TO/node /ABSOLUTE/PATH/TO/noracu/src/server.mjs

Find your node path with which node (Apple Silicon: /opt/homebrew/bin/node; Intel: /usr/local/bin/node; nvm: the versioned path). Convenience, only if Node is already on the GUI PATH: "command": "npx", "args": ["-y", "noracu"].

Then quit the host app completely (including any menu-bar icon) and reopen it. A loaded tool list means the transport works — it does not need any permission yet.

3. Grant macOS permissions to the host app, not to node. System Settings → Privacy & Security → Accessibility and Screen Recording (macOS 15: "Screen & System Audio Recording") → enable the host (Claude Desktop / Cursor / your terminal). noracu never appears here. Quit and reopen the host after granting.

4. Verify. With Finder in front, call mac_ax_state (no args) — it returns an indexed accessibility tree; indices last 30 s. Browser smoke test (no Accessibility needed): browser_launch (first run downloads Chromium unless NORACU_DISABLE_AUTO_INSTALL=1) → browser_navigate → browser_snapshot → browser_click by index.

Troubleshooting

Node 18 is rejected at startup — use Node ≥ 20. Claude Desktop may start one noracu process per surface (e.g. chat and a second surface); they share the browser profile, so launch the browser from only one at a time.

Full worked examples: docs/usage.md.

iPhone layer (experimental)

noracu can operate a real iPhone through Apple's iPhone Mirroring. The agent observes with OCR on the Mac (phone_screenshot, phone_ocr, phone_find), acts at normalized 0–1 coordinates (phone_tap, phone_swipe, phone_scroll), types by pasting (phone_type) and opens apps by name (phone_open). This is not element_index: the phone exposes no accessibility tree to the Mac, so a tap lands where OCR placed the text. The driver (imctl, Python, MIT, about 1,000 lines) ships in imctl/.

Apple's requirements (support.apple.com/120421): a Mac with Apple silicon or a T2 chip on macOS 15 or later; an iPhone on iOS 18 or later with a passcode; both signed in to the same Apple Account with two-factor authentication; Bluetooth and Wi-Fi on; the Mac not sharing its internet connection or using AirPlay or Sidecar. The iPhone must be locked and near the Mac — unlocking it, or quitting iPhone Mirroring, ends the session. iPhone Mirroring is currently unavailable in the EU. One Mac and one iPhone at a time; camera and microphone are not mirrored.

Unattended sessions: in iPhone Mirroring → Settings, "Authenticate automatically" lets a session start without a password prompt. Trade-off: anyone who can use your unlocked Mac can then open your phone.

Python: the driver needs a python3 that imports pyobjc (Quartz, Vision, AppKit). Set NORACU_PYTHON, or noracu probes /opt/homebrew/bin/python3, /usr/local/bin/python3, python3, /usr/bin/python3 once, on the first phone_* call. Apple's /usr/bin/python3 usually lacks pyobjc; Homebrew's python3 plus python3 -m pip install pyobjc-framework-Quartz pyobjc-framework-Vision pyobjc-framework-AppKit is the common working setup.

Mirror state: phone_status reports live (the phone's screen is being streamed), overlay (the text iPhone Mirroring shows instead, e.g. "Connection Paused" or "iPhone in Use"), paused (that overlay offers a Resume button) and setup (a setup or welcome screen the user must finish — noracu never clicks it). Detection reads the window's structure and shape (the mirror is a tall or wide phone-shaped window; a setup screen is nearly square), not its text. While the mirror is not live, phone_tap, phone_longpress, phone_swipe, phone_scroll, phone_type, phone_key and phone_open refuse with mirror_not_live instead of tapping the overlay, and phone_ocr / phone_find start their output with a warning. The mirror pauses after inactivity; phone_launch brings it forward and clicks Resume.

Privacy: whatever the phone shows — notifications, messages, mail, one-time codes — becomes tool output that your MCP client sends to its model. Turn on a Focus, open a neutral app before you start, and begin with NORACU_READ_ONLY=1 (phone_status, phone_screenshot, phone_ocr and phone_find stay allowed; all phone input is refused). iPhone notifications also reach the Mac once Mirroring is set up.

First success: phone_status → phone_launch if live is not true → phone_screenshot → phone_ocr → phone_find + phone_tap.

Known limits: a tap goes where OCR saw the text, so repeated or misread labels can be hit wrong (phone_open cannot tell an app from a contact or website that Spotlight's Top Hit shows under the same name); phone_scroll moves lists with trackpad-style scroll events (a vertical phone_swipe drag does not scroll a list — use it for gestures such as Home Screen paging); on one tested setup (iPhone 17 Pro Max on iOS 27.0.1, Mac on macOS 26.6.2) iPhone Mirroring draws the phone about 17% larger than its window at every View size, so the right column and bottom row are cut off and cannot be reached; moving the window between displays, changing the main display and relaunching made no difference, and the iOS 27 / macOS 26 version mismatch is the main suspect — check phone_screenshot if taps near the edges miss; phone_open types the app name into Spotlight and taps it only when Spotlight's Top Hit card shows exactly that name (web suggestions, websites and Settings rows of the same name are never tapped), otherwise returns to the Home Screen — it reads Spotlight's English "Top Hit" label, so on an iPhone set to another language it finds nothing until NORACU_TOPHIT_LABELS names that label; phone_home / phone_app_switcher use the iPhone Mirroring menu (English macOS) and work without python while connected.

Environment variables

Security

noracu runs with the privileges of your MCP client. It clicks, types, pastes, and drives a real browser on this machine — there is no sandbox and no remote approval prompt. Set these on the server entry, then quit the host completely and reopen it:

"env": { "NORACU_READ_ONLY": "1", "NORACU_DISABLE_EVAL": "1" }

• NORACU_READ_ONLY=1 is the safe first-run mode. It refuses every tool not on a 12-name allowlist (the observe tools plus browser_launch/browser_navigate/browser_close); every click, type, paste, and eval tool is denied. browser_navigate is on the allowlist, so it can still load a URL.
• NORACU_DISABLE_EVAL=1 makes browser_evaluate return an error when called (it stays listed in tools/list). Snapshot, click, and type still work.
• file:// is refused unless NORACU_ALLOW_FILE_URL=1, so a browser agent can't read ~/Documents by navigating to it. (The initial navigation URL is scheme-checked; redirects are not — see SECURITY.md.)
• Focus guard: pass app to mac_type / mac_paste / mac_key and input is refused unless that app is frontmost. Omit app and the guard is off.
• Stale-element protection: an element_index is not a coordinate. It expires after 30 s (element_no_longer_valid); a browser click re-checks tag/name/href/position, and a Mac click is refused if the frontmost app changed since the snapshot.
• iPhone screen is model input: phone_screenshot / phone_ocr / phone_find return whatever the phone shows — notifications, messages, one-time codes. Use a Focus and a neutral app, and start with NORACU_READ_ONLY=1 (phone observation allowed, phone input refused). Phone input is also refused while the mirror is paused (mirror_not_live).
• Injection-safe JXA: string parameters are passed to osascript as argv data, never built into the script; the one numeric value (AX read depth) is clamped to 1–10 before use. Regression-tested.

These switches do not make observed content trustworthy. Screenshot text, accessibility names, and page content are untrusted input to your model — do not let them override the task. Full threat model and per-threat status: SECURITY.md.

Tools & docs

• Full tool reference: docs/tools.md (auto-generated from the running server)
• Usage walkthroughs: docs/usage.md
• Architecture: docs/architecture.md

Roadmap

• Browser: tab management (list / select / close)
• Act vocabulary (browser): hover, set_value, wait_for
• Snapshot diffing to cut token use — pending measurement
• iPhone: element_index over OCR regions, re-checking the text before a tap

Maintenance

Solo-maintained, no SLA. macOS 15+ for the Mac layer; the iPhone layer also needs Apple silicon or a T2 Mac, iOS 18+ and iPhone Mirroring (not available in the EU); the browser layer runs wherever Playwright does.

License

MIT — see LICENSE.