SimMirror

SimMirror

Mirror and drive iOS Simulators, real iPhones and Android devices from Claude Code, CLI agents and the browser.

[CI] [License: Apache-2.0] [Release]

[A Claude Code session driving a calendar app on an iPhone 17 simulator in the viewer: it opens an event, scrolls through its notes, then fills in and saves a new one, the agent's cursor gliding to each spot before it taps]

A real session, sped up four times. The whole recording runs 1m 39s.

A live iOS Simulator -- or your own iPhone, over its cable, or an Android emulator or phone -- in any browser tab or web page, an animated cursor that shows exactly what your AI agent is about to tap, and token-efficient UI snapshots so agents read the screen as compact text instead of screenshots. Record any screen as an MP4 or a GIF for a pull request. Works with Claude Code, Codex, Cursor and any other MCP client.

Status: 3.0. The agent tools, the viewer, the protocol and the embedding API are stable, and a check holds every release to what SimMirror promises not to break. 2.0 brought real iPhones and iPads beside the simulators -- live over the cable, driven through WebDriverAgent -- and records any screen as an MP4 or a GIF. 2.1 reads and touches apps turned to landscape, keeps taps off what a tab bar covers, builds the worktree the agent works in, and adds observe mode for walkthroughs on accounts an agent must not change. 2.2 adds Android emulators and phones, with the same viewer, cursor and tools. 2.3 attaches a real device again by itself once a pulled cable is back, tells an agent why it cannot touch one, and checks one before a run with sim_device health. 2.4 leaves a phone a person took back alone, records as long as it ran, reads long screens without locking WebDriverAgent up, waits for exact words, and lets one agent at a time drive a real device. 2.5 adds concise agent workflow skills and reliable tool outcomes. 2.6 records at 30, 60 or 120 frames a second, at a real device's own width, says when a recording stutters, crops one to a region, keeps refs through a selection change, and guards switches in observe mode. 3.0 adds macOS apps -- one app and window at a time, captured and driven from inside the app, never the person's pointer or keyboard -- and a CEF host adapter that lets a Chromium Embedded app be read and driven fully, in the background. It removes nothing 1.x or 2.x promised.

Why SimMirror

• See the simulator anywhere. A live H.264 or JPEG stream in a Chrome tab, an iframe or a <sim-mirror> web component, with touch, scrolling, typing, hardware buttons and light and dark mode.
• Watch what the agent does. Every agent gesture is announced before it lands, and the viewer's cursor glides to the spot first -- so you can follow along, and step in: an agent waits for your hand to be still.
• Spend fewer tokens. Agents read accessibility snapshots with stable element refs and diffs between screens, act in batches with waits, and reach for a screenshot only when a question is visual.
• Read screens accessibility cannot. A game or a canvas reads from its pixels, and an app under development can share its own views, so icon-only buttons and hand-drawn controls read by name.
• Test on a real iPhone. Plug it in and it sits beside the simulators: live over the cable at up to 60 fps, touched and typed on from the browser or by an agent, and given back everything SimMirror changed on it.
• Check connection recovery. Open Control Center on a real iPhone to manage Wi-Fi and Bluetooth connections or toggle airplane mode, then return to your app to check how it reconnects to a BLE or Wi-Fi accessory. See the connectivity workflow.
• Show what happened. Record a flow as an MP4 or a GIF with every touch drawn in -- at 60 or 120 frames a second for a short animation, or only the part of the screen that matters -- and put it in a pull request.

Install

Claude Code, with the plugin (tools, a skill that teaches snapshot-first use, and /sim-mirror:open):

/plugin marketplace add AndrewKochulab/sim-mirror/plugin install sim-mirror@sim-mirror

Any MCP client, as a stdio server:

claude mcp add sim-mirror -- uvx --from sim-mirror==3.0.1 sim-mirror mcp

Configurations for Codex, Cursor and other clients. The plugin includes four workflow skills; the release also packages them for Codex and Cursor.

The command, for sim-mirror open, doctor and the rest:

uv tool install sim-mirror==3.0.1# orbrew install andrewkochulab/tap/sim-mirror

The viewer library, for your own pages: npm install @andrewkochulab/sim-mirror (npm). SimMirror itself is on PyPI and in a Homebrew tap.

Requirements: an Apple Silicon Mac, Xcode 26 or later, and uv or Homebrew. Nothing else: SimMirror drives a simulator with its own helper, which comes in the package. A real iPhone needs Developer Mode, and an Apple development team to be touched. Details in Installation.

Quickstart

sim-mirror doctor     # check this Mac, ending with a real tap on a simulatorsim-mirror open       # this project's simulator, live in a browser tab

Then ask your agent:

Open Settings → General → About and tell me which iOS version this simulator runs. Use sim_snapshot rather than screenshots.

With an iPhone plugged in, unlocked and in Developer Mode:

Switch to my iPhone, build and run the app on it, walk through sign-up and record it as a GIF.

The first time, press Set up touch in the viewer: it builds WebDriverAgent with your project's own signing team.

Getting started explains what happens along the way.

Features

Live viewer. H.264 through WebCodecs where the page can decode it, JPEG otherwise, negotiated per connection. Touch, drag, scroll, typing and paste, Home and Lock, light and dark, and a device picker. It reconnects by itself after a blip and says plainly when the simulator is off.

Agent cursor. Every agent gesture -- tap, long press, swipe, drag, typing -- reaches the viewer before it lands, and the viewer's own pointer glides there first (agent.cursor_lead_ms). Never the Mac's pointer.

Screen understanding. A snapshot is a few lines -- e2 button "General" (201,319) -- with refs that stay with their elements across scrolls, a digest, and diffs. sim_act plays up to 20 steps in one call and waits for text, its absence, or the screen to settle -- past a spinner or a pulsing dot that never stops. An app turned to landscape is read and touched in its own points, sim_device rotate turns a simulator, and a tap on what a tab bar, a toolbar or the keyboard covers is refused with how to reach it. See Screen understanding.

Observe mode. For a walkthrough on an account the agent must not change, agent.observe refuses a tap on what reads like Delete, Save, Send or Sign out -- in English and a dozen other languages, or styled destructive by its app -- unless the call confirms it. See Security.

Reading pixels. A screen whose accessibility says nothing -- a game, a canvas, an app still loading -- reads as the text macOS's Vision finds in it, with refs to tap and text to wait for, and the viewer can outline what was read (perception.ocr). See Read from pixels.

App SDK. An app under development can link SimMirrorKit, a small Swift package, and call SimMirror.start(): in a Debug build on the simulator it shares its own UIKit and SwiftUI views, so an icon-only button, a card with a tap gesture or a hand-drawn control reads by name, and .simMirror("Settings") names a SwiftUI view. Optional, and nothing of it is in a Release build. See The app SDK.

Real devices. A cabled iPhone or iPad shows up beside the simulators, and an agent can switch to it (sim_device list and choose): its live screen over the cable; its apps, logs, look and location through Xcode's devicectl; and touching, typing and its element tree through WebDriverAgent, set up once with a click and signed by your project's own team -- a tap lands in about a third of a second. It is reached only over the cable, and what SimMirror changes on it is put back when it is let go. See Real devices.

Android. Android emulators and phones sit beside the simulators, with the same viewer, cursor and tools: adb lists, starts and drives them, scrcpy's server -- set up once with a click -- shows the screen live, UI Automator reads it for snapshots, and sim_build_run and sim_test build a Gradle project with its own wrapper. A phone is never started or shut down, and what SimMirror changes on it is put back. See Android.

Recording. The viewer's Record button, sim_record or sim-mirror record keep a device's screen as an MP4, a GIF or both, with each touch drawn where it landed and a 9:41 status bar, and preset: "pr" keeps one small enough for a pull request. See Recording.

Connectors. SimMirror's own native helper gives full control with nothing to install; idb_companion still can, and the simctl connector mirrors the screen view-only when neither is there, and says why. auto uses the fastest that works and falls back when one fails. More connectors plug in through an entry point. See Connectors.

Embedding. An iframe with a one-shot ticket, the <sim-mirror> element themed with CSS custom properties, or SimMirror inside a Python application with its own sign-in.

Settings panel. sim-mirror open --settings puts every setting a click away in the viewer, a tab per section, saved for one project or all of them and applied before it says so. Settings that decide what runs or who may reach the daemon wait for a code from sim-mirror settings confirm. See The settings panel.

Doctor. Checks Xcode, its Simulator frameworks, runtimes, the native helper and idb_companion, Device Hub and the desktop session, the text reader and an app's shared hierarchy, then proves a tap reaches a simulator. See the doctor.

Security. Loopback only, a Host allowlist, exact Origin checks, CORS and framing only for origins you list, hashed scoped tokens, and one-shot codes in URL fragments. See Security.

Build and test. sim_build_run and sim_test build with xcodebuild for the agent's simulator, install and launch, and answer with only what failed and where -- each failing test named so it can be run again alone, and on another simulator when asked -- and with the folder, branch and commit they built. They build the git worktree the agent works in, with the configuration its scheme runs. Off by default, since they run commands: sim-mirror config set build.tools true.

Measure the token cost

benchmarks/tool_budget.py calls each read-only tool through the same MCP relay a client uses and prints each answer's bytes, estimated tokens and latency on your own screens; see the token budget example. Measured with SimMirror 2.4.0 on an iOS simulator using the native connector, showing Settings → General: 30 calls per tool after a separate initial round and 3 warmup rounds (2026-10-29). Host and device details are omitted from the public report for privacy.

On a physical iOS device over USB with WebDriverAgent, the same 30-round run measured 199 estimated tokens for a full snapshot (p50 859.1 ms), 5 for an unchanged-screen diff (817.9 ms), and 469 for a 400-pixel screenshot (31.4 ms).

Bytes are the whole answer, an image's base64 included; tokens are estimates (text at about 4 characters a token, an image at about 750 pixels a token), not billed usage or whole-task costs. All measured calls succeeded. See the performance report for real-iPhone results, initial-call timings, raw samples and the older baseline. Different connectors and environments make these separate observations, not a before/after speedup claim.

Compatibility

Rows checked on real hardware are marked verified, with the date, in Compatibility.

From 1.0 the tools, the protocol, the embedding API, the viewer package, settings and the command line are held to what SimMirror promises not to break: a release adds, and never breaks; 2.0 broke nothing. Connectors are the exception, and pin a minor version.

Documentation

Everything: getting started, clients, embedding, screen understanding, connectors, security, architecture, and the reference for tools, configuration, the command line and the protocol.

Roadmap

See ROADMAP.md and the milestones. How SimMirror relates to other tools is in Comparison.

Contributing

Contributions are welcome. Read CONTRIBUTING.md and the Code of Conduct; AI coding agents read AGENTS.md. Report security issues privately as described in SECURITY.md. Questions go to Discussions.

License

Apache-2.0. See NOTICE and THIRD_PARTY_LICENSES.md.

SimMirror is an independent project, not affiliated with Apple or Anthropic. iOS, iPhone, Xcode and Simulator are trademarks of Apple Inc.; Claude and Claude Code are trademarks of Anthropic PBC.