droidsight

droidsight

[CI] [Security audit] [License: MIT] [crates.io] [npm]

An MCP server that drives a real Android device over ADB — and hands the agent back the screen its action produced.

One native binary. The server has no Python, Appium, scrcpy, or ffmpeg dependency and runs no Node at all — npx below is only a convenient installer — so the runtime is the binary plus adb. A background H.264 screenrecord stream is decoded in process, so the screen that follows an action is attached to the tool result automatically instead of costing a second round trip to ask what happened.

Every image carries the coordinate space it was produced in, so a model can tap what it just looked at without guessing a scale factor. When the accessibility tree comes back empty — Flutter, React Native, canvas, games — OCR and template matching still find the target.

This is a powerful local controller, not a sandbox. Run it only against a device and an MCP client you trust.

Install

npx -y @edgecasehuman/droidsight

That downloads a prebuilt binary for your platform and needs no build tools. The server is also listed in the official MCP Registry as io.github.edgecasehuman/droidsight.

To compile it instead, from crates.io:

cargo install droidsight

Or from source:

cargo build --locked --release --bin droidsight

Both compiling routes need NASM on the PATH. The H.264 decoder is built from C++ by openh264-sys2, which assembles its hot paths with NASM, and no platform ships it by default -- without it the build fails inside a transitive dependency rather than in this crate. Install it with apt install nasm, brew install nasm, or winget install NASM.NASM. The npx route above needs none of this.

The server is target/release/droidsight. It speaks newline-delimited JSON-RPC 2.0 over stdin and stdout.

Client configuration

{  "mcpServers": {    "android": {      "command": "npx",      "args": ["-y", "@edgecasehuman/droidsight"],      "env": {        "DROIDSIGHT_DEVICE_SERIAL": "<serial from `adb devices`>"      }    }  }}

ADB must be reachable through PATH, ANDROID_SDK_ROOT, ANDROID_HOME, or an explicit DROIDSIGHT_ADB_PATH. If more than one authorized device is visible the server refuses to guess; set DROIDSIGHT_DEVICE_SERIAL.

Tools

Thirty-one tools are published by default. Two more appear only when DROIDSIGHT_ALLOW_SHELL=1 is set, and are listed last.

Seeing the screen

Driving the device

Applications

Device and system state

Diagnostics and data

Capture and presence

Only with DROIDSIGHT_ALLOW_SHELL=1

Coordinate space

Screenshots are downscaled to 720 pixels wide by default, so a coordinate read off the returned pixels is usually not a device coordinate. Every tool that returns an image also returns a metadata.image object describing how to convert one:

{  "image": {    "width": 720, "height": 1600,    "device_width": 1080, "device_height": 2400,    "origin_x": 0, "origin_y": 0,    "scale": 1.5,    "coordinate_space": "image",    "note": "To convert a coordinate (x, y) read from this image into a device coordinate: device_x = 0 + x * 1.5, device_y = 0 + y * 1.5."  }}

When coordinate_space is device, the image is already 1:1 and coordinates can be used directly. Request max_width: 1440 to reduce or remove downscaling. Crops additionally report a non-zero origin_x/origin_y that must be added after scaling.

Taps, uiautomator hierarchy bounds, OCR boxes, and template matches are all in device coordinates. The background stream runs screenrecord without --size, so decoded frames are native resolution and share that one space.

You can avoid the arithmetic entirely. The mcp_android_vision_query tool's elements action returns an indexed snapshot with a precomputed center per element, and its tap_element action takes that snapshot_id and an index.

Environment variables

Continuous vision cache

One background H.264 screenrecord stream starts at process startup and the latest decoded RGB frame is kept in memory. Screenshot tools and input observations read that cache, so a snapshot is available immediately once the first frame decodes. A static screen's last frame stays valid while the stream runs; it does not expire merely because the encoder stopped emitting unchanged pixels.

The stream allows 20 seconds for the first decodable frame before reconnecting. After that, silence is treated as a static screen, while EOF and read errors still reconnect. Starting or stopping a stream synchronously clears its frame cache so a previous session's image cannot be reused. The mcp_android_vision_stream stop action suspends capture until an explicit start; shutdown drops the owned ADB stream and leaves no screenrecord process behind.

Continuous capture has privacy, battery, CPU, and wireless-bandwidth costs. The device screen may contain sensitive information even though cached frames stay in process memory until a tool returns one. Use the explicit stop action when capture should be suspended.

MCP transport

Stdout carries one JSON-RPC value per line; diagnostics go to stderr. The server supports initialize, tools/list, and tools/call, plus the initialized notification and legacy mcp.* aliases. Batch arrays are handled member by member and produce one response-array line; notification-only batches produce no output and empty batches are rejected. Clients must complete the ordered initialize request and notifications/initialized notification handshake before listing or calling tools; premature calls return JSON-RPC error -32002. Input frames are limited to 16 MiB; oversized or invalid UTF-8 frames are rejected without preventing the next valid request from being processed.

Published text from logs, events, notifications, device files, hierarchy and OCR results, crash and forensic reports, instrumentation, shell/macro output, and aggregate flows is limited to 256 KiB. Truncated responses carry metadata.truncation with the strategy, original size, returned size, and configured limit; chronological feeds retain their newest tail while file-like output retains its head.

Finite ADB subprocesses drain stdout and stderr concurrently and retain at most 4 MiB from each stream while counting all observed bytes, which prevents pipe deadlocks and unbounded capture. The optional event monitor owns its ADB child and reader thread and is cancelled, killed, joined, and reaped during transport shutdown rather than being left detached.

OCR is an optional integration with an external Tesseract executable. Discovery probes are limited to five seconds and recognition to 30 seconds; timed-out children are terminated and reaped before the tool returns an error.

Smoke test:

printf '%s\n' \  '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{}}' \  '{"jsonrpc":"2.0","method":"notifications/initialized"}' \  '{"jsonrpc":"2.0","id":2,"method":"tools/list","params":{}}' \  | cargo run --quiet --bin droidsight

Destructive operations

Some tools change device state irreversibly. pm clear deletes an application's databases, preferences, accounts, and credentials, and it is reachable two ways. Both refuse to run without "confirm_destructive": true:

• mcp_android_forensics_control with "action": "clear_app_data", named for what it does rather than for a cache eviction.
• mcp_android_app_manage with "action": "clear_data". The same tool gates uninstall, permission, enable, and disable behind the same argument, because each of them removes an application, its data, or a security control.

The guard is checked before any ADB command is issued, so a refused call does not touch the device. It is a guardrail against an unintended call, not a security boundary: a client that can invoke tools can also set the flag.

The gate covers unintended calls to those specific tools. It is not a containment boundary, and several operations of comparable impact are not behind it at all:

• mcp_android_system_control enables an accessibility service, which grants full screen-read and input-injection access across every app on the device, and grants the draw-over-other-apps permission.
• mcp_android_network_control sets a global HTTP proxy, forgets saved Wi-Fi networks, and places real outbound phone calls.
• mcp_android_sensor_control overrides sensor readings such as battery level and status; its GPS location mocking uses the emulator console and applies only to emulators, not physical devices.

Arbitrary shell access

DROIDSIGHT_ALLOW_SHELL=1 publishes two tools that pass command strings to the device shell verbatim: mcp_android_run_shell, and mcp_android_run_macro for batches of up to 100 commands.

No command filtering of any kind is performed. The environment variable is the entire control. Once set, these tools are a strict superset of every gated operation above — pm clear, pm uninstall, and file deletion are all reachable without confirm_destructive, because the shell path never consults it. Leave the variable unset unless the connected MCP client is as trusted as a local shell on the device.

Background enforcement

DROIDSIGHT_SENTINEL=1 starts a loop that wakes every five seconds and, for each watched package, re-applies the state it was told to hold: waking and unlocking the screen with a PIN supplied when the watch was registered, enabling an accessibility service, granting the overlay permission, and granting a list of runtime permissions. No watches exist until a client registers one, so an enabled sentinel with an empty watch list only ticks.

Two consequences are worth stating plainly:

• It reverses manual changes. Revoking a permission or disabling the accessibility service for a watched package is undone within five seconds. Remove the watch first.
• It does not ask for confirm_destructive. That gate covers an operator calling mcp_android_app_manage; the loop invokes the same underlying operation directly, because re-granting is the entire purpose of a watch. Registering a watch is the consent step for everything that watch will subsequently do on its own.

The loop performs no network I/O beyond the local ADB connection and contacts no remote service.

Build and test

The repository pins its Rust toolchain and keeps device-mutating scenarios out of the default test suite.

cargo fmt --all -- --checkcargo clippy --locked --all-targets -- -D warningscargo test --locked -- --test-threads=1cargo build --locked --release --bin droidsight

These gates run on Linux, macOS, and Windows and pass without warning suppressions. The compiler is pinned in rust-toolchain.toml; a separate CI job type-checks the crate against Rust 1.89, the declared minimum.

See CONTRIBUTING.md before changing device-affecting code. SECURITY.md documents private reporting, sensitive-evidence redaction, and the ADB, host, MCP, filesystem, logging, and shell trust boundaries. Participation is governed by CODE_OF_CONDUCT.md.

Maturity

The host-only suite covers protocol parsing, schemas, serialization, ADB command construction, concurrent discovery, reconnect invalidation, device selection, subprocess deadlines, JSON-RPC batches, path confinement, shell quoting, output budgets, event-monitor teardown, hierarchy parsing, flow validation, stream framing and lifecycle, static-frame cache behavior, image encoding and coordinate metadata, Samsung keyguard/window parsing, and intent failure handling.

The release binary has been exercised over wireless debugging against a single Samsung device running Android 13, covering device state, battery, apps, hierarchy and element snapshots, OCR, screenshots and sequences, continuous H.264 capture, input and navigation, intents, logs, notifications, Wi-Fi scanning, file round trips, hashing, sessions, gesture capture, recording, rotation restoration, crash listing, and cleanup. Destructive application, permission, accessibility, overlay, telephony, network-mutation, GPS, and battery-mocking operations are deliberately excluded from that matrix. Other vendors and Android releases still need their own qualification.

H.264 decoding

The vision cache decodes H.264 in process using openh264, which builds Cisco's BSD-2-Clause C++ implementation from source. Cisco's royalty coverage applies to the binaries Cisco itself publishes and does not transfer to a self-compiled build. If you redistribute binaries of this project, satisfying any applicable H.264 patent licensing is your responsibility.

License

MIT. See LICENSE.