Isaac Sim robot workbench

dex-isaac-mcp

[An agent driving the Allegro hand through dex-isaac-mcp]

Recorded headless with the server's own tools (sim_frame_robot, sim_set_backdrop, sim_record_start), driving the Allegro hand example. Each caption is the request and the tool call it became.

An MCP server that lets an AI agent (Claude Code, or any MCP client) drive a live, persistent Isaac Sim session and launch Isaac Lab training runs.

Kit takes tens of seconds to boot. If every experiment is a fresh launch, most of your time goes to waiting. Here Kit starts once inside a daemon and stays up. Each tool call lands in that running session, so changing a gain, stepping physics or taking a screenshot costs a frame, not a relaunch.

 MCP client ──stdio──▶ dex_isaac_mcp (host, plain python)                          │  newline-delimited JSON over a Unix socket                          ▼                       scripts/simd.py (Isaac Lab container, Kit stays up)                          └─ one articulation, described by a robot JSON

• Any articulated robot. Point it at a USD and a small JSON config. Franka and Allegro examples are included.
• Normalized joint control. Targets are 0..1, where 0 is a joint's lower limit and 1 its upper limit, so agents don't need to know radians or meters.
• Named poses per robot (home, fist, …), which can be blended part-way.
• Headless capture and GIF recording from an auto-framed camera, with captions. The clip above was made this way.
• Props: spawn a table, a ball or a USD into the running scene and read back where they settle.
• Measurements: joint state, per-joint travel and tracking error, a range test that finds blocked joints, and parameter sweeps inside one session.
• Training control: each run is a detached docker compose run. You can poll its status, TensorBoard scalars, checkpoints and logs.

How it differs from other Isaac Sim MCP servers

NVIDIA's official Isaac Sim MCP is a documentation search for coding assistants, and works well alongside this one. Servers like isaacsim-mcp-server and omni-mcp/isaac-sim-mcp run inside the Isaac Sim GUI and cover scene building broadly. This one runs headless, as a daemon in Docker, and focuses on tuning, measuring and training one robot.

Pick an in-GUI server to build a scene by talking to it, interactively, with a wide robot and asset library.

Pick this one to have an agent tune, debug and train a specific robot (your own, from a USD), unattended or on a remote box, with numbers it can act on instead of only screenshots.

Requirements

• Linux with an NVIDIA GPU that Isaac Sim supports, plus the NVIDIA Container Toolkit
• Docker with the Compose plugin
• An NGC login to pull the Isaac Lab image: docker login nvcr.io (user $oauthtoken, password: your NGC API key)
• Python ≥ 3.10 on the host, for the MCP server only

Quick start

git clone <this repo> dex-isaac-mcp && cd dex-isaac-mcp
# 1. Build the image (Isaac Lab 2.3.2 base, pinned by digest)cd docker && docker compose build && cd ..
# 2. Install the host-side server, editable so it finds this clonepip install -e .            # add [metrics] for train_metrics: pip install -e '.[metrics]'
# 3. Register it with Claude Codeclaude mcp add isaac -- python -m dex_isaac_mcp
# 4. Allow the container to open windows (once per login; needed for screenshots)xhost +local:docker

Then ask the agent something like "start the sim with the Franka, move it to ready and show me a screenshot." It will call sim_up, sim_set_pose, sim_step and sim_screenshot.

From PyPI instead: pip install dex-isaac-mcp. The daemon still runs from a clone (it needs docker/ and scripts/simd.py), so point the server at it:

claude mcp add isaac -e ISAAC_MCP_HOME=/abs/path/dex-isaac-mcp -- dex-isaac-mcp

Other MCP clients can launch python -m dex_isaac_mcp (or the dex-isaac-mcp script) over stdio.

The first sim_up takes several minutes: Kit builds its shader cache and downloads Nucleus assets. Later starts are much faster.

Running the daemon by hand

sim_up deletes its container (--rm) when the daemon exits, so a crash on startup takes its traceback with it. To see the error, run the daemon in the foreground:

cd dockerdocker compose run --rm simd scripts/simd.py --gui --robot examples/robots/franka.jsondocker compose run --rm simd scripts/simd.py --headless --usd /path/in/container/robot.usddocker compose run --rm simd scripts/simd.py --sliders --robot examples/robots/allegro_hand.json

--sliders opens an omni.ui panel with one slider per driven joint. The socket stays live alongside it.

Tools

Session

Inspect

Drive

Scene

Capture and record

These work headless, with no GUI or viewport. They use a dedicated camera that is independent of the GUI view.

Tune

Train

Training runs do not depend on the daemon or on the MCP session. They keep going after the client disconnects.

Robot config

A robot is one JSON file. Only usd is required. Unknown keys are rejected, so a typo fails loudly instead of quietly falling back to a default.

{  "name": "franka",  "usd": "{ISAACLAB_NUCLEUS_DIR}/Robots/FrankaEmika/panda_instanceable.usd",  "fix_root_link": true,  "spawn_pos": [0, 0, 0],  "init_joint_pos": {"panda_joint4": -2.81, "panda_joint6": 3.04, ".*": 0.0},  "driven_joints": ["panda_joint[1-7]", "panda_finger_joint.*"],  "passive_joints": [],  "couplings": [{"leader": "joint_a", "follower": "joint_b", "ratio": 1.0}],  "actuator": {"stiffness": 400, "damping": 40, "effort": 87, "velocity": null},  "solver": {"pos_iters": 32, "vel_iters": 4, "self_collisions": true},  "camera": {"eye": [1.8, 1.8, 1.4], "target": [0, 0, 0.4]},  "poses": {"ready": {"panda_joint[1357]": 0.5, "panda_finger_joint.*": 1.0}}}

Using your own robot without forking

Keep the robot config and assets in your own repo, and add them to the container with a compose override that you list in COMPOSE_FILE. Set it in the MCP server's environment, using absolute paths:

# my-robot/mcp-compose.yamlservices:  simd:    volumes:      - /abs/path/my-robot:/workspace/my-robot

claude mcp add isaac \  -e COMPOSE_FILE=/abs/path/dex-isaac-mcp/docker/docker-compose.yaml:/abs/path/my-robot/mcp-compose.yaml \  -e ISAAC_MCP_ROBOT=/workspace/my-robot/robot.json \  -- python -m dex_isaac_mcp

Training defaults

Out of the box, train_start runs Isaac Lab's stock skrl script inside the isaac-lab service. It tags the run name onto the log directory (logs/skrl/<experiment>/<timestamp>_ppo_torch_<run_name>/), which the other train_* tools use to find the run. To use your own launcher, set these in the environment the MCP server starts in:

The script must accept --task, --headless and, when given, --num_envs, --seed, --max_iterations and --checkpoint. Tasks from your own extension need to be importable inside the container, either installed into the image or mounted.

Design notes

These are the constraints the code is built around. Most were learned by breaking them.

• Every Kit call happens on the main thread. Kit, PhysX and USD are not thread-safe. Socket threads only parse JSON and queue requests, and the main loop executes them between physics steps. Answering from a reader thread appears to work, then corrupts the stage under load.
• Spawn properties are frozen. Replacing a spawned articulation needs SimulationContext.stop(), which blocks on a timeline event that only advances while the Kit loop pumps. A command runs on that loop, so the call never returns. omni.usd new_stage() has the same trap. So the USD, solver iterations and self-collision need a restart (sim_reload), and gains stay live.
• A Unix socket, not TCP. The repo is bind-mounted and the container runs as the host uid, so the host sees the socket file directly, with no port mapping. Paths are capped at 107 bytes (AF_UNIX). If your checkout is deep, set ISAAC_MCP_SOCKET.
• The host side imports no Isaac code. protocol.py, robot.py and training.py are stdlib-only. The MCP server adds only mcp. Nothing on the host needs isaaclab, torch or a GPU.
• Cache directories are committed with .gitkeep. If Docker auto-creates a bind-mount source, it is root-owned, and Kit then dies with registry cache path is not set before any script runs.
• The base image is pinned by digest. A re-pulled tag once shipped /isaac-sim as mode 750, and every non-root container lost its Python.
• Recorded GIFs are stabilized. The renderer's denoiser shimmers: between two frames of a motionless scene, about 9% of background pixels change slightly, and a GIF re-encodes every one of them. Holding sub-threshold changes and using one shared palette took a 9-second clip from 15 MB to 1.2 MB.
• The daemon always renders, even headless (enable_cameras). Without rendering, PhysX never registers a prop spawned at runtime. Prop poses are read from fabric, because the USD transform and the PhysX CPU query both stay at the spawn pose, and creating a PhysX tensor view mid-simulation crashes CUDA.
• No floor for range tests (ground=False) on anything whose links can reach the ground. Otherwise the test measures the floor, not the robot.

Development

python -m unittest discover tests   # host-side tests: no Isaac, no GPUruff check .

dex_isaac_mcp/protocol.Client is a handy debugging client:

from dex_isaac_mcp.protocol import Clientwith Client() as c:    print(c.call("status"))    c.call("set_pose", name="ready"); c.call("step", n=240)

Status

Tested against Isaac Lab 2.3.2 (Isaac Sim 5.x) and mcp 2.3, over the stdio protocol, with the GUI on:

• Franka and Allegro examples, plus a custom closed-linkage hand through a compose override: sim_up, poses, screenshots, sim_range_test, sim_sweep (restores the original value), sim_down.

• Headless daemon: gains, frozen-parameter rejection, wave.

• Headless capture and recording, Allegro and Franka: auto-framing, backdrop, captions, GIF output (the clip at the top).

• Props, GUI and headless: a sphere and a cylinder dropped onto a static table settle at exactly table height plus their radius and half-height.

• Training, against Isaac Lab's stock skrl script: Isaac-Cartpole-v0 launched, polled, logged, checkpointed and read back through every train_* tool, plus a run stopped mid-training. skrl's write_interval: auto writes no TensorBoard scalars on a very short run (5 iterations), so train_metrics comes back empty there; 50 iterations gives 18 tags.

Issues and PRs are welcome.

License

MIT, see LICENSE.