Histopathology Annotation MCP

Image, Histopathology Annotation MCP

Image Annotation, including Histopathology slides; From H&E slide to training data in one conversation: nuclei, outlines and slide QC, exported to QuPath, COCO or YOLO. No GPU needed.

[PyPI] [CI] [License: MIT] [MCP Registry] [M8ven Score]

[Loading an H&E breast cancer slide, outlining every nucleus with StarDist, Diagnose: an AI interpretation written as a report, problematic regions outlined on the image and the agent's report, then Ask AI: a question about the results answered with an illustration labelled from the answer]

Accuracy: see Benchmark. Your data: see Your data. Changes: see CHANGELOG.

An MCP server for annotating histopathology and other images in Image Label, a free annotation tool. It lets Claude and other AI assistants upload images, outline every nucleus in an H&E slide automatically (StarDist), trace regions from rough boxes (Segment Anything, or MedSAM for medical images), check slide quality, normalize stain colours, clean up annotations, and export training data as COCO JSON, YOLO labels, YOLO classes or QuPath GeoJSON. A LangGraph agent is included.

Built for: Whole Slide Imaging (WSI) · Digital Pathology · Computational Pathology · Bioinformatics · Tumor Microenvironment (TME) analysis (tumour, immune and stromal cell types, immune-to-tumour ratio) · MLOps Data Labeling (COCO, YOLO and QuPath training data) · Nuclei and Cell Segmentation · Cell Classification · Medical Image Annotation · Stain Normalization and Slide Quality Control · Cancer Research · Computer Vision datasets.

Try it in the browser: https://agent.sachin-kale.com/#annotate: upload a slide and annotate it, detect nuclei, check its quality or get an AI interpretation, with no setup.

Published as label-image-mcp (PyPI) and io.github.sachinbkale27/label-image-mcp (MCP Registry).

Use cases and setup guide: https://labels.sachin-kale.com/mcp

LangGraph agent (label-image-agent)      Claude Desktop / Claude Code / any MCP client              │  langchain-mcp-adapters             │              └──────────────┬──────────────────────┘                             ▼                    label-image-mcp (stdio or HTTP)                             │                             ▼                 Image Label API  (/api/images ...)

Account (optional)

No account needed to try it. Without credentials the server works as a guest: it gets a guest identity on first use, saves it in ~/.config/label-image-mcp/, and keeps using it, so your images are there next time.

The tools see, change and upload only the images of the identity they use (guest or account). Set LABEL_USERNAME and LABEL_PASSWORD to use an account, or LABEL_GUEST=0 to turn guests off.

Install

Claude Desktop: download label-image-mcp.mcpb from the latest release and open it. Leave username and password empty to start as a guest.

Claude Code:

claude mcp add label-image -e LABEL_USERNAME=... -e LABEL_PASSWORD=... -- uvx label-image-mcp

Any MCP client (needs uv):

{  "mcpServers": {    "label-image": {      "command": "uvx",      "args": ["label-image-mcp"],      "env": { "LABEL_USERNAME": "...", "LABEL_PASSWORD": "..." }    }  }}

Benchmark

detect_nuclei checked against expert outlines on NuInsSeg (CC BY 4.0): H&E images of 31 human and mouse tissues, 30,698 hand-segmented nuclei. The detection threshold was chosen on the even-numbered images of each tissue; these results are on the other 332 images (15,701 nuclei), which were not used for any choice. A nucleus counts as found when its outline overlaps the expert's by IoU ≥ 0.5.

• Better on human tissue (F1 0.67) than mouse (0.50 to 0.53). Best on pancreas, cerebellum, placenta and tongue (0.73 to 0.81); weakest on umbilical cord, brain, mouse heart and bone (0.13 to 0.34), where nuclei are sparse, thin or faint. Per-tissue results are in benchmarks/results.
• The model is StarDist's pretrained H&E model, not trained on NuInsSeg: models trained on a dataset score higher on it. Read these as what to expect on new images, and review outlines before training on them.
• Reproduce: benchmarks/nuinsseg.py runs the same method locally.
• Cell types (classify_cells) are not benchmarked here yet: NuInsSeg has no cell-type labels. HoVer-Net's published results on PanNuke are the reference until then.
• Outlining (find_regions, segment_objects, with an AI assistant choosing what to outline) is so far only checked on a small private test set: 5 H&E images with 43 reference findings (glands, cell nests, cysts, skin layers, pigment), of which 41 were found. That guides development; it is not a benchmark, and outline edges are not yet measured against expert outlines.

Your data

What happens to images you upload through this server (hosted on AWS in us-east-1):

• Private to your account. Only you can list, read or change your images; the storage bucket blocks all public access, and viewing links are signed and expire after 24 hours.
• Deleted after 90 days, automatically: files, slides, annotations and analysis results. Deleting an image removes it straight away.
• Encrypted at rest (AES-256) and in transit (HTTPS).
• Not used to train models, and not sent to other AI services by this server: StarDist, Segment Anything and MedSAM run on the service itself. Your AI assistant sees what its tools return (view_image returns the picture), under that assistant's own terms.
• Usage counts. Each call to the service names the AI client (for example claude-ai/0.1.0, from the MCP handshake) and the server version, so usage can be counted by client; the account is logged only as a hash. The service also logs the approximate place of each call (city, region, country and time zone, from its CDN) and the local hour, and counts distinct IP addresses by a keyed hash; the addresses themselves are not kept. Nothing from your conversation is sent.
• No patient identifiers, please. Remove names, record numbers and slide labels (crop the label area or use a de-identified copy) before uploading. Results are for research and education, not diagnosis.

Agent skill

skills/label-image/SKILL.md teaches the agent when and how to use these tools: which annotation types end up in exports, cleaning up labels safely, and merging per-image exports into one COCO or YOLO dataset with consistent class ids (bundled scripts/merge_exports.py). It follows the open SKILL.md format, so it works in Claude Code and other agents that support skills.

# Claude Code: install for all your projects (or copy into a project's .claude/skills/)git clone --depth 1 https://github.com/sachinbkale27/label-image-mcp /tmp/label-image-mcpmkdir -p ~/.claude/skills && cp -r /tmp/label-image-mcp/skills/label-image ~/.claude/skills/

Configuration

Tools

Every tool declares all four hints (read-only, destructive, idempotent, open-world) and describes every parameter. Clients can ask before the destructive ones: replace_annotations, remove_annotations, and detect_nuclei, which replaces its own earlier nuclei. The server also offers three prompts: Analyse a histology slide, Outline objects in an image and Build a training dataset.

Whole slides

After the analysis the slide's image in the library is its overview, with the analysed tiles marked (the three densest in red; with full coverage, a density heatmap and the ten densest tiles), so view_image and the web app show it. The three densest tiles are saved as images of their own, with their nuclei outlined, for a close look or further annotation. Every nucleus is in a QuPath GeoJSON in slide coordinates: open the slide in QuPath, then File > Import objects. Slides are read with tifffile (Aperio SVS including JPEG 2000, Hamamatsu NDPI, pyramidal TIFF and OME-TIFF), decoding only the tiles needed. Other images are up to 20 MB.

There is deliberately no tool for deleting images.

segment_objects calls the API's POST /images/{id}/segment, which runs SAM (facebook/sam-vit-base) or MedSAM (wanglab/medsam-vit-base) in a Lambda container (backend/segment). After a quiet spell the model takes a while to start; the client retries then. Results are for research and dataset building, not diagnosis.

How it outlines

Vision models read coordinates poorly: asked for boxes around glands, they often land on the stroma beside them. So the server does the precise part and leaves the model the choosing:

• find_regions runs Segment Anything with no prompts: a grid of points (coarse to fine, skipping flat background) and overlapping boxes of four sizes, which find large structures such as cell nests and lobules. Duplicates, unstable masks, cut-off pieces and the dark surround of a microscope's field are dropped, and a lumen is grown into its whole gland when a larger box outlines it cleanly. The regions come back numbered with exact boxes and a picture, so the model picks numbers instead of estimating coordinates. It runs as a background job (the API answers at once and the client polls), so it isn't cut short by the API's 30-second limit.
• For a box, Segment Anything offers a part, the object and something larger; the server keeps the confident outline that best fills the box (less what spills outside), so a box around a gland gives the gland, not its lumen. With a point as well, the point settles which thing in the box is meant.

The histopathology tools call POST /images/{id}/pathology, which runs StarDist (BSD-3; its pretrained 2D_versatile_he model was trained on public H&E datasets, some under non-commercial terms, so check them before commercial use), torchstain (MIT) and OpenCV quality measures in a second Lambda container (backend/pathology). Microscope photos come at many magnifications, so detect_nuclei also tries the image enlarged 2x and 3x and keeps the scale that finds the most nuclei.

HTTP mode

label-image-mcp --http [--host 127.0.0.1] [--port 8765] serves Streamable HTTP at http://HOST:PORT/mcp, so several clients can share one running server:

claude mcp add --transport http label-image http://127.0.0.1:8765/mcp

Hosted mode

label-image-mcp --http --hosted --host 0.0.0.0 is for running the server remotely, for example on AWS Lambda behind the Lambda Web Adapter. It keeps no session state, answers with JSON, and requires Authorization: Bearer <token> on every request; with no token configured it refuses all requests.

LangGraph agent

A terminal chat agent that uses these tools. The graph is START → agent ⇄ tools, with conversation memory kept for the session.

uv tool install "label-image-mcp[agent]"export ANTHROPIC_API_KEY=... LABEL_USERNAME=... LABEL_PASSWORD=...label-image-agent

Development

uv sync --extra agentuv run pytestnpx @modelcontextprotocol/inspector uv run label-image-mcp

In the Image Label app repo, ./start.sh runs this server in HTTP mode next to the web app. It reads credentials from mcp-server/.env. Releases are described in PUBLISHING.md.

License

MIT