webEmbedding

io.github.jongko54由 GitHub 账号 jongko54 发布v0.3.13更新于 Nov 8, 2026

Clone any web page from a URL into a verified offline HTML copy, embed, or rebuild.

已验证Streamable HTTP可网页运行网页搜索与抓取浏览器自动化开发者工具

概览

AI 生成的概览

把网页 URL 变成嵌入代码、离线 HTML 快照或有界重建,并提供只读的 URL 检查工具与本地抓取引擎。

可以做什么
  • 检查 URL 的标题、元数据、框架策略与复用路线,并报告运行时能力
  • 发现页面上可能的嵌入、预览、查看器、remix 与源码链接
  • 判断参考页面应嵌入、取源、抓取、重建还是阻止,并为可嵌入的 URL 生成 iframe 片段
  • 规划以来源为先的复现路径,区分精确复用与抓取、重建工作
  • 本地运行时:用 Playwright 抓取页面、回放网络证据、重建被阻止的页面,并对照桌面、平板和手机截图自检
适用场景
适合在用户给出 URL 并要求克隆、复制、重现或嵌入页面,或需要判断页面能否被 iframe 嵌入时使用。最适合营销页、文档页和落地页等静态或半静态公开页面。
运行要求
远程接入端点只需网络访问。本地 stdio 包(npx web-embedding mcp)需要 Node.js 18 或更高版本、Python 3.9 或更高版本,以及本机已安装的 Chrome 或 Chromium 供 Playwright 抓取使用;它不会自行下载浏览器。需要登录态的抓取须由调用方提供 storage_state_path 或 user_data_dir。
安装前请注意
本地运行会把抓取产物(包括会话存储状态)写入调用方指定的输出路径。带登录态的抓取需要你自行提供浏览器会话数据。该服务不绕过登录、付费墙或验证码,复现页面仍需具备相应权利。遥测默认关闭,启用后发送到你自行配置的端点。

安装

在 SourceWeft 中

  1. 打开 控制台中的 webEmbedding,将其添加到工作区。
  2. 为需要使用其工具的对话启用该服务。

Web executable,通过 Streamable HTTP。 远程服务在工作区中配置后即可从网页运行时运行。

交给智能体安装

把这段提示词粘贴到 Claude Code、Codex、Cursor 等智能体,或 SourceWeft 对话中。智能体会读取该服务器的安装指南,先向你展示发布者、端点和登录方式,经你同意后再把它添加到你的客户端。密钥不会写进任何命令。

阅读 https://sourceweft.com/mcp/io.github.jongko54%2Fweb-embedding/install.md,并按其中说明把这个 MCP 服务器接入我的客户端。修改任何配置之前,先告诉我它的发布者、端点和登录方式,等我确认。不要把 API 密钥写进命令或我会提交的文件;问我密钥应从哪里获取。

查看智能体所依据的安装指南

其他 MCP 客户端

把它添加到你客户端的 mcpServers 配置中。

{
  "mcpServers": {
    "webembedding": {
      "type": "http",
      "url": "https://webembedding-jongkos-mcp.vercel.app/mcp"
    }
  }
}

README

webEmbedding

webEmbedding clones any web page from its URL for AI coding agents. Give Claude Code, Codex, Cursor, or any MCP client a URL and it returns a verified clone: the original embed when the page allows it, otherwise a self-contained offline HTML snapshot (inlined CSS, shadow DOM, localized fonts and images) or a bounded rebuild, scored against desktop, tablet, and mobile screenshots of the live page.

Add it to your agent in one line:

bash
claude mcp add web-embedding -- npx -y web-embedding@latest mcp   # Claude Codecodex mcp add web-embedding -- npx -y web-embedding@latest mcp    # Codex CLI

Then ask: "Clone https://example.com into ./clone and tell me the fidelity score." (Machine-readable summary for LLMs: llms.txt.)

webEmbedding is a source-first website cloning engine: it captures live pages with Playwright, replays network evidence from HAR artifacts, rebuilds only when direct reuse is blocked, and self-verifies the result.

It ships as a Skill + MCP server. Instead of asking a model to "clone this site" from a screenshot, it inspects the URL, chooses a reuse or rebuild route, captures DOM/runtime HTML/styles/assets/network traces, generates bounded frontend reconstruction artifacts, and checks the output with visual, DOM, computed-style, interaction, and responsive-breakpoint verification.

[webEmbedding Skill and MCP workflow]

GitHub listing, social preview, and launch-copy recommendations are in docs/github-listing.md.

Current Status

The current pipeline is strongest for static and semi-static web pages:

  • company, brand, marketing, and documentation pages
  • public landing pages
  • iframe-blocked pages that need capture-based reconstruction
  • responsive page snapshots across desktop, tablet, and mobile

It is not a full backend or app-logic clone engine. Login-only screens, app-first or native-app-required services, captcha-heavy sites, maps, games, canvas/WebGL-heavy pages, real-time feeds, payments, booking flows, and private server behavior still need separate handling.

Operationally, the repo is now a production-candidate clone engine for URL-based capture and bounded reconstruction: jobs can be queued, network evidence can be replay-audited from HAR artifacts, authenticated dashboard runs can be driven from user-owned browser state, and local gates verify the route corpus, score checks, package contents, and CI wiring. The remaining hard boundary is server-side product behavior, not front-end evidence capture and reconstruction.

Measured Checkpoints

Local benchmark runs from this repo (scripts/check_clone_quality_bench.py, mobile + tablet breakpoints, 2026-10-28):

URLPathScoreWall time
https://developer.mozilla.org/en-US/runtime snapshotroot 99, visual 100, mobile 99, tablet 99~34 s
https://lit.dev/ (web components / shadow DOM)runtime snapshotroot 99, visual 100, mobile 98, tablet 98~40 s
https://www.mozilla.org/runtime snapshotroot 95, visual 100, mobile 95, tablet 95~43 s (was ~160 s)
https://www.python.orgruntime snapshotroot 98, visual 100, mobile 98, tablet 94~226 s (was ~1011 s, failed at 73)
https://www.example.comexact reuseready yes—

These are generated by the local self-verify pipeline, not manually assigned ratings. Earlier bounded-rebuild previews embedded the reference screenshot, which inflated their visual score; the runtime snapshot renders real DOM and CSS, and every row above was also checked by side-by-side screenshot review. Known gap: layouts that page scripts compute for viewports other than the captured one (for example masonry footers) can still drift at tablet width. The reproducible commands and score thresholds are tracked in docs/benchmark-evidence.json. Production readiness gates are tracked in docs/production-pipeline-gates.json.

Core Features

  • Source-first routing:
    • direct iframe or embed reuse when it is safe and frameable
    • original preview, export, remix, or source routes when available
    • bounded rebuild only when exact reuse is unavailable
  • Live browser capture:
    • DOM snapshot
    • runtime HTML
    • full-page screenshot
    • computed style summaries
    • CSS analysis
    • asset inventory
    • HAR-like network metadata
    • interaction states and replay traces
    • storage state export for session-aware flows
  • Blocked-site rebuild:
    • handles X-Frame-Options and CSP-blocked pages by rebuilding from captured evidence
    • generates reusable frontend reconstruction artifacts from captured page structure
    • preserves custom tags, shadow-root host structure, and semantic document structure where captured
  • Evidence limitation reporting:
    • separates directly captured artifacts from inferred or missing evidence in reproduction results and prompts
    • marks app-gated, auth-gated, and native-app-led surfaces as bounded evidence, with recommendations for user screenshots or authenticated session capture
  • Operational failure classification:
    • reports typed pipeline action codes such as network-replay-limited, auth-session-missing, public-app-gate, and canvas-visual-fallback
    • exposes HAR/network replay_readiness before treating captured network evidence as replay-grade
  • Production pipeline helpers:
    • filesystem-backed async clone job queue with durable JSON records, worker locks, retry scheduling, cancellation, and manifest annotation
    • deterministic HAR replay engine for standard HAR, near-HAR, and captured network/manifest.json artifacts
    • authenticated dashboard live corpus runner that accepts user-provided storage_state_path or user_data_dir outside the repo
  • Self-verification:
    • screenshot similarity
    • DOM snapshot similarity
    • computed-style similarity
    • hover/focus/click interaction state parity
    • interaction trace parity
    • desktop/mobile/tablet breakpoint reports
  • Responsive benchmark support:
    • primary desktop viewport: 1440x1200
    • tablet profile: 768x1024
    • mobile profile: 390x844
  • Repair loop:
    • bounded self-repair can run when the first scaffold misses the readiness threshold

Install

Requirements

  • Node.js 18 or newer
  • Python 3.9 or newer
  • Chrome or Chromium available locally for Playwright runtime capture

The package uses playwright-core; it does not download a browser by itself.

Installing this project adds the source-first-clone plugin bundle, the exact-clone-intake skill, and the MCP server that exposes the URL inspection, capture, rebuild, and verification tools.

Install From npm

bash
npm install -g web-embeddingweb-embedding installweb-embedding doctor

Clone a public URL after installing:

bash
web-embedding clone \  --url https://developer.mozilla.org/en-US/ \  --output-dir ./.tmp/mdn-clone \  --wait-seconds 2 \  --timeout-seconds 35 \  --breakpoints mobile tablet

If you already have an older local plugin installed, overwrite it with:

bash
web-embedding install --forceweb-embedding doctor

You can also run the installer without a global install:

bash
npx web-embedding install

Use As An MCP Server

For MCP clients that can launch npm stdio servers:

json
{  "mcpServers": {    "source-first-clone": {      "command": "npx",      "args": ["-y", "web-embedding@latest", "mcp"]    }  }}

For local smoke testing:

bash
npx web-embedding@latest mcp

The MCP Registry identity is io.github.jongko54/web-embedding; server.json and package.json#mcpName are kept in sync for registry ownership verification.

Hosted Apps SDK Intake Endpoint

The public remote MCP intake endpoint for Apps SDK Developer Mode is:

text
https://webembedding-jongkos-mcp.vercel.app/mcp

It exposes low-risk source-first routing tools such as URL inspection, embed candidate discovery, clone-mode classification, and embed snippet generation. Full browser capture, HAR replay, queues, bounded rebuilds, and one-pass clone execution remain local-first through the stdio MCP package.

Apps SDK review pages are hosted alongside the endpoint: https://webembedding-jongkos-mcp.vercel.app/privacy.html, https://webembedding-jongkos-mcp.vercel.app/terms.html, and https://webembedding-jongkos-mcp.vercel.app/submission.html.

Sandboxing And Approvals

webEmbedding has two different execution boundaries:

  • Hosted Apps SDK intake: read-only URL routing and classification only. It accepts absolute http and https URLs, does not run Playwright, does not read local files, does not use browser profiles or storage state, and does not persist capture artifacts.
  • Local stdio MCP and CLI: full capture, HAR replay, queues, rebuild scaffolds, and self-verify run on the user's machine under the user's local agent and filesystem permissions. Output is written only to caller-provided paths such as output_dir or queue_root.
  • Authenticated capture: session-aware runs require the caller to intentionally provide a storage_state_path or user_data_dir. webEmbedding does not collect credentials, perform login bypasses, or treat a public login shell as private app evidence.
  • Access-controlled surfaces: paywalls, captcha flows, private dashboards, payment/checkout/account/admin flows, and native-app-led screens should be blocked, marked needs_session, or sent to manual review unless the user has explicit authorization and supplies the needed evidence.

Local URL entrypoints reject non-HTTP schemes such as file:// so an agent cannot use clone/capture tools as a local file reader. Telemetry is disabled by default and, when enabled, excludes target URLs, local paths, captured HTML, screenshots, storage state, environment variables, API keys, and command output.

Agent Marketplaces

This repository includes marketplace metadata for the two local agent surfaces:

  • Codex: .agents/plugins/marketplace.json points to ./bundle/source-first-clone.
  • Claude Code: .claude-plugin/marketplace.json points to the same bundle and the bundle includes .claude-plugin/plugin.json.

Claude Code users can add the marketplace from GitHub with:

text
/plugin marketplace add jongko54/webEmbedding/plugin install source-first-clone@webembedding

AI auto-selection expectations and golden prompts live in docs/ai-distribution.md and evals/ai-selection/webembedding-golden-prompts.json.

Install From Release

bash
curl -fsSL https://github.com/jongko54/webEmbedding/releases/latest/download/install.sh | bash

Install From This Checkout

bash
git clone https://github.com/jongko54/webEmbedding.gitcd webEmbeddingnpm installnode ./bin/web-embedding.mjs installnode ./bin/web-embedding.mjs doctor

Install Into A Temporary Home

Useful for testing without touching your real agent home:

bash
python3 python/web_embedding/installer.py install --target-home ./.tmp/homepython3 python/web_embedding/installer.py doctor --target-home ./.tmp/homepython3 python/web_embedding/installer.py uninstall --target-home ./.tmp/home

Opt-in Telemetry

Telemetry is disabled by default. On an interactive first install, web-embedding install asks once and defaults to No. Non-interactive installs such as CI and curl | bash do not prompt. If you opt in, web-embedding sends a small anonymous command-completion event to a JSON POST endpoint you control. It does not send target URLs, local paths, captured HTML, screenshots, storage state, environment variables, API keys, or command output.

Enable it during install:

bash
web-embedding install --telemetry --telemetry-endpoint https://your-collector.example/events

Or manage it later:

bash
web-embedding telemetry enable --endpoint https://your-collector.example/eventsweb-embedding telemetry statusweb-embedding telemetry disableweb-embedding telemetry reset-id

Each event contains an anonymous install id, package version, command name, success/failure status, OS/runtime basics, and coarse option flags such as breakpoint_count or install_source.

Environment controls:

bash
WEB_EMBEDDING_TELEMETRY=1WEB_EMBEDDING_NO_TELEMETRY=1WEB_EMBEDDING_TELEMETRY_PROMPT=0WEB_EMBEDDING_TELEMETRY_ENDPOINT=https://your-collector.example/eventsWEB_EMBEDDING_TELEMETRY_LOG=./telemetry.jsonl

Run a local/self-hosted JSONL collector:

bash
npm run telemetry:collector -- --host 127.0.0.1 --port 8765 --out ./telemetry.jsonlWEB_EMBEDDING_TELEMETRY=1 \WEB_EMBEDDING_TELEMETRY_ENDPOINT=http://127.0.0.1:8765/events \web-embedding doctor

Summarize collected usage:

bash
npm run telemetry:summarize -- ./telemetry.jsonl

The summary includes install and clone executions, total command executions, unique anonymous install IDs, command counts, and version counts. See docs/telemetry.md for collector and analyzer details.

Quick Start

Inspect a URL and get route hints:

bash
node ./bin/web-embedding.mjs inspect \  --url https://developer.mozilla.org/en-US/

Run a safe preflight audit before capture or clone:

bash
node ./bin/web-embedding.mjs audit \  --url https://developer.mozilla.org/en-US/

The audit reports whether the reference is ready for exact/embed reuse, needs local capture, needs an authenticated session, requires manual review, or should be blocked before any browser capture or filesystem output runs.

Run the full clone workflow:

bash
node ./bin/web-embedding.mjs clone \  --url https://developer.mozilla.org/en-US/ \  --output-dir ./.tmp/mdn-clone \  --wait-seconds 2 \  --timeout-seconds 35 \  --breakpoints mobile tablet

Run a lightweight quality benchmark:

bash
python3 scripts/check_clone_quality_bench.py \  https://developer.mozilla.org/en-US/ \  --output-root ./.tmp/clone-quality-bench \  --wait-seconds 1 \  --timeout-seconds 35 \  --breakpoints mobile tablet

The benchmark prints compact rows for root, visual, and breakpoint scores. The full artifacts are written under the output directory.

CLI Commands

bash
node ./bin/web-embedding.mjs capabilitiesnode ./bin/web-embedding.mjs installnode ./bin/web-embedding.mjs doctornode ./bin/web-embedding.mjs uninstallnode ./bin/web-embedding.mjs pathsnode ./bin/web-embedding.mjs telemetry status
bash
node ./bin/web-embedding.mjs inspect --url https://www.mozilla.org/
bash
node ./bin/web-embedding.mjs audit --url https://www.mozilla.org/
bash
node ./bin/web-embedding.mjs capture \  --url https://www.mozilla.org/ \  --output-dir ./.tmp/capture-mozilla \  --breakpoints mobile tablet
bash
node ./bin/web-embedding.mjs reproduce \  --url https://www.mozilla.org/ \  --output-dir ./.tmp/reproduce-mozilla \  --breakpoints mobile tablet
bash
node ./bin/web-embedding.mjs clone \  --url https://www.mozilla.org/ \  --output-dir ./.tmp/clone-mozilla \  --breakpoints mobile tablet
bash
node ./bin/web-embedding.mjs verify \  --reference-bundle ./.tmp/reference/capture.json \  --candidate-bundle ./.tmp/candidate/capture.json

Output Artifacts

A clone run can produce:

  • capture.json
  • pipeline-run-manifest.json
  • dom/snapshot.json
  • dom/runtime.html
  • styles/computed-summary.json
  • styles/css-analysis.json
  • network/manifest.json
  • network/har.json
  • network/har-like.json
  • network/replay-report.json
  • assets/inventory.json
  • interactions/states.json
  • interactions/trace.json
  • screenshots/runtime.png
  • session/storage-state.json
  • reproduction/plan.json
  • reproduction/evidence-limitations.json
  • reproduction/rebuild-prompt.txt
  • reproduction/rebuild/starter.html
  • reproduction/rebuild/starter.css
  • reproduction/rebuild/starter.tsx
  • reproduction/rebuild/next-app/
  • reproduction/self-verify/summary.json
  • reproduction/self-verify/renderers/*/verification.json
  • reproduction/self-verify/renderers/*/visual-qa.json
  • reproduction/self-verify/renderers/*/breakpoints/*-verification.json

Quality Benchmark

Run the default small benchmark:

bash
npm run check:clone-bench:local

Run the universal route regression corpus and expectations gate:

bash
npm run check:benchmark-routes:local

Run a lightweight clone score gate:

bash
npm run check:clone-score-gate:local

Validate the committed benchmark evidence manifest:

bash
npm run check:benchmark-evidence:local

Validate production pipeline gates:

bash
npm run check:production-readiness:local

Run the operational smokes individually:

bash
npm run check:job-queue:localnpm run check:har-replay:localnpm run check:authenticated-corpus:local

Classify failure/action codes from a route report:

bash
npm run classify:pipeline-failures -- --report ./.tmp/universal-route-benchmark/universal-route-report.json

Find low-scoring persisted benchmark artifacts:

bash
npm run summarize:benchmark-scores -- --root ./.tmp --min-score 60 --max-score 70

Run specific URLs:

bash
python3 scripts/check_clone_quality_bench.py \  https://www.example.com \  https://www.mozilla.org/ \  --no-breakpoints

Run a responsive benchmark:

bash
python3 scripts/check_clone_quality_bench.py \  https://developer.mozilla.org/en-US/ \  --breakpoints mobile tablet

Development Checks

bash
python3 -m py_compile \  bundle/source-first-clone/mcp/source_first_clone/*.py \  scripts/check_integration_smoke.py \  scripts/check_clone_quality_bench.py
bash
npm run check:integration:local
bash
git diff --check

Repo Layout

  • bundle/source-first-clone Installed plugin bundle, MCP server, and exact-clone intake skill.
  • bundle/source-first-clone/mcp/source_first_clone Capture, planning, rebuild, repair, and verification engine.
  • bin/web-embedding.mjs Node CLI wrapper.
  • python/web_embedding/installer.py Shared installer and command dispatcher.
  • scripts/check_clone_quality_bench.py URL clone quality benchmark helper.
  • scripts/benchmark_routes.py Universal route/capture-depth regression benchmark helper.
  • scripts/check_benchmark_report.py Benchmark expectation validator for exact, minimum, and contains-style checks.
  • scripts/check_benchmark_evidence.py Benchmark evidence manifest validator.
  • scripts/check_job_queue_smoke.py Filesystem async clone job queue smoke test.
  • scripts/check_har_replay_smoke.py Deterministic HAR replay engine smoke test.
  • scripts/benchmark_authenticated_corpus.py User-provided authenticated dashboard corpus runner.
  • scripts/summarize_benchmark_scores.py Utility for finding low or high scoring persisted benchmark artifacts under an output root.
  • scripts/classify_pipeline_failures.py Operational failure/action taxonomy summarizer for reports and capture artifacts.
  • scripts/check_production_readiness.py Production readiness gate validator for corpus, failure taxonomy, CI wiring, and policy docs.
  • scripts/check_integration_smoke.py Release, install, and URL-only clone smoke test.
  • scripts/release_bundle.py Release artifact builder.
  • docs/ Architecture notes and universal benchmark documentation.

Positioning

The strongest claim for this project is:

A source-first website cloning engine that combines Playwright capture, HAR replay, MCP tools, and self-verification to rebuild iframe-blocked public pages with reproducible visual, DOM, style, interaction, and responsive scores.

Avoid treating the output as a legal or ownership bypass. The engine can reconstruct public page structure, but permission, licensing, and acceptable use still matter.

FAQ

How can an AI coding agent clone a website from a URL? Install the MCP server (npx -y web-embedding@latest mcp) and ask the agent to clone the URL. The agent calls clone_reference_url, which inspects the page, captures it in a real Chromium, picks reuse or rebuild, and returns the clone entrypoint with fidelity scores.

What does the clone look like on disk? For pages that block iframes, snapshot/index.html plus snapshot/assets/ is a self-contained copy of the rendered page: stylesheets inlined from source (including CSS-in-JS and adoptedStyleSheets), open shadow roots as declarative shadow DOM, canvas pixels, form state, and localized images, fonts, and SVG sprites. Serve it over HTTP (python3 -m http.server); file:// blocks external SVG <use> references.

How is this different from SingleFile, screenshot-to-code, or open-lovable? SingleFile saves a page from a browser extension or CLI; webEmbedding uses the same class of DOM serialization but runs as an MCP tool, decides between embed reuse, snapshot, and rebuild, and verifies the result at three breakpoints. Screenshot-to-code and open-lovable generate new code from an image or scrape with an LLM; webEmbedding starts from the live DOM and CSS, so text, fonts, and layout come from the source instead of being re-imagined.

What can't it clone? Server behavior (logins, payments, search, real-time data), captcha- or bot-protected pages, closed shadow roots, and canvas/WebGL apps beyond a static frame. Authenticated pages need a user-provided browser session, and you still need the right to reproduce the page.

License

MIT

来源:README.md,提交 e4b1ec8

工具

6

detect_runtime_capabilities

Detect Hosted Runtime Capabilities

Report hosted Apps SDK readiness, read-only runtime capabilities, unsupported capture behavior, and the local stdio command.

无需输入参数。

inspect_url

Inspect URL Reuse Route

Fetch a public or user-authorized URL and inspect title, metadata, frame policy, and likely source/embed candidates. Does not capture screenshots or persist artifacts.

输入参数url必填timeout_seconds

discover_embed_candidates

Discover Embed Candidates

Extract likely embed, preview, viewer, remix, and source URLs from a public or user-authorized page.

输入参数url必填timeout_seconds

classify_clone_mode

Classify Clone Mode

Decide whether a reference should be embedded, sourced, locally captured, bounded-rebuilt, or blocked before reproduction.

输入参数candidateslicense_textsite_profilesource_signalsexact_requested

generate_embed_snippet

Generate Embed Snippet

Generate an iframe snippet for a known frameable and authorized URL. Does not verify frameability by itself.

输入参数url必填titleframework

plan_reproduction_path

Plan Reproduction Path

Create a source-first plan that separates exact embed/source reuse from local capture and bounded rebuild work.

输入参数candidateslicense_textsite_profilecapture_bundlesource_signalsexact_requested

版本历史

3
  1. v0.3.13最新Nov 8, 2026
  2. v0.3.12Oct 29, 2026
  3. v0.3.8Sep 16, 2026

评分

已满足适用检查项的 100%

该评分衡量此收录条目的完整程度,依据注册中心条目、端点与公开来源计算得出。它不是对服务器本身的评价,也不表示服务器安全或当前可用:可连通性是带日期的检查结果,评分也不包含安装量或用户评价。

  • 近 30 天内可连通必需25 分
  • 已声明认证方式必需10 分
  • 已列出工具必需15 分
  • 描述必需15 分
  • 图标10 分
  • 文档15 分
  • 许可证10 分