io.github.mauricekleine由 GitHub 帳號 mauricekleine 發布v1.1.0更新於 Oct 24, 2026
An open-source benchmark of how well LLMs solve nonogram puzzles, from 5x5 to 20x20.
唯讀存取 Nonobench——一個評測大型語言模型解 5x5 至 20x20 數織（Nonogram）謎題能力的公開基準，提供排行榜、單一模型結果與答案驗證。
Web executable，透過 Streamable HTTP。 遠端服務在工作區中設定後即可從網頁執行環境執行。
把這段提示詞貼上到 Claude Code、Codex、Cursor 等代理程式，或 SourceWeft 對話中。代理程式會讀取該伺服器的安裝指南，先向你展示發布者、端點和登入方式，經你同意後再把它新增到你的客戶端。金鑰不會寫進任何命令。
閱讀 https://sourceweft.com/mcp/io.github.mauricekleine%2Fnonobench/install.md，並按其中說明把這個 MCP 伺服器接入我的客戶端。修改任何設定之前，先告訴我它的發布者、端點和登入方式，等我確認。不要把 API 金鑰寫進命令或我會提交的檔案；問我金鑰應從哪裡取得。把它新增到你客戶端的 mcpServers 設定中。
{
"mcpServers": {
"nonobench": {
"type": "http",
"url": "https://www.nonobench.com/mcp"
}
}
}A benchmark suite for evaluating LLM reasoning capabilities on Nonogram (Picross) puzzle solving across different grid sizes. Results are published at nonobench.com.
Built by Maurice Kleine.
Nonograms (also known as Picross, Griddlers, or Paint by Numbers) are logic puzzles where you fill in cells on a grid based on numeric clues for each row and column. The clues indicate consecutive groups of filled cells, separated by at least one empty cell. Solving these puzzles requires logical deduction and constraint satisfaction - making them an excellent test of LLM reasoning abilities.
Create a .env file in the bench/ directory (see bench/.env.example):
This is the only variable needed. The visualizer builds without any.
Runs are incremental and append-only: a model/puzzle pair that already has a successful result is never run again, and the database refuses to overwrite it. Failed runs are retried on the next invocation. Results are stored in bench/results.db (SQLite).
Useful flags:
--max-cost <usd> stops launching new puzzles once this session's spend reaches the amount. Requests already in flight still finish, so a session can overshoot by up to --parallel requests per selected model. Use the OpenRouter key's own limit as the hard ceiling.--parallel <n> sets concurrent requests per model (default 10); lower it for rate-limited providers.--limit <n> runs only the first n puzzles of each size, for pilots against a scratch database (NONOBENCH_DB=/tmp/copy.db).New runs ask for the answer as strict structured output (a JSON schema, only routed to endpoints that enforce it), so models cannot wrap the grid in prose. For a few models the schema-enforcing endpoints measurably hurt answers; those run in text mode instead (outputMode: "text" in bench/constants.ts, chosen by a 5x5 A/B with the benchmark prompt). Every run records its mode, and grading is identical for both: the answer must satisfy every clue. The runner also stops a model that solves none of the 5x5 puzzles with structured output, and one whose early runs mostly report zero reasoning tokens, since both point at the harness rather than the model.
After benchmarking, export results for the visualizer:
This writes visualizer/app/results.json (aggregates) and visualizer/public/results-raw.json (every run, including prompts and outputs).
Other scripts:
bun test - parser, grader, database-policy and puzzle checksbun run typecheck - TypeScript checkbun run regrade - read-only comparison of stored grades against the current graderThe runner can benchmark a model that you serve yourself. Any server with an OpenAI-compatible /v1 endpoint works: vLLM, SGLang, llama.cpp, LM Studio, and Ollama.
This setup uses two machines. The model server is the workstation that serves the model. The bench machine is the laptop that runs the benchmark.
Bind the server to all interfaces. The default bind address, 127.0.0.1, accepts connections only from the model server itself.
vLLM:
Ollama:
llama.cpp:
SGLang:
LM Studio: open the Developer tab, then select Start Server. Turn on the Network Server switch, so the server listens on all interfaces.
CAUTION: Do not expose the port to the internet. Most local servers have no authentication, so anyone who reaches the port can run the model.
Restrict the rule to your own subnet. Allow TCP traffic from your local network to the port:
On macOS, allow incoming connections for the server app in System Settings.
Find the LAN address of the model server. On Linux, run ip -4 addr show. On macOS, run ipconfig getifaddr en0.
From the laptop, request the model list:
Make sure that the answer lists your model. Copy the exact id value. You need that value for NONOBENCH_LOCAL_MODEL.
If the request fails, check the bind address from step 1. Then check the firewall rule, the address, and the port.
If the model server is not reachable on your network, create an SSH tunnel to it instead. Then use the address http://127.0.0.1:8000/v1:
Do a pilot first. The next command runs two 5x5 puzzles into a scratch database:
Make sure that the plan lists your model as [local, text].
The runner adds the local model to the plan only when you set both NONOBENCH_LOCAL_BASE_URL and NONOBENCH_LOCAL_MODEL.
bun run bench:local, which is bun run bench with NONOBENCH_DB=local-results.db. bench/results.db is the shared dataset, and the export-contract test checks it against the committed exports. The runner refuses to write a local model to the shared database.id against the server's GET /v1/models list. A typo fails with the models the server does serve, and an unreachable server fails before any run is recorded.NONOBENCH_OUTPUT_MODE=json_schema.--parallel 1. One GPU serves fewer requests at the same time than a cloud provider does.bench/.env to avoid the prefix on every command.bench/local-models.json (name, server model id, family, server URL), so a later bun run export and the visualizer pick it up without any env vars. To register a model you already benched, add one entry there: "<name>": { "baseURL": "http://host:port/v1", "modelId": "<server model id>", "family": "<name>" }. Omit modelId when the name is the server's id.Then open http://localhost:3000 to view the interactive dashboard.
Your own runs live in a separate database. Serve them with:
dev:local exports bench/local-results.db into the dashboard's data files, starts the dev server, and restores the committed files when you stop it. While it runs, the dashboard shows only the models in that database — your local runs, not the published leaderboard. Set NONOBENCH_LOCAL_DB to serve another database.
While that server runs, bun test in bench fails one test: the export-contract test compares the dashboard's data files against bench/results.db, and dev:local has put your runs in them. Stop the server, let the script restore the committed files, and the suite passes again.
The visualizer is a Cloudflare Worker (visualizer/wrangler.jsonc). Pages are prerendered at build time and served as static assets; the Worker serves the API, the MCP server and the agent routes, reading the exported JSON through its assets binding. Workers Builds owns production deploys on merge to main: it runs bun install --frozen-lockfile && bun run build, then bunx wrangler deploy, from visualizer/ with Cloudflare credentials. Local deploy scripts and Wrangler logins are retired; agents use cf for Cloudflare access. From visualizer/:
CLOUDFLARE_ENV=beta bun run build builds the beta Worker for local preview. Nothing deploys it.bun run deploy:verify <sha> waits until production runs a build containing that commit on main. Branch commits fail immediately unless DEPLOY_VERIFY_URL names a separately deployed target. The repository root runs the same script, so bun run deploy:verify <sha> works from either directory.nonobench.com exposes the benchmark data to agents, with no authentication:
REST API under /api/v1 (leaderboard, models, puzzles, a solution checker, individual runs). The spec is at /api/openapi.json, and /.well-known/api-catalog (RFC 9727) points to it.
MCP server at /mcp (stateless Streamable HTTP, MCP 2026-07-28 with 2025 client compatibility), described by /.well-known/mcp/server-card.json. Add it to a client with claude mcp add --transport http nonobench https://www.nonobench.com/mcp. Browser requests may use the site's origins or HTTP localhost/127.0.0.1 origins.
It's listed in the Claude Connectors Directory, the MCP Registry (io.github.mauricekleine/nonobench), Smithery, Glama and mcpservers.org.
WebMCP tools registered in the browser via navigator.modelContext.
Markdown: / and /puzzles return markdown when requested with Accept: text/markdown. /llms.txt gives an overview.
Discovery: robots.txt (with Content Signals), sitemap.xml, Link headers on the homepage, an agent skill at /.well-known/agent-skills/index.json, and an ARD manifest at /.well-known/ai-catalog.json.
All of it is read from the same exported files as the dashboard (visualizer/app/results.json and visualizer/public/results-raw.json), so bun run export updates it too.
Each model receives the same system prompt and the puzzle's row and column clues. Standard answers are the grid as one string of 1s and 0s. Hard mode answers are one row per line, because at 400 cells most models miscount a single string (see LEARNINGS.md). An answer is correct when it satisfies every row and column clue.
Ten of the 30 puzzles (one 5x5, four 10x10, five 15x15) have more than one valid solution, so answers are checked against the clues rather than compared with the stored solution. Correctness is derived from the stored raw outputs at export time; the database is never rewritten. The puzzle test suite pins which puzzles are ambiguous, and any new puzzle must have a unique solution.
The core tier has 30 puzzles (10 each of 5x5, 10x10 and 15x15), defined in visualizer/components/puzzles/ and shared by the runner and the dashboard. They were sourced from nono-dataset. Hard mode has 10 generated 20x20 puzzles. A puzzle's ID is a hash of its solution, so changing a puzzle's solution creates a new puzzle.
Default benchmark runs cover the three core sizes (Standard). Use --sizes 20x20 with a model selection to run Hard mode; comma-separated sizes also work. The runner's plan reports missing 20x20 work separately. Headline overall accuracy and best-variant selection use Standard runs only; Hard mode has its own results. Hard-mode requests get a 128,000-token answer budget, capped at the endpoint's maximum (bench/max-output-tokens.json).
From bench/, bun run generate-puzzles recreates the 20x20 set with a fixed seed. It fills grids at random (no pictures, so a model can't guess the image), keeps grids with at least three blocks per line and little mirror symmetry, and checks uniqueness with an exact solver (NONOGRAM_SOLVER). The set mixes five puzzles that row-and-column propagation solves with five where it stalls with 20–200 cells left. bun test verifies their clues, uniqueness flags and line solvability; the original ambiguity list remains pinned.
Edit bench/constants.ts to configure:
MODELS - Array of model configurations (OpenRouter model ID, display name, reasoning settings)MAX_PARALLEL_RUNS_PER_MODEL - Concurrent puzzle runs per model (default: 10)REQUEST_TIMEOUT_MS - Per-request timeout; a timed-out request is stored as a failed run (default: 30 minutes)NONOBENCH_DB, NONOBENCH_RESULTS_JSON and NONOBENCH_RESULTS_RAW_JSON override the database and export paths, which is useful for testing against a copy.
Benchmark Runner
Visualizer
Contributions are welcome! Feel free to:
MIT
來源：README.md，提交 796cedd
Get leaderboard
Models ranked by accuracy; defaults to all effort levels for compatibility.
sizeeffortfamilyversionproviderreasoningmin_correctopen_weightsList providers
Provider ids, names, families and variant counts.
無需輸入參數。
List families
Model families, available efforts and best variants.
無需輸入參數。
Compare models
Side-by-side core overall and per-size accuracy, cost, latency and token results for model or family names.
models必填Get model results
Accuracy, cost, latency and token use for one model, broken down by grid size.
model必填List puzzles
The benchmark puzzles with their ids and row/column clues.
sizeGet puzzle
One puzzle, including the clue text models were prompted with. The reference solution is only included on request; some puzzles have several valid solutions.
id必填include_solutionCheck solution
Check a grid against a puzzle's clues, using the same rule as the benchmark grader. Reports which rows and columns do not match.
id必填grid必填Get puzzle results
Per-model outcomes for one puzzle. Answers are omitted unless requested.
id必填effortfamilyproviderreasoningopen_weightsinclude_answersGet model puzzles
Which puzzles one model solved, missed, timed out on, or has not run.
model必填List runs
Individual benchmark runs (one model on one puzzle), optionally with the raw prompt and model output.
sizelimitmodeloffsetpuzzle_idinclude_output已滿足適用檢查項的 100%
該評分衡量此收錄條目的完整程度，依據註冊中心條目、端點與公開來源計算得出。它不是對伺服器本身的評價，也不表示伺服器安全或目前可用：可連通性是帶日期的檢查結果，評分也不包含安裝量或使用者評價。
nonobench/├── bench/ # Benchmark runner, results database and exporter└── visualizer/ # TanStack Start dashboard on Cloudflare Workers (nonobench.com), also home of the puzzle setgit clone https://github.com/mauricekleine/nonobench.gitcd nonobenchOPENROUTER_API_KEY=your_openrouter_api_key_herecd benchbun installbun run bench # prints the plan and exits, no API callsbun run bench --model <name> # run one model (repeat --model for more)bun run bench --all-missing # run every configured model with missing workbun run bench --model <name> --sizes 20x20 # opt in to the extended tierbun run exportvllm serve Qwen/Qwen3-32B --host 0.0.0.0 --port 8000ollama pull qwen3:32bOLLAMA_HOST=0.0.0.0:11434 ollama servellama-server -m ./qwen3-32b-Q4_K_M.gguf --host 0.0.0.0 --port 8080python -m sglang.launch_server --model-path Qwen/Qwen3-32B --host 0.0.0.0 --port 30000sudo ufw allow from 192.168.1.0/24 to any port 8000 proto tcpcurl http://192.168.1.20:8000/v1/modelsssh -N -L 8000:127.0.0.1:8000 [email protected]cd benchbun installNONOBENCH_LOCAL_BASE_URL=http://192.168.1.20:8000/v1 \NONOBENCH_LOCAL_MODEL=Qwen3-32B \bun run bench:local --model Qwen3-32BNONOBENCH_LOCAL_BASE_URL=http://192.168.1.20:8000/v1 \NONOBENCH_LOCAL_MODEL=Qwen3-32B \NONOBENCH_DB=local-pilot.db \bun run bench --model Qwen3-32B --sizes 5x5 --limit 2cd visualizerbun installbun run devcd visualizerbun run dev:local