[Penguin Statistics - Logo]
Penguin Statistics - V3 Backend
[License]
[Last Commit]
[GitHub Actions Status]
[go.mod Version]
[FOSSA Status]
[Alt]
The re-designed Penguin Statistics v3 Backend, built with Go.
Using the data
The drop data is public:
-
REST API at https://penguin-stats.io/PenguinStats/api/v2/: see the
developer docs and the
OpenAPI (Swagger 2.0) spec.
-
MCP server for AI assistants (Claude, ChatGPT, Cursor, VS Code, Cherry Studio and others) at
https://mcp.penguin-stats.io/mcp: Streamable HTTP, no authentication, read-only. In Claude Code:
The steps for every other app, in English and Chinese, are at https://mcp.penguin-stats.io/.
-
Plugin and skill for coding agents, the MCP server with a skill on reading, citing and querying the numbers:
https://github.com/penguin-statistics/agent-skills
The drop data is licensed CC BY-NC 4.0: non-commercial use, with
credit. Credit it as Data: Penguin Statistics (penguin-stats.io), CC BY-NC 4.0. The code of this repository is
MIT.
Architecture
This Go project mainly uses:
This project mainly follows a RESTful API design.
Operations
Report tasks in the dead-letter stream
The report worker moves a report task out of the penguin-reports stream into penguin-reports-dead (subjects
REPORT_DEAD.*) when it can never be processed (it is not a task, or a stage of it does not exist) or when it failed at
each of its 30 deliveries, which take about 97 minutes of retry delays. Each move is logged at error level
(reportwkr.malformed, reportwkr.parked or reportwkr.dead). The copy keeps the payload as published, with these
headers:
Penguin-Original-Subject: the subject it was published under, like REPORT.SINGLE
Penguin-Original-Sequence: its sequence in penguin-reports
Penguin-Deliveries: its deliveries when it moved
Penguin-Last-Error: the error that moved it
A task too close to the maximum payload to fit with these headers moves without them, and its subject under
REPORT_DEAD.* names its original subject; its move is logged as reportwkr.dead.bare, with the sequence and the error.
Nothing in the stream expires. With the nats CLI, once the cause is fixed:
A task that moved after failing at each delivery may have been stored by one of them whose commit outcome was unknown
(for example the connection dropped during the commit); check drop_reports for its account and time before replaying
it.
Benchmarks
Report consumer CPU
internal/workers/reportwkr/consumerbench_test.go (build tag consumerbench) drives the report consumer through its
real code path: each task is published to a JetStream stream of its own, fetched and handed to
Worker.ingestPreprocess (BenchmarkConsumer/ingest), which stores it in PostgreSQL and sets its recall key in Redis,
with logging and tracing configured as in production. BenchmarkConsumer/process runs Worker.process alone. Besides
wall time and allocations it reports, per task, the CPU time of the process (cpu-ns/op, user + system from
getrusage, which a shared machine disturbs far less than the wall time) and the time spent waiting on PostgreSQL
(pg-wait-ns/op, pg-wait-%) and Redis.
The tasks (internal/workers/reportwkr/testdata/consumerbench/tasks.jsonl.gz) are encoded as the report handler
publishes them, drawn from the stages, servers, times, sources, versions and drop patterns of recent production reports,
with synthetic accounts, IPs and task ids.
It needs a scratch database, Redis and NATS with JetStream. The scratch database holds the schema and the reference
tables of a restore of production (penguinprod, only read) and synthetic accounts; the benchmark truncates its
drop_reports and drop_report_extras:
The defaults point at penguin_cpubench on 127.0.0.1:5434, Redis DB 14 on 127.0.0.1:16379 and NATS on 127.0.0.1:4222
(CONSUMERBENCH_POSTGRES_DSN, CONSUMERBENCH_REDIS_URL, CONSUMERBENCH_NATS_URL). Then:
Outputs go to $OUT (default a new temporary directory). Each run logs the load average and the queries other sessions
had active on the cluster to $OUT/runs.log. The PostgreSQL connection runs over loopback, where a write system call
also does the receiving side's TCP work, so the system calls weigh more than against a remote server.
Maintainers
This project has mainly being maintained by the following contributors (in alphabetical order):
The full list of active contributors of the Penguin Statistics project can be found at the Team Members page of the website.
Contributors
This project is made possible by the following contributors. Contributions are always welcome!
[Contributors]
License
[FOSSA Status]