Event-driven decoupling with Pub/Sub
Break a distributed monolith or tightly coupled service chain apart with
asynchronous messaging on Pub/Sub so services scale and fail independently.
This skill uses progressive disclosure across Phases 0–5 (with Phase 0
inline below and modular references/ files for Phases 1–5): run the full
end-to-end migration sequence for architecture journeys, or load only the
single phase reference needed for a focused design, code, or troubleshooting
request.
Usage Attribution
MUST READ: Before executing any gcloud, REST API, or MCP command, you
MUST read and follow the instructions in
Usage Attribution [blocked].
Progressive disclosure: load only the phase you need
Routing rules:
- End-to-end requests: run Phases 0–5 sequentially, loading each phase's
reference only when that phase begins. Two exceptions apply:
- Greenfield consumers: run Phases 3 and 4 together so new consumers ship with distributed tracing from day one.
- Missing observability: if the user's primary problem is missing latency visibility or absent distributed traces, run Phase 4 (references/phase4-distributed-tracing.md [blocked]) first to establish a trace baseline before proposing Pub/Sub changes, then resume Phases 1–3.
- Focused single-phase requests: load only the matching phase
reference, name which phase file applies, and never dump unrelated phases:
- Flow-control / subscription configuration questions
(
MaxOutstandingMessages, ack deadline,max-delivery-attempts, backoff): load only Phase 2 (references/phase2-pubsub-topology-and-sizing.md [blocked]). Relate flow control to the database connection pool and ack deadline to the per-message processing timeout; do not present the Phase 1 coupling matrix, Transactional Outbox, or Phase 3 consumer handler code. - Consumer handler ack/nack questions: load only Phase 3 (references/phase3-consumer-and-autoscaling.md [blocked]). Provide the handler logic (never provisioning commands alone) and note that the subscription ack deadline must exceed the handler's processing timeout.
- Flow-control / subscription configuration questions
(
- Non-Go languages and other SQL engines: Phases 1, 3, and 4 provide
compiled Go (
cloud.google.com/go/pubsub/v2,database/sql, OpenTelemetry Go) and PostgreSQL reference code, while Phase 2 maps flow-control fields across Go, Python, Java, and Node.js. When the user's workload uses another language (such as Java, Python, or Node.js) or database (such as MySQL or Oracle), keep every architectural invariant and phase gate unchanged, translate the patterns into idiomatic SDK and SQL constructs for the requested stack, and state explicitly that you are adapting the Go reference patterns.
Prerequisites
Use the Google Cloud CLI (install guide:
https://docs.cloud.google.com/sdk/docs/install-sdk.md.txt) and, for GKE,
kubectl. Never install host tools or grant IAM bindings without explicit user
approval: both alter the user's host or production access. If gcloud is
unavailable, produce design artifacts, code, Kubernetes YAML, or Terraform
directly.
-
Billing is attached to
{project_id}. Pub/Sub, Cloud Trace, and Cloud Logging have free allotments, but an active billing account is still required (https://cloud.google.com/pricing, https://cloud.google.com/pubsub/pricing, https://cloud.google.com/run/pricing, https://cloud.google.com/kubernetes-engine/pricing, https://cloud.google.com/stackdriver/pricing). -
Project, region, authentication, and APIs (default
us-central1, https://cloud.google.com/about/locations): -
Least-privilege IAM: one service account per producer and consumer; never
roles/editor(why: basic roles grant project-wide write across every service).
End-to-end workflow and phase gates
- Phase 0 - Frame the goal: capture the business outcome (independent scaling, isolated failure, shorter p99, removable batch windows) and constraints (schema freeze, language, runtime GKE vs Cloud Run, data residency) as measurable acceptance criteria before designing.
- Phase 1 - Assess (phase1-coupling-assessment.md [blocked],
phase1-outbox-and-bulkheads-go.md [blocked]).
Gate: baseline p99 latency, connection saturation, and lock-wait
numbers recorded per hop; every outbound event has a name, owning team,
idempotency key, and Transactional Outbox vs state-marker decision; zero
network calls remain inside a database transaction. Gate enforcement:
if the user claims assessment is finished and asks for
gcloudtopic or subscription creation commands while only naming events without confirming those Phase 1 gate items, do not emitgcloud pubsub topics createorsubscriptions createcommands and do not invent payload schemas or owners; ask to confirm the missing Phase 1 gate items first and state that Phase 2 (references/phase2-pubsub-topology-and-sizing.md [blocked]) is the next step once the gate is met. - Phase 2 - Design messaging (phase2-pubsub-topology-and-sizing.md [blocked]). Gate: per-replica flow control derived from the database pool; dead-letter topic paired with its own pull subscription and both Pub/Sub service-agent IAM bindings; Cloud Storage claim-check decision for any payload that can exceed 1 MB (Pub/Sub hard limit: 10 MB).
- Phase 3 - Build consumers (phase3-consumer-and-autoscaling.md [blocked]).
Gate: unit tests for every branch of the ack/nack/quarantine matrix;
shutdown test proving the database pool closes strictly after
Receivereturns and in-flight commits finish; GKE HPAmaxReplicasor Cloud Run concurrency capped within the database connection budget. - Phase 4 - Observe (phase4-distributed-tracing.md [blocked]).
Gate: one continuous trace ID from producer request through Pub/Sub to
consumer database stages, structured logs correlated via
logging.googleapis.com/trace, explicit sampling policy per environment. - Phase 5 - Cut over incrementally: Strangler pattern per event type.
Dual-run (synchronous path plus event) behind
{feature_flag}so a single flip restores the synchronous call, compare outcomes, and remove the synchronous call only after{dlq_subscription_id}stays empty for{soak_period}(default: one full business cycle, at least 7 days). - Validate → fix → repeat: re-verify all phase gates post-cut-over using solution-checklist.md [blocked].
Cross-cutting rules and scope boundaries
- One default, one escape hatch per decision: every phase names one default and one escape hatch; do not add a third option.
- Idempotency is non-negotiable: Pub/Sub delivers at least once. Every consumer deduplicates on an event ID or natural key before side effects and acks duplicates (nacking duplicates loops valid data into the DLQ).
- Budgets flow downstream: database connection budget → per-replica pool
→
MaxOutstandingMessages→ HPAmaxReplicas. Enforce the invariantper_replica_pool * max_replicas <= database_connection_budget(derive whichever of the three values is unknown, tie HPAmaxReplicasto that budget, never leave client flow-control defaults of 1000 messages / 1 GiB in place, and derive per-replica flow control with Phase 2 (references/phase2-pubsub-topology-and-sizing.md [blocked]) and the HPA manifest with Phase 3 (references/phase3-consumer-and-autoscaling.md [blocked])). - Never leave a dead-letter topic without a subscription: it silently
discards messages; alert on
subscription/dead_letter_message_count. - Other brokers (Kafka, RabbitMQ, SQS, self-managed queues): state
plainly that this skill is specific to Google Cloud Pub/Sub, offer only
broker-agnostic principles explicitly labeled as such (or redirect to
broker-specific docs), never present Pub/Sub flags, flow-control settings,
or fabricated
gcloudcommands as Kafka config, and never suggest migrating to Pub/Sub as the answer to a Kafka tuning question. - Synchronous HTTP/gRPC-only services (no Pub/Sub): load only Phase 4
(
TracerProvider+ Cloud RunWithRemoteParentNotSampledsampler,otelhttp,otelgrpcstats handlers, DB stage spans,slogcorrelation); never introduce Pub/Sub topics or message carriers. - Querying existing traces ("find traces slower than 2s"): state that
this is a Cloud Trace query task rather than instrumentation; point to the
Cloud Trace console/API
(https://docs.cloud.google.com/trace/docs/finding-traces.md.txt) or the
cloud-trace-queryingskill. Never emit OpenTelemetry SDK setup code, never require re-instrumenting first, and never invent a trace query DSL. - Orchestration engines & sagas: Workflows, Managed Service for Apache
Airflow (
managed-airflow-*), Application Integration (application-integration-basics), and multi-service compensating sagas are out of scope; keep each aggregate's state change in one transaction.
Validation checklist
- Producer and consumer deploy, scale, and fail independently: with the
consumer stopped, producer p99 is unchanged,
num_undelivered_messagesgrows, and the backlog drains on restart with matching row counts. - No dual writes: a fault injected between commit and publish is recovered by the outbox relay or sweeper with zero duplicate side effects.
- A malformed message lands in
{dlq_subscription_id}aftermax-delivery-attempts(or in quarantine when chosen) while healthy messages keep flowing (subscription/dead_letter_message_count> 0). - Rolling consumer restarts under load produce zero
context canceledcommits and zero duplicate side effects. - One trace ID links producer request, Pub/Sub hop, consumer processing,
and database stages, with
logging.googleapis.com/traceon log lines. - Every acceptance criterion names an observable signal (metric, log field, row count, or test assertion), never a qualitative claim like "services are decoupled".
Resource cleanup
Remove only temporary test or dual-run resources in a non-production
project. Before running or proposing any delete command after a migration,
explicitly ask the user to confirm the exact test resource names and
{test_project_id}, distinguish test/dual-run resources from live production
topics and subscriptions, and never delete production resources carrying
live traffic:


