Operations
Live Dashboard
Live Dashboard
- Status: active observer and operator reference
- Owner: ClawSweeper maintainers and the designated Cloudflare operator
- Source of truth:
dashboard/worker.ts,dashboard/exact-review-queue.ts, - Last verified:
openclaw/clawsweeper@647503ec44b8e777dd172adf974a945367da0d19 - Update when: routes, public fields, queue projections, capacity, alerts,
dashboard/wrangler.toml, dashboard tests, and deployed read-only endpoints
deployment, or state-writer telemetry changes
Read when changing the Cloudflare status dashboard, status ingest contract, or operator-facing ClawSweeper observability.
The signed internal canonical-record export route serves up to 200 records per page (also the Worker and hydration-client default), bounded by 2 MiB of source content and 4 MiB of serialized entries. Large-record pages stop early and resume from the last returned record; the existing first-record progress exception remains. See canonical record export limits.
The live dashboard is observer-only. ClawSweeper still owns review, repair, apply, merge, comments, labels, and all GitHub mutations. The Cloudflare Worker reads GitHub workflow state, projects it into closed status views, and optionally accepts signed status events from workflows. The one identity exception is a bounded reference sample containing canonical repository and issue or pull-request numbers from the explicit verified-public PUBLIC_BAY_REPOS allowlist. Overview uses that sample for cards and search; clicking a card opens a local blade with the closed stage/source, canonical repository and issue/PR links, and, when available, canonical GitHub run/job links plus fixed action-step categories and states. Public responses still exclude workflow, item, and step titles, raw or source URLs, queries, raw failure keys and payloads, internal opaque keys, credentials, tokens, private or non-allowlisted repositories, and per-job diagnostic text.
Queue transport failures keep the fixed public exact_review_queue_unavailable response. Server logs for exact_review_queue_request_failed retain only the Cloudflare remote, retryable, and overloaded boolean flags; exception text, stacks, request payloads, and credentials are excluded. These flags are diagnostic signals and do not change retry or publication policy. Inside the Durable Object, exact_review_queue_handler_failed additionally records initialize or fetch and the first numeric [line, column] location in the deployed worker.js module (or null when unavailable). Match coordinates to that deployment’s bundle; remote transport replaces the original stack before the outer Worker sees it. These logs exclude error messages, SQL, private paths, and raw stacks.
#Deployment
Cloudflare account:
- account:
Services@openclaw.org - account id:
91b59577e757131d68d55a471fe32aca - zone:
openclaw.ai
Worker:
- name:
clawsweeper-status - current deployment:
https://clawsweeper.openclaw.ai/ - fallback workers.dev deployment:
https://clawsweeper-status.services-91b.workers.dev/ - machine ingest:
https://clawsweeper.openclaw.ai/api/events
Deploy with the OpenClaw Cloudflare token:
source ~/.profile
CLOUDFLARE_ACCOUNT_ID="$OPENCLAW_CLOUDFLARE_ACCOUNT_ID" \
CLOUDFLARE_API_TOKEN="$OPENCLAW_CLOUDFLARE_API_TOKEN" \
pnpm run dashboard:deploy
GitHub deploys use .github/workflows/dashboard.yml. Configure either OPENCLAW_CLOUDFLARE_WORKERS_API_TOKEN or OPENCLAW_CLOUDFLARE_API_TOKEN with Workers Scripts edit permission before enabling the workflow as the production deploy path. The deploy workflow injects the CLAWSWEEPER_STATUS_INGEST_TOKEN GitHub secret into a temporary Wrangler config as the Worker INGEST_TOKEN. The smoke test waits for the expected deployment revision and a successful exact-review queue response within the same default 600-second readiness budget. It then waits up to another 600 seconds for the signed review-admission response to report that same deployment revision from the Durable Object itself. The queue Durable Object can keep serving the previous version for several minutes after a deploy. A healthy outer Worker or a successful response from an older queue is not deployment readiness. The smoke also verifies the capability contract, queue schema, unsigned-request rejection, and Overview rendering. An exact-revision smoke run requires the existing CLAWSWEEPER_WEBHOOK_SECRET; it sends a signed empty JSON object without following redirects and validates scheduled pacing, replay, and manual publication policy. A valid disabled manual-publication policy passes. Local smoke runs without CLAWSWEEPER_EXPECTED_DEPLOY_SHA explicitly report this signed check as skipped. A persistently unavailable queue still fails readiness.
A status contract failure keeps its original error and nonzero smoke exit, with one bounded diagnostic summary: projection completeness, freshness, cache state, validated numeric diagnostic count, and fetch duration. It excludes response text, timestamps, and diagnostic error messages. These are observed fields; an unavailable projection does not identify which upstream request or cache entry caused it. The smoke adds no status retry.
When a change updates both the Worker and a GitHub Actions workflow, keep the cross-component protocol compatible in both deployment orders. The exact-review v2 rollout dispatches the immutable lease tuple under queue_claim plus a bounded v1 snapshot; the Worker accepts v1 claims/finalizers while the workflow can consume either v1 or v2 claim responses. Deploying the reviewed Worker first remains the preferred order, but this rollout does not require disabling or draining ClawSweeper:
gh workflow run dashboard.yml --repo openclaw/clawsweeper --ref <reviewed-branch>
gh api "repos/openclaw/clawsweeper/actions/workflows/dashboard.yml/runs?per_page=1" \
--jq '.workflow_runs[0] | {id, status, conclusion, html_url}'
#Access Model
The intended browser reader policy is Cloudflare Access with GitHub login restricted to the openclaw organization. The dashboard Worker does not implement GitHub OAuth itself. Keep auth at the Cloudflare edge, but do not use Access as the privacy boundary: every public status and observability response must remain safe when treated as public.
The current local Services token can identify the account, but cannot deploy the Worker or edit Cloudflare Access/DNS. Add the Workers deploy secret, the openclaw.ai routes, and the Access policy after the Services token has Workers Scripts edit, Zone DNS/route, and Zero Trust Access permissions.
Workflow events are sent with a bearer secret without a browser login. Ingest requires the INGEST_TOKEN Worker secret. Events and CI status persist only through the private STATUS_STORE binding. If that binding is absent, ingest remains available but the Worker deliberately skips persistence; raw event and item metadata never falls back to the shared edge cache.
curl -X POST https://clawsweeper.openclaw.ai/api/events \
-H "Authorization: Bearer $CLAWSWEEPER_STATUS_INGEST_TOKEN" \
-H "Content-Type: application/json" \
--data '{"event_type":"status.test","mode":"e2e","stage":"probe","status":"ok"}'
#CI Status
The dashboard does not fan out from the browser to GitHub check APIs. Active pipeline rows use the ClawSweeper workflow run status as an immediate fallback, then .github/workflows/dashboard-ci.yml refreshes target pull request check state and posts compact ci.status events into KV:
CLAWSWEEPER_STATUS_URL=https://clawsweeper.openclaw.ai \
CLAWSWEEPER_STATUS_INGEST_TOKEN=... \
GITHUB_TOKEN=... \
pnpm run dashboard:refresh-ci
The UI renders run pending/green/red until stored target checks arrive, then switches to checks pending/green/red with failing/pending/total counts. CI snapshots expire after two hours so old PR head state does not stick to fresh pipeline rows. Production also enables a bounded live fallback for the first few active PR rows so visible rows do not remain on workflow-only status when KV is absent or a cache event lands in another Cloudflare colo.
#Workflow health source
Workflow health uses bounded GitHub run/job polling exclusively. The retired workflow-source setting is no longer read. Workflow read-model reads, repairs, and missing-subscription warnings are not part of dashboard health, even when the shared webhook signing secret is configured. Snapshot and job-cache TTLs, pagination, freshness, genuine GitHub errors, and the public status contract are unchanged.
#Operator dashboard pages
The operator dashboard at / (Overview, Agent performance, Throughput, Review assessments, Queue, Operations, Coverage, Repositories) is a static Astro site in dashboard-site/. The Dashboard site workflow (.github/workflows/dashboard-site.yml) rebuilds it every 15 minutes and publishes it to the Cloudflare Pages project clawsweeper-dashboard (https://clawsweeper-dashboard.pages.dev). This Worker serves those page paths and their assets from Pages for GET and HEAD (dashboard/dashboard-site.ts; override the origin with DASHBOARD_SITE_ORIGIN). A data refresh never redeploys this Worker, so webhooks, the queue Durable Object and /api/* are not restarted by it.
Each build reads four sources:
- the corpus aggregate (totals, decisions, quality, closes, per-repository
- the public
/api/exact-review-queueand/api/review-coverageresponses; - the public 7-day
/api/health-historysamples and the 24-hour and 7-day - GitHub Actions run history for this repository, kept as per-hour buckets in
counts), cached in the workflow state. It is rebuilt twice a day by the first scheduled build in the 00 or 12 UTC hour (the quietest review hours) that finds the corpus over 11 hours old, by any scheduled build that finds it over 13 hours old (GitHub delays and drops scheduled runs), and when the workflow is started by hand with refresh_corpus: that run hydrates every canonical record read-only with scripts/hydrate-state.ts, reads front matter only and keeps repositories that GitHub reports as public. The cache keeps each repository's contribution apart, so every build checks visibility again, drops a repository that is no longer public and recomputes every total without it; a failed check fails the build, so nothing is published. A close counts as a ClawSweeper close only when the record has a close decision and no GitHub close time (current_item_closed_at), which apply writes only when it archives an item that someone else closed. The record export runs inside the exact-review queue Durable Object and slows reviews while it runs (a full export on 2026-10-12 failed 26 review runs in 14 minutes; after the queue fixes, the 2026-10-14 and 2026-10-15 refreshes saw 1 failure in 100 review runs), so the schedule uses the quietest hours. Until the first such run, scheduled builds skip with a notice;
/api/apply-observability summaries, for the Operations page. Samples are reduced to hourly points (counter growth between consecutive 5-minute samples; a counter reset or a sample gap leaves that step out) and to handoff incidents (degraded or stalled runs of 10 minutes or more; shorter runs count as blips; an idle or healthy sample ends a run). The healthy share counts only observed samples, and a value the APIs do not report shows as unavailable, never as zero. The apply figures cover only the scheduled close sweep (apply-existing), not closes made right after a review;
the workflow cache. Each run rereads from three hours before its last refresh, so a late or failed build leaves no gap; a missing cache bootstraps two days with the ClawSweeper app token, and the window then grows to 30 days. A manual run with backfill_days fills older days of sweep history: success/failure counts from the API's per-conclusion totals, and review-run durations from a sample of about 600 runs per day (the newest 100 in each 4-hour slice). About 20 requests per day, on this repository's GITHUB_TOKEN, never the app token, whose rate limit production reviews share.
Open pages check for a newer build every five minutes and when the tab becomes visible, and reload when one is published. The header shows both the build time and the review-corpus time. A page warns when its build is more than an hour old (the 15-minute rebuild has stopped) or when the corpus is more than 14 hours old (the scheduled refresh has stopped). Unknown paths return the site's 404 page with status 404. Pull requests that touch dashboard-site/ build the site from dashboard-site/fixtures/dashboard.json, without secrets or publishing.
Metric labels distinguish latest-record snapshots from event history. Items refreshed count each item once on its latest review date; re-review moves that date and does not preserve historical daily throughput. Maintainer-authored items describes the item author, not who performed a review. Activity deltas are neutral. Model time and creation-to-latest-review age show sample counts across the stored corpus; neither measures first response or end-to-end service latency. Sweep run success shows its numerator, denominator and recorded days in the requested seven-day window; period comparison requires all seven days in both windows. Execution success does not establish decision correctness.
Each detailed chart has one page owner. Agent performance owns execution timing, failed attempts by stage and repeated-claim investigation. Its run-time chart shows daily p50 and p90 together in duration units; these are run timings, separate from the model-only latest-review snapshot. Operations owns review flow, publication backlog, actual deliveries and batch handoff. Throughput owns volume and workflow activity; Coverage owns freshness. Overview keeps compact summaries and links.
Publication pending excludes superseded revisions retained for command acknowledgements; those records do not establish failed delivery. Repeated-claim alerts count recorded claims, which may end before model execution. The bounded public target sample links to GitHub for revision and trigger investigation; it is not a ranking or proof of wasted execution.
The site shows no report bodies, author identities or free-text concerns. Local preview: in dashboard-site/, install dependencies, run dashboard-site/scripts/build-data.ts with GITHUB_TOKEN, --records <records dir> and --state <state dir>, then start the Astro dev server.
#Review usage metrics
POST /internal/exact-review/review-usage accepts schema-1 review-attempt summaries signed over the exact request body with CLAWSWEEPER_WEBHOOK_SECRET in x-clawsweeper-exact-review-signature (sha256=<hex HMAC>). The Worker validates the payload and writes one Analytics Engine point directly to REVIEW_USAGE, dataset clawsweeper_review_usage; it does not access either Durable Object. A valid write returns 202, malformed or invalid data returns 400, invalid signatures return 401, and an absent dataset binding returns 503 with review_usage_unconfigured. No model name, prompt, response, or raw error is stored. This private operator dataset adds no public API or browser integration.
The payload contains schema: 1, target_repo, positive item_number, item_kind (issue or pull_request), decimal-string run_id from GITHUB_RUN_ID, positive numeric run_attempt, optional trigger_lane, service_tier, and reasoning_effort, outcome (ok, failed, or timeout), nonnegative integer duration_ms, and usage. Repository owner/name are bounded to 39/100 ASCII characters, optional labels to 64 ASCII letters, digits, underscores, dots, or hyphens, and run IDs to 32 digits. Unknown fields, including model names, are rejected. usage is either null or an object with four required nonnegative safe integer counters: input_tokens, cached_input_tokens, output_tokens, and reasoning_output_tokens. cache_write_input_tokens is a nonnegative safe integer or null. The receiver accepts unknown cache writes without treating them as a measured zero. The current collector remains strict: a missing or invalid counter makes its whole usage null. For rollouts that omit cache writes, preserving their four measured counters requires the separate producer rollout below.
The source-to-ingest flow has two separate steps:
- The review runtime synchronously appends token-only review-attempt records to the JSONL
- Immediately after review, the trusted
Submit review usageworkflow step runs
file named by CLAWSWEEPER_REVIEW_USAGE_PATH, configured in the workflow as ${{ runner.temp }}/clawsweeper-review-usage.jsonl. Records contain the bounded metadata above and the five counter fields below (or null usage), never turn counts, prompts, responses, or model names. The append completes before a failure or timeout can exit the review process. The review step receives neither CLAWSWEEPER_WEBHOOK_SECRET nor EXACT_REVIEW_QUEUE_URL.
with if: always() and continue-on-error: true. If the file exists, node dist/review-usage-submit.js "$CLAWSWEEPER_REVIEW_USAGE_PATH" validates its records and submits signed requests using EXACT_REVIEW_QUEUE_URL and CLAWSWEEPER_WEBHOOK_SECRET, which are scoped to this trusted step. Collection and submission are best effort and do not change the review result.
Review output handling is unchanged. Usage comes from $CODEX_HOME/sessions/YYYY/MM/DD/rollout-*.jsonl: the last event_msg / token_count event's info.total_token_usage is cumulative per session, so only that total is used and separate session files are summed. Files must have modification times within the captured review-process start/end window. The hosted setup isolates CODEX_HOME by run ID/attempt, and each review process reviews items sequentially. The reader scans backward in 64 KiB chunks to find each session's latest token count. Missing homes/files, unreadable or changing files, malformed usage, and files larger than 64 MiB produce null usage rather than a partial sum. Receiver compatibility does not change that collection behavior or begin sending nullable cache writes.
Roll out the shared validator, Worker writer, and updated queries before the separate producer change starts sending nullable cache writes. Qualify the deployed receiver with legacy, nullable, and missing-usage records and verify their persisted columns and the weighted queries; HTTP acceptance alone is not persistence proof. The schema remains 1 and older five-number payloads remain valid. Roll back the producer before the receiver, and drain pending nullable handoffs before reverting receiver support. Keep the updated mask-aware queries for the lifetime of already-written partial rows; producer rollback does not remove those rows. A legacy query must explicitly restrict its input to double7 = 1. Existing complete rows have double7 = 1 but no double8; queries must recognize either presence flag when selecting measured core usage. This is a source rollout order, not a deployment performed by this change. Preserve the selected base revision’s review-admission policy and raw admission guards throughout the rollout. Do not enable the producer until receiver and query qualification is complete.
REVIEW_USAGE_LAYOUT in dashboard/review-usage.ts is the canonical source of the ordered tuples used by the writer. Analytics Engine positions are one-based within each tuple:
| Columns | Ordered values |
|---|---|
index1 | target_repo |
blob1–blob8 | target_repo, item_kind, outcome, trigger_lane (default empty), service_tier (default standard, including empty input), reasoning_effort (default empty), run_id, string item_number |
double1–double8 | input_tokens, cached_input_tokens, cache_write_input_tokens, output_tokens, reasoning_output_tokens, duration_ms, usage_present, core_usage_present |
double7 is 1 only when all five counters are known. double8 is 1 when the four core counters are known, including when cache writes are unknown. Unknown cache writes use zero in the numeric column with double7 = 0, double8 = 1; that stored zero is not a measured zero. Null usage zeros the token columns and both presence flags but preserves duration. input_tokens includes cached reads and cache writes; output_tokens includes reasoning output. Neither subset should be counted twice. Run attempt is validated in the payload but is not a separate dataset column. Points represent review attempts, not unique issues or PRs; the route does not deduplicate repeated submissions.
#Querying daily tokens and estimated cost
Use the Analytics Engine SQL API with an operator token permitted to read account analytics:
curl "https://api.cloudflare.com/client/v4/accounts/91b59577e757131d68d55a471fe32aca/analytics_engine/sql" \
-H "Authorization: Bearer $CLOUDFLARE_API_TOKEN" \
--data-binary @review-usage.sql
Save this query as review-usage.sql. Multiply sums by _sample_interval to account for Analytics Engine sampling. The estimate uses $2/M uncached input, $0.10/M cached input, $2.50/M cache writes, and $10/M output tokens:
SELECT
toStartOfInterval(timestamp, INTERVAL '1' DAY) AS day,
blob1 AS repo,
sum(_sample_interval) AS reviews,
sum(double7 * _sample_interval) AS reviews_with_full_usage,
sum(if(double7 = 0 AND double8 = 1, _sample_interval, 0)) AS reviews_with_partial_usage,
sum(if(double7 = 0 AND double8 = 0, _sample_interval, 0)) AS reviews_without_usage,
sum(double1 * _sample_interval) AS input_tokens,
sum(double2 * _sample_interval) AS cached_input_tokens,
sum(double3 * _sample_interval) AS known_cache_write_input_tokens,
sum(double4 * _sample_interval) AS output_tokens,
sum(double5 * _sample_interval) AS reasoning_output_tokens,
sum(((double1 - double2 - double3) * 2
+ double2 * 0.10 + double3 * 2.50 + double4 * 10)
* double7 * _sample_interval) / 1000000 AS full_usage_estimated_usd
FROM clawsweeper_review_usage
WHERE timestamp >= NOW() - INTERVAL '30' DAY
AND blob1 != 'openclaw/__review_usage_canary__'
AND blob4 != 'operator-canary'
GROUP BY day, repo
ORDER BY day, repo
This is a reference-price estimate, not an invoice: service-tier adjustments and the greater-than-272K per-request surcharge are not included. These are review sums, so they cannot determine whether an individual request crossed that threshold. Cost covers only the full-usage cohort; partial and missing attempts have unknown cost and are not estimated as free. Core token sums include both full and partial usage. Known cache writes cover only full usage. Compare the weighted cohort counts before interpreting totals.
#Per-review duration and average tokens by lane/repository
This includes failed and timed-out attempts in duration percentiles. Average tokens use attempts with measured core usage, including older complete rows. Groups without measured core usage are omitted. Duration percentiles still cover all attempts in those groups. The blank lane means no lane was provided, not an inferred workflow lane.
SELECT
blob1 AS repo,
blob4 AS lane,
sum(_sample_interval) AS reviews,
sum(if(double7 = 1 OR double8 = 1, _sample_interval, 0)) AS reviews_with_core_usage,
quantileExactWeighted(0.50)(double6, _sample_interval) AS duration_p50_ms,
quantileExactWeighted(0.95)(double6, _sample_interval) AS duration_p95_ms,
sum(double1 * _sample_interval)
/ sum(if(double7 = 1 OR double8 = 1, _sample_interval, 0)) AS avg_input_tokens,
sum(double4 * _sample_interval)
/ sum(if(double7 = 1 OR double8 = 1, _sample_interval, 0)) AS avg_output_tokens,
sum((double1 + double4) * _sample_interval)
/ sum(if(double7 = 1 OR double8 = 1, _sample_interval, 0)) AS avg_total_tokens
FROM clawsweeper_review_usage
WHERE timestamp >= NOW() - INTERVAL '30' DAY
AND blob1 != 'openclaw/__review_usage_canary__'
AND blob4 != 'operator-canary'
GROUP BY repo, lane
HAVING sum(if(double7 = 1 OR double8 = 1, _sample_interval, 0)) > 0
ORDER BY repo, lane
See the sampling guidance and weighted aggregate functions for interpreting approximate aggregates.
#Bounded private operator workflow
- Status: active operator interface; live permission and deployed-receiver qualification remain separate gates.
- Owner: ClawSweeper maintainers.
- Source: maintenance workflow,
- Verified base:
104113c2b611525c4103a3e3fb6900cfaea72a8f. Update this contract when the workflow inputs, fixed resources,
operator helper, and its focused CLI tests.
request limits, credential reference, Analytics layout, or receiver contract changes.
The existing maintenance workflow has three private usage modes. They do not change existing queue/storage modes, review policy, or the one-time PR 2002 checkpoint. The public dashboard remains observer-only. This consumer does not match any push path in the dashboard deployment workflow; merging only these operator files does not deploy the Worker.
Use only a reviewed consumer landed on canonical main. Each job checks out its immutable workflow event SHA with persisted Git credentials disabled. Record that source SHA separately from the deployed Worker source SHA and version UUID. A squash changes commit identity: rebind the reviewed source tree to the landed commit, and obtain exact-head checks before dispatch. The consumer verifies Worker source through the approved version's resources.bindings and current settings' bindings; the latest deployment must contain exactly that version at 100%. These are point-in-time guards, not a deployment lock. Coordinate against concurrent deployments; any drift before or after the operation fails qualification.
Only read jobs receive OPENCLAW_CLOUDFLARE_WORKERS_API_TOKEN as CLOUDFLARE_API_TOKEN, the existing Foundation credential reference used for this Worker/account. No general-token fallback exists. This reference does not establish effective permissions: version content/bindings access and Account Analytics Read are independently required. Missing credentials, HTTP 401/403, Analytics denial, unexpected shapes, redirects and limits stop with a redacted error. Do not request additional access or switch credentials automatically. Read jobs receive no HMAC secret. The canary job receives only the existing CLAWSWEEPER_WEBHOOK_SECRET and native read-job identity outputs.
All requests use the fixed account, clawsweeper-status Worker, clawsweeper_review_usage dataset and canonical usage route. There is no production URL, SQL, resource, credential or command override. Each request has a 30-second deadline and rejects redirects. Streaming response limits are 256 KiB for metadata and 64 KiB for Analytics/ingestion; each invocation runs at most three fixed SELECTs with LIMIT 8 FORMAT JSON. Metadata guards run before and after SELECTs. Only approved identity and numeric result fields are printed; plaintext bindings, API errors and credential values never appear. UInt64 decimal strings are normalized only within JavaScript's safe numeric range.
The modes are:
operator-read: metadata guards plus fixed weighted cohort, duration andcanary: a read guard job runs first. Bothexecute=trueandread-back: a separately approved original canary run ID selects the four
historical-full-row SELECTs over the last 30 days. It performs no ingestion.
authorize_usage_canary=true are required for the separate write job; workflow run attempt must be 1. Exactly four fixed schema-1 records cover null usage, unknown cache writes, measured zero and legacy five-number usage. Each uses duration 100 ms, the workflow run ID and run attempt 1. The write job signs the exact body and requires HTTP 202 plus ok: true. Another isolated read job rechecks the deployment after success or failure.
records within the last day. Weighted counts, all five counters, both masks and each record’s 100 ms duration must match; missing, duplicate, sampled or corrupted evidence fails. The same aggregate SELECT checks full/partial/missing/core cohorts, full-mask reference cost and input/output/total averages over measured core usage. Actual curried quantileExactWeighted SQL executes, plus a 30-day historical query restricted to double7 = 1 AND double8 = 0. No writes occur.
Before any canary, both global queries above and the helper's global SELECTs exclude repository openclaw/__review_usage_canary__ and lane operator-canary. The predicates exclude a row when either identity matches; reserving a repository alone would contaminate other aggregations. Focused SQLite predicate proof verifies both documented queries with ordinary, repository-only, lane-only and combined synthetic identities. These synthetic records do not represent GitHub items and trigger no reviews or model runs.
The fixed payloads use input/cached/cache-write/output/reasoning counts of null usage; 100/20/unknown/30/10; 0/0/0/0/0; and 100/20/5/30/10. Expected weighted counts are 4 attempts, 2 full, 1 partial, 1 missing and 3 with core usage. Input/output totals are 200/60; core averages are 200/3 input, 20 output and 260/3 total tokens. Full-only reference cost is $0.0004645 at the prices above, not a billing claim. These 100 ms fixtures do not use earlier illustrative payload quantities. Per-item duration checks reject a corrupted duration even if both aggregate percentiles remain 100.
No write retry or idempotency assumption exists. A failure reports attempted and acknowledged counts, then stops. A timed-out request may already have written. Preserve the original run ID and inspect it through read-back; do not rerun the canary job or create replacement writes. HTTP 202 establishes only acceptance. A successful read-back establishes bounded persistence, masks, counters and SQL execution. Constant-duration canaries do not establish nontrivial weighted-quantile or sampling correctness. An empty historical cohort establishes query execution only, not observed historical compatibility.
#Staged dispatch packet
These are future operator commands, subject to exact-head review and explicit stage authorization. No workflow dispatch, receiver merge/deploy or canary is a side effect of preparing this PR.
- Verify the reviewed consumer's landed source/tree, canonical main SHA and
- After the trusted consumer lands and its named credential route is verified,
exact-head checks. This PR's files do not trigger Worker deployment. Leave receiver PR 2011 separate; its future merge matches dashboard/** and can automatically deploy the entire Worker. Rebind its squash tree/source and qualify that whole deployment before selecting its expected SHA/version.
authorize a read-only dispatch with the approved current deployment identity:
``bash ghx workflow run exact-review-queue-maintenance.yml --repo openclaw/clawsweeper --ref main -f mode=operator-read -f usage_deploy_sha=APPROVED_DEPLOY_SHA -f usage_version_id=APPROVED_VERSION_UUID ``
- Inspect the run's immutable workflow source SHA, approved deployment identity,
- Only after separately authorized receiver deployment and source/deployment
before/after guards, numeric outputs and terminal conclusion. Permission failure is a stop, not authorization to try a different credential.
qualification, approve one post-receiver canary run:
``bash ghx workflow run exact-review-queue-maintenance.yml --repo openclaw/clawsweeper --ref main -f mode=canary -f execute=true -f authorize_usage_canary=true -f usage_deploy_sha=APPROVED_RECEIVER_DEPLOY_SHA -f usage_version_id=APPROVED_RECEIVER_VERSION_UUID ``
- Keep the original run ID even after partial failure. Authorize read-back
once that data is queryable; this consumer has no polling or automatic retry:
``bash ghx workflow run exact-review-queue-maintenance.yml --repo openclaw/clawsweeper --ref main -f mode=read-back -f usage_deploy_sha=APPROVED_RECEIVER_DEPLOY_SHA -f usage_version_id=APPROVED_RECEIVER_VERSION_UUID -f usage_canary_run_id=APPROVED_ORIGINAL_RUN_ID ``
Nullable writes require the receiver capability from PR 2011, independently qualified on its actual deployed identity. Stop/drain nullable production before any receiver downgrade. Keep mask-aware queries while partial rows remain retained; legacy-only consumers must explicitly filter double7 = 1.
#What the status API shows
- bounded active-work counts and closed workflow, worker, status, stage, and
- six aggregate activity stages from arrival through repair, split into disjoint
- a budget-sized capacity rail plus aggregate counts for review, commit, assist,
- queued and waiting run counts
- operational health derived from queue age and running age: queued runs from
- 24-hour and seven-day health trends for total queue depth, over-age queue
- aggregate worker-attempt error, recovery, and unresolved-failure counts,
- active pipeline rows grouped as exact review, hot review, apply, or background
- CI state for active PR rows when available
- explicit workflow status events posted to the ingest API when KV ingest is
- global apply-lane queue, result, lease, and closed failure-category counts;
- lane-level apply health in status JSON so closure processing and durable
- skip next-action buckets in apply health JSON so stale reviews, missing close
- scheduled close-cycle telemetry in apply-health JSON, including current
- fleet-wide review coverage and review-lane health for normalized time ranges;
- exact-review queue backlog, retry-ready backlog, target-admissible backlog,
outcome categories; arbitrary titles, targets, source links, keys, and job text are omitted
queued and live counts only when both producer censuses are complete, plus a bounded verified-public repository/item/action reference sample used by Overview cards, search, overflow lists, and client-side public-reference blades; sampled queue and live references may additionally carry one validated queue-start or public action-start clock with its closed kind
and other worker classes
30 through 1440 minutes old degrade operational health; queued runs older than 1440 minutes are reported separately as zombies; pre-queue pending reruns older than 60 minutes are reported separately as wedged; approval-gated runs are also reported separately; in-progress runs become stalled after 150 minutes
depth, and the oldest queued/running ages
including failures hidden by workflow continue-on-error
review
enabled
repository inventory, run links, and failure text remain private
review-comment sync are reported separately even when they share the same applicator
proof, protected labels, stable skips, invalid reports, and open closing PRs are discoverable without reading individual item records
apply-ready candidate count and an estimated number of cursor windows to revisit the close queue; scheduled cadence time is explanatory only because successful windows can dispatch immediate continuations
repository rows and identifying filters are not public
fixed backoff and parked reason counts, handoff health, and pressure from the current durable queue snapshot
The Worker fetches job details only for the bounded active-run set, limits that GitHub fanout to 12 concurrent requests, and caches each run's jobs for 60 seconds. It separately samples up to 40 recent completed worker runs with twenty-way fanout and caches error/recovery telemetry for 120 seconds. This bounds telemetry pressure independently of the 128-worker fleet budget. Worker details paginate up to 300 jobs per workflow run so retained large matrix runs contribute to a complete internal census. Titles, job names, raw URLs, opaque target keys, and raw errors are removed before the status snapshot is persisted or returned. Only the allowlisted canonical repository/item reference tuple and a validated action descriptor are retained for the bounded Overview sample. The action descriptor contains canonical repository/run/job identifiers, a canonical start time, and closed step kind/status/conclusion values; raw step names and URLs are discarded. If the census is incomplete, the public activity projection fails closed instead of presenting partial counts as complete.
Automatic issue-build lifecycle events are retained privately for seven days so completed and blocked work can be reconciled after the worker leaves the active Actions set. The public recent-publication route revalidates that state into bounded outcome counts without returning private repositories, revision identifiers, target keys, facts, titles, raw URLs, or failure detail.
Status responses use stale-while-revalidate delivery. After the configured fresh window expires (60 seconds in production), the Worker immediately returns the last good snapshot, marks it with X-ClawSweeper-Cache: stale, and coalesces one background refresh per isolate. With the production Durable Object binding, StatusStore owns both cached reads and publication. It atomically retains the snapshot with the newer collection timestamp; delayed older writes and equal-timestamp replays cannot replace it. Reads retain the original age. Existing edge bodies are bypassed, so an older edge write cannot undo the durable fence. This adds one Durable Object read per cached status request. Deployments without that binding retain the edge-cache path. The Worker applies the public status projector before publication and again on every cache/store read. Legacy cache bodies are reprojected, nested or unexpected fields are dropped, and stale, future-dated, or malformed store documents are rejected. Diagnostics retain a bounded error count and the fixed telemetry_unavailable category, never upstream exception or API error text.
#Public projection contract
Public observer routes validate a fixed response schema rather than forwarding their backing store. Unsupported identifying query parameters are ignored; a malformed or inconsistent backing document fails closed with a fixed unavailable response.
Run-level observer writes validate and retain the first terminal tuple per run attempt without reading queue items or rescheduling work. Their existing 30-day retention cleanup runs on telemetry writes and queue status computation; actual queue and auxiliary work retain ownership of alarm scheduling.
The removed Bay observer no longer materializes lifecycle events, tides, or StatusStore journey/terminal snapshots. Recent durable publication events still use direct and batch outcome telemetry, and Overview still uses the shared bay_projection queue sample. Neither read path requires Bay telemetry writes.
/api/review-observabilityreturns the four closed review lanes and global/api/review-coveragereturns fleet-wide inventory status and coverage/api/apply-observabilityreturns global queue, result, retry, lease, and/api/exact-review-queue/itemdoes not perform a public per-item lookup, and
health, completeness, run counts, item counts, and timestamps for a normalized 6h, 24h, or 7d range. It does not return repository filters or labels.
totals. Per-repository rows remain in the private queue store.
closed failure-category counts for a normalized range. It omits repository inventories, run links, and failure messages.
/api/exact-review-queue/reviews returns a stable empty aggregate envelope.
Rich workflow, queue, item, repository, session, revision, and failure records may remain in the queue and status Durable Objects when binding-authenticated workers need them. They must pass through the corresponding public projector before they reach a public response, edge cache, or ordinary status snapshot. The projector may retain only the narrow PUBLIC_BAY_REPOS reference tuple described above; this is not permission to expose arbitrary public-repository metadata or untrusted text.
#Exact Review history
A Cloudflare Cron Trigger records one Exact Review queue sample every five minutes. It reuses the queue status read that also performs scheduled queue maintenance and makes no GitHub Actions request. The sample contains each lane's pending backlog plus cumulative counts for newly enqueued and successfully completed work. Review samples also retain the cumulative shed count so overload demand remains visible even when no queue item was admitted.
Samples are stored in the existing StatusStore Durable Object under daily UTC keys named health-history:YYYY-MM-DD. Writes replace the current five-minute slot, making retries and overlapping triggers idempotent. Buckets expire after the seven-day retention window plus one day of boundary margin. No health history is written to openclaw/clawsweeper-state.
GET /api/health-history?range=6h returns the dashboard's default chart range; range=24h and range=7d return the longer windows. The endpoint still accepts and returns legacy operational samples, but new samples omit those unused chart fields. Existing buckets expire naturally; no migration or manual cleanup is required.
Each lane renders one signed net-rate value and curve: successfully completed work minus newly incoming work, expressed per hour. A positive rate means the lane is catching up, a negative rate means it is falling behind, and zero is balanced. Incoming counts newly created queue work units; review demand also includes shed recovery work. Pending merges, delivery replays, retries, and source-drift requeues do not create new demand. Completed work is counted only when a successful item actually leaves its lane, in the same queue storage transaction as the deletion. A help icon beside each net-rate label exposes this definition on hover, keyboard focus, or activation; the review explanation explicitly notes that incoming demand includes shed work.
After two continuous samples roughly five minutes apart, the dashboard scales the observed net change to an hourly rate and labels it provisional with the actual window length. Once an hour of continuous counters exists, it uses a trailing hourly rate. Zero incoming or zero completed work remains valid data. A gap over 12 minutes, a cumulative counter reset, or a legacy sample without flow counters starts a new rate segment; a latest rate point older than 12 minutes is stale. Pre-deployment counter history cannot be backfilled, so the first provisional rate appears about five minutes after deployment.
Operational health remains a current-snapshot alert rather than a historical chart. /api/status classifies the already-fetched active workflow runs as:
healthy: complete telemetry and no non-zombie over-age runs;degraded: at least one queued run is from 30 through 1440 minutes old;stalled: at least one in-progress run is 150 minutes old;unknown: one or more actionable-status reads failed.
Webhook workflow snapshots retain a per-run delivery-or-poll confirmation time. An over-threshold queued run whose confirmation is older than the five-minute workflow TTL is re-read through the exact GitHub run endpoint before it can degrade health. One refresh checks at most the ten oldest stale rows, matching two waves of the five-way request fanout within the 20-second refresh cadence. Omitted unconfirmed rows cannot contribute to queue pressure and make health unknown until later refreshes reach them; batch telemetry records both the selected and omitted counts. Completed or missing runs are removed from the snapshot and emit structured eviction telemetry; a failed recheck makes the snapshot unknown. Until workflow_run subscription coverage has actually been observed, repair-fed rows remain unusable and this path uses the same bounded live status polls as before the webhook read model. Runs beyond the existing 24-hour zombie boundary remain separately visible and do not spend exact verification requests.
Healthy status stays hidden. A non-zombie queued run at least 30 minutes old, an in-progress run at least 150 minutes old, or incomplete Actions telemetry opens the expandable “Work execution needs attention” alert. This live diagnostic reuses the status snapshot's Actions reads; the history cron no longer stores queue pressure or oldest-run values.
Active-run collection reads up to three pages of 100 runs for each active status. When counts or rows change during a census, it discards that attempt and reads the affected status again once, bypassing the refresh-local memoized pages. The ceiling is 30 page reads across five statuses, including fresh rebuilds. A full final page with additional runs, a failed page, or a second inconsistent census keeps operational telemetry unknown. A complete 100-run page does not itself make telemetry unavailable. The advisory ETag cache has a one-second deadline per operation; a stalled lookup, confirmation, or write falls back to the authoritative GitHub response without consuming the entire refresh deadline.
Queued runs older than 1440 minutes are reported separately as zombies and do not contribute to queued_over_threshold or oldest_queued_minutes; they therefore do not make an otherwise healthy snapshot degraded. The API exposes them as zombie_queued_runs and oldest_zombie_queued_minutes. Runs waiting on deployment approval are also excluded from queue pressure and exposed as approval_gated_runs and oldest_approval_gated_minutes.
Pre-queue pending reruns older than 60 minutes are also excluded from queue pressure because GitHub cannot cancel or rerun them. The API exposes aggregate counts and ages only, as wedged_rerun_runs and oldest_wedged_rerun_minutes; it does not add run identifiers to the public status payload.
#Boundaries
Do not move these into the dashboard:
- maintainer authorization
- PR branch writes
- labels/comments/closes/merges
- final merge safety gates
The dashboard Worker owns durable exact-review admission only: it deduplicates webhook deliveries, coalesces each repository/item pair, and leases at most 80 Actions executors, with up to 64 active leases per target repository. It does not decide review outcomes or perform target repository mutations. For command-triggered reviews, the queue retains the bounded review prompt and command-status identifiers so the leased GitHub Actions executor can update the original acknowledgement through completion. GitHub Actions remains the executor and the existing review/apply safety model remains unchanged.
The Worker's Durable Object bindings separate three storage owners: STATUS_STORE (StatusStore) holds dashboard state, EXACT_REVIEW_QUEUE (ExactReviewQueue) holds queue state, and GITHUB_ETAG_CACHE (GithubEtagCache) holds disposable GitHub response bodies and validators. The cache uses idFromName("owner/repo") from the store-validated, normalized /repos/ route, with repository names lowercased. Requests without a valid repository key use the single fallback shard and retain the existing validation errors. Credential pool, media type, query, and page remain part of the cache key within each shard. The 2,048-entry cap and 30-day retention now apply per shard.
The additive v3 migration creates only GithubEtagCache; existing classes, queue storage, leases, fences, and alarms are unchanged. Runners keep the same HMAC-authenticated /internal/exact-review/github-etag-cache/{lookup,store,confirm} routes and response contract. Both these routes and dashboard health reads use the new binding directly, with no queue fallback. Deploy the binding and code together through Wrangler. The namespace starts empty; misses use the existing GitHub fetch/revalidation path. Old queue cache tables are deliberately left inert: they are not read, migrated, recreated, or pruned by the new code, avoiding extra work in the overloaded queue. A rollback may reuse them, subject to the existing validator/digest checks; no cached body is served without confirmation.
Cache transport failures retain generated trace IDs and the existing github_etag_cache_lookup, github_etag_cache_store, and github_etag_cache_confirm endpoint labels, under github_etag_cache_* log events. Each shard's binding-only GET /stats returns its trailing 24-hour telemetry counters; no public cache-stats route is added. The public /api/exact-review-queue payload is unchanged. After deployment, compare queue canceled invocations and bare 5xx rates against the pre-deploy window, and confirm etag requests are served by GithubEtagCache.
The singleton queue Durable Object stores each delivery receipt and queue item in its own SQLite row. Receipt insertion and item coalescing commit in one transaction, so a crash cannot record a duplicate-suppression receipt without its queued work. Receipts retain the seven-day idempotency window and expire through the indexed timestamp path in bounded batches. Delivery receipts, queue items, storage schema and rollback metadata, item keys, workflow/job metadata, dispatcher details, credential circuits, and diagnostic records are private binding-only state. /api/exact-review-queue does not serialize them.
On the first upgraded request, the Worker transactionally imports the former exact-review-queue value. For 24 hours it maintains a generation-marked legacy shadow containing the queue and the complete active seven-day receipt set. Receipt timestamps are translated by two days so the immediately previous Worker's five-day pruner preserves their original seven-day expiry, and the reserved generation marker cannot expire. SQL state, its generation, and the synchronous KV shadow update in one SQLite transaction, so no committed generation can leave an older shadow readable. A later re-upgrade uses the generation plus deterministic timestamp translation to distinguish unchanged shadow receipts from receipts accepted or refreshed by the rolled-back Worker. It imports authoritative queue and receipt changes; a surviving generation is reconciled before deletion even when the rollback outlives the ordinary window. A divergent stale generation fails closed instead of discarding either side.
The Worker publishes that compatibility shadow only when the complete active set stays within 20,000 receipts and 1 MiB. If it cannot publish the complete shadow, it deletes any stale copy, reports rollback unavailable, and keeps the normalized queue serving; it never emits a lossy rollback state or retries the oversized write. The rollback bridge therefore cannot recreate the normalized queue's intake failure.
Before each dispatch batch, the queue reads the sweep.yml workflow state once. If the workflow is disabled, or GitHub cannot confirm its state, due items stay pending and retry after EXACT_REVIEW_WORKFLOW_PAUSED_RETRY_MS (60 seconds by default). The private queue state retains the dispatcher and workflow check needed to resume admission. The public queue projection exposes only the closed handoff status/reason and bounded phase counts and ages; it does not expose raw dispatcher state, workflow state, check timestamps, or retry detail. Re-enabling the workflow does not require a queue mutation; the next private status check resumes normal admission.
Scheduled hot and normal-backfill decisions have already passed the queue's fleet-wide rate and burst controls, so they skip the webhook-churn debounce and become ready immediately unless the dispatcher itself is paused or blocked. Review items parked after retry exhaustion or a permanent dispatch rejection retry automatically after 5, 10, and 20 minutes. A successful newer decision resets that recovery budget; after three unsuccessful recovery cycles the item stays parked for operator inspection. Publication dead-letter-capacity parks retain their separate operator-controlled recovery path.
Review items parked as source_drift_loop have no automatic recovery ladder. They used EXACT_REVIEW_SOURCE_DRIFT_REQUEUE_LIMIT consecutive automatic source-drift review generations without organic input; see Automation limits for the counter and its reset rules. The next organic webhook event, explicit command, or manual review admits the item normally and resets the counter. A scheduled offer with a newer sourceUpdatedAt admits one review without resetting it, so another drift requeue re-parks the item at once. Operators can list, resolve, or recover-fresh these rows through the signed parked-review routes, and the periodic terminal check still removes closed or head-advanced targets. The lane breakdown counts them under parked_reasons.source_drift_loop.
Review items parked as locked_conversation, oversized_pull_request, or maintainer_request_required are holds for deterministic no-ops, not failures. A review run for an open but locked conversation completes without Codex, and a run for a PR above the size limit takes the metadata-only size path. Maintainer items without an explicit request or automation opt-in also skip review. None writes a completed review the scheduled planners can see, so without the hold the same unchanged item is offered and claimed again every tick. The queue keeps the completed row parked instead, scheduled offers dedupe against it with that reason, and automatic recoveries dedupe with dedupe_scope: review_hold. Any other admitted input releases the hold: an organic webhook event such as unlocked or synchronize, an explicit command, or a manual review. The periodic terminal check removes closed and head-advanced targets, and the parked-review reconciler recovers a row whose source identity changed (for example an unlock or a size: accepted-large label). Command reviews are never held; oversized PR holds also require a pinned head.
review_runaway_health is the companion alert for any self-feeding loop the breaker does not stop, such as a command continuation. The queue records each newly claimed review run per item (publication and finalizer-only claims excluded) and reports degraded with reason review_runaway when any item claimed more than EXACT_REVIEW_RUNAWAY_REVIEWS_PER_DAY (production: 24) reviews in the trailing 24 hours; otherwise it is healthy. A failed history read reports unknown with reason telemetry_unavailable. The object carries window_hours, threshold_reviews_per_day, the runaway_items count across all repositories, and sample_item_keys: at most five owner/repo#number keys, highest review count first, limited to repositories in PUBLIC_BAY_REPOS. Private-repository runaways count but are never named. The dashboard health summary raises review_runaway (or review_runaway_telemetry_unavailable) to amber. Snapshots without the object, and malformed objects, project as null and do not change health, so the field is optional for older cached statuses.
Every failed exact-review completion first records one durable, deduplicated attempt keyed by its claim tuple. The record contains only closed stage/reason, retryability, source and failure fingerprints, immutable source identifiers, and workflow coordinates; raw diagnostics and scanner findings are never stored. The public review_failure_health projection groups the last hour into fixed stage buckets and marks a repeated target/source/failure identity critical. The dashboard raises recent failures to amber and repeated failures or exhausted review retries to red. A signed operator inventory provides the corresponding target and run identities for investigation.
/api/exact-review-queue is an explicit, closed aggregate projection. It contains generated_at, ready_pending, admissible_pending, pressure, handoff_health, review_failure_health, the optional review_runaway_health, and bounded counts and oldest timestamps or ages for the pending, dispatching, and leased phases. ready_pending excludes retry-delayed items. admissible_pending further excludes ready items blocked by their target's exact-review cap. pressure is a deterministic observation from that same queue snapshot: it reports congested or saturated only when capacity is full, handoff telemetry is known, and target-admissible backlog remains. The projection adds no GitHub API fanout, and no workflow, planner, admission, continuation, or dispatch decision consumes the pressure value. New dispatch and claim transitions carry explicit phase timestamps. Rows written by an older deployment derive their phase start from the active dispatch or execution lease; a stale timestamp left by a rollback cannot override that newer lease, and a wholly unknown legacy age stays non-alarming. A claim is degraded after one third of the dispatch lease (bounded to 30-120 seconds) and stalled after two thirds (bounded to 31-300 seconds), so operators see the failure before the lease expires and requeues. During its retry delay, a blocked dispatcher with pending work is stalled and a paused dispatcher is degraded. After the deadline, eligible work shows degraded dispatcher_retry_due until a successful preflight records recovery. If capacity, target cooldowns, review throttling, or debounce prevents admission, the expired block does not override current handoff health. Missing retry deadlines retain the warning. /api/status includes this snapshot and the live dashboard renders the three phases, oldest age, available exact-review slots, and the current classification without changing queue capacity or storage schema. Fleet snapshots may use the longer stale fallback during a GitHub API outage, but /api/status attaches queue telemetry after selecting that snapshot so handoff recovery stays live. If the optional queue read fails or is malformed, the public response uses an unavailable/unknown aggregate and a bounded telemetry_unavailable diagnostic; it never returns the underlying error text.
The object memoizes its private stats response for 10 seconds by default (EXACT_REVIEW_STATS_CACHE_MS; set 0 to disable). Concurrent polls with the same public sample parameters share the computation and its original generated_at. Queue mutations and auxiliary SQLite writes invalidate the memo; a missing or overdue alarm also bypasses it so polling still repairs the queue heartbeat. Only completed snapshots whose mutation generation and storage change counter remain unchanged are memoized; polls waiting on invalidated work recompute. This only bounds observation freshness: the public projection's fields and meaning, admission, and publication fences are unchanged.
The composed status response retains the dedicated closed queue projection through every cache and store read. Its parked-reason counts, including source_incompatible and the aggregate unknown bucket, must survive together with the parked total. Dropping a reason during a second generic sanitation pass makes a valid queue appear malformed on the next read and hides Overview's public references. The dedicated projector remains the privacy boundary; no private queue fields or mutation controls are exposed.
Producer and publication fences have independent revision counters. For newly admitted protocol-v2 publications, the lifecycle projection keeps an immutable producer fence/revision/generation link in its existing JSON. Audit uses that exact link to display publication completion on the corresponding producer journey after the queue item is removed. Missing or conflicting lineage fails closed; an older publication cannot complete a later producer revision. Physical receipts and terminal telemetry remain on the publication fence, so replay does not emit a second producer completion. The private link is not serialized by public endpoints.
For capacity displays, /api/exact-review-queue also exposes compatible lanes.review and lanes.publication objects. Each lane reports its own pending, ready, backoff, dispatching, leased, capacity, active, available-slot, oldest-pending, and next-attempt values. backoff_reasons and parked_reasons count the causes represented by those lane totals, and the dashboard renders the same breakdown beside the lane counts. The publication lane reports a retained stale-revision row as parked with reason stale_revision, not as pending or ready, so it no longer pins oldest_ready_at or next_attempt_at (see the stale-revision retention paragraph below). The existing top-level aggregate fields remain available for older consumers. Both lanes additionally report enqueued_total and completed_total; the review lane's existing shed_since_reset supplies overload demand. The public response omits item samples, ownership maps, per-target statistics, raw dispatch or failure detail, adaptive capacity-control internals, and all unexpected fields. A malformed required count or health value returns an unknown projection with HTTP 503 instead of a plausible empty queue.
Production overrides publication minimum, base, and maximum capacity to 8, 32, and 32, while source fallback values are 4, 24, and 48. The controller records failure, cooldown, recovery, and demand telemetry and scales within the production range. The private publication state also tracks batches, direct, and adaptive capacity control: production enables up to 8 concurrent size-8 batches, reserves two fresh-lane members per batch, and enables direct publication with retry/batch fallback. These controls affect the aggregate counts but are not serialized by the public projector. Document effective production values from dashboard/wrangler.toml, not only fallback constants in dashboard/exact-review-queue.ts.
Canonical record snapshots for openclaw/openclaw are produced every six hours by worker-records-ops.yml, at minute 9 UTC. The same scheduled run then runs node scripts/worker-records.ts snapshot-bootstrap-cold, which lists every canonical repository slug and uploads a first snapshot for each one that has none; one slug's failure is reported without stopping the rest, and the step fails if any slug failed. Manual dispatch remains available for a selected repository; scheduled and manual snapshots share a concurrency group so snapshot runs do not overlap. Full record hydration replays changes since the latest snapshot.
The workflow runs node scripts/worker-records.ts snapshot-upload --repo-slug <slug>: it hydrates the latest snapshot plus export delta into a temporary tree, streams the same ustar/gzip layout to disk on the runner, then uploads and registers the archive. The temporary tree and archive are removed on success or failure. The snapshot watermark is exportStartRevision, the revision returned by the first delta page. Later pages may observe concurrent writes, so using the highest revision observed would risk claiming changes not present in the archive. As with object-side production, later record versions may be included; future hydration replays every change after the initial watermark.
The outer Worker's signed POST /internal/state/records/snapshots/upload/ start, part, manifest, complete, and abort routes reuse the blob upload pattern: bounded JSON/base64 requests, each HMAC-signed over the exact body with CLAWSWEEPER_WEBHOOK_SECRET. Every signed body includes operation matching its route; a mismatch returns 400 upload_operation_mismatch. Each body also includes ISO issuedAt: requests at least one hour old or more than five minutes in the future return 400 upload_request_expired before accessing upload state. Start receipts survive the entire validity window, including allowed clock skew, so an expired signed start cannot recreate an upload. start accepts repoSlug, revisionWatermark, bytes, gzip sha256, fileCount, uncompressedBytes, identityDigest, and a client-generated operationId; the runner uses <repoSlug>:<GITHUB_RUN_ID>:<GITHUB_RUN_ATTEMPT> or a UUID outside Actions. A dedicated instance of the existing StatusStore serializes metadata operations and deduplicates starts: an identical retry returns 200 and the original session, while conflicting metadata returns 409. A new start generates createdAt, an upload ID, and the R2 key <repoSlug>/<revisionWatermark>/<createdAt>-<uuid>.tar.gz. part accepts uploadId, partNumber, base64 data, and per-part sha256. Parts are 6 MiB decoded (above R2's 5 MiB minimum), except the final part. Requests are bounded before parsing, with at most 200 parts and 1 GiB per archive. Descriptors and part receipts use the existing StatusStore and expire after one hour (extended for an accepted future issuedAt). Multipart bytes pass from the outer Worker directly to R2. The runner retries transport failures with identical bytes and part numbers. Aborts and expiry alarms clean up unfinished multipart uploads and completed objects that have no snapshot descriptor. Cleanup failures retain the session and its part/manifest receipts. Explicit retry alarms back off from one minute to one hour; after eight failed cleanup attempts a bounded warning is logged and the receipt remains for the next upload-triggered cleanup. The bucket's multipart lifecycle policy remains a backstop.
complete accepts uploadId, ordered {partNumber, etag} receipts, and the sorted packed identities as [section, id] pairs (at most 250,000). If the signed completion body would exceed the outer JSON cap, the runner first stages final signed manifest chunks of at most 10,000 identities in R2. Each chunk includes uploadId, sequential partNumber, and identities; StatusStore keeps only its digest receipt. Completion reads these chunks and registration checks the combined list. Manifest chunks are removed with session cleanup. The StatusStore assembles the R2 multipart upload, verifies the per-part SHA-256 receipts and total R2 object size, and calls the object's /records/snapshots/register. It does not download the entire object to recompute gzip SHA-256; R2 metadata marks that digest as a runner claim verified by parts and length. Registration checks the descriptor watermark against the current export revision, object key prefix, and R2 existence/size. A cheap aggregate SQL count rejects a smaller fileCount with 422 snapshot_coverage_incomplete. Registration then recomputes identityDigest, SHA-256 over the JSON encoding of the pair list sorted by section/id, and scans the live export index once for identities with store_revision <= revisionWatermark, reading ids only. Every required identity must appear in the supplied list, or registration returns 422 snapshot_coverage_incomplete; extra entries are allowed because records may have been deleted after the watermark. Later updates can move identities above the watermark, so the required set is a lower bound. The verified digest is stored for audit; membership verifies the manifest claim, not archive content. Registration inserts into the existing snapshot table and retains the newest two snapshots. Retrying completion or registration after response loss is safe. Cleanup checks all descriptor references and serializes deletion with registration, preserving registered objects even after a lost completion response.
The small signed /internal/state/records/snapshots/register route is also available for descriptor registration. The existing /internal/state/records/snapshots/trigger endpoint remains for explicit object-side production, but neither scheduled nor manual ops invokes it. Cold hydration retains its existing record bound; a large repository without any snapshot still needs an initial snapshot before normal hydration can run, and the scheduled cold bootstrap above provides it. The runner's snapshot-upload producer can build that first snapshot directly from the paginated export, bounded by 250,000 journal identities and the existing 1 GiB archive limit. Ordinary cold hydration still refuses above 2,000 records; only snapshot production uses the larger bound. Once a snapshot exists, both paths reuse the same snapshot-plus-delta materialization and first-page watermark. No bindings or Durable Object migrations change. The existing descriptor table gains a nullable identity_digest column; existing snapshots remain readable. OpenClaw Bay is unaffected because its public observer contract does not use these internal maintenance routes.
Batch publication heartbeats retry network errors, timeouts, and HTTP 5xx responses (including exact_review_queue_unavailable) up to three attempts within 45 seconds, with at most 20 seconds per attempt. Heartbeat retries preserve the signed bytes, use jittered exponential backoff, honor Retry-After up to 10 seconds, and stop at the last confirmed server lease expiry. Validation, authentication, and fence rejections are never retried. Periodic workflow heartbeats use --tolerate-until-lease to tolerate exhausted transport failures only while the confirmed lease has more than EXACT_REVIEW_BATCH_HEARTBEAT_SAFETY_MS (default 180,000 ms) remaining. Claim, fetch, and heartbeat responses include an internal server_time; the manifest saves leaseTtlMs as lease_expires_at - server_time, leaseTtlSource: "server", and leaseConfirmedAtLocal. Retry deadlines and tolerance subtract only local elapsed time from that TTL, so a runner clock offset cannot extend or shorten the lease. Older Workers without server_time use the local clock at confirmation and mark leaseTtlSource: "local"; tolerance is forbidden once that confirmation is older than one safety margin. Manifests without confirmation metadata require a successful heartbeat before tolerance is available. Each pre-loop heartbeat stays strict to establish live ownership, and 4xx/fence rejections remain fatal. The public projection and OpenClaw Bay are unchanged.
Batch post-effect router-receipt and terminal-disposition POSTs retry transient transport failures up to three times within 45 seconds, preserving the signed bytes and stable operation identity. Durable Worker replay handling preserves newer requeues. Publication enqueue remains one-shot. Other client callers, including operator retirement, retain the single-attempt default. Legacy lifecycle payloads without their route's replay identity also remain one-shot.
Lifecycle receipt replays are no-ops for the whole operation, including terminal transitions and acknowledgement drivers. Terminal-disposition requests from the batch workflow carry a stable operation_id derived from the run, attempt, and fence; applied IDs are retained with the lifecycle revision's receipt history. A replay after a newer requeue preserves that requeue. Older workflows without an operation ID retain their existing terminal-transition behavior. Failed queue requests report the HTTP status and, when present, a validated short server error code; raw response bodies are never included.
Batch claims carrying the current dispatch reservation consume only the still-valid subset of the key/revision pairs checked before departure. Fresh arrivals wait for the next departure; changed or removed members are skipped. An empty subset retires that reservation and requests another preflight.
The Worker preserves structured retryable Durable Object 5xx responses, including 503 {error: "target_visibility_unverified", retryable: true}, through /github/webhook item enqueue, /github/target-dispatch direct intake, and the /internal/exact-review/* proxies. The status, JSON body, and optional Retry-After header reach the caller unchanged; unexpected exceptions still produce 500 and the structured server-response telemetry remains intact. Visibility admission precedes delivery persistence, so a refused admission does not consume its enqueue delivery ID. Credential-bearing paths continue to require live visibility probes. Workflow shell retries are documented in the scheduler.
Only signed intake (/enqueue, /command-intake, /branch-authority, /source-authority) may reuse durable KV public-admission observations; credential-bearing paths require live probes. EXACT_REVIEW_HOSTED_TARGET_ADMISSION_FRESH_MS (60,000 ms) permits intake without probing; retryable probes allow fallback until EXACT_REVIEW_HOSTED_TARGET_ADMISSION_MAX_STALE_MS (1,800,000 ms; zero disables caching). Terminal visibility or eligibility revokes admission with a unique token fencing older probes, including across tombstone expiry. Tombstones expire lazily after at least 30 minutes; probes outliving retention must retry.
Alarms prune up to EXACT_REVIEW_STALE_PUBLICATION_PRUNE_LIMIT stale revisions (default 100), oldest first, without GitHub reads. Pending/parked rows require no active batch ownership or terminal finalization. A command row with a missing terminal requires one exact authoritative successor and matching lifecycle admission: the same marker and comment requeues without a finalizer, while a changed address or non-command successor records superseded with an acknowledgement driver. Missing, ambiguous, or mismatched successor state stays retained fail-closed; duplicate-lineage and legacy terminal cleanup remains operator-owned. New publication admissions retain one predecessor-bound successor witness, so the successor's normal completion cannot erase command ownership evidence. Same-source conflicts remain blocked through removal, restart, and redelivery; only a unique, verified admission beyond both the durable source head and the retained conflict can establish newer authority. Direct publication conversion can block authority but cannot establish it. Malformed or mismatched evidence fails closed. Historical rows whose successor disappeared before evidence was retained are not automatically repaired. This private queue state changes no Bay response or action contract. While batching is enabled, stats classify a pending batchable row that claim and departure exclude as superseded, and that no active batch owns, as parked with the stats-only reason stale_revision. The durable row stays pending; retention, pruning, supersession, the 80-day artifact refresh, and claim fences are unchanged. The row leaves the queue only through a newer revision's fenced cleanup, the closed-target retirement runbook below, or that refresh. Publication health keeps the age-based severity these rows had as pending work: degraded stale_revision_over_1h and critical stale_revision_over_6h. Real dead letters keep dead_letter_capacity. Authenticated reconciliation samples report successor_fence_state only for stale_revision rows whose acknowledgement is unavailable because the terminal disposition is missing. verified is diagnostic evidence, never mutation permission; other rows report null. The maintenance client exposes this field as successorFenceState and rejects missing, unknown, or incoherent values. If the Worker must roll back below e4d0e82050300cafb9459a6d9cf8a2041f4e62cb, roll back the strict client first so it never reads a response without this field. CLI samples omit target, item, retained-item, and producer identities. Their identity_hash is SHA-256 of the unmodified UTF-8 item key, without a newline, so operators can correlate a row without logging its key. A different hash or an empty sample does not establish what happened to a previously observed row. Successful queue handoff expires the workflow's review-start lease to permit another exact-head review; terminal publication still deletes the placeholder.
The binding-only publication state retains additional diagnostics:
credential_circuitsrecords the pool class, optional target owner,github_request_metricscontains cumulative counters keyed by poolflow.last_15_minutes.causesandflow.last_60_minutes.causesreconcile
observation time, raw credential reset as blocked_until, latest per-member reset-plus-jitter boundary as recovery_until, reset source, authority flag, active state, and affected pending count. A circuit remains active through recovery_until; active: true with free publication slots means credential-blocked, not capacity-starved or healthy-idle.
class, endpoint category, operation class, outcome, and whether the item revision was already retried.
publication retry, backoff, supersession, refresh, and dead-letter exhaustion against durable flow counts. The surrounding flow window exposes refreshed and refreshed_rate_per_hour as the independent refresh denominator. Cause rows use only closed stage, completion, reason, revision-relation, pool-class, recovery-cause, backoff, and attempt buckets. attribution_complete=false or a failed per-transition reconciliation explicitly marks a legacy or truncated denominator; the Worker never invents attribution for an old aggregate.
Those records are available only through the Durable Object binding and are not part of /api/exact-review-queue, /api/status, or Bay. The public projection keeps useful closed lane totals and fixed reason counts without exposing credential, target, revision, reset, member, or transition rows.
An unattempted credential-circuit member appears as transition: backoff and does not increment retried. A retry-exhausted dead letter preserves the underlying completion reason in the cause row while the operator-facing dead letter retains its established retry_exhausted reason. A terminal coverage deferral is reported as transition: deferred, never as a publication. When a batch publishes its owned revision while a newer local revision is already queued, the ledger records both the completed publication and the follow-on backoff; the published row still reconciles to the durable publication total. Cause rows persist privately in SQLite for the same 48-hour window as publication flow buckets. They are not a public dimension-row surface.
The durable handoff's handoff_health.recovery_reasons counts bounded claim_timeout, execution_timeout, workflow_cancelled, and workflow_failed recovery causes. These are observed queue and workflow facts; they do not infer why GitHub or a runner cancelled or failed a workflow.
GET /api/github-egress-observability?hours=6 adds the publication transport denominator as revision-independent closed aggregate rows. It separates durable members, gh invocations, and observed HTTP wire attempts and retains bounded pool, method, normalized route, page, source, stage, claim-generation, and first/repeat categories. Deployment/configuration revisions are withheld and otherwise-identical rows are combined across them. Private 403/429 observations become counts by status, pool class, operation, reset-authority category, resource category, and header presence; raw reset values and observation rows are not public. See GitHub publication egress telemetry for exact semantics, retention, privacy, and known opaque boundaries.
Bay renders closed aggregate health context only. Its retired proof/batch switch filters the already-projected cards and selects between closed timing aggregates; it does not change queue admission or execution. Bay has no circuit reset, workflow dispatch, queue retry, replay, acknowledgement, or gate control, and it does not expose credential circuits or per-member recovery boundaries. Private circuit state continues to control automatic recovery.
The standalone State writer panel separates the repo-wide serialization boundary from exact-review materialization telemetry. After the coordinator cutover, state_writer.coordinator is authoritative for the active writer, FIFO queue depth, completed turns, recovery counters, and coordinator wait. Private state may retain Git lease and revision fences for crash recovery; the public panel omits them. Exact-review terminal telemetry still owns aggregate item/commit throughput and fence timing; an idle or failed publisher can make that telemetry stale without making the coordinator unavailable. The panel therefore shows bounded queue and throughput counts, uses five-minute coordinator queue depth for its primary chart, and never renders stale terminal zeroes as current throughput.
Executors report the GitHub job outcome from their finalizer. Failure or cancellation clears the lease and requeues the item. Finalizer success remains provisional because GitHub can still cancel the run or fail a post-action; only the signed terminal-run backstop removes the item after GitHub confirms the exact attempt succeeded. A newer revision can requeue immediately. A signed POST /internal/exact-review/publications/reconcile defaults to apply:false. It classifies stale and legacy publication candidates without reconciliation-driven queue, lifecycle, batch, or Bay mutations; expired batches are excluded from active ownership without reclaiming their rows. Existing admission-cache revocation, request cleanup, and cold-start initialization still run. apply:true retains normal expiry recovery and terminalization. This is an authenticated operator route, not a Bay action.
#Retire One Closed-Target Publication
Active operator runbook. Owner: exact-review maintainers. Owning sources: .github/workflows/exact-review-queue-maintenance.yml, src/repair/closed-publication-retirement.ts, and dashboard/exact-review-queue.ts. Last source verification: ff0156fb plus the retirement maintenance route. Update this section when artifact provenance, plan fields, terminal disposition, or acknowledgement ownership changes.
Use Maintain exact review queue on main, mode retire-closed-publication, only for an explicitly approved publication whose public target PR is merged. Supply its exact producer run ID, artifact ID, publication-key SHA256, and publication queue revision. The source-review revision in the artifact is not the publication queue revision.
- Leave
executefalse. Preview fetches the exact artifact and attempt - Review the emitted plan SHA256 and numeric tuple against the approved
- Run the same inputs with
executetrue andreviewed_plan_sha256set to
metadata, verifies the archive digest, reads only its bounded root manifest, and checks the public merged target. It neither constructs a Worker client nor requests a Worker route. An expired artifact or absent GitHub archive digest fails closed.
publication. The plan also binds the immutable workflow SHA, historical producer source SHA, archive identity, and merged PR identity. Raw target keys, command identities, artifact contents, and signed URLs stay private.
that reviewed hash. Preparation remains secretless with respect to the Worker signing credential. Only the final assertion step receives it. A changed workflow SHA or evidence requires a new preview and review.
Execution records one new operator target_closed fact through the existing terminal-disposition route. It is not a replay of a historical completion. This is deliberately not an ownership CAS: concurrent active or newer queue ownership does not reject the assertion. The normal owner removes only a matching pending or parked publication revision and uses the original admission's command marker for acknowledgement. It does not retire another publication, reject a concurrent owner, or publish a stale review verdict.
The normal finalizer must observe the original command acknowledgement as Complete, with detail The item is closed; no stale verdict was published., and record its verified receipt. HTTP success or acknowledgementState: pending is not completion proof. Check the exact original receipt, available matching queue cleanup evidence, and preservation of later commands. Do not force a driver dispatch. On timeout, redirect, malformed response, or error, stop and inspect the exact operation; never blindly retry or substitute a new operation ID.
POST /internal/exact-review/reconcile backstop accepts at most 32 exact run IDs and intersects them with currently claimed leases. The Worker checks those IDs and attempts with an Actions-read GitHub App token and reconciles only runs whose immutable GitHub attempt status is completed; queued and in-progress runs remain leased. A per-claim generation check prevents a terminal decision sampled before a rerun claim from releasing that newer attempt. The request body is { "runs": [{ "run_id": "<run-id>", "run_attempt": 1 }] }, signed over the exact bytes with CLAWSWEEPER_WEBHOOK_SECRET in x-clawsweeper-exact-review-signature: sha256=<hmac>.
Do not disable or drain the sweep workflow for this protocol rollout. A v2 Worker sends the strict tuple under queue_claim plus the immutable v1 event snapshot, accepts legacy lease-id claims/finalizers only for claims recorded as protocol v1, and keeps tuple/generation CAS mandatory for protocol v2. A v2 workflow falls back to the v1 event snapshot only when the claim response identifies or implies a v1 Worker. Keep this mixed-version coverage until every in-flight v1 dispatch has drained naturally. The dashboard deployment smoke test must still observe HTTP 401 from an unsigned reconciliation request; HTTP 404 means the old Worker is serving that route.
#Exhausted command review records
The queue activity projection counts retained exception records alongside live repair activity, not running repair workers alone. Its public references may carry the bounded queue_disposition values parked_exhausted, parked, or retry_scheduled. parked_exhausted is reserved for exhausted review attempts; dispatch-rejected work uses neutral parked attention even after its recovery budget ends, because no review may have run. The public sanitizer retains only these values, and only for queue references. Exhausted records show operator attention instead of an increasing queued-worker clock. This remains an observer-only surface with the existing public repository allowlist and sampling/freshness limits.
Terminal scanner holds retain a scanner_refused queue reason count, but do not contribute an active or waiting review card. The failed review remains in the existing terminal lifecycle projection. A fresh explicit retry or an intentional scanner-policy epoch change is required; ordinary source changes do not revive it. Overview exposes no release control.
The queue's globally bounded parked-terminal check also observes exhausted command producers. An eligible still-open exhausted review may receive a separate, acknowledgement-only stopped-status explanation. Its producer stays parked and visible for operator attention after that receipt; acknowledgement settlement does not reset the budget, dispatch another review or repair, or claim that review succeeded. Current command, revision and canonical source (title/body/review labels/lock plus PR head/base/draft) fences apply before the comment update. Repeated settlement is idempotent despite bot-comment timestamp churn. Missing recorded source identity or live source drift keeps the legacy record parked without a status write; it is not guessed or backfilled. New commands capture that canonical source identity during verified command intake; clients cannot supply those server-owned identity fields. A stored closed failure category can explain the observed failure; legacy records without it retain an explicit unavailable historical reason.
Closed-target cleanup remains distinct. An explicit closed GitHub item, observed twice with the same node/head/closure identity, may create a separate acknowledgement-only driver. The producer remains parked until its own receipt is observed or its trusted receipt is explicitly missing/locked. The driver rechecks the live closed identity and current producer revision/decision before authorizing a status write. Reopened, superseded, unknown, and still-open work is not restarted; operator bulk resolution/recovery still refuses command-context records. Closed failed commands retain a failure acknowledgement, not a fabricated successful review. Ordinary reconciliation now includes command exclusions in its bounded skip-reason accounting.
The stopped-review and source-fenced acknowledgement proof uses real local Worker/SQLite/HTTP; see docs/proof/review-failure-attention/README.md.
Parked-command finalizers reserve status-write ownership while their receipt is looked up, then re-fence immediately before the status PATCH. A successor may be recorded but cannot dispatch through that owned write window; ownership is released after a successful bounded PATCH or expires under the acknowledgement lease. The opted-in finalizer does not prune duplicate comments during lookup.
When a parked-command target reopens after acknowledgement authorization, the queue atomically cancels the unobserved closure plan in its existing terminal operation ledger before deleting the driver. The producer remains exhausted. Cancellation retains a failed/attention lifecycle state, not a requeue claim; no new review is scheduled by canceling a closure plan. That cancelled revision cannot accept a late receipt or recreate an unfenced closure finalizer after a later webhook or Worker restart; a genuinely new command/source revision retains its separate acknowledgement ownership. Losing hosted/public repository eligibility also leaves exhausted command obligations parked: it is neither a verified receipt nor a missing/locked comment skip, and does not authorize generic deletion or finalizer dispatch. An already-planned parked-command driver is deferred with its original fence rather than superseded or discarded, so eligibility can recover safely. Such auxiliary drivers never replace their exhausted producer in the canonical queue projection; actual workflow activity remains a separate live overlay. Claimed parked-command writes also re-probe hosted/public eligibility before target credentials and after the target read. Revocation defers the original fenced driver; it cannot authorize a status PATCH on a now-ineligible target.
A later stable closure can re-arm an exhausted command whose earlier closure plan was cancelled. It receives a fresh lifecycle revision while the cancelled revision and its receipt-rejection tombstone remain immutable. The producer stays parked with its exhausted attempt/recovery budgets; no review is restarted. If a status writer retries after a token or write failure, it releases only its exact recorded successor coordination deferral before clearing the lease. A concurrently changed successor revision, reason, or deadline is not overwritten.
Attention lists from a complete corpus refresh sort by reviewer-assigned priority (P0 through P3), then oldest review, with unranked items last. These are historical flags, not verified unresolved work. Cached corpora built before priority collection retain newest-first ordering until the next refresh_corpus run; discarded older rows cannot be reconstructed by sorting the cache.