Node scale to zero and wake (R2)

Node scale to zero and wake (R2)

Standalone catcher collection — local development checkpoint, 2026-11-08

The production sleeper's recoverCatchers poller also discovers never-bound candidates on nodes with an already persisted wake credential and no sleeping cohort. Collection runs after demand scheduling, at most one job at a time, with a five-minute interval per node. It creates neither a wake credential nor a replacement catcher. Unsupported, unsealed or opted-out agents are refused before a fresh admission barrier is created.

A durable collecting maintenance phase blocks new sleep/wake admission while bounded, exact-key inventory is retained as retirement-only identities without service metadata. The existing ordered loaded-routing acknowledgement, drain and immutable-ID retirement fence still govern every removal. Lost replies keep targets in the journal for replay after restart, including targets absent from the next inventory. A foreign-key container is not an inventory member. The existing 64-entry inventory and 256-member retained-snapshot limits still apply; uncertainty holds the journal. Ordinary sleeping-cohort maintenance keeps its readiness and replacement requirements.

This extends the existing text phase and JSON snapshot representation without a SQL migration. An older panel refuses the unfamiliar phase rather than completing the barrier. Local component and native collection evidence are distinct from hosted automatic idle/Stop/wake acceptance and publication; those broader requirements remain open.

Lost preparation replies — local development checkpoint, 2026-11-07

Catcher maintenance now requires the separately advertised, sealed wake.catcher.inventory operation before preparing another generation. An agent without that capability keeps maintenance held until upgraded; ordinary demand polling and wake retain their existing requirements.

Inventory reads at most 64 managed catcher identities for the exact node wake key, including older images and stopped candidates. It never starts or removes a container. The panel records unknown identities as retirement-only members of its durable maintenance intent before preparing again. Actual retirement still requires readiness, confirmed ordered routing, drain and the agent's immutable-ID fence. An uncertain or incomplete inventory holds the operation. More than 64 inventory entries or 256 retained snapshot members requires operator review.

This closes the lost-reply/image-change identity gap within an existing maintenance intent. Standalone collection without a sleeping cohort is described above. Local tests and native Docker/sealed-HTTP inventory checks are separate from hosted end-to-end retirement, upgrade and release acceptance; those broader gates remain required.

Current release checkpoint: v0.18.17 was published from 91017c3405398666aa78f168751e0baa62a9b42f. Its release run completed with all nine runtime cohorts and 18,373 source tests passing. The sleep/wake cohort passed on attempt 2; the unchanged image also passed isolated automatic acceptance. The first automatic idle/Stop timeout remains unresolved. Preserve that failed attempt alongside the passing evidence.

Diagnose an automatic idle timeout

Run the original manual-wake → lost-routing-acknowledgement → automatic-sleep sequence on one isolated GitHub-hosted runner, using an immutable published image:

gh workflow run serving-traffic.yml --repo ninedeploy/ninedeploy --ref main -f sleep_cohort=true -f immutable_image=ghcr.io/ninedeploy/ninedeploy@sha256:c19bafb5a4c04508994b4ce3009319dc15dd1882233a509e40ac6b88be2d1dd0

The diagnostic job retains the original per-case deadlines (15/12/20 minutes) and real five-minute idle requirement. It pulls the digest without rebuilding. Its script revision and the image's embedded source revision are separate identities and must both be recorded.

Download the automatic-sleep-cohort-<run-id> artifact. The automatic case includes bounded public status observations, a journal snapshot after the actual Stop, traffic/audit records and owned cleanup. On failure, automatic-sleep-failure.json records the automatic journal phase, idle decision age, remaining lease and a fixed reason class. Raw exception text, wake keys, worker tokens and agent credentials are excluded. Failure to collect diagnostics preserves the original test failure.

Automatic transitions live in nodeSleepTransitions; /lifecycle reports manual intents. Use the automatic journal to distinguish missing idle coverage, expiry before Stop, restoration and failed wake readiness. These diagnostics read the owned fixture's state and preserve the production timers and readiness checks. Local Docker execution remains refused by the smoke's hosted-runner guards.

Historical pre-release checkpoints

Status, 2026-11-03: durable node authority, ordered sleep/wake effects, guarded demand and staged catcher retirement have component acceptance. The two-service cohort 41616036959 passed actual five-minute idle, Stop/Start, failed strict HTTP readiness, new-process recovery, Basic Auth/IP denials, agent opt-out and Stop during wake, with 2,328 valid responses. The pre-Stop transport fault 41627779632 restored the application after a lost real loaded-routing acknowledgement, with 640 valid responses. The latest public automatic-cycle cohort 41634736108 passed ordinary API settings, production timers, genuine idle after a real traffic reset, automatic Stop, readiness-held demand wake and panel restart, with 791 valid responses and settled audit events. All three had zero 502/connection errors. The public automatic candidate passed all required CI 41643863613 and serving-traffic checks on frozen head 65d15f69f71eb0e63e59dd1600e7455089e331a8. PR #25 merged to main as 977390d7279268b3ab8dffd0dde133fa40668c01; normal main plugin timers now activate eligible configured nodes. It is not in a numbered release. Manual Sleep now continues to refuse node placement. Final combined release/upgrade/artifact checks remain open. Sections below retain exact source identities; DEPLOYMENTS.md documents the released activation limit.

Why the panel-host mechanism does not carry over

• Idle evidence. The panel host counts requests from its own Traefik access log (traffic analytics). A node's access log never reaches the panel. The node serving-traffic windows used by RPS autoscaling (NODE_RPS_AUTOSCALING.md) are bounded reservations of at most 60 s with gaps between them. A zero there proves only that window was idle, not the minutes before it. Sleeping on that would stop a service that is in use.
• Wake path. A sleeping panel-host domain routes to the panel's own wake endpoint (/_nd/wake, modules/wake.ts) over the local Docker network. On a node, the panel calls the agent and the agent never calls the panel. A node proxy cannot assume a route to the panel, and the panel's Traefik does not know node domains.

Decisions

1. Continuous idle watch (new sealed agent operations). proxy.idleWatch.open/sample/finish/close is a separately advertised capability under the existing owner opt-in NINEDEPLOY_AGENT_PROXY_TRAFFIC=on. It reuses the serving-window engine with purpose: 'idle' and windows of 1 minute to 1 hour (idleWatchOpen). A window counts requests to the selected serving leaves continuously between its start and end sequence markers. The engine releases a window that is not sampled within 30 s, so the panel samples every 15 s; that is the lease. Because sleepAfterMinutes can reach 24 hours, the panel chains windows. It opens the next window before finishing the previous one, so the next start counter is below the previous end counter, and requires the same source, proxy and agent epoch. Overlapping chained receipts therefore leave no unobserved gap. A proxy restart, source change, routing change, lease lapse, agent restart or any refusal ends coverage, and the idle clock starts again. A service sleeps only after a verified zero count over continuous coverage of sleepAfterMinutes. Unavailable coverage never counts as idle, the same rule RPS uses.

2. Node wake catcher. The agent prepares private generation names, ninedeploy-wake-g-<nonce>, on the node's ninedeploy network. A previous selected catcher is retained through stable readiness, ordered route acknowledgement and retirement proof; names fenced as retired are never reused. Legacy ninedeploy-wake remains a compatibility identity. Each catcher runs the agent's own pinned image with no host mounts or Docker socket. It answers valid keyed requests with the "waking up" page: 503, Retry-After, a self-reloading HTML body and HEAD support. Opaque wake tokens map to services only on the panel. Bounded demand counters emit bounded Docker-log records, and reads bind the immutable container plus its current process start time.

3. Routing. A sleeping node domain routes to the catcher after the domain's own middlewares, so basic auth and IP allow lists still run before anything wakes, as on the panel host. Routes are written through the panel-rendered node dynamic config and confirmed with proxy.routingAck before any container stops.

4. Wake. For each node with sleeping services, the panel polls the sealed wake.demand operation every 2 s. On demand it acquires the service runtime owner and starts the recorded serving count by immutable ID through the ordered node lifecycle. It then waits for readiness, restores the routes with routing confirmation, and records service.wake. The visitor's page reloads into the app. A cross-host redirect is not used, because a non-GET body cannot be replayed anyway.

5. Sleep effect order. Sleep follows a strict sequence:

   1. reserve the runtime owner;
   2. switch routes to the catcher;
   3. confirm routing;
   4. stop the serving members by immutable ID with a sleep intent, distinct from Stop;
   5. record service.sleep.

   A failure at any step before the containers stop restores the routes, and the service stays awake.

6. Authority. Explicit Stop, deploy and Restart supersede sleep and wake, as on the panel host. The autoscaler holds while a node service sleeps; nothing sleeps during a deploy, canary or autoscale transition. Fan-out and Swarm services stay refused with their own reasons.

Ordered implementation

1. Agent: the proxy.idleWatch contract and capability, on the bounded log transport and sequence markers. Cover lease lapse, restart, source change and refusal with tests, and add a hosted observation case (continuous zero, a single request breaks the zero, proxy restart resets coverage). Implemented 2026-11-02:

   • lib/nodeIdleWatch.ts and agentOps/idleWatch.ts.
   • Agent tests: test/agentOps/idleWatch.test.ts (two-minute verified zero under a 15 s lease, a single request, lease lapse, overlapping chain, owner disable).
   • Hosted cases in scripts/fixtures/serving-traffic-agent.mjs: a verified zero over one minute, a chained window broken by one request, and a proxy restart ending coverage.
   • The agent now also closes serving and idle windows when it shuts down.

2. Agent: the ninedeploy-wake catcher (an idempotent ensure/remove operation, pinned image, no mounts) and the sealed wake.demand operation. Cover token bounds, expiry and missing-catcher refusal. Implemented 2026-11-02:

   • src/wakeCatcher.ts, lib/nodeWakeCatcher.ts and agentOps/wakeCatcher.ts, under the capability wake.catcher (same owner opt-in; NINEDEPLOY_AGENT_CONTAINER_MUTATIONS=off also turns it off).
   • The catcher runs the agent's own image by ID as 65534:65534, read-only, with all capabilities dropped and no published port. It has no volume or bind mount. The image declares VOLUME /data, which Docker would otherwise satisfy with a writable anonymous volume. The first hosted run (41511827062) caught exactly that, so /data is shadowed by an empty 64 KB tmpfs with mode 0555. The smoke asserts that writes to /data, / and /app fail. Removal uses rm -fv.
   • Service containers share the proxy network, so demand counts only when a request carries the node's wake key. The x-nd-wake-key header is added by the panel-rendered Traefik middleware, and the key comes from wake.catcher.ensure { key }. The container gets it in its environment, and its label holds only a hash. Any other request gets a 404 and writes no line.
   • It writes at most one ND-WAKE line per token per second and 20 a second in all. wake.demand aggregates those lines from the catcher's Docker log since a cursor. A read that fills the 2,000-line tail is flagged truncated, so the panel never treats it as all the demand.
   • A missing, stopped or unlabelled catcher is refused rather than reported as no demand, and a same-name container it did not create is never touched.
   • Hosted check: scripts/smoke-wake-catcher.mjs in CI and on the release image.

3. Panel: the node idle clock in the sleep tick, using the idle watch. Report hold reasons (capability, coverage, opt-in) through sleep status. Never sleep on unavailable coverage. Read-only scheduling, transport and coverage calculation implemented 2026-11-02; public status and runtime effects remain open:

   • lib/nodeIdleWatchClient.ts uses the shared sealed observation client with its own idle schemas, capability and receipt frame. It retains before/after runtime, owner, route, topology and conflict checks, cancellation cleanup, source continuity and one-use finalization. Idle windows do not enter the autoscale receipt ledger.
   • A normal node deployment does not require autoscale spare capacity to be observed. Its serving names/count/port and database placement are checked in the panel; the sealed agent verifies the live immutable serving IDs and deployment labels. RPS observations still require their complete capacity receipt.
   • lib/nodeIdleCoverage.ts accepts only completed zero-request windows. A chain requires overlapping sequence markers, unchanged source/proxy/epoch, serving IDs, routing and runtime context. A gap, traffic, replacement, lease loss or validation failure resets coverage.
   • Chained durations are not added together: monotonic panel timestamps bracketing each opening RPC give a conservative union-duration lower bound. Opening latency and overlapping seconds cannot inflate the idle proof. Repeated and superseded completions are refused, with exact uint64 counter comparisons.
   • lib/nodeIdleWatchService.ts resolves all serving members through sealed docker.identity before opening each window. Both docker.recovery and proxy.idleWatch are required. Runtime owner/configuration checks surround discovery, and the bound transport rechecks deployment, topology and route eligibility before reserving any window.
   • lib/nodeIdleScheduler.ts samples every 15 seconds independently per service. It opens a successor before finishing its predecessor, resets on traffic/refusal/lease lapse, closes ineligible or superseded generations, and aborts/cleans up during server shutdown, including late opening replies. One slow node cannot block another service's lease tick. Missing current service state discards all coverage.
   • Each node reserves paired slots for at most eight active services within the agent's sixteen-window limit. Other services report capacity waiting with no idle value. Backing-off failures do not reserve empty slots or starve healthy services. Refusals retry after one minute instead of generating a request and warning at every lease tick.
   • plugins/sleeper.ts starts the node read-only observer separately from the existing one-minute panel-host sleep tick. The scheduler's internal status distinguishes unavailable coverage (idleMs: null), a verified traffic reset (idleMs: 0) and a completed idle proof. idleSatisfied is observation evidence, not mutation authority. Public status, wake reachability, route changes and runtime mutation effects remain pending; node sleep stays refused until those steps and hosted acceptance pass.
   • Transport/coverage foundation validation: 222 server tests passed in ten files covering idle transport/coverage, existing RPS observation/consumption/sample/controller/admission, agent idle watch and the Swarm runtime-reader inventory. Server typecheck and targeted lint passed. Agent transport is mocked in panel adapter tests; the agent idle tests use the real observation engine with a fake Docker I/O boundary. This is local source validation, not hosted node sleep/wake acceptance. Local Docker was not accessed (DOCKER_HOST=tcp://127.0.0.1:9).
   • Scheduler/helper validation: 230 focused tests passed in eleven server files, including the real Fastify plugin's 15-second cadence and shutdown. Scheduler tests use the actual idle-watch engine, counter and coverage calculator with fake Docker I/O; they verify five-minute continuous coverage, pending unverified tails, traffic, lease/source loss, late replies, concurrency, paired capacity and refusal backoff. Identity tests use migrated SQLite, real panel/client checks and a fake sealed-agent boundary. These checks do not accept real node sleep or wake effects.

4. Panel: the node sleep effect (decision 5), with its routing render for sleeping node domains pointing at the catcher with the domain middlewares kept. Durable authority foundation implemented 2026-11-02; runtime effects remain disabled:

   • Migration 0120_node_sleep_transitions records the selected immutable serving IDs, runtime owner, configuration binding, idle evidence, wake token, catcher identity, ordered phase and a millisecond worker lease. It survives deletion of deployment history and is removed with the service.
   • The scheduler issues opaque, one-use idle evidence, valid for less than 30 seconds. Traffic, lost coverage and changed sources revoke outstanding evidence. Sleep admission rechecks the selected generation, routes, database placement, conflicts and current configuration transactionally; a caller-provided numeric idle value cannot admit sleep.
   • lib/nodeSleepAuthority.ts owns admission, checkpoints, phase transitions and exact immutable target receipts. Recovery retains the original command identity and replaces only the expired worker lease. Expired unpublished idle evidence is abandoned instead of putting the service to sleep after a restart.
   • A fresh wake demand must match the current sleeping token, node and catcher. Explicit Stop, Restart, deploy and limits supersede sleep/wake; autoscaling and automatic Start hold. Catcher recreation uses a current-owner identity comparison so an older catcher cannot wake a superseded cycle.
   • A persisted fallback-routing intent is retained when an explicit command supersedes a partially published sleep. This is journal state only: the renderer and lifecycle recovery still need to consume it and restore app routing only after readiness and routing acknowledgement. The new authority is not connected to container mutations, route publication or the wake poller, and the existing node sleep refusal remains in place.
   • Related server validation passed 338 tests in thirteen files without coverage; the database package passed all 77 tests in 31 files. A separate-process SQLite case proves worker recovery and rejection of the old worker. Server typecheck passed. An earlier combined Windows run ended with native worker exit 3221225477; the isolated scheduler and serialized thirteen-file rerun passed, but the native failure's cause is unresolved. These checks are not full release certification or hosted sleep/wake acceptance. Local Docker was not accessed.
   • Fallback renderer implemented 2026-11-03: a same-generation fallbackRouting journal routes node domains to ninedeploy-wake:8080, after their auth, IP and rate-limit middleware. A per-domain prefix carries the opaque token and a shared header middleware overwrites the wake credential. The per-node credential is stable, encrypted in settings and included in master-key rotation. Missing credentials or malformed catcher identities refuse rendering; a stopped service withdraws its route, and an older generation's fallback never follows a newly selected generation or another node. This does not start sleep effects: readiness-authorized route promotion, ordered controller effects, polling and hosted Traefik acceptance are still required.
   • Wake route publication plumbing implemented 2026-11-03: a worker in the wake journal's publishing phase can obtain an opaque permission valid for less than 30 seconds. It revalidates the immutable journal, configuration, current owner and worker lease during rendering and before/after each agent call. Only the selected node can use it; confirmed routing and adoption of ordered proxy writes are mandatory. An ordinary render retains the durable fallback until wake completes. A wrong acknowledgement, changed owner, expired permission or unsupported agent never reports publication success. The controller still must prove stable readiness before entering publishing, repeat readiness on recovery and complete wake only after the exact routing acknowledgement; these effects are not connected yet. Panel tests simulate sealed agent I/O and do not replace real Traefik acceptance.
   • Publication plumbing validation: 225 tests passed across ten related files in two serialized runs without coverage, including supersession during static write, ordered dynamic write and routing acknowledgement, older-agent refusal, expired/manufactured permissions and an unmatched acknowledgement. Server typecheck and scoped lint passed. These results do not certify the complete release gate or live wake readiness.
   • Internal sleep controller and production adapter implemented 2026-11-03: nodeSleepController.ts fences the admitted service revision, ensures the pinned catcher with the persisted node credential, verifies three stable TCP samples of its immutable identity, confirms ordered catcher routing, drains and only then stops the captured serving IDs. Recovery of an uncertain Stop reconfirms reachable fallback routing and retries the same Stop command before marking asleep. No normal proxy container recreation is requested.
   • A pre-Stop failure never resumes an old idle decision on recovery. Unpublished failures cancel without stopping; uncertain published fallback enters the durable restoring phase. Restoration verifies configured application readiness plus listener reachability, publishes the app under a current short-lived permission and clears fallback only after routing acknowledgement. An unready restoration stays held. Expired idle evidence or sleep being disabled before Stop takes this same path. Once a Stop attempt may have landed, its recorded phase is recovered instead of assuming the containers are still running. New explicit ownership always supersedes old effects.
   • restoring extends the TypeScript phase enum over the existing SQLite text column. Database build and Drizzle generation confirm no SQL migration or schema drift. Application readiness is shared with wake, preserving the original common deadline and three-sample policies. Automatic sleep, catcher recreation recovery across sleeping services, demand polling, explicit-command fallback settlement and real sealed-HTTP/Traefik acceptance remain open; the existing node sleep refusal stays in place.
   • Sleep-controller validation: 110 tests passed in seven related files without coverage, including both production adapters through queued rendering and ordered proxy modules with fake agent I/O. Server typecheck and scoped lint passed. The earlier foundation CI job (41532831737) passed 12,995 server tests but failed its retention inventory because node_sleep_transitions was unclassified. The same failure reproduced locally; recording its actual one-row-per-service primary-key/cascade bound made all five retention guards pass. The fix is propagated through the stacked branches and full CI is rerunning. This is not a complete release gate or real sleep/wake traffic acceptance.

5. Panel: the wake poller and wake effect (decision 4), with deploy/Stop/Restart supersession and Stop arriving during a wake. Internal wake controller and production adapter implemented 2026-11-03; polling and automatic activation remain open:

   • lib/nodeWakeController.ts checkpoints the admitted wake owner and immutable journal around every effect. Sealed mutation, Start-set, readiness, ordered-proxy, routing-ack and catcher capabilities are required before effects. The service fence confirms the recorded deployment watermark; Start addresses only captured immutable IDs and requires an exact successful receipt.
   • recordNodeWakeStarted atomically validates that receipt before entering readiness; the generic phase transition cannot bypass it. This phase durably acknowledges Start without storing a second mutable target set. A lost Start reply retries the same command and targets. After an acknowledged Start, recovery from readiness or publishing repeats readiness and routing without repeating Start.
   • Every serving member must pass three consecutive configured readiness samples. Default, running and command policies also require three application TCP samples within the original shared readiness deadline. Strict HTTP retains the configured path/query. An unready application stays behind the catcher, with asleepAt retained.
   • The production adapter uses the queued dynamic-config writer with the short-lived wake publication permission, mandatory ordered proxy adoption and exact loaded-routing acknowledgement. Only afterward does completion clear the asleep flag and durable fallback. Explicit Stop/deploy ownership supersedes the continuation. Refusals retain the phase/reason and release the worker lease for recovery; an old worker cannot release a recovered worker's lease.
   • Validation: 102 tests passed in seven related files without coverage, followed by an 18-test controller rerun including actual explicit Stop admission; server typecheck, scoped lint and diff checks passed. The controller suite covers all five readiness policies, a closed listener, lost Start/routing replies, fresh readiness on recovery, immutable receipts, capability/cancellation refusal and Stop during Start/readiness/routing. One real-clock test traverses the production adapter, queued render and ordered node proxy modules with fake agent I/O. Its initial fake-time driver ended before dynamic import completed and is not counted as passing evidence. No local Docker was accessed. Real catcher reachability, sealed HTTP/Traefik traffic acceptance, sleep effects, explicit-command fallback settlement and the durable wake poller remain required before activation.
   • Demand transport and agent process boundaries hardened 2026-11-03: nodeWakeCatcherClient.ts performs bounded sealed capability-checked ensure/read exchanges. It requires one typed frame, the pinned immutable catcher, a complete tail, unique bounded token counts, a monotonic cursor and fresh panel receipt time. Foreign, future, pre-process, truncated or cancelled observations never become empty/valid demand. No cursor is returned on refusal.
   • Three agent regressions reproduced before their fixes: concurrent ensure calls entered the shared creation path together; a restarted same-ID catcher replayed old-process demand; and a process restart during log reading returned the prior epoch. Ensure/remove now serialize within the agent process. Demand is bounded by StartedAt and a second identity/start-time inspection confirms the process after reading logs. The typed panel client independently checks process timestamps. All 42 related tests in four files passed without coverage; server typecheck and scoped lint passed. Tests use fake Docker/agent I/O; the existing catcher HTTP test uses an isolated local HTTP listener, not Docker.
   • Automatic recovery/polling is still unconnected. Per-agent ensure/remove serialization is not a panel maintenance fence: catcher replacement must coordinate with node sleep/wake admission and any active Stop before it is wired into the poller. Sleeping bindings must be refreshed only after the new catcher is reachable and fallback routing is acknowledged. Active-transition recovery, explicit-command settlement and hosted traffic acceptance remain open.
   • Private demand poller implemented 2026-11-03: nodeWakePoller.ts reads the pinned existing catcher's last minute through the sealed client and admits matching demands with the SQLite wake authority. It dispatches the actual wake controller independently of node polling, bounds node polls and wake jobs to four each, rotates node polling order, coalesces overlapping polls and backs off refusals for 30 seconds. It never creates or replaces a catcher. No timer/plugin activates this factory yet.
   • The one-minute stateless lookback survives a poller restart; journal admission and changing sleep tokens prevent rereads from becoming another wake. An expired/held wake journal recovers without a new demand and repeats readiness without repeating acknowledged Start. Recovery claims now compare the captured request/action/binding, so an old poll snapshot cannot claim a newer cycle. Shutdown aborts exchanges, waits for loading/polling and admitted jobs, and discards late demand replies. settle() drains current work when callers stop scheduling ticks.
   • Poller validation: 75 tests passed in five related files without coverage, with server typecheck and scoped lint clean. Poller tests use real in-memory SQLite, actual admission/controller/renderer logic and simulated agent/routing acknowledgement boundaries. They cover object recreation over the same journal, repeated/foreign/truncated/unknown demand, Stop during observation, refusal backoff, stale recovery identity and late shutdown replies. They do not prove a full panel-process restart or multi-node capacity under load. Initial fixture runs failed on single-connection transaction reads and Windows file-handle cleanup; neither is counted as passing. The tests now drain admitted work before inspecting in-memory state. Seven closed failed-run fixture directories under the OS temp directory remain because automatic approval rejected their cleanup. Existing separate-process journal recovery remains a separate proof. Automatic catcher recovery, sleep activation and hosted traffic acceptance remain open.
   • Durable node maintenance authority implemented 2026-11-03: migration 0121_node_wake_maintenance stores one intent per historical node ID, with a monotonic generation, request identity, bound sleeping-service snapshot, worker lease and ordered preparing/routing/rebinding/completed phases. Claiming refuses any active sleep/wake transition on that node. An unresolved intent transactionally blocks new sleep and wake admissions even after worker expiry; recovery takes the same intent with a new worker rather than removing the barrier. Other nodes remain independent. Holds release only the worker, and old workers cannot release recovered leases.
   • After the caller has confirmed catcher reachability and loaded fallback routing, settlement atomically refreshes still-owned sleeping bindings and completes the barrier. A newer explicit Stop is skipped; changed membership or configuration refuses settlement. A real SQLite rollback proof observes the first binding UPDATE execute, then verifies it was rolled back when a later member changed. Deleted nodes do not age-evict pending remote authority. This is the coordination foundation only: the actual replacement/reachability/routing controller and poller integration are not connected, and node sleep remains disabled.
   • Maintenance validation: 54 covering server tests passed in five files without coverage, plus a separate-process test persisting and recovering the same maintenance intent, rejecting its old worker and blocking wake before settlement. The entire database package passed 78 tests in 32 files, including upgrade from migration 120, unique request identity, millisecond lease precision and persistence after node deletion. Server typecheck and scoped lint passed. No local Docker was accessed; no real catcher replacement or production traffic acceptance is claimed.
   • Maintenance execution and private poller integration implemented 2026-11-03: the controller and production adapter ensure the catcher using the persisted node credential, require three stable TCP samples of the returned immutable ID, publish/confirm ordered fallback routing, then atomically refresh sleeping bindings. Every attempt re-proves readiness and routing, including a recovered rebinding phase; recording a catcher invalidates the earlier acknowledgement. Lost replies, unsupported/unreachable resources and old workers leave the durable barrier pending.
   • The poller uses this path only with explicit internal recoverCatchers opt-in. A refused/foreign catcher observation can reserve maintenance; a pending maintenance journal is resumed on the next poller instance without another demand. Normal polling still never replaces catchers by default. New admission remains blocked through the entire maintenance intent, while explicit Stop membership is excluded from rebinding. No production timer or public setting enables this flow yet.
   • Execution validation: 59 related tests passed in five files without coverage; server typecheck and scoped lint passed. Tests traverse the production maintenance adapter, typed sealed client, queued renderer and ordered proxy modules with fake agent I/O. They cover reachability refusal, lost route acknowledgement, repeated readiness on recovery, old-worker late responses, Stop supersession and opt-in poller recovery. They are not a zero-502 replacement certificate. Real traffic must be sampled throughout catcher recreation to prove non-disruptive replacement before activation; explicit-command fallback settlement and the full node sleep/wake acceptance remain open.

6. Surfaces: lift the node refusal in sleepHoldReason, the service API validation and the panel/CLI copy, but only for nodes that confirm both capabilities. Older agents keep the refusal.

7. Hosted smoke (smoke-node-canary.mjs --sleep-only) on isolated daemons:

   • an idle service sleeps after the window;
   • a request through the node proxy wakes it, and the reloaded page is served by the app;
   • a basic-auth domain asks for credentials before waking;
   • a request during the idle window prevents sleep;
   • agent opt-in off never sleeps;
   • Stop during a wake leaves the service stopped.

   Run it in CI and on the release image before node sleep is enabled.

Foundation integration update: PR #12 passed all source/runtime jobs in CI 41538310932 after the retention classification fix and merged as b75fb070. PR #13 passed CI 41538382781 and merged as f9b43f45. PR #14's internal wake controller passed CI 41538384727 and merged as 1262e8b9. These certificates cover their frozen heads and existing runtime cohorts, not the later sleep/poller/maintenance work, full hosted sleep/wake acceptance or a new release.

Catcher replacement: real traffic RED — 2026-11-03

Hosted run 41547605441, source d40b14bb56c94a0f21298faae4d050cd50b6bcdc, reproduced the replacement gap with real Docker, sealed HTTP and Traefik. The fixture seeds a dormant journal with simulated idle/Stop receipts; it does not accept real idle duration or shutdown. A derivative image with identical binaries but a different immutable image ID forces the production maintenance controller to replace the catcher. A Docker-boundary shim pauses creation for three seconds without modifying application code.

The public trace independently retained 948 requests: 89 HTTP 502, one connection error (status 0, not classified more precisely), and 858 HTTP 503 responses. Both first and last samples returned 503; this version sampled status only, not page contents. The catcher inherited the image's panel/agent healthcheck, which probes a different port; maintenance remained held because catcher readiness was unconfirmed. All owned containers, the fixture network and the derivative image were removed. The JSONL and array traces independently agree. Evidence: .temp_files/wake-recovery-red-41547605441/independent-verification.json and the run's public wake-recovery-41547605441 artifact.

Earlier runs 41546537854 and 41546963312 were fixture diagnostics, not product traffic REDs: a required production JWT was missing; then Node 26 fetch did not send the custom Host header and child environment scrubbing removed shim-only variables. The fixture now supplies an ephemeral production secret, uses node:http for the Host-bound request and stores shim controls in its private directory. Private configuration, credentials and SQLite files are excluded from uploaded artifacts. The second diagnostic's failed cleanup is not accepted as a cleanup pass.

Commit 91eb7f75 disables the inherited image healthcheck only for catcher creation; real catcher readiness still probes port 8080. Its unchanged regression was seen failing before the fix, then all 25 catcher/agent tests passed; server typecheck and scoped lint passed. Run 41548044955, frozen source 2dfb90de4dc64c1115e64fb35db5d992aa1ea77e, confirmed native healthcheck NONE and completed real sealed maintenance, but independently reproduced 135 HTTP 502 and 413 HTTP 503 across 548 requests. All owned cleanup passed. Thus the healthcheck fix is verified separately; replacement acceptance remains RED.

The next harness revision also requires Retry-After: 5, the waking-page text and its refresh marker on every 503, and records bounded network error codes. Five real HTTP controls distinguish the valid page, generic 503, missing retry header, oversized body and reset connection. Earlier status-only traces are not retroactively treated as body verification. The creation gap remains open: replacement must keep the old catcher reachable until a ready new generation has confirmed routing, then retire it safely. Node sleep and automatic replacement remain disabled until that traffic acceptance and explicit-command fallback settlement pass.

Run 41549047503, source 945b0f094340bb3a340846921aa6ca20e75309cb, repeated the unchanged replacement path with the stronger page checks: 554 requests, 135 HTTP 502 and 419 confirmed waking pages. Maintenance completed, the before/after waking-page controls passed and all owned cleanup passed. JSONL and array traces were independently recounted. This is the frozen RED baseline for generation promotion; it is not a successful replacement acceptance.

The agent has a separate sealed wake.catcher.prepare operation and capability. It creates an agent-owned generation without removing the legacy catcher or any earlier generation. Lost-reply retries reuse that generation, a stopped matching candidate restarts with the same immutable ID, and conflicting identity/credential/ownership refuses without deletion. The typed client requires the new capability; old agents cannot accidentally use the destructive legacy ensure path as preparation. The later retirement increment below replaces deterministic instance names with non-reused names and image/key labels for lookup.

Preparation validation: the old backend's disappearance was reproduced at the Docker creation boundary with 22 passing controls, then the same regression passed. The final covering server run passed 443 tests in 23 files, including registered operation dispatch, opt-in/kill-switch refusal and the typed sealed client; the schemas contract file passed 24 tests. Server typecheck, schemas build and scoped lint passed. Docker was blocked locally; these tests fake only agent/Docker I/O and do not certify hosted generation promotion.

Staged catcher routing — hosted handoff accepted, retirement open

Migration 122 preserves existing bindings as ninedeploy-wake and records generation names for new bindings. Sleep and maintenance now prepare separate catchers and prove TCP readiness against the returned immutable ID/name. Maintenance receives a process-local, short-lived promotion permission bound to the current worker, candidate and captured service memberships. The renderer and every proxy exchange recheck it; old-worker responses cannot publish or settle another worker's intent. Exact ordered routing acknowledgement and binding settlement execute inside the same routing queue slot, so the next queued render observes the confirmed new binding. Lost acknowledgement retains the old binding and both backends. Demand for a named generation is read by immutable catcher ID instead of the legacy fixed name.

Local validation: 140 tests in eight handoff/authority/client/poller files and 162 tests in eight neighbouring proxy/wake/recovery/Swarm files passed without coverage. The full DB package passed 79 tests in 33 files, including an upgrade from a populated migration-121 schema. Server typecheck and scoped lint passed; a subsequent schema generation found no drift.

Hosted run 41551341118, frozen source 077823b9ae5b7bb0b0d7ccba17dcf6a5dc5c0c42, passed the real Docker/sealed HTTP/Traefik handoff with the same three-second creation delay: 583 requests, all verified waking-page HTTP 503, zero HTTP 502 and zero connection errors. The new immutable catcher recorded real visitor demand, durable maintenance completed and the old backend remained available. Independent JSONL/array recounting agreed; all four owned containers, the network and derivative image were removed by fixture cleanup. Public artifact: wake-recovery-41551341118; local independent receipt: .temp_files/wake-recovery-staged-41551341118/independent-verification.json. This closes the demonstrated handoff gap against the earlier 135-502 baseline only for this cohort.

This handoff intentionally retains previous catchers. The hosted fixture now also checks that actual requests produce demand on the new generation and reports whether the old backend remains. It proves staged traffic handoff, not safe retirement, actual idle shutdown, public activation or the complete sleep/wake cohort. Retirement requires the remaining explicit-command fallback settlement and loaded-route/drain safeguards. Production node sleep remains disabled.

Explicit Start/Restart fallback settlement — local validation

The actual lifecycle controller and runtime reproduced a second defect: explicit Start and Restart reached completed with a running app but continued routing visitors to the superseded wake catcher. Both contract assertions failed while explicit Stop remained correctly withdrawn. The fixture's initial incomplete fake-clock scheduling attempts are diagnostics, not product REDs; the settled production-path run reproduced both routing failures with one passing Stop control.

Readiness-confirmed Start/Restart now acquire an app-route promotion permission bound to the current lifecycle owner and the exact superseded fallback. Ordered app routing must be acknowledged before the routing queue clears that fallback, and a newer Stop or worker invalidates the permission. The unchanged Start/Restart regressions and Stop control passed after the fix; the expanded seven-test file also covers readiness refusal, lost acknowledgement/retry, Stop supersession and the next queued ordinary render. The final covering run passed 116 tests in five files without coverage; server typecheck passed and scoped lint reported no warnings or errors (four existing style information notices remain). These checks fake agent I/O and do not accept the full real sleep/wake cohort or retirement.

CI and the published-image agent-runtime release cohort now require the staged handoff smoke. Both preserve only its public/ evidence directory; credentials, SQLite state and generated private routing remain excluded. The release matrix guard first failed on the missing cohort, then all ten matrix/runner-refusal checks passed after the workflow addition. The added mandatory steps still require a complete run on the later source and a future published candidate; the earlier dedicated handoff run does not certify those workflow executions.

Agent retirement protocol — hosted verification pending

The sealed wake.catcher.retire operation binds removal to the exact acknowledged ordered head and immutable managed container ID/name. It refuses a requested route that still names that backend, checks all observed HTTP/TLS definitions (including unexpected extra definitions), drains for two seconds and repeats both live probes. A persistent retired-name fence commits before removal; a lost Docker reply cannot roll it back. Only exact missing-container evidence counts as confirmed absence, and a foreign or renamed target is retained.

Both ordered-write phases reject retired backend references. Connection-local SQLite writer admission also prevents an older agent implementation from bypassing that validation after retirement. Preparation shares the native proxy journal lock and now finds retry candidates through image/key labels. Every newly created instance receives a fresh private name; deleted or fenced names are never reused. This permits safe retries without reviving a retired endpoint.

The previous writer reproduced acceptance of a higher-revision retired-backend route, with a valid new-backend control passing. The same proof now rejects that route; a separate check confirms the older writer connection is fenced while the current writer still accepts valid routing. Native journal, operation, capability and client checks passed 150 tests in seven files without coverage; the full schemas package passed 803 tests in 40 files, and server typecheck/scoped lint passed. No local Docker was invoked.

The hosted fixture has been extended to refuse retirement of the actually serving legacy catcher under a valid loaded ordered head, then explicitly retire that catcher after handoff while sampling requests continuously. This exercises the agent RPC explicitly: automatic panel retirement/recovery integration, full sleep/wake, operator status and activation remain unfinished. A hosted result for this increment is still required.

Run 41557117837 on 5df1c165739abfcb32e111c0b0b5b6236ff55aae stopped in the negative-control fixture because agentOp throws on non-zero exit by default. The fixture must explicitly tolerate the expected refusal before inspecting its exact error frame. This is a fixture diagnostic, not a traffic RED or a successful retirement certificate. All owned containers, network and derivative image were removed. The refusal assertions remain unchanged.

The raw panel transport now also rejects plaintext for the entire wake.* and proxy.idleWatch.* families, even when an operator enabled the global legacy cleartext fallback. Typed preflight alone is insufficient because agentOp reselects transport. Six transport regressions reproduced a successful unsealed exec before the fix (69 other controls passed), then all 107 covering transport/client tests in three files passed. Sealed positive controls verify operands remain inside the envelope. This change sends no production credentials and was verified with mocked network I/O.

Run 41599729256, frozen source c3c367d266646b79d959cddb17a3405808cb12c4, passed the extended agent retirement cohort: the active catcher refused retirement under a valid loaded ordered head, then the old immutable ID was physically removed after handoff/drain. All 700 requests were verified waking pages, with zero 502/connection errors. Independent JSONL/array recounting, real new-generation demand, exact retirement receipt and owned cleanup all passed. Artifact: wake-recovery-41599729256; local independent receipt: .temp_files/wake-retirement-41599729256/independent-verification.json. Retirement was invoked explicitly by the fixture; this certificate does not cover the later automatic panel retirement path.

Reference validation now examines actual server/auth addresses rather than every string in a proxy document. A new regression reproduced rejection of valid CSP/Link/descriptive response headers after a retirement fence (11 controls passed), then all 21 native retirement/journal tests in three files passed after the fix. Current writer admission remains compatible with these headers while retired server addresses remain refused. The older-writer check covers prior ordered-writer implementations, not arbitrary panel/agent downgrades.

Automatic panel retirement — hosted component verified

Production maintenance now retains its worker/admission barrier in retiring after confirmed binding settlement. It rechecks catcher readiness and selected fallback references, confirms a fresh ordered route inside the routing queue, then invokes the typed sealed retirement RPC. Completion requires an exact receipt set for every captured older ID. A lost removal response keeps the same durable intent; takeover repeats absence/fence proof without repeating binding publication. A superseding explicit Start that still selects an old fallback retains that catcher until its own route settles.

If the ready candidate changes during interrupted maintenance, rerouting/retirement_rebinding recover current sleeping memberships while preserving all older IDs as retirement-only targets. The intent identity survives; a new binding and worker token invalidate old continuations. Retirement-only entries never regain service ownership or override explicit-command routing. These are additional values in the existing SQLite text/JSON fields, with no schema DDL drift.

Local validation passed 62 server tests in seven files, including real production adapters, changed-candidate recovery, lost replies, late workers, selected-reference protection, separate-process persisted recovery and the Swarm reader inventory. The full DB package passed 79 tests in 33 files. Earlier incomplete fixture classifications were corrected: stopped/superseded membership must keep its original binding, and the new SQL-only runtime reader is classified explicitly. Typecheck and scoped lint passed; schema generation found no DDL changes.

The hosted cohort requires controller-driven retirement, idempotent replay of an already absent ID, successful ordered routing after the native gate is adopted, CSP response-header compatibility and measured traffic after confirmed retirement. This does not certify an actual idle window, application Stop/wake, full public status/activation, or cleanup of every never-bound orphan candidate.

Run 41607197631, frozen source cee91bda7282b5bfcaa0373c417a5a7e9ce1f409, passed all eight automatic-retirement cases. The production maintenance controller removed the old immutable ID before reporting completion. An explicit replay then confirmed its absence with removed: false and fenced: true; a subsequent ordered write and loaded-route acknowledgement succeeded. Independent JSONL/array recounting confirmed 1,099 verified waking pages, zero 502/connection errors, and 48 requests after confirmed retirement. Live-target refusal, actual demand on the replacement and all five owned cleanup receipts passed. Evidence: .temp_files/wake-auto-retirement-41607197631/independent-verification.json. This fixture seeds dormant state; actual idle/Stop/wake and public activation remain unverified.

Wake publication settlement

A permanent production-runtime regression reproduced a completed wake whose next queued ordinary proxy writer restored the catcher. Wake completion now runs inside the acknowledged writer's serialized routing slot. Both automatic wake and explicit Start/Restart settlement require the exact acknowledged ordered generation (node, revision, request ID and content digest) still to own the DB head inside the metadata transaction. The acknowledgement comes from the actual successful node sync, not a later read of the current head.

Reservations precede rendering. If another writer reserves before settlement, even without binding content yet, completion holds with fallback retained; retry repeats readiness and publication. A writer reserving after settlement renders the cleared fallback. Additional interleaving cases cover both bound and unbound competing reservations for automatic wake and explicit Start. These are controller/SQLite tests with network I/O substituted, not a separate-process or real-Docker sleep/wake certificate.

The original queued-writer regression failed on the final catcher-route assertion while its ready-app publication control passed. After the fix, 94 tests in eight files passed across wake, explicit settlement, authority, poller and maintenance/recovery. A second run passed 179 tests in four files covering the shared proxy, Swarm runtime inventory, lifecycle controller and maintenance authority. Server typecheck and scoped lint passed. One intermediate test expectation required the newly returned routing authority; its final assertion checks the exact DB generation and written-content digest rather than omitting the field.

Full hosted CI 41602638625 passed for the earlier native-retirement source c3c367d266646b79d959cddb17a3405808cb12c4: source checks, real Docker/PostgreSQL integration and all image/runtime smoke steps passed. This does not certify the later automatic-maintenance or wake-settlement source; their final combined CI/release gates remain separate.

Aborted sleep restoration settlement

The same queue interleaving independently reproduced on a pre-Stop failure: the catcher ACK was lost, the controller restored a ready application and marked sleep aborted, but a queued ordinary writer restored the catcher. Its ready-app control passed; the final routing assertion failed. Restoration now settles its fallback inside the acknowledged routing slot and compares the exact ordered generation transactionally, using the same guard as wake and explicit Start/Restart. A newer bound or unbound reservation holds restoration for retry rather than completing an obsolete ACK. Recovery restores the app and never resumes the old idle decision or issues Stop.

The unchanged queued restoration regression passed after the fix. The covering run passed 99 tests in six files across sleep, wake, poller, explicit settlement, authority and the Swarm reader inventory; server typecheck and scoped lint passed. The first filtered reproduction invocation had incorrectly escaped Windows arguments and was stopped without being counted as evidence; the corrected invocation failed on the intended routing assertion with 12 other tests skipped. These tests substitute agent I/O and do not certify the real-Docker pre-Stop failure path.

Real idle acceptance: concurrent metadata readers

The dedicated real-idle fixture uses the production scheduler/client and a real five-minute window, real immutable Stop/Start, strict HTTP readiness failure, an independent traffic observer and recovery through a new Node/HTTP-agent process. It keeps credentials outside public artifacts and does not seed dormant state. Run 41609229034 on 4a65b21c7bcbd0d89ec6806358cc1eb4f19eae86 failed: the serving-request reset control passed, but four SQLITE_BUSY errors prevented continuous idle coverage. All four owned containers and the network were removed. Artifact: .temp_files/node-sleep-wake-41609229034/. No Stop/wake or zero-502 certificate follows from that run.

An isolated file-backed subprocess reproduces the failure without Docker: one observer obtains the expected stable context, but a pair of concurrent production context reads yields one successful result and one SQLite lock refusal. Drizzle's libSQL adapter starts write transactions even for the read-only snapshots. Serialize only the short context-capture operations per DB handle, preserving both transactional snapshots and before/after route checks; agent RPCs and observation windows remain independent. The unchanged concurrency assertion now passes, and a refused routing context does not poison later queued readers. This does not claim cross-process database serialization or a passing hosted idle/Stop/wake cohort; the corrected real cohort must run again.

The covering metadata/idle scheduler/client/coverage run passed 120 tests in five files without coverage; server typecheck and scoped lint passed. The file-backed proof runs in a child process and removes its owned directory only after child exit. Local Docker remains disabled.

Real single-service idle/Stop/wake — hosted component verified

Run 41611317840 passed all ten cases on frozen source f71e074cf7da56ec46c33fe7b18899fc87cc5038. The actual scheduler established 311,749 ms of continuous zero-request coverage after proving that real serving traffic resets it. The production sleep controller stopped the captured immutable app, with the verified catcher route already serving. Real visitor demand caused Start; intentional strict HTTP readiness failure retained the fallback. A new Node/HTTP-agent process reopened persisted readiness, published the app after readiness passed, and did not repeat Start. The ordinary follow-up writer retained the app; its container and the proxy kept their initial IDs.

Independent JSONL/array recounting confirmed 1,788 requests: 494 application responses and 1,294 verified waking pages, with zero 502/connection errors. All 396 samples inside the failed-readiness interval remained waking pages; 59 samples after completed recovery were application responses. The two controller PIDs differ, no observation/controller error was recorded, and all five owned containers plus the network were removed. Evidence: .temp_files/node-sleep-wake-41611317840/independent-verification.json. This is real sealed HTTP/Docker/Traefik behavior with actual elapsed time, but internal factories rather than public API activation. Basic Auth/IP restrictions, agent opt-out, concurrent services, Stop during wake and pre-Stop restoration faults remain additional acceptance work.

Shared managed transaction ordering

Preparing concurrent services exposed the same ownership conflict beyond metadata reads. Two controlled DB regressions demonstrate that a second legitimate managed transaction is rejected while the first awaits commit or rollback, even though a transaction on an independent DB succeeds. Queue complete Drizzle transactions per DB handle through commit/rollback, preserving nested savepoints and the existing connection PRAGMAs. The earlier observation-only queue is removed; the transactional snapshots and route comparisons remain intact. This covers managed transactions on one handle, not raw client calls or serialization across different processes/handles. The two regressions were RED before the change and GREEN afterward. The full DB package passed 82 tests in 34 files, including rollback, independent handles and nested savepoints. The server covering run passed 245 tests in 14 files, including file-backed read/write concurrency, scheduler/client/coverage, sleep/wake, explicit commands, maintenance recovery and Swarm inventory. DB build, server typecheck and scoped lint passed. This later source still requires full hosted CI and the expanded real cohort.

The source job in CI 41608355751, frozen wake-settlement source b1796e82, reported 13,153 server tests passing and one failure in proxyNodeSleep.test.ts: its exact successful-result expectation omitted the new acknowledged routing authority. The expectation now also checks the exact DB head, revision/request identity and written-content digest; all 18 tests in that file pass locally. This is a contract-expectation correction, not a successful full-CI certificate. The rest of that workflow was still running when its completed source-job log was inspected; do not discard or restart it to recover observation.

Main integration and expanded guard acceptance

PRs #15–#18 passed their frozen full CI gates and were merged to main on 2026-11-03: 8560ea42, 62d78012, b6b8e2ef and c1777241. PR #22 then passed full CI 41616043392 on f86d10a53779ae50c55df2c24709f1f5d266f799, including source checks, real Docker/PostgreSQL integration and all image/runtime smoke steps, and merged as c3e0214a. Image publication was skipped for the PR as expected. Git ancestry verifies that all three complete heads from #19–#21 are present in #22; those intermediate PRs were closed as superseded, without dropping their changes or claiming they merged independently. The local main history was reconciled with both integration batches while preserving the prepared source tree. PR #23 is the next direct-to-main manual-wake increment. No production activation or release followed from these component merges.

Expanded run 41614144107, source 3259a5a8b604e4a5cdfc5a4cfb7af47a54170e19, stopped at the opt-out fixture's expected-error wording. The operation correctly refused, but the fixture expected an environment hint while the development image still reports 0.18.16 and returns its 0.19.0 upgrade refusal. This is a fixture diagnostic; none of the later concurrent-idle/auth/Stop cases was accepted. Five owned containers and the network were removed. The corrected control requires the advertised capability to be present before opt-out, absent during opt-out, restored afterward, plus the exact observation-refusal family and an untouched runtime/journal. Syntax and scoped lint passed; a new real result remains required.

Follow-up run 41615171431, source 14544a66536137d59e5e6a521f5f24e340472a72, then stopped on another fixture defect before idle qualification: its per-service journal helper used services.id, which Drizzle rendered as the nonexistent nodeSleepTransitions.id. It now selects nodeSleepTransitions.serviceId. A local preflight executes the actual fixture helper expressions against fully migrated SQLite for both service IDs; all four current queries succeed, while the prior wrong-column control reproduces the error. Syntax/scoped lint pass. Five owned containers and the network were removed; no expanded acceptance is claimed for this failed run.

Guarded two-service idle/Stop/wake — hosted component verified

Run 41616036959 on frozen source f86d10a53779ae50c55df2c24709f1f5d266f799 passed all 14 cases. Two service entries in the production scheduler established 322,069 ms and 324,936 ms of real idle evidence, then their sleep controllers ran concurrently. The agent opt-out control verified advertised capability before/off/restored and no runtime/journal effect. Unauthorized Basic Auth and disallowed-IP requests returned 401/403 without recording the protected service's token or starting it. An authenticated request subsequently started its wake; a real explicit Stop superseded it while strict HTTP readiness was in flight. Its immutable application remained stopped after a fresh poller, while the other service continued serving.

Independent JSONL/array recounting confirmed 2,328 valid responses: 900 application responses, 1,428 verified waking pages, zero 502/connection errors, 400 waking pages during failed readiness, and 59 application responses after all cases. New controller/agent PIDs differ, no observation/controller errors were recorded, and all six owned containers plus the network were removed. Evidence: .temp_files/node-sleep-wake-41616036959/independent-verification.json. These are isolated real sealed HTTP/Docker/Traefik effects through internal factories. Public API/plugin activation and later source changes are outside this certificate.

Manual node wake entry point and client feedback

The public manual-wake route previously passed a node's sleeping row to the panel-host wake implementation. Three route regressions reproduced the planned local docker start web-9 effect through mocked process I/O; seven local-route controls passed. Ordinary node Docker wake now admits the existing durable remote Start command and returns 202 with pending: true, leaving readiness and loaded-route confirmation to the ordered worker. Unsupported node Compose/Swarm placement and disabled/unsealed agents refuse without local Docker effects or a new intent. This does not enable automatic sleep or bypass the idle evidence needed for sleep admission.

SDK wake results reuse ServiceLifecycleResult. Panel and CLI treat accepted commands as pending rather than reporting the service awake. The panel observes the existing lifecycle query, retains newer completed state, disables repeated wake during verification and binds a delayed acceptance to its original service after navigation. Three API, two initial UI and one CLI regression were RED before the corresponding fixes; two additional UI checks caught repeat submission and delayed-response handling during implementation. The covering runs passed 1,264 server tests in six files, 164 web tests in four files, the full SDK package (384 tests in 34 files) and 20 CLI tests. SDK build, server/web/CLI typechecks and scoped lint passed. These checks substitute process/network I/O; real public manual-wake acceptance and final full CI remain required for this later source.

Public manual wake — native before/after verified

Identical fixtures ran on before source befc77e269d35016992eff4424b50d3893476e76 (run 41622494732) and fixed source 5587e972ab0da9bebc09c15e891c6e64f808ed9e (run 41622507659). Each used a real authenticated panel, its normal lifecycle worker, standard sealed node enrollment, separate owned panel/node Docker daemons and genuine idle/Stop through a private production factory. Observed idle was 315,614 ms before and 315,775 ms after the fix.

The before run failed on the harmful-effect assertion: public manual wake changed the stopped same-name panel decoy's StartedAt from its never-started value to a real process timestamp. This was an actual isolated Docker mutation, not a fixture setup/import failure or merely an unexpected HTTP code. Its normal deployment and real idle/asleep controls had passed. The fixed run kept that immutable panel decoy never started, acknowledged public 202/pending, held failed strict HTTP readiness behind the catcher, retained the command ID on explicit retry, and completed readiness/routing through the normal worker with the original node container ID preserved.

Independent recounting confirmed 675 valid responses: 600 verified waking pages, 75 application responses, zero 502/connection errors. Both cohorts removed their three outer containers, both owned volumes and the owned network. The dedicated upload accepts only enumerated public JSON files; tokens, DBs and raw logs are excluded. Evidence: .temp_files/public-manual-wake-fixed-41622507659/independent-verification.json. This accepts the public manual-wake entry point; idle admission still uses a private real-observation factory. Automatic public sleep/status/plugin activation and full CI for PR #23 remain open.

Lost pre-Stop acknowledgement — native acceptance in progress

The next isolated cohort is run 41627779632, frozen source 9eec752efe773c3ae0af1ef6766b44d979506587. Normal authenticated API deployment and sealed agent enrollment establish the healthy control. A private scheduler must observe and admit genuine five-minute idle before the independent traffic observer starts. A loopback-only forwarder sends the original sealed bytes unchanged to the actual enrolled agent. After verifying its nonce, successful result and exact loaded-routing hash, it holds that response for one second, then drops the panel-facing connection. Production sleep restoration must abort the old idle decision, issue no Stop/Start, clear fallback metadata, confirm an ordinary subsequent writer and leave the immutable application's start/finish timestamps unchanged. Traffic must include at least 20 verified waking pages during the lost acknowledgement and remain on the application after restoration, with no unexpected response or connection failure.

This is a real transport-fault acceptance requirement, not a replacement for the existing queued-writer regression. The fixture's network/sampler checks and covering controller checks passed locally: 22 tests in three files, then seven helper tests in two files after the final observer/delay change. Script syntax, generated sampler syntax, workflow parsing, enumerated JSON-only upload paths and all three pre-Docker hosted-runner guards passed. Local Docker remained disabled. The hosted run is still in progress; no native restoration result is claimed yet. PR #23 retains its frozen head while this separate proof branch runs, without opening another integration PR.

PR #23's full CI 41622799651 passed on 5587e972ab0da9bebc09c15e891c6e64f808ed9e: 18,294 tests in 1,160 files, all 12 tasks, typecheck/lint/build, post-test checks, real Docker/PostgreSQL integration and the full image/runtime job including RPS source-loss acceptance. The separate serving-generation check also passed. The PR merged to main as 9e79b931e82922cbc47ba23e2949c95d87774110 on 2026-11-03; image publication was skipped for the PR as expected. No new release was published.

Lost pre-Stop acknowledgement — native transport fault verified

Run 41627779632 completed successfully on 9eec752efe773c3ae0af1ef6766b44d979506587. Its private production scheduler observed 315,721 ms of genuine idle before admission. The loopback-only forwarder verified the real sealed agent's successful nonce/hash-bound loaded-routing acknowledgement, then dropped the panel-facing connection. The sleep controller restored the healthy application, completed its journal with fallbackRouting: false, and confirmed a later ordinary writer. No runtime Stop/Start was issued; the original immutable application's start/finish timestamps remained unchanged.

Independent artifact verification counted 640 valid responses: 216 verified waking pages, 424 application responses, zero 502/connection errors, and the last 40 responses all from the application after restoration. All three owned outer containers, both volumes and the network were removed. Evidence: .temp_files/node-sleep-restoration-41627779632/independent-verification.json. This proves the stated real pre-Stop lost-acknowledgement path; it does not certify every possible fault timing or activate public automatic sleep. The existing production-renderer queued-writer regression remains the proof for settlement ordering. Integration CI for this later fixture source and the automatic scheduler/coordinator/status/plugin path remain separate work.

Bounded sleep coordinator — local component checkpoint

createNodeSleepCoordinator now reserves at most four execution slots, consumes an opaque idle token only inside a reserved slot, prioritizes expired pending sleep commands and leaves wake journals to the wake poller. It recovers the same Stop identity after a lost acknowledgement, backs off held work for 30 seconds, skips ineligible placements before consuming evidence, and aborts/drains owned work on close. Shutdown before publication completes the unused sleep decision; shutdown with fallback routing leaves restoration recoverable, so the next coordinator restores the application instead of resuming the old idle Stop.

The covering six-file run passed 55 tests. After the placement filter was added, the coordinator's two-file run passed 17 tests, including the eight ineligible-placement controls. Server typecheck and scoped lint passed. Network calls are substituted; the actual admission/controller/renderer and managed SQLite transactions are used. The four-slot concurrency proof runs with a real file-backed database in a child process, whose exit precedes Windows cleanup. The memory-only diagnostic reproduced TRANSACTION_ACTIVE: its single connection cannot execute an unscoped query while borrowed by a transaction. The identical file-backed proof admitted four services, consumed none of the other two tokens and recorded no error. That diagnostic is not claimed as a production defect or fixed by weakening concurrency.

The coordinator is not yet connected to the production sleeper plugin. Public node sleep/status reporting, automatic wake/maintenance scheduling and full public automatic-cycle acceptance remain required before activation. PR #24 contains only the earlier hosted transport-fault fixture and documentation, with its full CI running on frozen head feeffe8f6156a55e39e766f992aab320089028c0; it does not contain this coordinator checkpoint.

Public automatic cycle candidate — local validation only

The unreleased candidate connects the idle observer, bounded sleep coordinator and wake/maintenance poller to the sleeper plugin's lifecycle. Idle observations run every 15 seconds; pending sleep recovery and wake demand run every two seconds. Close unregisters status, stops timers, aborts and drains the three components. Each service still requires its own sleepAfterMinutes setting, eligible ordinary node Docker placement and the sealed agent's advertised capabilities/traffic opt-in. NINEDEPLOY_NODE_SLEEP_AUTOMATION=off pauses automatic node sleep, wake and recovery while retaining explicit Start/Wake. The private-factory manual/transport-fault fixtures set this pause so the normal plugin cannot compete with their explicitly controlled admission.

GET /v1/services/:id/sleep is a viewer-scoped read-only status endpoint shared by SDK, CLI and panel. It exposes observation duration and coarse observing/sleeping/asleep/waking/held state; no wake credential, immutable target, lease token, runtime owner identity or raw worker reason is returned. App-local observer registration cannot leak coverage across DB handles. The panel removes unsupported manual node Sleep now, shows verified idle coverage and pending wake, and keeps manual wake available for held verification. The CLI's node status does not infer awake from an accepted command. Older panels returning 404 keep an explicit unverified node state.

Local checks passed: 77 server checks in seven files before the later authorization classification, then 1,064 authorization/environment checks in two files; 170 panel tests in four files; all 384 SDK tests in 34 files; 21 CLI checks. Schema/SDK builds, server/web/CLI typechecks and scoped lint passed. The reviewed API snapshot and authorization matrix include the new viewer/read endpoint. A dedicated hosted-only cohort is prepared to prove ordinary public settings, a real traffic request resetting idle, five more idle minutes, automatic Stop, visitor-demand wake, failed readiness, panel restart and automatic recovery with sampled Traefik traffic. Its preflight syntax, YAML/JSON-only artifacts and all hosted guards passed. That native cohort and full CI for this later source are still required; no production/public acceptance or release is claimed here.

The public automatic cohort is now running as 41633395833, frozen source 8f26e19c4af2428807d271ef5a2067e5b3eef854. Its source tree matches the prepared local candidate before this documentation update. PR #24's separate CI 41630104741 is still running on feeffe8f6156a55e39e766f992aab320089028c0; it does not certify the public automatic-cycle candidate. Neither frozen branch was updated to restart its checks. No additional integration PR or release was opened.

Public automatic cohort control correction and activity settlement

Run 41633395833 failed at the healthy serving-request control after normal API deployment and the first public idle observation; it did not accept the later automatic sleep/wake/restart cases. All three owned outer containers, both volumes and the network were removed. Its control used fetch with a custom Host. A local Node 26.12.1 server proved that this sends the transport host instead, whereas node:http sends the requested application Host. The control now uses bounded node:http, verifies status and exact application body, and saves a public status/boolean receipt. The real-HTTP regression failed with the prior fetch implementation, then passed unchanged with the corrected transport. Four controls now pass, including unhealthy status, a different backend and an oversized body.

Automatic sleep/wake settlement also now calls the normal audit/activity fan-out only after the Stop receipt or ready application routing is committed. Held, superseded and restoration-aborted work does not claim sleep/wake completion; recovery emits the completion only when it settles. The covering coordinator/poller/audit checks passed 28 tests in three files; server typecheck and scoped lint passed. Audit remains the application's existing best-effort fan-out, not a new durable notification outbox. The hosted cohort now additionally requires one persisted sleep event, no wake event during failed readiness, and one persisted wake event after panel recovery and confirmed routing. Native acceptance for this corrected later source remains required.

The corrected public cohort is run 41634736108, frozen source cd2ca25fbc1ea2cb9974d915155fb9241b8b55be; it is still running. An independent verifier is prepared to recount its traffic, require the public control/reset/idle and persisted audit receipts, verify its source identity and owned cleanup, and reject private-factory admission. No result is claimed before those artifacts exist. Separately, the Turkish catalog check reproduced 13 missing node-state translations and the obsolete node-unsupported key. The catalog correction passed all 35 checks in the catalog/SleepState/SleepCard run. These translation-only changes are outside the frozen native source and do not restart it. PR #24's source job passed 18,298 tests in 1,161 files; its final image/RPS job remains in progress.

Public automatic cycle — hosted API/plugin acceptance verified

Run 41634736108 completed successfully on frozen source cd2ca25fbc1ea2cb9974d915155fb9241b8b55be. The independent verifier confirmed ordinary public API settings and production plugin timers, with no private factory admission. A healthy application request reset the public idle coverage to zero. Only after 402,743 ms since that request, with 344,815 ms of conservatively verified idle coverage, did the normal coordinator complete automatic Stop on the captured immutable node container. Visitor demand then ran the normal wake poller. Failed strict HTTP readiness retained the catcher and sleeping marker; a panel restart recovered the pending wake without changing the acknowledged application's start timestamp. Clearing the readiness fault let automatic retry confirm the application route.

Independent artifact recounting confirmed 791 valid responses: 677 verified waking pages, 114 application responses, zero 502/connection errors, and the final 40 all from the application. The public activity API returned exactly one service.sleep event after Stop, no service.wake during failed readiness, and exactly one service.wake after confirmed routing and recovery. All three owned outer containers, both volumes and the network were removed. Evidence: .temp_files/node-automatic-sleep-41634736108/independent-verification.json. This accepts the stated single-node public automatic cycle and restart path; it does not certify public DNS/TLS, every fault timing, never-bound orphan catcher collection or the complete R2–R6/release scope.

The actual new React source components were also rendered in an owned headless Chrome fixture at 1280px and 375px, English and Turkish. Observing, waking, held and paused states had no page overflow or browser errors; automatic waking disabled repeated wake, held/paused kept manual wake available, and awake nodes exposed no manual Sleep now. The long held message wrapped in the mobile render. This substitutes the client boundary and is component visual acceptance, not a real panel/backend browser journey. Evidence: .temp_files/node-sleep-ui-20261103/result.json and four screenshots. The owned Chrome and Vite processes were closed.

PR #24 passed full CI 41630104741, including 18,298 tests in 1,161 files, real Docker/PostgreSQL integration and the final image/RPS cohort, and merged to main as 1b9334f1b07e3b966fd50252b2d7f3c6c112a26a. Its source excludes this public automatic candidate. Local main history was reconciled while preserving the prepared source tree exactly. No release was published.

Public candidate integration gate — reader inventory correction

PR #25's first CI 41637057912, head 49e5bd3a84e37a0edafce6d5f7475f419b23d3f5, passed 13,198 server tests but failed the single static Swarm runtime-reader inventory check. The new nodeSleepCoordinator and serviceSleepStatus readers had not been classified. The identical check reproduced locally. The inventory now states their actual boundary: new sleep admission skips Swarm before consuming evidence, recovery revalidates ordinary node Docker authority before effects, and public status returns unsupported for Swarm before consulting node observations or journal identities. An additional status behavior control proves that a Swarm row never consults the node observer. All 34 covering checks in three files passed after the correction. Runtime source and coverage floors did not change. Fresh full CI on the corrected PR head remains required; the old image/runtime job is allowed to finish without being canceled by this test-only update.

The old image/runtime job subsequently passed every step, including PostgreSQL autoscaling and the final RPS source-loss control; the real Docker/PostgreSQL integration job also passed. The workflow remains failed because of the explicit source inventory failure. CI now runs the existing 13-test Swarm runtime guard immediately after build and before the long full suite, with Docker explicitly unavailable for that guard. All 13 checks passed locally. The existing full suite still follows it. The corrected PR update contains only tests, this early CI guard and evidence documentation; it preserves the public automatic-cycle runtime source accepted by run 41634736108.