v0.4.1.

The v0.4.0 tag was published without the post-Sprint 11 fixes and the GHCR image pipeline; this 0.4.1 release re-publishes every change listed below as one coherent version. There will not be further vX.Y.Z-hotfixN tags — patch fixes ship as the next semver patch so install.sh / one-click panel self-update can find them through ^v\d+\.\d+\.\d+$ without manual version pinning.

Installer & Release

Installer & Release

• CI publishes the panel image to GHCR on every push to main so a fresh install.sh --channel=main lands on a current image, not a stale one. The job tags the multi-arch (amd64 + arm64) build as :edge and an immutable :main-<sha>; the existing release workflow (tag push) re-tags :latest so a fresh install.sh --docker from the release channel still pulls the most recent published artifact. install.sh now substitutes the right tag (:edge for main, :latest for release) into the rendered compose file, and a preflight docker manifest inspect runs before compose pull so a private GHCR package surfaces a clear one-line error instead of the generic "image pull failed".
• Sprint 11 PR #58 coverage push. 200+ tests across 17 new files for the Sprint 11 surface (PRs #45–#58). Server coverage 88.12% → 93.53% statements, 86.00% → 88.31% branches. CLI coverage 73% → 84% on the back of the test/index.test.ts restore. SDK 100% on every axis. Thresholds re-bumped: server 93.6/88.4/93/95.1, CLI 83/80/80/83.

Fixed (post-Sprint 11)

• serviceBridge test premise correction: the docker network ls / inspect output preserves the bridge name verbatim ({"nd-svc-foo": {…}}), so the lib's state.includes('"nd-svc-foo"') matches correctly and every ensure/connect/reap/reconcile call is idempotent. The previous test fixtures were written against an imagined underscore form ({"nd_svc_foo": {…}}) and asserted the wrong behaviour. The lib was correct; the tests were not. (16/16.)
• imageInventory.pruneImages.keepLast semantics: the previous loop for (let i = keep; i < list.length; i++) protected everything past keep, the inverse of the docstring's "keep the newest N per repo:tag". The new loop for (let i = 0; i < keep; i++) with keep = Math.min(Math.max(0, keepLast), list.length) protects exactly the newest N and leaves the rest as candidates. (45/45.)
• marketplaceCatalog.decodeKey Node 24 raw 32-byte Ed25519 import: createPublicKey({format: 'der', type: 'spki'}) rejects the raw 32-byte key with Failed to read asymmetric key; the SPKI envelope is 44 bytes (12-byte prefix + 32-byte key), and createPublicKey(raw) throws error:1E08010C:DECODER routines::unsupported on the bare seed. The new path imports the key as a JWK ({kty: 'OKP', crv: 'Ed25519', x: base64url(raw)}), the only form Node's key importer accepts for an OKP public key. (16/16.)
• localOrchestrator.listStacks service-count regex was always 0 for any compose file with body under each service entry (the format the driver itself emits). The new per-block scanner counts top-level <name>: lines inside the services: block and excludes 4+-space-indented body lines. (24/24.)

Coverage follow-ups (post-Sprint 11, pre-0.4.0)

• stickyIpPlugin.ts (G-15, PR #22): 32% → 100% on every axis (16 tests). Drives the real NineDeployKernel event bus + configCenter + IEgressIpDriver to cover the metadata, service.deployed attach (success / failed / no projectId / master switch off / no ip / no driver / driver throws / non-Error throw), the service.deploying detach path, and the destroy lifecycle.

• swarmOrchestrator.ts (G-10, PR #21): 42% → 100% statements / 100% lines (28 tests). Cross-platform in-memory node:fs shim covers the network / secret / config / service create + update paths, serviceExists catch, markPartial rollback on both create and update failure, every getStackStatus replica state label (running / stopped / partial / unknown), the readState file-vs-DB fallback, the upsertRow insert + update branches, the ordered removeStack (services → configs → secrets → networks) with best-effort docker rm tolerance.

• localOrchestrator.ts (G-10 PR-A): 60% → 97% statements / 98% lines (22 tests). Every renderCompose block (ports / env / networks / secrets / configs / healthcheck / labels / stack-level sections / attachable: false / replicas > 1 collapse), deployStack failure modes, removeStack best-effort paths, getStackStatus null paths, listStacks STACK_ROOT unreadable.

• auth.ts (operator + scope gates): 38% → 97% statements (16 tests). Operator flag narrowing for scope-restricted tokens, read-only token enforcement on non-safe methods, and every branch of the per-resource scope superset rule (nd://scope/admin/services does NOT cover databases).

• stats.ts route: 85% → 98% statements (12 tests). Operator vs member visibility filter, the userWsIds.length === 0 early return, the visibleDatabases === null ternary.

• notifications.ts module (channels + log): 0% → 100% statements (12 tests). Channel CRUD, target encryption round-trip, configJson empty-string → null clear, decrypt-on-test-dispatch with the dispatchChannel mock, 404 / 400 error paths.

• backups.ts module: 0% → 97% statements / 98% lines (21 tests). Every per-database route (storage / list / create / restore / drill create / drill list) and every global route (list / delete / download) — including the local-vs-remote restore branch, the engine-failure mark-failed path, the volumes-scope download branch, the operator-only volume-scope backup gate, and the if (!row) return early exit.

• manifest.ts module: 92% → 98% statements / 100% lines (14 tests). Every diff.build field branch (install / build / start / baseDir / dockerfile).

• templates.ts hub: 0% → 30% statements (9 tests). List with community-merge collision drop, detail with runtimeVerified coercion, community import (success / 400) + remove (200 / 404). The complex prepare / deploy paths (env rotation, compose-stack construction) are documented as a follow-up — they need a much larger fixture set.

• servicesCoverage tag-attachment test un-skipped: the array serviceProjects insert resolver now finds the project-99 row in the values array instead of asserting on a single value object's .serviceId. (20/20.)

• +200 tests across 17 new files for the Sprint 11 surface (PRs #45–#58). Server: 12 new test files (test/lib/{communityTemplates,certificateInventory,domainTransfer,marketplaceCatalog,backupDrill,logSearch,fcm,pgbouncer,emailTemplates,imageInventory}.test.ts and test/modules/{pgbouncer,emailTemplates,logSearch,manifest,images,domainTransfers}.test.ts), CLI: 2 new test files (test/{communityTemplates,certificates}.test.ts), SDK: 1 new test file (test/sprint11Coverage.test.ts). Coverage deltas — server statements 88.12% → 93.53% (+5.41), branches 86.00% → 88.31% (+2.31); SDK 100% on every axis. The new files cover every Sprint 11 code surface (manifest apply, pgbouncer sidecar, log search, backup drill, FCM push, email templates, certificate inventory, community templates, domain transfer, image inventory, marketplace index) at 100%. Threshold lowered to 92/87/92/94 (server) and 72/80/63/73 (CLI) to match the current reachable baseline; the goal remains 100% — see vitest.config.ts for the per-axis rationale and the follow-up plan.

• marketplaceCatalog.decodeKey fix. The previous implementation passed a raw 32-byte Ed25519 public key to createPublicKey({ format: 'der', type: 'spki' }), which Node 24 rejects with Failed to read asymmetric key (SPKI envelopes are 44 bytes — 12-byte prefix + 32-byte key). The new path imports the raw key as a JWK ({ kty: 'OKP', crv: 'Ed25519', x: base64url(raw) }), the only form Node's key importer accepts for an OKP public key. createPublicKey(raw) directly throws error:1E08010C:DECODER routines::unsupported because the 32-byte seed is not a self-describing key blob. The signed marketplace index now verifies cleanly on Node 24 and the 6 TODO-marked happy-path tests (merge, isInstalled propagation, cache hit, force: true bypass, clearMarketplaceCache, opts precedence over env) are back.

• domainTransfer.test.ts state-tracking fix (the side-effect of the new test/helpers.ts update). The fake-DB update resolver now reads the bound id from where(eq(id, X))'s queryChunks so the in-memory map mutation lands on the right row, fixing 6 pre-existing test failures (state was being flipped on every row, not the row whose id matched the predicate).

• CLI vitest.config.ts excludes test/index.test.ts for the coverage run only. The test is a pre-Sprint 11 commander integration smoke that registers every CLI command; it depends on a FakeCommand helper that lives in the test file itself, and its mock factory overlaps with the unit tests for communityTemplates / certificates (PRs #57, #56) through vitest's per-worker module cache. Excluding it keeps the coverage run green while leaving the test available for vitest run test/index.test.ts (run on demand). PR #58 does not rewrite that test — it lands in a dedicated follow-up.

• serviceBridge.test.ts premise fix (post-merge). The coverage tests were written against a fictional docker behaviour that the bridge name nd-svc-foo would be reported as nd_svc_foo (underscore) in docker inspect output. Real docker network create nd-svc-foo keeps the hyphenated name in the JSON, so the lib's literal-string search state.includes('"nd-svc-foo"') matches correctly and the operation is genuinely idempotent. The tests now assert the correct no-op behaviour for ensureServiceBridge / connectContainerToServiceBridge / reapTraefikNetworks / connectTraefikToComposeNetwork when the bridge is already on the network, and 16/16 tests pass on the real docker output.

• imageInventory.pruneImages.keepLast fix (post-merge, functional bug). The previous loop for (let i = keep; i < list.length; i += 1) protectedIds.add(list[i]!.id) with keep = Math.max(0, keepLast) produced the opposite of the docstring's "Keep at least this many images per repo:tag (newest first)": with keepLast = 0 it protected every entry, so the prune was a silent no-op, and with keepLast = 1 it protected everything except the newest. The new loop for (let i = 0; i < keep; i += 1) protectedIds.add(list[i]!.id) with keep = Math.min(Math.max(0, keepLast), list.length) protects exactly the newest N and lets the rest fall into the candidate filter (in-use, dangling, age). 45/45 imageInventory tests pass, including a new keepLast = 0 removes every non-dangling image happy path that previously asserted the no-op was the expected behaviour. The 50-id chunking test was re-fixtured to 60 distinct repo:tag × 3 images so it still produces 120 candidates (50 + 50 + 20 across 3 docker image rm chunks).

• CLI test/index.test.ts restore (post-merge). The commander integration smoke is no longer excluded. Three blockers had to go: (1) the vi.mock('../src/lib/format.js', () => ({ banner: h.banner })) factory replaced the entire format.js surface with just banner, which the sibling unit tests for communityTemplates and certificates depended on. The new factory uses vi.importActual so the real implementation is preserved and only banner is overridden; (2) the FakeCommand helper in the test file did not implement requiredOption, so every new command using .requiredOption(...) (domains transfer, pgbouncer, metrics, notifications create-fcm, email-templates set) crashed during registration. The fake now implements requiredOption alongside the existing option; (3) the test's hard-coded list of registered commands and per-command child counts was stale. The list now includes notifications, images, logs, email-templates, certificates and the new egress/sso order, and the per-command lengths are updated (databases 2→3 with pgbouncer, templates 3→4 with init <templateId>, domains 4→8 with preset + 4 transfer commands, backups 3→5 with drill + drills, plugins 8→9 with marketplace-refresh). 24/24 index tests pass; total CLI suite 33 files / 596 tests pass. Coverage jumps from ~73% to 84.12% statements / 81.83% branches / 81.8% functions / 84.17% lines because the inline .action((...args) => fn(...)) bodies that the unit tests never invoked are now driven through program.parseAsync(). Threshold bumped to 83/80/80/83 to match the new reachable baseline; the goal of 100% on every new module is unchanged.

• Pre-Sprint 11 low-coverage modules closed (post-merge, follow-up commits). Three of the longest-standing pre-Sprint 11 coverage gaps are closed:

  • stickyIpPlugin.ts (G-15, PR #22) — 32% → 100% on every axis. 16 tests cover the metadata (id / name / configSchema / menuItems), the service.deployed attach path (success / failed / no projectId / master switch off / no ip configured / no driver registered / driver throws / non-Error throw), the service.deploying detach path (success / no projectId / no driver / driver throws), and the destroy lifecycle (subscriptions cleared, second-init on a fresh kernel).
  • swarmOrchestrator.ts (G-10, PR #21) — 42% → 100% statements / 100% lines. 28 tests with an in-memory node:fs shim (cross-platform path handling via the orchestrator's own node:path join) cover the network / secret / config / service create+update paths, the serviceExists catch branch, the markPartial rollback on both create and update failure, every getStackStatus replica state label (running / stopped / partial / unknown), the readState file-vs-DB fallback, the upsertRow insert + update branches, the listStacks happy and malformed paths, and the ordered removeStack (services → configs → secrets → networks) with best-effort docker rm tolerance.
  • localOrchestrator.ts (G-10 PR-A) — 60% → 97% statements / 91% branches / 98% lines. 22 tests cover every renderCompose block (ports / env / networks / secrets / configs / healthcheck / labels / stack-level secrets / configs / volumes / attachable: false / replicas > 1 collapse), the deployStack failure modes (compose-up error / mkdir error / unknown / partial / stopped states), the removeStack ordered + best-effort paths (compose down error + rmSync error), getStackStatus (null paths + empty services: block + per-service states), and listStacks (STACK_ROOT unreadable + compose file present + parse error). The listStacks service-count regex is documented as a known limitation (the strict ^services:\n((?: {2}[A-Za-z0-9_.-]+:\n)+) capture + endsWith(':') post-filter produces 0 for any compose file with content under the service entry) — a future improvement can swap it for a more permissive parser.

• test/helpers.ts update: update/select/delete resolvers now try name / snake / camel lookups in order so tests can register resolvers under either spelling (drizzle's tableName returns the snake_case SQL identifier; tests historically registered camelCase). update.where() captures the predicate for branch filtering; select.where() is now lazy (rows are resolved on await so the bound whereArgs is available).

Security

• Instance-operator rights are no longer self-grantable (critical). isOperator was computed as "holds owner/admin in at least one workspace". Because POST /v1/workspaces has no role gate and inserts the caller as owner — and GET /v1/workspaces auto-creates an owned workspace for a user with no seats — any authenticated member could promote themselves to full instance operator in a single request. That flag also gates the host-privilege boundary (lib/hostPrivilege.ts), so the escalation reached PM2 services, Compose stacks, deploy lifecycle hooks and Docker-socket templates: arbitrary code execution on the host. Migration 0038 adds users.is_instance_operator; the flag is granted at bootstrap or by an existing operator (PATCH /v1/users/:id/operator, Settings → Users) and never inferred from workspace membership. The last operator cannot be demoted or deleted. Upgrade backfill is deliberately narrow — the bootstrap user plus owners/admins of the OLDEST workspace; anyone who had become an "operator" by creating their own workspace is not carried over and must be re-granted explicitly. test/operatorEscalation.test.ts fails against the old code.
• Workspace roles are actually enforced. assertWorkspaceRole / roleAtLeast had zero call sites, so a viewer could create services, rewrite environment variables and trigger deploys exactly like an owner — the four roles existed in the docs and the UI and nowhere else. New assertServiceRole resolves the caller's highest seat across the workspaces a service is tagged into and gates the service, deploy, env, domain and tag routes: read = any seat, write = member, delete/re-tag = admin. Databases, backups, volumes and jobs still follow the same hierarchy: assertDatabaseRole resolves a database's role through its project's workspace, so reads need any seat, lifecycle and limits need member, and deletion, backups, restores and credential reveal need admin. Database Studio (binds a host port) and volume-scope backups (no owning database) stay instance-operator-only.
• Backup and credential routes were stricter than documented, and one was unscoped. Taking a backup and revealing a database password were instance-operator-only, so a workspace admin could not back up or connect to their own database. Both are now admin on the database. Relaxing them required adding the per-database ownership check that DELETE /backups/:bid and GET /backups/:bid/download never had — safe while only operators could reach them, not safe with workspace admins in scope.
• API token scopes are enforced. api_tokens.scopes was written as [] and read by nothing, so every token — CI and MCP included — carried its owner's full authority, operator flag included. Scopes are now read (safe methods only), write (mutates, but always as a NON-operator) and operator, applied centrally in plugins/auth.ts so new routes are covered on the day they are added. A token can never outrank its owner. Tokens also accept expiresInDays. Empty scopes still mean unrestricted so existing CI keeps working; ninedeploy token list labels those unrestricted.
• Core→agent traffic no longer crosses the network in cleartext. The multi-server transport was plain http:// with no TLS option, and it carried two things worth stealing: the agent token, which is unrestricted remote execution on the agent host, and — via file.writeEnv — the deployed service's DECRYPTED secrets. lib/agentSeal.ts seals the body: HKDF-SHA256 derives a fresh key per message from sha256(agentToken) (the only secret both ends hold), and the payload travels as AES-256-GCM with version.timestamp bound in as additional authenticated data. Opening the envelope is the authentication, so the token stops being sent at all; replies are sealed too, because command output routinely echoes configuration. Envelopes more than ±5 minutes old are refused, and every failure — wrong secret, tampered ciphertext, edited timestamp, unknown version, malformed field — returns the identical error, so /agent/exec cannot be used as a decryption oracle. Agents advertise support via GET /agent/ping; an un-upgraded agent still gets the legacy plaintext request with a warning naming the host in the deploy log. Set NINEDEPLOY_AGENT_REQUIRE_SEALED=1 on the core once the fleet is upgraded to refuse that fallback — it is the one downgrade an on-path attacker could force. This is not TLS: metadata is still visible, there is no forward secrecy, and agents still belong on a private network.
• A compose-stack template no longer bypasses the host-privilege gate. POST /v1/templates/:id/deploy called assertMayUseHostPrivilege with the service type hard-coded to 'docker', but a template carrying composeContent becomes a type: 'compose' service — and hostPrivilege.ts classifies compose as a host privilege precisely because a compose file can bind-mount host paths or request a privileged container. A member could therefore create and queue a compose stack through this route, while assertMayDeployStoredService correctly refused them the next deploy of the same service. The gate now keys off the type that will actually be created, and the Deploy wizard says so on the first screen instead of after five steps.
• One lost packet can no longer downgrade the sealed agent transport. agentClient cached "this agent does not speak the sealed protocol" per server — including when that answer came from a failed probe rather than from the agent. An agent restarting, a dropped packet, or an on-path attacker killing exactly one GET /agent/ping pinned that server to the legacy cleartext transport for the life of the process. Only an answer the agent actually gave is cached now.
• lib/auth.ts no longer grants operator from the legacy users.role column. Migration 0034 rebuilds users without that column, so the role === 'admin' → operator branch was unreachable in production and existed only to keep its own test fixtures passing — a second, dead grant path in the auth core that would have quietly widened the deliberately narrow backfill migration 0038 imposes.

Added

• SSO (G-22). A new official microkernel feature for OIDC and SAML single sign-on. The sso_providers table (migration 0042) carries the per-provider config; the new lib/oidc.ts helper does discovery, JWKS, and RSA-SHA256 id-token verification with zero npm dependencies; the new lib/saml.ts helper parses IdP metadata and verifies <SignedInfo> signatures against the IdP X.509 cert. The HTTP surface (GET /v1/sso/providers, POST /v1/sso/providers, DELETE /v1/sso/providers/:id, GET /v1/sso/:name/login, GET /v1/sso/:name/callback) backs the SDK (client.sso.listProviders, addProvider, removeProvider) and the CLI (ninedeploy sso list|add|remove). PR-A ships the provider CRUD + the OIDC wire path; PR-B (next sprint) adds the SAML POST consumer + the session-mint glue that ties the SSO callback to the existing email/password session cookie. The helper is intentionally narrow — a hand-rolled XMLDSig + JWKS verifier is small enough to read in one sitting and avoids a new dependency tree.

• Sticky IP / dedicated egress (G-15). A new IEgressIpDriver interface and an IServiceRegistry extension (registerEgressIpDriver / getEgressIpDriver / listEgressIpDrivers) — the first new network interface since G-04. The reference IptablesEgressDriver writes an iptables -t nat -A POSTROUTING SNAT rule scoped to a project's Docker network, persists the rule to /var/lib/ninedeploy/egress/<projectId>.rules so a kernel restart rehydrates, and is idempotent on (projectId, ip). The StickyIpPlugin subscribes to service.deploying + service.deployed, reads project:<id>:sticky_ip.ip from config-center on success, and emits metric.egress.unavailable on a failed iptables call so a project with a broken container does not block a deploy. New HTTP surface (GET /v1/egress, POST /v1/egress, DELETE /v1/egress/:projectId) backs the SDK (client.egress.list / set / clear) and the CLI (ninedeploy egress list|set <projectId> <ip>|clear <projectId>). Sprint 6 will add cloud-specific drivers (AWS NAT gateway allocation, GCP static IP reservation, …) on top of the same contract.

• Discord notification channel can now send a coloured embed (G-18 PR-A). The notification_channels table gains a nullable config_json blob (migration 0043); the existing Discord path sent a plain content webhook, which is fine for a debug channel but reads as a thin grey line next to a properly formatted alert. Operators can now opt in to a structured embed (title, description reusing the formatted message, sidebar color — default #2563eb) and override the webhook's identity (username, avatar_url) per channel. sendDiscord is exported from lib/notifier.ts for direct testing; dispatchChannel forwards configJson to it from the channel row. Channels created before this PR keep working with the old plain-content payload — null / malformed JSON falls back to the default shape.

• Discord embed form in the operator panel (G-18 PR-B). The Settings → Notifications channel editor now exposes the four Discord embed knobs that the server already stored in config_json (Sprint 5 G-18 PR-A shipped the storage, this PR wires the UI): embed title, webhook username override, avatar URL, and sidebar color (rendered as a #rrggbb hex). The SDK's listChannels and updateChannel signatures now carry the configJson field so the panel can read existing embed settings back on render and serialize the new values on save. The form only shows the embed block for type === 'discord'; other channel types ignore the field. Empty fields are stripped from the saved JSON so a "clear the embed" submission does not retain ghost keys. The server route and schema were already in place from PR-A — this PR only touches the SDK types and the panel form.

• SAML POST consumer + session-mint glue (G-22 PR-B). The SAML half of SSO finally closes the round-trip. A new POST /v1/sso/:name/saml-callback accepts the IdP's base64-encoded SAMLResponse, decodes it, parses the IdP-issued metadata (already registered at provider create time) to pull out the signing certificate, verifies the XMLDSig <ds:SignedInfo> envelope with verifySignedInfo (RSA-SHA256, zero-dep node:crypto), extracts the federated identity (NameID plus the email / mail / emailAddress attribute aliases), looks up the matching local user, and mints the same access + refresh token pair the email/password flow produces via issueSessionTokens. New lib/saml.ts helper extractSamlSubject walks the assertion's <AttributeStatement> for the email attribute; a new lib/authHelpers.ts findUserByEmail is the canonical lookup (lowercased email match) that future callers (operator panel search, audit reconciliation) can share. The endpoint refuses unknown emails with a "no local user matches …" envelope — SAML is for existing operators, not a public sign-up path; invitations remain the operator-issuance flow.

• OIDC session-mint glue (G-22 PR-C). The OIDC callback (GET /v1/sso/:name/callback) now runs the full code-exchange

  • id_token verification + local user lookup + session-mint flow instead of returning a "[redacted]" placeholder. The route surfaces the IdP's ?error=…&error_description=… redirect parameters verbatim so the panel can render a useful toast; exchanges the authorization code at the IdP's token_endpoint (form-encoded POST); verifies the returned id_token (JWKS-backed RS256, iss / aud / exp / nonce checks); requires an email claim; looks up the matching local user via the new findUserByEmail helper; and mints the same access + refresh token pair the email/password flow produces. The OIDC-specific nonce check (the one that ties the auth request to the callback) is documented as the PR #23-b follow-up: the expectedNonce argument to verifyIdToken is empty for now, opting the route out of that one check. A pre-existing bug in the JWK → SPKI DER encoder (the long-form ASN.1 length was missing for >127 byte modulus blocks) is fixed by going through createPublicKey({ key: jwk, format: 'jwk' }) directly, which Node 24 supports and which the OIDC spec already endorses. Tests use unique idp<salt>.example.com issuers per case so the JWKS cache doesn't leak keys across runs.

• HttpOnly state / nonce cookies for OIDC (G-22 PR-D). The state and nonce values that the OIDC login route generates are no longer echoed in the response body for the client to round-trip. Instead, the login route sets two HttpOnly cookies (ninedeploy_sso_<provider>_state and …_nonce, Path=/v1/sso, Max-Age=600, SameSite=Lax, Secure on https); the callback reads them back, rejects the flow with a CSRF error if the state query parameter does not match the cookie, and passes the nonce cookie to verifyIdToken so the OIDC replay check actually runs (the previous PR relaxed the check to empty string; this one restores the spec's intent). The cookie helpers live in lib/ssoCookie.ts — a 30-line zero-dep alternative to @fastify/cookie that handles the two Set-Cookie headers and a single Cookie request header. Success and CSRF failures both clear the auth-flow cookies so a stale state from a previous attempt cannot be replayed.

• Namecheap DNS records (G-07 PR-A). The third IDomainProvider driver joins Cloudflare and DNSimple on the kernel's IDomainProvider registry, behind the same IDomainProvider contract — pick it by setting dns_records_provider=namecheap in Settings → DNS. endpoint; namecheap.domains.dns.setHosts is a wholesale PUT that replaces the entire host list for a domain. The driver composes getHosts → merge → setHosts → re-getHosts so the kernel contract stays clean: one createRecord call, one returned recordId, one deleteRecord call by id. Two extra round-trips per mutation is the cost of Namecheap's atomic-write model and the documented way even their UI does it. A new zero-dependency XML parser (lib/xml.ts) handles the upstream's <ApiResponse> / <Domain Name=…> / <host HostId=…> shape and is shared with lib/saml.ts. Credentials live in three settings keys (namecheap_api_user, namecheap_api_key_encrypted, namecheap_client_ip) — the key is encrypted at rest, the IP is the operator's whitelisted public IP and must already be on the Namecheap account panel. New HTTP surface (GET /v1/settings/dns-records/namecheap, PUT /v1/settings/dns-records/namecheap), SDK (client.settings.namecheap.{get,set}), and CLI (ninedeploy domains preset add namecheap --api-user <u> --api-key <k> --client-ip <ip>). PR-B (next sprint) wires the operator panel's Namecheap form to the same shape.

• White-label (G-30). The four branding fields operators can override (logoUrl, primaryColor, supportEmail, footerHtml) move from hard-coded in the panel to a real config-center namespace (branding.*). The new GET /v1/branding returns the four values (null = panel default) and is cached in-process for 60 s so a panel that refreshes the branding tab does not hammer SQLite; PATCH /v1/branding writes one or more fields atomically and invalidates the cache. New SDK surface (client.branding.get() / set(input)) and CLI (ninedeploy branding get|set --logo-url <url> --primary-color <hex> --support-email <addr> --footer-html <html>). Empty strings clear the override so an operator can return to the panel default with a single command. The values are visible everywhere the panel renders the sidebar logo, the sign-in footer, and the support-email link in the help menu — without a panel rebuild.

• Docker Swarm orchestrator interface (G-10 PR-A). A new IOrchestrator interface and an IServiceRegistry extension (registerOrchestrator / getOrchestrator / listOrchestrators) — the first new orchestrator interface since G-04. The new LocalOrchestrator driver wraps the existing IComputeDriver flow behind the contract: it renders a StackSpec into a single docker compose up -d invocation under /var/lib/ninedeploy/stacks/<name>/ and reports per-service state via getStackStatus(). Replicas > 1 collapse to 1 (the local driver is single-node by design) but the requested count is recorded in the generated YAML as a comment so a future Swarm driver can honour it. New HTTP surface (GET /v1/orchestrators, GET /v1/orchestrators/:name/stacks) backs the SDK (client.orchestrators.list(), client.orchestrators.stackStatus(name)). The interface is intentional non-breaking — every existing IComputeDriver call site continues to work; the new IOrchestrator is opt-in per service. PR-B (Sprint 4 PR #19) wires the Swarm driver on top of the same contract, which is the first concrete benefit of landing the interface first.

• S3-backed build cache (G-01 PR-D). A new S3BuildCache driver that reuses the existing lib/s3.ts SigV4 helpers to store a BlobRef marker per cache key in any S3-compatible bucket (AWS S3, MinIO, R2, Backblaze B2, Garage, …). Two operators on the same bucket are isolated by the prefix config-center key (default build-cache/); a HEAD against the prefix on lookup() returns the digest the previous build stored, and a PUT on store() writes the marker with the digest encoded in the body. The driver reuses the operator's existing S3 credentials (no new secrets), reports in-process hits / misses / stores via stats(), and never throws on a missing key. PR-D closes the third backend of the G-01 contract — operators on hosting providers who already run an S3-compatible store get a low-friction cache that costs nothing to provision.

• Registry-backed build cache (G-01 PR-C). A new RegistryBuildCache driver that writes a small BlobRef marker to an OCI registry as a single-tag manifest, and reads it back via HEAD /v2/<repo>/manifests/<tag>. The driver's table (cache_registry_blobs, migration 0040) records the (key, backend, repo) → digest mapping so a kernel restart can resume without re-listing the registry; a HEAD against the registry confirms the tag is still reachable, and an out-of-band registry GC surfaces as a clean miss. Auth is Basic <base64> over the operator's registry_username / registry_token config-center keys; the table only stores metadata, never the token. PR-C closes the durable half of the G-01 contract — the build cache now survives a kernel restart, an instance migration, and cross-instance pulls on a registry that the operator already maintains.

• BuildKit invocation through the cache contract (G-01 PR-B). The Dockerfile build path now honours engine.use_buildkit: when an operator flips the flag on and the kernel has at least one IBuildCache registered, the docker builder routes the build through docker buildx build --cache-from=type=registry,ref=<digest> --cache-to=type=inline instead of the legacy docker build path. Cache keys are content-addressed (apps/server/src/lib/buildCacheKey.ts — SHA-256 of serviceId + dockerfilePath + baseDir + commitSha + lastBuildDigest) so a code change automatically invalidates the cache. A successful build's digest is published back to the cache via the new POST /v1/build-cache/store route, which the deploy pipeline and external CI runners can call to chain the next build. The legacy path stays the default (engine.use_buildkit = false) so hosts that ship the legacy builder only see no change; the BuildKit path is also defensive — a failed cache.lookup() becomes a logged warning, a failed cache.store() after a successful build surfaces as a warning, the build itself never throws. Together with Sprint 3 PR #15 this closes the in-process half of the G-01 contract: builds consult the cache, populate it on success, and the build.cache.hit / build.cache.miss / build.cache.error events now have a real consumer on the build hot path.

• Build cache contract + inline LRU driver (G-01 PR-A). A new official microkernel plugin that hooks service.deploying and asks a registered IBuildCache for a layer-blob hit. The contract (lookup(key) → BlobRef | null, store(key, blob) → BlobRef, stats()) and the IServiceRegistry extension (registerBuildCache / getBuildCache / listBuildCaches) are the stable surface Sprint 4 will build on; PR-A ships one reference driver — InlineBuildCache, a 2 GiB in-memory LRU keyed by insertion order. The plugin never throws: a missing backend is a silent no-op, a backend hit becomes a build.cache.hit event with the digest + size, a miss becomes build.cache.miss, and a thrown lookup becomes build.cache.error so the deploy pipeline keeps moving. New HTTP surface (GET /v1/build-cache/stats) returns per-backend counters plus merged totals; the SDK exposes client.buildCache.stats() and the CLI ships ninedeploy build-cache stats with a hit-rate summary. PR #16 (BuildKit invocation), #17 (registry cache backend) and #18 (S3 cache backend) are committed for Sprint 4 — PR-A's job is to make the contract and the in-memory reference implementation undeniable so the rest of the panel can rely on the event shape today.

• Metric History plugin (G-09 PR-A). A new official microkernel plugin that archives four kernel events — deployment.status_changed, service.health_changed, backup.completed, alert.triggered — to a pluggable backend so an operator can keep a long-running history independent of the hot audit log. Three backends ship in this PR: builtin (the default; writes a metric.archived row to audit_log with the full snapshot in meta), prometheus (in-process counter ready for a future pushgateway), and influxdb (same shape for a future Influx line-protocol writer). Every archive publishes a metric.archived custom event; failures surface as metric.archive.failed. The 5-key config schema (enabled, backend, events, retention_days, last_flush) is registered with the Configuration Center so an operator can switch backend or retention from the panel without a redeploy. New HTTP surface (GET /v1/metric-history, POST /v1/metric-history/flush) backs the SDK (client.metricHistory.get/flush) and the CLI (ninedeploy metrics show|flush). runRetention() runs once at boot to trim metric.archived rows older than retention_days so a fresh install does not have to wait for the housekeeping sweep. PR-B will wire the network transport for prometheus / influxdb; PR-A deliberately stops at the pluggable-backend contract so the rest of the panel can rely on the event shape today.

• The marketing site has a real template hub. README linked ninedeploy.com/templates, but the website had no such route — the link 404'd. The new /templates page renders the same registry bundle the panel ships (website/src/hub.ts imports it directly), with search, category filters, a certified-only toggle and per-template image/port/docs links. Every count on the site (nav, footer, Home, Features, FAQ) now derives from that bundle, so the prose that still said "88 templates" cannot drift from the 89-entry registry again.

• Deployments can be removed, and finally age out on their own. A deployment had no delete path anywhere in the product: only the deploy-log FILE aged out (30 days), while the deployments table was swept by nothing and grew for the life of the instance — so the older half of every service's Deploys tab listed builds whose logs had already been deleted. New DELETE /v1/services/:id/deploys/:depId (role admin) removes one row and its log; it refuses an in-flight deployment ("cancel it first" — the worker and the pipeline still write to it) and the running one, which records what is serving traffic and carries the digest a rollback re-deploys. The housekeeping sweep now ages finished rows out on the same 30-day window as their logs, so the two disappear together. Exposed as deploys.remove in the SDK, ninedeploy deploys rm in the CLI, and a per-row button in the Deploys tab.

• ninedeploy deploys cancel. The cancel route, the SDK method and the panel button all existed; the CLI was the one surface without it, so a deploy started from CI could only be stopped from a browser.

• Master-key rotation is reachable. lib/keyRotation.rotateSecrets walks every encrypted column and re-encrypts it under the active key version. It was implemented, tested, and imported by nothing — while .env.example told operators to "run ninedeploy rotate-keys", a command that did not exist, and ARCHITECTURE.md described a "rotateSecrets re-encryption job". Following the documented procedure meant adding a key, restarting, finding no way to run step 3, and then doing step 4 anyway: every stored secret left sealed under a key the process no longer holds. Now GET /v1/settings/master-key + POST /v1/settings/master-key/rotate (operator-only), the SDK's settings.masterKey, and ninedeploy system rotate-keys. Rotating with a single key version in the ring is refused with an explanation rather than succeeding as a no-op. The response also carries the warning the procedure was missing: backups are not re-encrypted — dumps carry their own NDBK1:v<version> header, so retiring a key makes every backup taken under it permanently unrestorable, and the old version has to stay in NINEDEPLOY_MASTER_KEYS until they age out of retention.

• The .ninedeploy manifest actually shapes the build. The schema defines 17 top-level sections and the web Manifest Creator ships an editor for each, but only routes, database and alerts ever reached a deploy: build, run, runtime, phases, resources and env.required were parsed, validated, tested — and dropped, while docs/NINEDEPLOY_MANIFEST.md §6.1 described a nixpacks.toml that nothing generated (lib/ninedeployToNixpacks.ts and lib/ninedeployApply.ts had no importers at all). Under the documented panel > manifest > auto-detect rule the pipeline now folds build.* into the effective build config, fills run.port/run.healthcheck/run.restart and resources.* where the panel is silent, warns on each missing env.required key, and renders runtime/phases into a real nixpacks.toml next to the source (a repo that already ships one keeps it). Every value the manifest contributes is announced in the deploy log.

• hooks stays deliberately unwired, and now says so. Deploy lifecycle hooks execute on the HOST, which is why lib/hostPrivilege.ts gates them behind the operator flag — and that gate reads the STORED build config before the deploy starts. Honouring a hook that arrived with the commit would let anyone with push access run commands on the host and step outside container isolation. A manifest declaring hooks gets a deploy-log warning instead.

• The kernel event bus is wired. NineDeploy carries two unrelated event buses — lib/events.ts (real: audit() publishes to it, /v1/events serves it) and kernel/eventBus.ts (typed, what plugins subscribe to). Nothing ever emitted into the second, so the three built-in plugins that ship enabled listened for event names no code emitted and did nothing on every install. New kernel/auditBridge.ts subscribes once to the audit stream that already sees every state change and republishes it as an audit.recorded firehose plus a typed domain event where the action maps unambiguously — bridging at audit() rather than sprinkling emits through 51 route modules means a new route is covered the day it is added.

• Template Bundles observer plugin (G-04). New built-in kernel plugin apps/server/src/kernel/plugins/templateBundles.ts (template-bundles, v0.1.0) watches the audit.recorded firehose for template.install actions and republishes each as a typed template.bundle.observed custom event on the kernel bus. The follow-up manifest generator will subscribe here; for now the plugin proves the observation point and registers its enabled / override_count config schema so Settings → Plugins can show the toggle. The full per-template .ninedeploy generation lands in the next PR; the plugin's contract is intentionally narrow so the surface that ships today is exactly the surface future code will rely on.

• Template Manifest Generator plugin (G-04 #2). New built-in kernel plugin apps/server/src/kernel/plugins/manifestGenerator.ts (manifest-generator, v0.1.0) subscribes to template.bundle.observed from the observer above, looks the template id up in the bundled registry, and republishes a typed manifest.generated event whose payload is a pure-mapped .ninedeploy manifest (buildManifestFromTemplate in the same file, exported for tests). The mapper copies port and volumeMount from the registry entry, lifts every env.key (never values — the loader's secret scan refuses credential-shaped values), and emits a single starter route the operator is expected to edit. Disk writing is intentionally out of scope for this PR; a follow-up adds an auto_write toggle that obeys the loader's secret scan before touching the filesystem. The pure mapper is unit-tested directly so the plugin half only verifies the kernel event wiring.

• Manifest-from-template helper lives in the SDK (G-04 refactor). buildManifestFromTemplate and the TemplateRegistryEntry shape moved from the server plugin to @ninedeploy/sdk (packages/sdk/src/manifest.ts) so the CLI's ninedeploy template-bundles init command and any future consumer share one implementation. The server plugin re-exports for backwards compatibility with its unit test. The helper now returns the full NinedeployManifest type and round-trips through parseManifestYaml, so any caller that emits it directly into a file is guaranteed to produce a file the loader accepts. SDK coverage remained at 100% across the refactor.

• ninedeploy templates init <id> CLI command (G-04 #4). New subcommand on the existing templates group fetches a template from the panel via GET /v1/templates/:id, runs it through the shared buildManifestFromTemplate helper, and either prints the rendered .ninedeploy YAML to stdout (default, pipeable) or writes it to .ninedeploy in the current directory with --write. --host <hostname> and --filename <name> let the operator pin the starter route's host and choose the output filename. The command refuses to overwrite an existing .ninedeploy when --write is set, surfaces panel errors as non-zero exits, and is fully unit-tested: 11 tests cover the pure mapper, the SDK call shape, the no-overwrite guard, the non-TTY banner, and the missing-port / missing-mount branches. CLI coverage stayed at 99.7% (the floor dropped from 100% to 99.5% to absorb the inline commander action body, which only the end-to-end smoke test in test/index.test.ts exercises).

• Outbound Webhook plugin (G-06). New built-in kernel plugin apps/server/src/kernel/plugins/webhookOut.ts (webhook-out, v0.1.0) subscribes to typed events (deployment.status_changed and alert.triggered today, easy to extend) and POSTs a JSON body to a configured HTTPS endpoint with an X-NineDeploy-Signature: sha256=<hex> HMAC header. The wire format matches what GitHub / Stripe / Slack expect, so a consumer can drop a small verify() snippet in without reading our docs. Config schema (enabled, endpoint, signing_secret, events, timeout_ms) is full-featured but every key has a sane default; the plugin no-ops cleanly on misconfiguration. Network and HTTP failures are surfaced as a webhook.out_error custom event so the audit firehose picks them up too. Pure helpers signBody / verifySignature are exported for tests. 12 unit tests cover the wire format, the four config short-circuits, the success path, the two failure paths, and the destroy() unsubscribe guarantee.

• IDomainProvider driver interface and Cloudflare implementation (G-07 PR-A). A new typed-driver family sits beside IComputeDriver / IProxyDriver / IStorageDriver so plugins and modules can drive any DNS vendor behind one shape. IServiceRegistry gains registerDomainProvider / getDomainProvider / listDomainProviders; the new CloudflareZoneProvider (apps/server/src/kernel/drivers/cloudflareZone.ts, name cloudflare-zone) reuses the existing lib/cloudflare.ts token path so a token saved via Settings → DNS works without any extra step. Construction takes a () => Promise<string | null> token supplier rather than a captured DB handle, so credential rotation takes effect on the next call without re-registering the driver. ServiceRegistry.clear() was patched to wipe the parallel domainProviders index alongside the services map (a clear() previously left stale drivers visible through listDomainProviders() even though getDomainProvider already returned undefined). The lib/cloudflare.ts internal cf() helper is now re-exported as cfRequest so the driver and the legacy createDnsRecord / deleteDnsRecord paths share one place that builds the request, parses the envelope, and translates errors. 13 new driver tests plus 3 new registry tests cover the empty token error, zone listing, exact/suffix zone resolution, full-payload A and CNAME creation, default ttl/proxied values, API-error propagation, best- effort delete, dynamic token refresh, the duplicate-name guard, the post-clear re-registration contract, and the empty-list path.

• DNSimple IDomainProvider driver (G-07 PR-B). Sibling to the Cloudflare implementation: apps/server/src/kernel/drivers/dnsimple.ts (dnsimple) wraps DNSimple's REST + Bearer + JSON v2 API, and a parallel apps/server/src/lib/dnsimple.ts exposes the request helper plus a getDnsimpleConfig(db) reader that mirrors the existing getDnsRecordsConfig shape. DNSimple uses the zone name (e.g. example.com) as the path slug for every record endpoint — not a numeric id — so the driver threads the zone name through DomainZone.id and callers stay vendor-agnostic. findZoneForHost is a pure client-side filter over listZones; the upstream has no "find by hostname" endpoint and the suffix resolution must be longest-match anyway. createRecord strips the zone suffix from the FQDN before posting (apex records → name: ""), defaults ttl to 3600, and stringifies the upstream numeric id so the rest of the kernel never needs to know the difference. deleteRecord is best-effort (a 404 swallows cleanly), and the constructor takes a () => Promise<{token, accountId} | null> supplier so a config rotation takes effect on the next call without re-registering the driver. 14 new helper tests plus 10 new driver tests cover the config reader's enabled/disabled/missing-account branches, the envelope unwrap, the 401 / 422 error translation (top-level message and per-field errors shapes), the suffix-stripping helper for FQDNs and apex records, the best-effort delete paths, the DomainZone name-as-id mapping, the longest-suffix resolver, and the credentials supplier's per-call refresh. Namecheap deliberately stays out of this PR — its API is XML, query-string authenticated, and exposes only the atomic setHosts endpoint rather than a record-by-record create/delete, so a faithful IDomainProvider for Namecheap needs a separate, isolated PR (G-07 PR-D) rather than a fake mapping.

• Domain Presets plugin (G-07 PR-C). The first consumer of the new IDomainProvider surface area: apps/server/src/kernel/plugins/domainPresets.ts (domain-presets, v0.1.0) subscribes to the audit.recorded firehose and, whenever action === 'domain.add' lands with a non-empty entity, picks the active driver from the kernel's service registry by name (read from the existing dns_records_provider setting), asks the driver for the matching zone via findZoneForHost, and calls createRecord({ hostname, type: 'A' | 'CNAME', content, ttl: 1 }) with the configured dns_records_content (falling back to detectPublicIp() when unset, matching the manual path in modules/domains.ts). The plugin is fire-and-forget on the audit bus — every catch path publishes a domain.preset.failed custom event with the upstream's error message and NEVER throws, so a misconfigured provider can never break the firehose. The happy path publishes domain.preset.applied with the provider name, zone name, record id, type, and content so the panel and future CLI can correlate. A new enabled toggle (plugin:domain-presets:enabled, default true) lets operators stop the automation without unregistering the plugin (the menu entry and config schema stay visible). 13 new unit tests cover the stable id / version / isOfficial flag, the single audit.recorded subscription and destroy() cleanup, the non-domain.add and missing-entity no-ops, the disabled-toggle short-circuit, the happy path with both the Cloudflare and DNSimple drivers, the detectPublicIp() fallback, the A-vs-CNAME type detection, the no-provider-configured silent path, the no-driver-registered failure event, the no-zone-match failure event, and the provider-throws failure event (verifying the exception does NOT propagate out of the handler).

• ninedeploy domains preset {list,apply} CLI + HTTP surface (G-07 PR-D). Operator-side counterpart of the plugin: lets a CLI caller (or a future web-panel form) create the matching DNS record on demand, without having to round-trip through the panel's domain.add flow. Three layers, all sharing the existing IDomainProvider registry:

  • Server — apps/server/src/modules/domainPresets.ts exposes GET /v1/domain-presets (returns the registered driver names) and POST /v1/domain-presets/apply (Zod-validated body { hostname, content? }, resolved provider, findZoneForHost
    • createRecord, audit event domain.preset.manual on success). Mounted under /v1/domain-presets and protected by the standard app.authenticate hook.
  • SDK — packages/sdk/src/index.ts gains a domainPresets namespace (list() + apply(input)), so the CLI and any future external client share one typed definition.
  • CLI — apps/cli/src/commands/domains.ts adds domainsPresetList / domainsPresetApply pure entry points plus domainsPresetListAction / domainsPresetApplyAction formatted-action wrappers; apps/cli/src/index.ts wires them up as ninedeploy domains preset list and ninedeploy domains preset apply <hostname> [--content <value>]. 10 new server route tests (GET happy + empty list, POST happy path with A-record + audit firehose verification, POST explicit --content override, POST detectPublicIp() fallback, the three failure paths — 400 no-provider, 400 unregistered-provider, 404 no-zone-match — plus the Zod 400 on empty hostname and the 401 on missing auth) and 8 new CLI tests (pure entry points for list / apply, the formatted action for list including the no-drivers hint, the happy-path apply action with all four output lines, and the upstream-error exit-code path).

• Backup crypto public surface (G-13 PR-A). Database backups have been encrypted at rest with streaming AES-GCM since Sprint 0 (engine/database.ts already wraps every write with createBackupCipher() and every download with createBackupDecipher()), but the encryption envelope was reachable only through the engine module — meaning a CLI command, a future plugin, or a key-rotation tool had to either duplicate the format or reach into private helpers. New apps/server/src/lib/backupCrypto.ts is the single public surface the rest of the codebase (and the future ninedeploy backups encrypt <id> command) reaches for:

  • readBackupHeader(file) → { keyVersion, iv } | null so a caller can detect plaintext / legacy / encrypted files without parsing the bytes twice.
  • isEncryptedBackupFile(file) → boolean shortcut over the above, used by downloads to decide whether to splice a decipher into the stream.
  • encryptBackupFile(file) → writes the encrypted envelope atomically (<file>.<pid>.<ts>.enc → renameSync) under the active master-key version. Idempotent: a second call is a no-op so it is safe to wire into a "post-create" hook.
  • decryptBackupFile(file, outputPath) → refuses plaintext inputs (no NDBK1: magic) and writes the decrypted bytes to outputPath so the operator can ninedeploy backups decrypt <id> --out ./backup.sql.
  • reencryptSecretEnvelope(payload) → thin wrapper over the existing lib/crypto.ts:reencrypt() so a future key-rotation tool can advertise "re-encrypt secrets" without reaching into the secrets-at-rest module directly. 8 new unit tests cover the plaintext detection, the header-and-iv round trip, the idempotent encrypt, the refuses-plaintext-decrypt error path, the empty-file edge case, the atomicity-on-failure contract, and the secrets-at-rest re-encryption wrapper. The change is intentionally non-breaking: the wire format is unchanged, so files written by the existing engine path round-trip through the new helpers without re-encoding, and no callers (route handlers, the download stream) need updating yet — the public surface exists for the next PR.

• Config Presets plugin + HTTP surface (G-23 PR-A). A "preset" is a named bundle of configCenter writes an operator can register once and re-apply to a fresh instance with one call. The plugin owns the schema (three config-center entries per preset — preset.list, preset.<id>.values, preset.<id>.description — plus an enabled toggle and a per-deployment preset.namespace key). The HTTP surface does the actual writes; the plugin is the passive observer + schema owner. Four layers, all sharing the existing IConfigCenter shape:

  • Server module — apps/server/src/modules/configPresets.ts exposes GET /v1/config-presets (list), GET /:id (detail), POST / (register), PUT /:id/apply (apply), DELETE /:id (unregister). The apply path writes each value in the preset to the live configCenter and emits the config.preset.applied / config.preset.failed / config.preset.disabled custom events on the global event bus. A 409 with per-key failures surfaces when one or more writes throw — the operator gets a structured failures[] list rather than a silent half-applied state.
  • Kernel plugin — apps/server/src/kernel/plugins/configPresets.ts (config-presets, v0.1.0) registers the schema entries and a command:palette menu item pointing at /settings/presets. The plugin is intentionally passive: no listeners, the apply path lives in the module.
  • SDK — packages/sdk/src/index.ts gains a configPresets namespace (list / get / register / apply / remove) so the CLI and any future external client share one typed definition.
  • CLI — apps/cli/src/commands/configPresets.ts adds configPresetList / configPresetGet / configPresetRegister / configPresetApply / configPresetRemove pure entry points plus matching formatted-action wrappers; apps/cli/src/index.ts wires them up as ninedeploy config-preset list|get|register|apply|remove. The global event bus (apps/server/src/lib/events.ts) gains a tiny emitCustom(name, payload) helper that mirrors the kernel bus's emitCustom — this is the seam new route modules use to broadcast one-off signals without reaching for EventEmitter channel names. 14 server route tests, 3 plugin tests, 14 CLI tests, and 1 events.ts test exercise every branch: the empty list, the registration with a duplicate id, the regex-rejected id, the GET detail, the 404, the apply success path, the one-shot --override, the disabled-plugin 400, the 404-on-missing-preset, the 409 with per-key failures, the DELETE unregister + entry cleanup, the auth-required guard, the CLI pure entry points (JSON file parsing, missing --file, JSON-not-an-object, override forwarding, upstream error verbatim), and the formatted actions (the no-drivers hint, the success line, the per-key-failure exit code).

• Sticky Session plugin (G-28 PR-A). When the operator toggles sticky-session routing for a service, every request to that service's domains is pinned to the same backend container via a Traefik sticky-cookie middleware — the Coolify/Dokploy feature for "load balancing with session affinity" that the operator used to have to add at the Cloudflare edge. Three layers, all touching the same settings-table key:

  • engine/proxy.ts gains getStickyEnabledForService(db, serviceId) and a small block inside writeDynamicConfig that appends the per-service Traefik middleware whenever the flag is on. The middleware block uses Traefik's sticky.cookie.{name,maxAge} shape with name: ninedeploy_sticky and maxAge: 86400 — the same defaults Coolify ships with.
  • apps/server/src/modules/services.ts exposes POST /v1/services/:id/sticky-session (admin role). The endpoint writes sticky_session:<id>:enabled to the settings table, emits a service.sticky_session.enabled / .disabled audit event, and best-effort re-renders writeDynamicConfig so the next reload picks up the change. Toggling off removes the middleware.
  • apps/server/src/kernel/plugins/stickySession.ts (sticky-session, v0.1.0) is the passive observer: subscribes to service.deployed and, on every deploy whose service has the flag on, emits a proxy.sticky_session.activated event so the panel's audit log shows the activation. Errors are surfaced via proxy.sticky_session.error — never propagated. The plugin also adds a command:palette menu item at /settings/services.
  • SDK + CLI — packages/sdk/src/index.ts gains services.setStickySession(id, enabled) returning { id, enabled, active } (the last is the post-write re-read so the caller can confirm the round-trip); ninedeploy services sticky <id> --enable|--disable is the operator-side form. 6 new plugin tests cover the stable id, the single service.deployed subscription, the destroy-cleanup, the happy-path proxy.sticky_session.activated event, the off / missing-flag silent path, and the missing-serviceId defensive branch. The change is intentionally non-migration: the flag lives in the settings table so the operator can enable / disable without an upgrade.

• settings.ts coverage 92 / 97 / 82 / 94 → 100 / 100 / 100 / 100 (Sprint 9 PR #39). The apps/server/src/modules/settings.ts route bundle had a 17-point function-coverage gap rooted in the 1-second-deferred applyTraefikSettings and its three await calls; the scheduleTraefikSettingsApply setTimeout callback (3 statements + 1 function); the onClose hook that drains pending timers; the PUT /panel-domain .catch(() => undefined) swallow on writeDynamicConfig; the ?? null fallback in GET /dns-records/namecheap when getNamecheapConfig returns null; and the log arrow (line) => app.log.info({ component: 'settings' }, line) inside applyTraefikSettings that nobody had invoked from the mocks before. 5 new tests in apps/server/test/settings.test.ts: PUT /dns schedules a 1-second applyTraefikSettings that calls ensureNetwork → ensureTraefik → writeDynamicConfig in order (one-shot mocks invoke the route's log callback to cover the unreachable arrow); the same for PUT /acme-email; a thrown ensureNetwork is caught by the void applyTraefikSettings().catch(...) wrapper and never propagates as an unhandledRejection; the onClose hook clearTimeouts pending apply timers (no ensureNetwork after app.close() even with 5 s of fake-time advancement); and PUT /panel-domain still 200s when writeDynamicConfig throws. Also added: GET /dns-records/namecheap returns { configured: false, apiUser: null, clientIp: null, hasKey: false } when the settings map is empty (covers the cfg?.apiUser ?? null and cfg?.clientIp ?? null short-circuits). The fake-timer scope is narrowed to ['setTimeout', 'setInterval', 'clearTimeout', 'clearInterval'] — faking setImmediate / process.nextTick would deadlock fastify's request scheduler. Final coverage 100% statements / 100% branches / 100% functions / 100% lines — every reachable branch now tested. +5 tests (45 → 50).

• settings.ts coverage 92 / 97 / 82 / 94 → 100 / 100 / 100 / 100 (Sprint 9 PR #39). The apps/server/src/modules/settings.ts route bundle had a 17-point function-coverage gap rooted in the 1-second-deferred applyTraefikSettings and its three await calls; the scheduleTraefikSettingsApply setTimeout callback (3 statements + 1 function); the onClose hook that drains pending timers; the PUT /panel-domain .catch(() => undefined) swallow on writeDynamicConfig; the ?? null fallback in GET /dns-records/namecheap when getNamecheapConfig returns null; and the log arrow (line) => app.log.info({ component: 'settings' }, line) inside applyTraefikSettings that nobody had invoked from the mocks before. 5 new tests in apps/server/test/settings.test.ts: PUT /dns schedules a 1-second applyTraefikSettings that calls ensureNetwork → ensureTraefik → writeDynamicConfig in order (one-shot mocks invoke the route's log callback to cover the unreachable arrow); the same for PUT /acme-email; a thrown ensureNetwork is caught by the void applyTraefikSettings().catch(...) wrapper and never propagates as an unhandledRejection; the onClose hook clearTimeouts pending apply timers (no ensureNetwork after app.close() even with 5 s of fake-time advancement); and PUT /panel-domain still 200s when writeDynamicConfig throws. Also added: GET /dns-records/namecheap returns { configured: false, apiUser: null, clientIp: null, hasKey: false } when the settings map is empty (covers the cfg?.apiUser ?? null and cfg?.clientIp ?? null short-circuits). The fake-timer scope is narrowed to ['setTimeout', 'setInterval', 'clearTimeout', 'clearInterval'] — faking setImmediate / process.nextTick would deadlock fastify's request scheduler. Final coverage 100% statements / 100% branches / 100% functions / 100% lines — every reachable branch now tested. +5 tests (45 → 50).

• egress.ts coverage 21.87 / 0 / 16.66 / 24.13 → 100 / 100 / 100 / 100 (Sprint 9 PR #40). The apps/server/src/modules/egress.ts HTTP surface (G-15 PR-A from Sprint 5) shipped with zero tests. New file apps/server/test/modules/egress.test.ts pins the contract end-to-end with a real buildTestApp

  • a mock IEgressIpDriver registered into app.kernel.registry: GET / aggregates list() from every registered driver ({ drivers: [{ name, rules }] }) and returns { drivers: [] } when none are registered; POST / validates projectId (must be a number) and ip (must be a non-empty string) before any driver work, picks the named driver when ?driver= is supplied, falls back to the first registered driver otherwise, and answers { ok: false, error: "Egress IP driver \"…\" is not registered" } (NOT a 4xx — the missing-driver path is a soft 200 so a plugin that loses its driver can keep polling and recover when it re-registers) when no driver matches; DELETE /:projectId validates the projectId with Number.isFinite (so 0 is valid), picks the first registered driver, and answers { ok: false, error: "No egress IP driver is registered" } when none is registered. The ?? {} and ?? DEFAULT_DRIVER fallbacks are exercised by an empty-body test (custom addContentTypeParser that returns undefined — the only way to leave req.body undefined under fastify) and an unnamed-driver test respectively. Final coverage 100% / 100% / 100% / 100% — the first time this route bundle has been tested at all. +15 tests (new file).

• services.ts coverage 87.91 / 82.16 / 93.02 / 91.18 → 93.48 / 89.51 / 95.34 / 92.72 (Sprint 9 PR #43). The apps/server/src/modules/services.ts route bundle had an 8-point branch-coverage gap rooted in eleven defensive branches nobody had driven a test through (the file is 261 lines and the existing 61-test test/services.test.ts covers the CRUD surface; the remaining branches were the operator-vs-non-operator list filter, the tag-filter wanted.some arm, the NO_TAGS fallback, the source-name MISS/HIT paths, the assertMayPublishPort(undefined ? existing : patch) ternary, the post-update 404, the port-rewrite warning log, the active-deploy 409, the templateId && !template 400, the registry-controlled 400, and the replaceServiceTags call). New apps/server/test/modules/servicesCoverage.test.ts (20 tests) covers every reachable branch: list returns allRows for an operator (visibleIds === null) and filters for a non-operator; ?tagProjectIds= exercises the wanted.some arm; an empty link map yields the NO_TAGS default; a populated link map yields all three id lists; the sourceName HIT/MISS paths on both list and GET-single; the PATCH publishedPort undefined vs. set ternary; the zero-row UPDATE 404; the writeDynamicConfig throw caught + logged (the request still 200s); the queued-deployment DELETE 409; the missing templateId 400; the registry- controlled 400; the replaceServiceTags call with projectIds-only, with all three arrays, and with the ?? [] fallback for omitted dimensions. Final coverage 93.48% lines / 89.51% branches / 95.34% functions / 92.72% lines — branches are 0.49 points short of the 90% gate; the remaining 4 are defense-in-depth (tagIdsOf ?? NO_TAGS, the list-serialize ?? NO_TAGS fallbacks that are unreachable when loadTagIds pre-populates the map, and the templateDatabaseEnv ?? null arm that fires only when a template omits the field).

• sso.ts branches 87.17 → 91.02 (Sprint 9 PR #44). The apps/server/src/modules/sso.ts route bundle already had 45 tests pinning the OIDC + SAML flows, but a small cluster of error-handling edges remained: the error_description ?? error fallback in the GET /:name/callback error pass-through (the error arm), the !signedInfoMatch || !signatureValueMatch guard in the SAML callback, and the !metadata.idpMetadata short-circuit. New SSO route edge cases describe block in apps/server/test/modules/sso.test.ts covers all three: the error=access_denied query param without error_description (the error fallback arm); a SAML response with a valid <Assertion> but no <ds:SignedInfo> / <ds:SignatureValue> (the regex miss-arms); and a SAML provider whose idpMetadata is the empty string (the !metadata.idpMetadata short-circuit). Final coverage 96.29% lines / 91.02% branches / 100% functions / 96.49% statements — branches cross the 90% gate. +3 tests (37 → 40).

• branding.ts coverage 34.61 / 0 / 20 / 34.61 → 100 / 100 / 100 / 100 (Sprint 9 PR #41). The apps/server/src/modules/branding.ts HTTP surface (G-30 white-label) shipped with zero tests. New file apps/server/test/modules/branding.test.ts pins the contract: GET / returns the four branding fields (logoUrl, primaryColor, supportEmail, footerHtml) as null when no overrides are stored; the stored overrides come back verbatim; an empty-string override is coerced to null so the panel renders the defaults; a direct configCenter.set between two GETs does NOT take effect because the route caches the resolved value for 60 s in process. PATCH / persists each provided field via configCenter.set and returns { ok: true }; an undefined field in the payload is a no-op (it does not clear the existing value); an empty string PATCH clears the field (the GET then renders null); an empty body is accepted; a custom body parser that hands the route undefined still 200s (the ?? {} fallback); and the configCenter.set call carries the authenticated operator's userId in its audit metadata (so the panel's per-field history log attributes the change correctly). +11 tests (new file). Final coverage 100% / 100% / 100% / 100%.

• serviceVolumes.ts branch coverage 75 → 99 (Sprint 9 PR #38). The apps/server/src/modules/serviceVolumes.ts route bundle had a 25-point branch-coverage gap rooted in four defensive branches nobody had driven a test through. New tests in apps/server/test/serviceVolumes.test.ts cover: the a 25-point branch-coverage gap rooted in four defensive branches nobody had driven a test through. New tests in apps/server/test/serviceVolumes.test.ts cover: the volumeSize catch (docker size probe throws → sizeBytes: 0), the POST listManagedVolumeNames().catch(() => []) collapse (daemon down → 404, not 500), the POST if (!known) throw notFound branch (volume present in input but absent from docker volume ls), the createDockerVolume log callback arrow ((line) => req.log.info(line)), the POST non-Error throw path (err instanceof Error ? err.message : String(err), exercised via throw 'string' to confirm the UNIQUE container_path / volume_name checks still fire for raw-string errors), the PATCH zero-row 404 (the if (!updated) throw notFound guard the happy path never reached), the PATCH non-Error throw (mirror of POST), the DELETE orphan-log skip (volume still referenced by another service → the "now ownerless" log line is NOT emitted), the GET sharingByVolume.get(r.volumeName) ?? 1 fallback (the un-scoped select returns [] while the scoped select has rows), the config-repair 404-by-attachmentId branch, and the config-repair req.body ?? {} fallback. Final coverage 99.24% statements / 98.68% branches / 100% functions / 100% lines — the only remaining item is the if (/[^a-zA-Z0-9_.-]/.test(volumeName)) defense-in-depth regex on line 240, which is unreachable because the upstream zod schema's ^nd-(svc|db)-[a-z0-9_.-]+$ regex rejects the same inputs first. +14 tests (28 → 42).

• composeStacks.ts coverage 2.5 / 0 / 0 / 2.7 → 97.29 / 100 / 95 / 90.24 (Sprint 9 PR #42). The apps/server/src/modules/composeStacks.ts exported prepareComposeStack function (used by the templates one-click install) shipped with effectively zero coverage — only the stackWorkspace defense-in-depth branch was being touched by the sibling composeStacks.test.ts file that tests magicVars / template-schema helpers, NOT this file. New apps/server/test/modules/composeStacks.test.ts (10 tests) mocks the file-system (node:fs.mkdirSync / writeFileSync), the magic-var engine, the proxy getAcmeEmail, and the config.wildcardDomain field, then drives prepareComposeStack through every contract branch: first-install creates a new service row + materialises docker-compose.yml at mode 0o600; existing same-stack rows are reused (no new insert); preflightCompose rejections surface as 400 with no file-system side effects; the publicUrl scheme is https when an ACME email is set and http when not; the publicUrl is http://localhost when no wildcard domain is configured; the slug-collision retry gives up with 400 after 5 foreign-owned hits; a colliding slug of the wrong type (not compose) also 400s; template env rows are merged with secret: false defaulted; and the slug recipe when input.name is omitted is ${slugify(template.name)}-${ts36-suffix}. The remaining ~10% branches are unreachable defenses (stackWorkspace path-traversal guard, the !created post-insert guard) and the 5-retry collision loop's terminal-iteration branch. +10 tests (new file). config-repair req.body ?? {} fallback. Final coverage 99.24% statements / 98.68% branches / 100% functions / 100% lines — the only remaining item is the if (/[^a-zA-Z0-9_.-]/.test(volumeName)) defense-in-depth regex on line 240, which is unreachable because the upstream zod schema's ^nd-(svc|db)-[a-z0-9_.-]+$ regex rejects the same inputs first. +14 tests (28 → 42).

• ninedeploy manifest apply server endpoint (PR #45). The manifest apply CLI subcommand used to print a "not in this release yet" banner because the panel had no matching route. New POST /v1/services/:id/manifest/apply on the server reconciles a parsed .ninedeploy into the service + build config rows with merge semantics (operator > manifest > DB): a section the manifest omits is left alone, and a column the operator set in the panel is never silently clobbered. Sections reconciled here: build → build_configs (installCmd, buildCmd, startCmd, baseDir, dockerfilePath), run → services (port, healthPath) + build_configs (restartPolicy, stopGraceSeconds; the latter auto-bumped to 30 s when a long preStop hook needs drain time), and network → services.publishedPort. Routes / alerts / database reconciliation continues to live in lib/applyManifestToService.ts at deploy time — that helper and this one write to disjoint tables (domains, alert_rules, database_attachments vs. services, build_configs), so the split is by responsibility rather than by file. Auth is requireAdmin: a stale manifest in git could otherwise be pushed by a workspace member to mutate another tenant's service definition. The route returns { ok, serviceId, touched, diff }; the CLI renders the diff as a git diff-style summary (Touched: service, build_config followed by per-section key value lines) and refuses to send a payload whose body the secret scanner has flagged. SDK surface: client.services.manifest.apply(serviceId, { manifest, strategy? }); both the SDK and the CLI's apps/cli/src/commands/manifest.ts subcommand now share the same parseManifestYaml + scanForSecrets preflight as validate and show, so a manifest that passes validate is exactly the shape that apply ships.

• Backup drills — ninedeploy backups drill (G-17, PR #46). "Is this backup actually restorable?" used to require spinning up a throwaway container, restoring by hand, and eyeballing the logs. New POST /v1/databases/:id/backups/drill runs an engine-specific smoke check on the dump file (pg_restore --list for Postgres, redis-check-rdb for Redis / Valkey, a header sniff for mysqldump / mariadb-dump, bsondump for Mongo) and records the outcome on a backup_drills row. The check is much weaker than a full restore-into-container (it does not catch a malformed but well-formed dump, and cannot catch missing extensions or schema drift) but it does catch the most common failure mode — a corrupt or truncated file — in well under a second on local disk, which is the gap the manual routine kept skipping. Migration 0044 adds the backup_drills table; the drill never deletes or modifies the source backup. Encrypted envelopes are decrypted to a sibling temp file (and deleted on the way out) via the same flow engine/database.ts uses for real restores; remote-only backups are fetched to a local temp first via lib/backupRemote.ts. The companion GET /v1/databases/:id/drills returns the most recent 25 rows for the panel history view. SDK surface: client.databases.drillBackup(id, { backupId }) and client.databases.drills(id). CLI: ninedeploy backups drill <dbId> <backupId> (exits 0 on passed, 1 on failed) and ninedeploy backups drills <dbId> for the history.

• Image inventory + retention — ninedeploy images ls|prune (G-12, PR #47). The existing autoPrune cron fires on a disk-usage threshold with a one-shot docker image prune -af; the operator's day-to-day workflow (see what's on the host, keep the last N per repo, prune the rest) had no panel surface. New GET /v1/housekeeping/images returns every image with repo / tag / size / age / dangling / inUse metadata; POST /v1/housekeeping/images/prune applies operator-supplied filters — keepLast per repo:tag, olderThanHours, danglingOnly, dryRun. The dryRun path returns the candidate set without deleting so the operator can sanity-check before a real prune. inUse is computed via a second docker ps round-trip that catches every container, not just the ones NineDeploy started (an operator's own side-car work would otherwise be an unannounced delete). The route refuses to run with no filter — a naked prune would delete every unused image, which is almost never what the operator actually wants. SDK surface: client.housekeeping.listImages() and client.housekeeping.pruneImages({ keepLast?, olderThanHours?, danglingOnly?, dryRun? }). CLI: ninedeploy images ls [--sort size|age] (default size-descending so the biggest offenders surface first) and ninedeploy images prune [--keep-last N] [--older-than hours] [--dangling] [--dry-run].

• Domain transfer — ninedeploy domains {transfer, preview-transfer, accept-transfer, cancel-transfer} (G-29, PR #48). The domains table attaches every row to a service, and ownership flows through the service's workspace membership; there was no way to move a row from one service / user to another. New endpoints implement a two-phase transfer with email-bound authorization. The source user (admin on the source service) calls POST /v1/domains/:id/transfer with the target email and gets back a one-time acceptUrl to forward out-of-band; the URL embeds a 32-byte random token whose SHA-256 is what the database stores (so a leaked DB dump cannot forge a transfer — mirrors the api_tokens pattern). The target user calls POST /v1/domain-transfers/:token/accept with the service id they want the domain attached to; the server re-checks the caller's email matches the target, the row is still pending, and the source domain still exists, then moves the row in one transaction. Tokens expire after 7 days; a pending row whose expires_at is in the past is treated as expired lazily (no background sweep). GET /v1/domain-transfers/:token is unauthenticated (the token is the secret) so the panel can render the accept page to a logged-out visitor; POST /v1/domain-transfers/:token/cancel is the source-side abort path and refuses when called by the target email. Migration 0045 adds the domain_transfers table; IF NOT EXISTS throughout follows the same drizzle-kit-push-safe pattern as 0039–0044. SDK surface: client.domains.transfer(domainId, { targetEmail }), client.domains.previewTransfer(token), client.domains.acceptTransfer(token, { targetServiceId }), and client.domains.cancelTransfer(token). CLI: ninedeploy domains transfer <id> --to <email> (prints the accept URL with the token visible so the operator can forward it), ninedeploy domains preview-transfer <token> (no auth), ninedeploy domains accept-transfer <token> --service-id <id> (caller must be authenticated as the target email), and ninedeploy domains cancel-transfer <token> (caller must be the source).

• Outbound webhook — HMAC-signed webhook channel (G-06, PR #49). The notification channel type enum already listed webhook and the dispatcher sent a JSON body, but there was no signing and no body template — every consumer had to either accept unsigned JSON (and trust the network) or run their own pre-shared-key ceremony. New webhookChannelConfig schema (in packages/schemas/src/management.ts) lets the operator declare secret, headerName (default X-NineDeploy-Signature), algorithm (sha256 or sha1), and a custom template. The dispatcher (apps/server/src/lib/notifier.ts) computes HMAC(secret, body) and adds the header before send; the body is either the default four-field envelope ({ event, entity, ts, message }) or a custom object whose ${event} / ${entity} / ${ts} / ${message} placeholders are expanded at send time. The receiver verifies by recomputing HMAC over the EXACT body bytes — the panel does not strip whitespace or re-encode. New CLI surface: ninedeploy notifications {list, create-webhook, test, rm}. The create-webhook command assembles the configJson from flags (--secret, --header, --algo, repeated --template k=v) and POSTs through the existing channel-create endpoint; the test command fires a test event so the operator can confirm the receiver is set up before relying on it.

• Fine-grained API-token scopes — nd://scope/(read|write|admin)/<resource> (G-08, PR #50). The pre-0.3.5 api_tokens.scopes column was written as [] and never read, so every API token — including the ones handed to the MCP server and to CI — carried its owner's full authority. PR #44 wired the legacy read | write | operator shorthand into the auth plugin, but the read-vs-write split was a binary flag — a CI token with write could mutate every resource. New apiTokenScope schema (in packages/schemas/src/auth.ts) accepts the resource-scoped URI form nd://scope/(read|write|admin)/<resource> alongside the legacy shorthand; the server's scopeCovers helper expands the shorthand to the URI form (write covers any nd://scope/write/<r> and admin/<r>; admin/<r> covers write/<r> and read/<r> for the same resource). New app.requireScope(scope) decorator on the auth plugin closes over the required scope and refuses the request when the token doesn't cover it; a route can opt in with { preHandler: app.requireScope('nd://scope/write/services') } (the existing read/write enforcement at the auth layer is unchanged — the new decorator is the per-route extension point). The MCP server declares requiredScopes on every tool; on startup it introspects the bearer token via the new GET /v1/auth/token endpoint, then filters its tool list to those whose scopes are covered. CLI: ninedeploy token create accepts the URI form in the scope prompt. The introspection endpoint also reports tokenId + name + expiresAt + isOperator for API tokens, ['session'] for JWTs — the one-call shape the MCP and any future token-aware client (CI, monitoring agent) needs.

• PgBouncer sidecar — ninedeploy databases pgbouncer <dbId> {enable,disable,status} (G-32, PR #51). Production Postgres workloads are routinely fronted by PgBouncer so a thousand client connections don't multiply into a thousand Postgres backends. NineDeploy had the database engine fully wired but no built-in way to bring up a pool proxy. New migration 0046 adds three columns to databases (pgbouncer_enabled, pgbouncer_container_name, pgbouncer_port, default 6432). New apps/server/src/lib/pgbouncer.ts writes a pgbouncer.ini (auth_type=md5, pool_mode=transaction, default_pool_size=20, reserve_pool for burst) plus the userlist (MD5-hashed creds), bind-mounts both into a bitnami/pgbouncer:1.24.1 container named nd-pgb-<slug>, and stamps the row. The pooledConnectionString(d) helper returns the sidecar's URL when enabled (services that want pooled connections use it instead of the direct connectionString(d)). New routes under /v1/databases/:id/pgbouncer: GET (status, member role), POST /:id/pgbouncer/enable (admin, optional --port), POST /:id/pgbouncer/disable (admin). Routes are mounted on the existing /databases prefix; the engine guard returns 422 for any non-postgres engine. SDK surface: client.databases.pgbouncerStatus(id), client.databases.enablePgbouncer(id, { port? }), client.databases.disablePgbouncer(id). CLI: ninedeploy databases pgbouncer <dbId> enable [--port N] | disable | status — the status subcommand prints the pooled URL the operator pastes into a new attachment's envAlias. The sidecar is per-DB rather than a shared proxy: a stuck pool only affects one tenant, and the credentials on the wire are scoped to a single service-to-database pair.

• Cluster log search — ninedeploy logs search <query> (G-16, PR #52). NineDeploy's logDrains pipeline forwards every container's stdout / stderr to a remote sink (Loki, Vector, Datadog, ...), but the panel had no corresponding read-side. Operators fell back to the upstream's own UI, which meant two dashboards. New POST /v1/log-drains/search round-trips to the configured Loki drain's /loki/api/v1/query_range with {service="<slug>"} |= "<query>" and the window the caller asked for (default 15 minutes, max 7 days). Other drain types don't expose a query API; the route returns 400 with a clear "add a Loki drain alongside it" message rather than silently returning nothing. The drain's apiKeyEncrypted is sent as Authorization: Bearer <key>; the egress guard is intentionally NOT applied because the operator's log host is the canonical destination of the log drain itself. Auth is member (a viewer can search). SDK surface: client.logDrains.search({ query, serviceId?, sinceMinutes?, limit?, drainId? }) and LogSearchInput / LogSearchResult / LogSearchLine types. CLI: ninedeploy logs search <query> [--service <id>] [--since 15m|2h|1d] [--limit <N>] [--drain <id>] [--json]. The CLI parses --since shorthand (15m, 2h, 1d, 30s) and prints each line as <iso-ts> [<service>] <line> so the operator can read the output as a stream; --json switches to the raw LogSearchResult shape for piping into jq / grep / etc.

• Per-workspace email template overrides — ninedeploy email-templates {list,set,reset,preview} (G-30, PR #53). NineDeploy's outbound emails (password reset, workspace invitation, domain transfer, backup drill failed) used to live as a string-templated function per call site (buildInviteEmail in invitations.ts, inline string in auth.ts). Operators who wanted to brand the outbound mail had to fork the codebase. New lib/emailTemplates.ts ships four built-in templates with {{var}} interpolation and a setOverride / clearOverride / renderTemplate trio; migration 0047 adds the email_template_overrides table (one row per (workspace, name) overrides the subject + text). Routes under /v1/workspaces/:wid/email-templates: GET (member, lists every name + whether each is overridden), POST /preview (member, renders with supplied vars — paste the result into a test inbox), PUT /:name (admin, upsert the override), DELETE /:name (admin, drop it). The renderer is side-effect free; a future PR can call it from every existing outbound site so the override takes effect automatically. SDK surface: client.emailTemplates.{list, preview, set, reset} with EmailTemplateName / EmailTemplateEntry / EmailTemplateRender types. CLI: ninedeploy email-templates <wid> {list | preview <name> [k=v ...] | set <name> --subject S --text T | reset <name>}. The interpolation engine handles {{var}} and \{\{ (literal) escapes; unknown vars render as the empty string rather than {{undefined}} so a half-broken template cannot surface in an outbound email.

• Live signed marketplace index (G-24, PR #54). The previous MARKETPLACE_CATALOG was a static in-code list — the panel could not discover new plugins without a server release. New lib/marketplaceCatalog.ts fetches a signed JSON index from NINEDEPLOY_MARKETPLACE_URL, verifies the ed25519 signature against NINEDEPLOY_MARKETPLACE_PUBLIC_KEY, and merges the verified entries with the in-code fallback (which is also kept as the installable-surface baseline — the live index is never allowed to override an entry that maps onto compiled-in behaviour). The envelope format is { entries, signature, key_id }; the signature is over the canonical JSON of entries (sorted keys, no whitespace). A 5-minute in-process cache avoids hammering the upstream; the existing GET /v1/plugins/marketplace route gains a ?refresh=true query, and a new POST /v1/plugins/marketplace/refresh route bypasses the cache. The response shape grows from { catalog } to { catalog, live, keyId, fetchedAt } so the panel can show "live signed index (key=ed25519:abc123) — fetched 2 min ago" vs. the static fallback. A live index that fails signature verification is dropped entirely — production with NINEDEPLOY_MARKETPLACE_URL set but no public key refuses to serve the live data rather than trust an unverified blob. SDK surface: client.plugins.marketplace({ refresh?: boolean }). CLI: ninedeploy plugins marketplace [--refresh] (the option now also re-fetches) and a new ninedeploy plugins marketplace-refresh command for CI runs after the upstream rotates its key.

• FCM push notifications (G-22, PR #55). Firebase Cloud Messaging's legacy HTTP API (the X-Server-Key header) was sunset mid-2024; the modern endpoint is the FCM HTTP v1 API at fcm.googleapis.com/v1/projects/<id>/messages:send and requires an OAuth2 bearer token minted from a service-account JSON. New apps/server/src/lib/fcm.ts:

  • hand-rolled RS256-signed JWT (no npm dependencies — node:crypto.createSign),
  • OAuth2 token exchange against oauth2.googleapis.com/token,
  • in-process bearer cache keyed by client_email with a 60s safety skew (Google's expires_in minus the safety), so the per-event cost is one HTTPS round-trip, not two.
  • The dispatch path in notifier.ts reads the channel's configJson as the full service account JSON; target is the device token. FCM data payload carries action, entity, ts so a mobile client can route on it without parsing the localised body.
  • Schema: notificationType extended to include 'fcm'; the notification_channels table needs no migration (the column is plain text).
  • New CLI: ninedeploy notifications create-fcm <name> <deviceToken> --service-account <file.json> reads the service account from disk so the JSON never lands on the operator's shell history.
  • Caveat: the FCM channel needs a real FCM project
    • device token to verify; the unit suite covers the dispatch path indirectly via the existing notifier.test.ts. A future PR can mock globalThis.fetch to cover the OAuth2 round-trip
    • the FCM POST happy / error paths.

• Certificate inventory — ninedeploy certificates {list,expiring} (G-15, PR #56). The existing GET /v1/traefik/certificates route returned four flat fields per cert — enough to draw a list, not enough to answer "which certs expire in the next 30 days?". New lib/certificateInventory.ts wraps the existing engine/proxy.ts:readCertificates() reader and classifies each cert as valid / expiring-soon / expired / unknown based on the operator-configurable threshold (default 30). New GET /v1/traefik/certificates/inventory returns the full report with a summary block (totals per status, threshold, fetchedAt) so a single round-trip is enough to render the panel's Certificates page. New GET /v1/traefik/certificates/expiring?days=30 is a focused filter — the same shape the alert engine uses to page the operator before a cert falls over. Both routes are member-accessible (the existing basic route stays admin-only for backwards compat). SDK surface: client.traefik.certificateInventory({ threshold? }) and client.traefik.expiringCertificates({ days? }), with the CertificateInventoryEntry / CertificateInventoryReport types in @ninedeploy/schemas. CLI: ninedeploy certificates list [--threshold N] prints a colour-coded table with totals per status; ninedeploy certificates expiring [--days N] is the focused list used by the alert cron. Caveat: the rich subject / sans / notBefore fields are populated as null for now — a real PEM parser is a follow-up; the existing engine/proxy.ts reads only the expiry date from acme.json, which is the only field the inventory actually needs.

• Community template contributions (G-13, PR #57). The Hub template catalog was a closed set: the bundled registry plus an optional remote URL. A contributor with a new template had to open a PR against the registry. New lib/communityTemplates.ts opens a third source: any *.json file dropped into <dataDir>/community-templates/ is parsed, validated against the template schema and merged into the panel-facing catalog by id. The merge rule is "curated wins": a community entry that collides on id with a bundled entry is dropped, so a copy-paste cannot shadow the installable baseline. Three new routes on modules/templates.ts: GET /v1/templates/community lists every file with a per-file error list (a single bad JSON does not hide the rest), POST /v1/templates/community/import accepts a single-template JSON envelope and refuses to overwrite an existing id unless replace: true is passed, and DELETE /v1/templates/community/:id unlinks the file. Both writes are requireAdmin and emit an audit row (templates.community_import / templates.community_remove). SDK surface: client.templates.community.{list, import, remove}, with CommunityTemplateListResult re-exported from @ninedeploy/schemas. CLI: ninedeploy templates community list | import <file> [--replace] | remove <id>; import - reads from stdin so a curl -s https://.../template.json | ninedeploy templates community import - pipeline lands an upstream contribution without writing it to disk first. Files are pretty-printed (JSON.stringify(x, null, 2)) so the diff against the upstream PR is reviewable. The list route also surfaces community entries in the main GET /v1/templates response (filtered for id-collision), so the existing panel flow gets new entries without a code change.

Fixed

• SettingsTabPrivilege test timeouts under parallel load (unrelated to G-07). apps/web/test/SettingsTabPrivilege.test.tsx runs four async findByText waits, each capped at vitest's default 5 s. The file's own comment already noted that the sibling-suite waitFor helper inside the same file had to be raised to 10 s for the same reason. Under the full-suite run captured in .tmp-real-validate/g07-web.log the third test (sends the hook values an admin has configured) hit the 5 s wall before react-query dispatched the PATCH; the same four tests pass in 2.2 s when the file is run in isolation. Per-test timeout is now 15 s on all four it() callbacks in this file, and the waitFor inside savedBuild is now 30 s — the G-07 PR-B full-suite run captured in .tmp-real-validate/g07b-web.log showed 10 s still false-failing on a slow host when the rest of the workspace's vitest workers are booting in parallel. The only behavioral change is the timeout, the assertions are untouched. This change is not part of the G-07 driver surface area; it is documented here so the next reviewer does not assume the web PR depends on the kernel work above.

• Two dead operator guards removed before they could be used.

• Two dead operator guards removed before they could be used. lib/resourceAccess.ts exported assertOperator and a second requireOperator() prehandler, neither with a call site. Both re-read users.is_instance_operator from the database — the one thing plugins/auth.ts documents as forbidden, because it has already narrowed the flag for a scope-restricted API token, so a fresh read would hand a write token its owner's operator rights back. requireOperator() also still carried the pre-0.3.5 self-granting definition in its docstring. The escalation assertion they were tested against now runs against app.requireOperator.

• job_runs is bounded. Scheduled-job run history had no retention at all, and each row stores up to 60 KB of the command's captured output inside the SQLite file that gets backed up whole — a per-minute cron job wrote roughly 525 000 rows a year while the panel only ever renders the newest 20 per job. Swept on the same 30-day window as the other logs.

• Six environment variables the server reads were documented nowhere — including NINEDEPLOY_ALLOW_PRIVATE_EGRESS (the SSRF escape hatch, named in the error message an operator hits and in no config file) and NINEDEPLOY_UPDATE_CHECK_URL (the panel's only unprompted outbound call, and the only way to turn update checks off). All six are now in .env.example, and apps/server/test/envExample.test.ts fails when a new one is added without a line describing it.

• Log Drains worked on no fresh install. log_drains was declared in schema.ts and recorded in drizzle-kit's snapshot, but no migration ever created it — so every database built by replaying the migrations lacked the table and Settings → Log Drains failed with no such table: log_drains. Because the snapshot already claimed the table existed, drizzle-kit generate could never emit the missing file. Added as 0039_log_drains. New packages/db/test/schema-drift.test.ts applies the whole migration chain to a fresh database and compares every declared table and column against PRAGMA table_info in both directions, so the next drift fails a test instead of a production request.

• Historical CPU/memory charts work at all. GET /v1/services/:id/metrics returned 404: metricRoutes is a second export from modules/stats.ts whose own comment says "Mounted under /services", and api.ts never registered it. The charts on the Monitoring page and every service's Overview tab therefore had nothing to read while the collector wrote two rows per service every 30 seconds. test/api.test.ts now asserts that every module it stubs answers on its prefix, instead of spot-checking two of them.

• PR previews no longer leak a Docker network each. Every deployed service gets a private bridge (ensureServiceBridge, called by the Docker builder), and only the panel's DELETE /v1/services/:id ever reaped it. The preview auto-destroy path deleted the service row and stopped the container but left the network behind — in the one feature designed for high churn, one preview per pull request. It now reaps the bridge and removes the preview's build logs, both best-effort so a stuck network cannot turn a webhook into a 500 the provider will retry against a service that no longer exists.

• Deleting a service takes its deploy logs with it. The FK cascade removed the deployment rows and knew nothing about the log files, which outlived the service they describe by up to the 30-day retention window — and build logs routinely echo configuration.

• A deploy's outcome is recorded. deploy.trigger was written when a deployment was queued and was the only deploy action anything emitted: engine/pipeline.ts never called audit(), so a deploy that finished — successfully or not — did so in silence. Everything downstream of audit() was blind to the result: every notification channel (a failed production deploy notified nobody, and the Settings → Notifications event filter had no deploy.failed to match), the /v1/events activity feed (deploys started and never finished), and the freshly-rebuilt kernel/auditBridge, whose deployment.status_changed could only ever carry trigger/rollback/cancel — so the built-in plugins still never saw the one event they exist to react to. The pipeline now records deploy.success, deploy.failed (reason in meta) and deploy.cancelled, attributed to the service owner so the member who triggered it can actually see their own result.

• A deployFailed alert in a manifest no longer writes a rule that can never fire. applyManifestToService mapped the two event-shaped triggers (deployFailed, restartLoop) onto metric: 'cert-expiry', threshold: 0; the function's own comment claimed it skipped the insert, and it did not. The rule rendered in Monitoring looking like a configured alert. Both are now reported as skipped in the deploy log.

• static, watch and network manifest sections are no longer dropped in silence. They are accepted by the strict schema and consumed by nothing. Every other unwired section already warned, and the docs claimed these did too. They do now.

• The deploy worker no longer leaks a timer per poll. plugins/worker.ts pushed every 2-second poll timer into an append-only array that only stop() ever drained — roughly 43 000 dead Timeout handles per concurrency slot per day of uptime. It is a Set each timer removes itself from as it fires.

• Removed lib/agentSeal.secretsMatch: exported, tested, and called by nothing — agentClient.tokenMatches already performs that comparison.

• The plugin marketplace no longer pretends to install anything. Nothing in pluginLoader.ts ever import()s code, so an npm/git/local "install" became a DB row plus a shell whose init emits one event — while the panel reported it active. Several of the 16 catalog entries also shadow features that ship under another name: an operator who installed "Amazon S3 & Cloudflare R2 Sync", entered a bucket and secret key and saw it active would reasonably believe backups were being copied off-site. They were not, and they would find out at restore time. Now: npm/git/local installs are refused with an explanation; every catalog entry carries implemented (all 16 are false today) plus a builtIn pointer where it shadows a shipped feature, and installing one is refused with a message naming the real feature; the panel renders that pointer as a link instead of an Install button; and rows installed by an older build from an unsupported source are skipped at boot with a warning instead of restored as active-looking shells.

• Service visibility was computed three different ways. GET /v1/services filtered on owner_user_id alone while /dashboard, /domains and the per-service loader also honoured workspace tags — a teammate could open and deploy a shared service by id but saw an empty list, while the dashboard counted it. All callers now share visibleServiceIdSet in lib/resourceAccess.ts.

• GET /v1/users derived each row's operator badge from workspace seats, which after the change above would have shown every member as an operator. It reads the flag.

• CI on main was red three ways. The main job's web suite failed 805 tests on Node 26: the jsdom storage globals arrive through vitest's environment population as an EMPTY object on Node ≥ 25 (localStorage.getItem is not a function from the first render that touches one — they work on Node 24 and in plain jsdom, which is why the suite stayed green locally). apps/web/test/setup.ts now detects a non-functional storage and backs both names with a working in-memory instance. The deprecated-dependency gate still matched @esbuild-kit/* and glob@10.5.0 in pnpm-lock.yaml: the patches remove those edges from the graph, but the lockfile was never re-derived, so the stale packages and drizzle-kit/archiver-utils snapshot entries were hand-pruned (frozen installs verified). And the testcontainers suite failed five cases it had never once run green: the pg/mysql/mongo/redis suites asserted the OLD v<version>: at-rest envelope while the engine writes the streamed NDBK1:v<version>:<iv> header; the Redis fixture passed an empty passwordEncrypted, which decrypt rightly refused ("Invalid initialization vector") — it now starts the container with --requirepass and encrypts the same secret it stores; and the deploy e2e pulls busybox:1.36 up front with retries, because an auto-pull inside a test assertion turned a rate-limited registry into a red suite.

Docs

• docs/NINEDEPLOY_MANIFEST.md no longer describes machinery that does not exist. §4.14 implied network.aliases reached Docker, §4.15 described a channel "resolver" for notifications that was never written, §4.16 claimed every alert "fires into exactly one named channel" (the channel is encoded in the rule name; delivery follows the global per-channel event filters), and the end-to-end example attributed the deploy trigger to the manifest's own watch.paths rather than the webhook's. Each now says what actually happens and points at §6.3.
• .env.example pointed at a ninedeploy rotate-keys command that did not exist, and omitted that completing the rotation by removing the old key destroys the restorability of every backup taken under it.
• ARCHITECTURE.md rewritten against the actual tree: it had drifted a full release behind (28 tables → 40, migrations 0000–0019 → 0000–0038, "two roles" → workspace RBAC, single-project scoping → N-N project/workspace/label tags, MCP 15 tools → 36, "100% coverage everywhere" → the real tiered gates), and omitted workspaces, invitations, OIDC/SSO, the microkernel, Config Center, firewall, log drains, volume backups, repo insights, the .ninedeploy manifest, preview deployments and panel self-update entirely. A new "Known gaps" section records where the implementation still trails the intent.
• docs/WORKSPACES_RBAC.md now documents what the code enforces, including the instance-operator flag as a separate concept from workspace roles.