- deletion flow emitted job/instance envelopes before the transaction that
closed the job rows; subscribers refetching on the notification saw
status 'running' with no further event. Emits now happen after commit,
including the failure path
- interrupted-deletion sweep now emits job envelopes with the job's real
request_id (it previously stayed silent while the secure-execution sweep
already emitted)
- verified during the sweep: backup restore validation re-derives the
expected schema via migrateDatabase, so migration 18 is automatically
covered; no tables reference audit_events or operation_preparations, so
the retention deletes cannot trip FK constraints
Overview cold start:
- InstanceResourceService persists every live fetch into status_snapshots
(category 'resources') and serves a snapshot younger than 2 minutes
before calling upstream, so the first fleet overview after a restart
costs zero device requests; instance deletion cleans up via FK cascade
Bounded retention for the four fastest-growing tables (15-minute sweep):
- connection_logs: one row per probe per beat had no automatic cleanup
- audit_events: new pruneBefore (90d)
- operation_preparations: every prepare/retry attempt inserted a row,
terminal rows now expire after 7 days
- notification_queue: terminal rows pruned after 30 days
SSE events now cover instance lifecycle:
- create/update emit instance envelopes from the routes; the two-phase
delete emits a job envelope on every terminal transition plus an
instance envelope when the node actually disappears, so fleet and
instance views invalidate in real time
Linux ops:
- 'install.sh unit' writes systemd user units for API and gateway with
Restart=on-failure, 0600 secret injection, and
MULTI_SIMADMIN_SYSTEMD_UNIT wired so console self-update restarts via
systemctl -- closing the self-update loop on Linux
Fleet overview N+1:
- InstanceResourceService gains a 30s TTL cache with single-flight
coalescing; the overview no longer fires six live upstream requests per
device on every render (plus the re-login storm), while the per-device
detail route probes live via force:true
Event journal becomes live:
- job terminal transitions (manual executions and the interrupted sweep)
now append to the journal, so /api/v1/events SSE feeds the frontend's
invalidation controller that was built but never received events
- journal pruning moves off the append hot path (was an unindexable
full-table json_extract scan per insert) onto the retention timer
Console auth hardening:
- scrypt upgraded from N=16384 to N=2^16 (OWASP interactive guidance);
a new password_kdf column records the derivation per row and legacy
hashes rehash transparently on the next successful login without
invalidating sessions (migration 18)
Legacy stack:
- instance URL validation blocks IPv4-compatible IPv6 after WHATWG
canonicalization (::a9fe:a9fe metadata, ::7f00:1 loopback slipped past)
- status polls cool down auto-login for 60s after a failed attempt so a
stale saved password cannot hammer the device into an account lockout
Build hygiene:
- web bundle splits app (410kB) from vendor (212kB) so framework code
stays cacheable across releases; stale root package-lock.json removed
(pnpm is the only lockfile)
Legacy panel:
- upstream body reads now carry their own deadline and a 10 MB byte budget;
a stalled modem can no longer hang /api/status fan-out forever nor OOM
the proxy (request headers alone had the timeout, bodies had none)
- add X-Frame-Options DENY / CSP frame-ancestors none / nosniff; the panel
(delete-instance and confirmed-write dialogs) is no longer clickjackable
- only send a JSON content-type when the API console request has a body, so
payload-less dangerous writes stop failing with 400 and burning the
one-use confirmation token
- register a form-urlencoded parser (the proxy branch was unreachable) and
drop the multipart parser that buffered up to 60 MB before rejecting;
bodyLimit drops to 2 MB; framework-level 415 keeps the stable error body
- remove /api/sms/send from readable paths: GET bypassed the write
confirmation for a send endpoint
- /api/instances/:id/login maps upstream failures to a stable 502 instead
of leaking raw error text
- guard MULTI_SIMADMIN_TIMEOUT_MS parsing (NaN aborted every request);
prune dead code (buildClients, cookie expando no-op)
Control plane:
- deleting a notification channel detaches it from rules instead of leaving
dangling ids that made every referencing rule unreadable and silently
dropped future notifications; rule reads tolerate unknown ids
- startup sweep resets notification_queue rows stranded in 'sending' by a
crash (mirrors the sms outbox sweep); terminal outbox rows are pruned on
the retention timer
- /api/v1/metrics no longer emits operator-assigned node names on the
session-free scrape; login limiter map is bounded and pruned; Secure
cookie honors the gateway-declared x-forwarded-proto
- webhook delivery sets redirect: manual (signed payloads are not replayed)
- SMTP envelope sender is validated against CR/LF smuggling
- scheduled reboots with delaySeconds != 3 fail fast at the dispatcher with
a clear reason instead of burning every retry; contract narrowed to the
pinned baseline
Linux deployments under a systemd unit can now use the online update
panel: MULTI_SIMADMIN_SYSTEMD_UNIT (+ optional _SCOPE=user|system) hands
the restart to systemctl; createPlatformRestartLauncher picks launchd on
macOS and systemd on Linux, degrading to 'unsupported' otherwise.
The production composition hardwired the macOS Keychain (/usr/bin/security),
which made Linux deployment impossible. Now:
- SecretStore gains a stable provider identity persisted in
secret_references.provider; services stop hardcoding 'macos-keychain'
- shared reference codec (secret-reference.ts) understands both
keychain:// and secret-file:// references
- FileSecretStore: single 0600 JSON map under the data root, atomic
temp+rename writes, serialized in-process, same validation envelope
- production composition picks the backend via
MULTI_SIMADMIN_SECRET_BACKEND (default: Keychain on darwin, file store
elsewhere) and readiness probes the matching backend
- cutover readState: enforce 0600 mode bits only on POSIX (Windows ACLs
govern access; chmod is a no-op there)
- backup activation: skip the read-only-handle fsync on win32; the staged
rename-over-open-WAL tests keep running on the POSIX deployment targets
- test-fixtures: normalize fixture paths to POSIX separators before
comparing with manifest entries
- cutover-orchestrator: restart via 'node --import tsx' (matching install.sh)
instead of a hardcoded .pnpm/tsx@4.22.4 loader path that breaks on tsx bumps
- production-gateway: use process.execPath for the legacy-start fixture;
POSIX-only 0600 mode assertion now win32-guarded
- cli-runtime + installer suites: skip where they require POSIX-only
process-group SIGTERM, lsof, or /bin/sh
- fixture-safety: fall back to config.example.json when config.json is absent
on a fresh clone
- eslint: respect the repo's existing _-prefix convention for unused vars
- phase-one-blockers: derive projectRoot via import.meta.dirname so Windows
checkouts stop producing C:\C:\... paths
- backup/release-evidence/cutover-orchestrator: skip POSIX-only directory
fsync on win32 and fsync read-only handles through a writable handle
- Wire identity, metrics and update services into the assembled API.
- Add connection authorization modes and scoped console authorization.
- Preserve notification group IDs for filtered delivery.
- Extend maintenance, backup, resource and module flows with runtime metadata and upstream safety.
- Add console update planning, download validation and staged installation.
- Create a component backup before applying online updates.
- Expose controlled update routes from the production control plane.
- Allow scheduled operations to resolve multiple device groups.
- Add reusable SMS time and random-value macros.
- Add China operator PLMN metadata shared by API and UI.
Deletion now follows the published contract: prepare returns the one-time
token and the console spends it on DELETE /api/v1/instances/:id with
If-Match, where the generic execute endpoint had been rejecting it as an
operation the safe transport cannot replay.
An online device is told to release its own binding first. A refusal keeps
the record and reports UNBIND_FAILED on the job item so the console can say
why the node is still there; a node that never answers a heartbeat is
forgotten locally.
Add central notification channels, rules, queue and delivery logs, fleet
organization groups and tags, device discovery, the device action catalog,
instance module reads, the log centre, connection settings and system
maintenance as native /api/v1 routes backed by the existing secret store,
audit trail and pinned upstream transport.
Drop the redundant advanced table so fleet stays resource-first cards.
Keep multi-select batch service/system restart via prepare→execute, card
restarts, overview system ops, and progressive fleet loading.
chick