Compare commits

..
39 Commits
Author SHA1 Message Date
chick 61a5aee48d fix(events): close review gaps found in the follow-up sweep
- deletion flow emitted job/instance envelopes before the transaction that
  closed the job rows; subscribers refetching on the notification saw
  status 'running' with no further event. Emits now happen after commit,
  including the failure path
- interrupted-deletion sweep now emits job envelopes with the job's real
  request_id (it previously stayed silent while the secure-execution sweep
  already emitted)
- verified during the sweep: backup restore validation re-derives the
  expected schema via migrateDatabase, so migration 18 is automatically
  covered; no tables reference audit_events or operation_preparations, so
  the retention deletes cannot trip FK constraints
2026-09-07 05:57:35 +08:00
chick 70e598208a perf,feat(events),ops: third optimization pass
Overview cold start:
- InstanceResourceService persists every live fetch into status_snapshots
  (category 'resources') and serves a snapshot younger than 2 minutes
  before calling upstream, so the first fleet overview after a restart
  costs zero device requests; instance deletion cleans up via FK cascade

Bounded retention for the four fastest-growing tables (15-minute sweep):
- connection_logs: one row per probe per beat had no automatic cleanup
- audit_events: new pruneBefore (90d)
- operation_preparations: every prepare/retry attempt inserted a row,
  terminal rows now expire after 7 days
- notification_queue: terminal rows pruned after 30 days

SSE events now cover instance lifecycle:
- create/update emit instance envelopes from the routes; the two-phase
  delete emits a job envelope on every terminal transition plus an
  instance envelope when the node actually disappears, so fleet and
  instance views invalidate in real time

Linux ops:
- 'install.sh unit' writes systemd user units for API and gateway with
  Restart=on-failure, 0600 secret injection, and
  MULTI_SIMADMIN_SYSTEMD_UNIT wired so console self-update restarts via
  systemctl -- closing the self-update loop on Linux
2026-09-07 02:43:11 +08:00
chick c1be714ba4 perf,feat(events),hardening: second review pass
Fleet overview N+1:
- InstanceResourceService gains a 30s TTL cache with single-flight
  coalescing; the overview no longer fires six live upstream requests per
  device on every render (plus the re-login storm), while the per-device
  detail route probes live via force:true

Event journal becomes live:
- job terminal transitions (manual executions and the interrupted sweep)
  now append to the journal, so /api/v1/events SSE feeds the frontend's
  invalidation controller that was built but never received events
- journal pruning moves off the append hot path (was an unindexable
  full-table json_extract scan per insert) onto the retention timer

Console auth hardening:
- scrypt upgraded from N=16384 to N=2^16 (OWASP interactive guidance);
  a new password_kdf column records the derivation per row and legacy
  hashes rehash transparently on the next successful login without
  invalidating sessions (migration 18)

Legacy stack:
- instance URL validation blocks IPv4-compatible IPv6 after WHATWG
  canonicalization (::a9fe:a9fe metadata, ::7f00:1 loopback slipped past)
- status polls cool down auto-login for 60s after a failed attempt so a
  stale saved password cannot hammer the device into an account lockout

Build hygiene:
- web bundle splits app (410kB) from vendor (212kB) so framework code
  stays cacheable across releases; stale root package-lock.json removed
  (pnpm is the only lockfile)
2026-09-07 02:20:41 +08:00
chick 29a0eee854 style: apply prettier to the repo's declared format surface 2026-09-07 01:59:24 +08:00
chick 2977f75129 fix: apply security and correctness review findings across both stacks
Legacy panel:
- upstream body reads now carry their own deadline and a 10 MB byte budget;
  a stalled modem can no longer hang /api/status fan-out forever nor OOM
  the proxy (request headers alone had the timeout, bodies had none)
- add X-Frame-Options DENY / CSP frame-ancestors none / nosniff; the panel
  (delete-instance and confirmed-write dialogs) is no longer clickjackable
- only send a JSON content-type when the API console request has a body, so
  payload-less dangerous writes stop failing with 400 and burning the
  one-use confirmation token
- register a form-urlencoded parser (the proxy branch was unreachable) and
  drop the multipart parser that buffered up to 60 MB before rejecting;
  bodyLimit drops to 2 MB; framework-level 415 keeps the stable error body
- remove /api/sms/send from readable paths: GET bypassed the write
  confirmation for a send endpoint
- /api/instances/:id/login maps upstream failures to a stable 502 instead
  of leaking raw error text
- guard MULTI_SIMADMIN_TIMEOUT_MS parsing (NaN aborted every request);
  prune dead code (buildClients, cookie expando no-op)

Control plane:
- deleting a notification channel detaches it from rules instead of leaving
  dangling ids that made every referencing rule unreadable and silently
  dropped future notifications; rule reads tolerate unknown ids
- startup sweep resets notification_queue rows stranded in 'sending' by a
  crash (mirrors the sms outbox sweep); terminal outbox rows are pruned on
  the retention timer
- /api/v1/metrics no longer emits operator-assigned node names on the
  session-free scrape; login limiter map is bounded and pruned; Secure
  cookie honors the gateway-declared x-forwarded-proto
- webhook delivery sets redirect: manual (signed payloads are not replayed)
- SMTP envelope sender is validated against CR/LF smuggling
- scheduled reboots with delaySeconds != 3 fail fast at the dispatcher with
  a clear reason instead of burning every retry; contract narrowed to the
  pinned baseline
2026-09-07 01:57:47 +08:00
chick 236e78327c feat(console-update): systemd restart launcher for Linux supervision
Linux deployments under a systemd unit can now use the online update
panel: MULTI_SIMADMIN_SYSTEMD_UNIT (+ optional _SCOPE=user|system) hands
the restart to systemctl; createPlatformRestartLauncher picks launchd on
macOS and systemd on Linux, degrading to 'unsupported' otherwise.
2026-09-07 01:38:23 +08:00
chick 3c5971e167 feat(installer): support Linux hosts with XDG data root and ss port probe
- platform gate accepts Darwin and Linux; other platforms are refused
- default install root follows each platform's convention
  (~/Library/Application Support vs ~/.local/share)
- LAN IP discovery uses ip -4/hostname -I on Linux, ipconfig on macOS
- port occupancy checks fall back from lsof to ss
- API process gets MULTI_SIMADMIN_SECRET_BACKEND pinned per platform
- README documents the per-platform secret storage
2026-09-07 01:36:04 +08:00
chick e4c4bce75b feat(secrets): add a 0600 file secret backend so production runs on Linux
The production composition hardwired the macOS Keychain (/usr/bin/security),
which made Linux deployment impossible. Now:

- SecretStore gains a stable provider identity persisted in
  secret_references.provider; services stop hardcoding 'macos-keychain'
- shared reference codec (secret-reference.ts) understands both
  keychain:// and secret-file:// references
- FileSecretStore: single 0600 JSON map under the data root, atomic
  temp+rename writes, serialized in-process, same validation envelope
- production composition picks the backend via
  MULTI_SIMADMIN_SECRET_BACKEND (default: Keychain on darwin, file store
  elsewhere) and readiness probes the matching backend
2026-09-07 01:32:07 +08:00
chick c57dc81e42 fix: finish cross-platform test parity while keeping POSIX deployment semantics
- cutover readState: enforce 0600 mode bits only on POSIX (Windows ACLs
  govern access; chmod is a no-op there)
- backup activation: skip the read-only-handle fsync on win32; the staged
  rename-over-open-WAL tests keep running on the POSIX deployment targets
- test-fixtures: normalize fixture paths to POSIX separators before
  comparing with manifest entries
2026-09-07 01:18:44 +08:00
chick 96dd9e2ba6 chore: pin LF line endings via .gitattributes
Windows checkouts previously materialized CRLF, breaking every pinned
sha256 snapshot check in operation-registry and test-fixtures.
2026-09-07 01:10:19 +08:00
chick 4f42a6f835 fix: make cutover restart argv upgrade-safe and tests platform-honest
- cutover-orchestrator: restart via 'node --import tsx' (matching install.sh)
  instead of a hardcoded .pnpm/tsx@4.22.4 loader path that breaks on tsx bumps
- production-gateway: use process.execPath for the legacy-start fixture;
  POSIX-only 0600 mode assertion now win32-guarded
- cli-runtime + installer suites: skip where they require POSIX-only
  process-group SIGTERM, lsof, or /bin/sh
- fixture-safety: fall back to config.example.json when config.json is absent
  on a fresh clone
2026-09-07 01:09:49 +08:00
chick eed134a25c fix: honor underscore-unused convention and make durability fsyncs cross-platform
- eslint: respect the repo's existing _-prefix convention for unused vars
- phase-one-blockers: derive projectRoot via import.meta.dirname so Windows
  checkouts stop producing C:\C:\... paths
- backup/release-evidence/cutover-orchestrator: skip POSIX-only directory
  fsync on win32 and fsync read-only handles through a writable handle
2026-09-07 01:07:16 +08:00
chick f6319374cd feat(web): add console update panel, system maintenance path and connection settings UI
- Add online update workflow with plan, backup and install states.

- Expose maintenance paths, component backup jobs and connection authorization modes.

- Align auth and automation screens with the fused control plane.
2026-09-07 00:45:44 +08:00
chick 6d251dab7b feat(web): fuse Hub device modules into instance panels with SMS aggregation
- Consolidate cellular, eSIM, network and device module interactions.

- Add aggregate SMS workflows and identity-aware action controls.

- Extend SMSC privacy handling and module action icons.
2026-09-07 00:45:38 +08:00
chick 864e1bd90c feat(web): rebuild fleet page with Hub-style identity guard and context actions
- Surface device identity state, carrier details and upstream link health.

- Add fleet-level identity monitoring and restart/baseband context actions.

- Apply consistent notification scoping across fleet views.
2026-09-07 00:45:28 +08:00
chick 635527dd26 feat(api): integrate identity, notification scope and connection settings into control plane
- Wire identity, metrics and update services into the assembled API.

- Add connection authorization modes and scoped console authorization.

- Preserve notification group IDs for filtered delivery.

- Extend maintenance, backup, resource and module flows with runtime metadata and upstream safety.
2026-09-07 00:45:18 +08:00
chick 7168e9dba2 feat(api): add online console update with pre-install backup
- Add console update planning, download validation and staged installation.

- Create a component backup before applying online updates.

- Expose controlled update routes from the production control plane.
2026-09-07 00:44:58 +08:00
chick 961eb928c4 feat(api): add device identity guard and Prometheus metrics
- Persist device identity observations and expose fleet identity summaries.

- Block device control actions when identity verification fails.

- Add metrics collection and a Prometheus scrape endpoint.
2026-09-07 00:44:51 +08:00
chick 8406f14469 feat(contracts,automation): add multi-group scopes, SMS macros and operator registry
- Allow scheduled operations to resolve multiple device groups.

- Add reusable SMS time and random-value macros.

- Add China operator PLMN metadata shared by API and UI.
2026-09-07 00:44:33 +08:00
chick 8b7df8cb65 fix(instances): release the hub binding before forgetting a node
Deletion now follows the published contract: prepare returns the one-time
token and the console spends it on DELETE /api/v1/instances/:id with
If-Match, where the generic execute endpoint had been rejecting it as an
operation the safe transport cannot replay.

An online device is told to release its own binding first. A refusal keeps
the record and reports UNBIND_FAILED on the job item so the console can say
why the node is still there; a node that never answers a heartbeat is
forgotten locally.
2026-09-05 19:52:27 +08:00
chick 5016586922 test: give process-spawning suites room and pin the fused behaviour
The runtime executables bind fixed ports, so the production command check now
skips instead of failing against a live deployment, and the canary check takes
a reserved free port. Raise the default test timeout for the SQLite and browser
suites that were only slow under full-suite parallelism.
2026-09-05 18:53:05 +08:00
chick a75badeb58 feat(web): rebuild the console around the fused control plane
Render the Hub feature set in the existing Animal Island styling: fleet
messages, notification centre, log centre, organisation panel, per-instance
device module panels and the settings backup, connection and maintenance
surfaces.
2026-09-05 18:53:04 +08:00
chick f9186bd851 feat(api): absorb the Hub control plane into the local instance model
Add central notification channels, rules, queue and delivery logs, fleet
organization groups and tags, device discovery, the device action catalog,
instance module reads, the log centre, connection settings and system
maintenance as native /api/v1 routes backed by the existing secret store,
audit trail and pinned upstream transport.
2026-09-05 18:53:04 +08:00
chick f11877f13e feat(contracts): share notification, organization and device capability models
Move the Hub channel table, organization tags and device capability definitions
into the workspace packages so the control plane and the console validate the
same contract, and refresh the frozen upstream evidence for the new paths.
2026-09-05 18:52:54 +08:00
chick 03fa2e6c7b chore: keep local runtime data out of version control 2026-09-05 18:52:53 +08:00
Codex 9bc0f27f12 fix: canonicalize browser navigation after commit 2026-07-30 22:35:37 +08:00
Codex 95abd09c39 style: format navigation tests 2026-07-30 22:21:29 +08:00
Codex 4fbc902325 test: cover persistent browser navigation 2026-07-30 22:13:05 +08:00
Codex 25502a52bb fix: flush proxied event stream headers 2026-07-30 22:11:48 +08:00
Codex 144e17c220 feat: navigate workspaces without page reloads 2026-07-30 22:10:59 +08:00
Codex 3a6652be0f fix: simplify top-level settings 2026-07-30 22:08:34 +08:00
Codex af5abb4b82 docs: plan settings navigation performance fix 2026-07-30 22:06:28 +08:00
Codex 3bbd227c58 docs: define settings navigation performance fix 2026-07-30 22:01:31 +08:00
Codex d60cff1dd8 feat(web): protect phone data and polish release UI 2026-07-30 20:16:50 +08:00
Codex 6ec8e4f6a6 docs: plan phone privacy release polish 2026-07-30 19:52:37 +08:00
Codex cf2cf6827d docs: define phone privacy and release polish 2026-07-30 19:49:17 +08:00
Codex 570edf6bb2 feat: rebuild warm operations workbench 2026-07-30 14:49:33 +08:00
Codex 25c853dea8 docs: define compact fleet cards 2026-07-30 14:25:34 +08:00
Codex f57ec1f48d docs: define warm operations workbench 2026-07-30 00:53:59 +08:00
314 changed files with 80304 additions and 18462 deletions
+1
View File
@@ -0,0 +1 @@
* text=auto eol=lf
+3
View File
@@ -7,3 +7,6 @@ npm-debug.log*
coverage/
apps/web/dist/
.hermes/
.local-run/
artifacts/
data/
+47 -5
View File
@@ -11,9 +11,10 @@
- 支持无密码实例状态读取;有密码实例可通过本地服务代登录并保持会话。
- 统一代理 SimAdmin API,内置 API 工作台,便于读取/调试设备、SIM、网络、短信、eSIM、OTA 等接口。
- 保留原始 SimAdmin 页面 iframe 嵌入;若目标站禁止 iframe,可一键在原站打开。
- 融合 [SimAdminHub](https://github.com/3899/SimAdminHub) 的管理思路:跨节点短信、通知、自动化与跨域健康自然融入 Fleet 节点总览体系,无需部署或接入外部 Hub。
- 敏感配置留在本地 `config.json`,仓库只提交 `config.example.json`
## 一键安装(macOS
## 一键安装(macOS / Linux
```bash
curl -fsSL https://gitea.chickliu.fun/Hermes/multi-simadmin/raw/branch/main/scripts/install.sh | sh
@@ -27,7 +28,7 @@ curl -fsSL https://gitea.chickliu.fun/Hermes/multi-simadmin/raw/branch/main/scri
安装完成后按终端输出访问 `http://<本机局域网 IP>:8788/fleet`。首次密码可在当前管理台直接设置。Gateway 默认监听所有网络接口,因此必须确保主机仅接入可信内网或已通过防火墙限制 8788 的来源;HTTP 部署不得直接暴露到公网,公网开放必须由前置代理提供 HTTPS 和访问控制。
> 当前生产秘密存储使用 macOS Keychain因此一键安装脚本暂只支持 macOS。脚本不会覆盖已有源码目录或数据库,也不会占用已被其他进程监听的 8788/8790 端口。
> 秘密存储按平台自动选择:macOS 使用 KeychainLinux 使用数据目录下 0600 权限的 `secrets.json` 文件;也可用 `MULTI_SIMADMIN_SECRET_BACKEND=macos-keychain|secret-file` 显式指定。脚本不会覆盖已有源码目录或数据库,也不会占用已被其他进程监听的 8788/8790 端口。
### 服务管理
@@ -42,7 +43,46 @@ sh /tmp/multi-simadmin-install.sh start
sh /tmp/multi-simadmin-install.sh uninstall
```
`uninstall` 默认只移除程序源码,保留数据库、Gateway token 和日志。默认安装位置为 `~/Library/Application Support/multi-simadmin`,可通过 `MULTI_SIMADMIN_HOME` 覆盖;额外 LAN Host 可通过 `MULTI_SIMADMIN_ALLOWED_HOSTS`(逗号分隔)配置。
`uninstall` 默认只移除程序源码,保留数据库、Gateway token 和日志。默认安装位置 macOS `~/Library/Application Support/multi-simadmin`Linux 为 `~/.local/share/multi-simadmin`(遵循 XDG),均可通过 `MULTI_SIMADMIN_HOME` 覆盖;额外 LAN Host 可通过 `MULTI_SIMADMIN_ALLOWED_HOSTS`(逗号分隔)配置。
#### Linux systemd(可选,但推荐)
`sh install.sh unit` 会生成 `~/.config/systemd/user/` 下的 API 与 Gateway 用户单元(`Restart=on-failure`、开机自启、密钥经 0600 的 `service-env` 注入)。随后:
```bash
systemctl --user enable --now multi-simadmin-api.service multi-simadmin-gateway.service
```
单元内声明了 `MULTI_SIMADMIN_SYSTEMD_UNIT`,因此 Web 控制台的“在线更新”在 Linux 上会把重启交给 systemd 完成,形成自更新闭环。由 systemd 托管后请用 `systemctl` 管理进程,不要再用本脚本 `start/stop`,避免双进程争抢端口。
### Fleet 节点总览与跨域健康
`/fleet` 读取本机 Control Plane 中的节点状态、资源快照与设备号码,形成节点总览。侧栏在实例总数 / 在线 / 需处理之外,还聚合跨域健康指标:
- 短信异常:标记哪些节点的短信接口不可用,点击直达 `/fleet/messages`
- 通知异常:汇总跨节点通知队列的 pending + failed 数量,点击直达 `/fleet/notifications`
节点卡片直接进入对应实例详情,查看 CPU、内存、温度、SIM 号码、平台与版本。
### 跨节点短信中心
`/fleet/messages` 融合官方 SimAdminHub 的短信中心形态:它通过本地 Control Plane 按节点读取 SimAdmin 的 `/api/sms/list`,再跨设备聚合搜索、展示与发送。
- 跨设备记录:汇总各节点最新短信,支持按号码、内容或节点名称搜索,并分页加载。
- 会话视图:按号码分组为会话,节点栏 / 会话栏 / 消息栏三栏工作流,默认打开最新会话。
- 发送短信:从可用节点下拉选择目标设备,提交到 `/api/v1/fleet/messages/send` 后由 Control Plane 转发到对应 SimAdmin。
- 批量删除:可在短信记录中勾选选择并确认删除,Control Plane 通过严格白名单的 `/api/sms/batch-delete` 按节点批量清理,未成功项会以逐条失败原因返回。
- 安全边界:短信正文与号码只在本地页面展示;批量删除请求最多 500 条,且不接收外部 Hub 地址、Webhook 或任何可注入字段。
### 跨节点通知中心
`/fleet/notifications` 融合官方 SimAdminHub 的通知中心能力:它按本地节点聚合阅读 SimAdmin 的通知接口,再跨设备汇总转发通道、转发规则、转发日志与失败重试队列。
- 设备可用性:自动登录后读取 `/api/notifications/config``/api/notifications/logs``/api/notifications/queue`,无法读取的实例单独标记。
- 转发通道:统计通道总数、启用数以及常见的 webhook、Bark、PushPlus、企业微信、钉钉、飞书、Telegram、邮件等类型。
- 转发日志:展示成功、失败、免打扰、未匹配、无可用通道与最近记录。
- 失败重试:展示待发送、定时、重试中、发送中和失败的队列摘要,并支持整队列重试、单条重试或删除积压项。
- 安全边界:只返回经过白名单清洗的计数与最近条目,不暴露通知原文、webhook/token/secret 或模板配置。
## 旧版开发入口
@@ -50,10 +90,12 @@ sh /tmp/multi-simadmin-install.sh uninstall
```bash
cp config.example.json config.json
npm install
npm start
corepack pnpm install
corepack pnpm start
```
> 仓库统一使用 pnpm`pnpm-lock.yaml`)作为唯一 lockfile;陈旧的 `package-lock.json` 已移除。
## 配置
编辑 `config.json`
+1
View File
@@ -16,6 +16,7 @@
"@multi-simadmin/contracts": "workspace:*",
"@multi-simadmin/operation-registry": "workspace:*",
"better-sqlite3": "12.11.1",
"cron-parser": "^5.6.2",
"drizzle-orm": "0.45.2",
"fastify": "5.10.0",
"tsx": "4.22.4"
@@ -1,6 +1,7 @@
import type Database from 'better-sqlite3';
import {
AUDIT_OUTCOMES,
type AuditOutcome,
type AuditEvent,
type AuditPage,
type AuditPageQuery,
@@ -19,6 +20,26 @@ export class AuditQueryError extends Error {
}
}
export interface ControlPlaneAuditSummary {
readonly total: number;
readonly succeeded: number;
readonly failed: number;
readonly partiallySucceeded: number;
readonly denied: number;
readonly recent: readonly ControlPlaneAuditSummaryEvent[];
}
export interface ControlPlaneAuditSummaryEvent {
readonly id: string;
readonly occurredAt: string;
readonly actorId: string;
readonly action: string;
readonly outcome: AuditEvent['outcome'];
readonly requestId: string;
readonly instanceId?: string;
readonly jobId?: string;
}
interface AuditRow {
id: unknown;
instance_id: unknown;
@@ -118,6 +139,74 @@ function parameterSummary(value: unknown): readonly RedactedParameterSummaryItem
export class AuditQueryService {
constructor(private readonly db: Database.Database) {}
/** Retention hook: drops audit records older than the cutoff. */
pruneBefore(cutoffIso: string): number {
if (typeof cutoffIso !== 'string' || !Number.isFinite(Date.parse(cutoffIso)))
throw new RangeError('cutoffIso must be a valid date-time');
return Number(
this.db
.prepare('DELETE FROM audit_events WHERE created_at < ?')
.run(new Date(cutoffIso).toISOString()).changes,
);
}
summary(limit = 5): ControlPlaneAuditSummary {
if (!Number.isSafeInteger(limit) || limit < 1 || limit > 20) validation('limit is invalid');
const totalRow = this.db.prepare('SELECT COUNT(*) AS total FROM audit_events').get() as
| {
total: unknown;
}
| undefined;
if (
!totalRow ||
typeof totalRow.total !== 'number' ||
!Number.isSafeInteger(totalRow.total) ||
totalRow.total < 0
)
fail('Persisted audit count is invalid');
const outcomeRows = this.db
.prepare('SELECT result_code, COUNT(*) AS count FROM audit_events GROUP BY result_code')
.all() as Array<{ result_code: unknown; count: unknown }>;
const counts = new Map<string, number>();
for (const row of outcomeRows) {
const outcome = boundedString(row.result_code, 'result_code');
if (!(AUDIT_OUTCOMES as readonly string[]).includes(outcome))
fail('Persisted audit result_code is unrepresentable');
if (!Number.isSafeInteger(row.count) || (row.count as number) < 0)
fail('Persisted audit outcome count is invalid');
counts.set(outcome, row.count as number);
}
const count = (outcome: AuditOutcome): number => counts.get(outcome) ?? 0;
const recentRows = this.db
.prepare(`SELECT ${COLUMNS} FROM audit_events ORDER BY created_at DESC, id ASC LIMIT ?`)
.all(limit) as AuditRow[];
const recent = recentRows.map((row) => {
const outcome = boundedString(row.result_code, 'result_code');
if (!(AUDIT_OUTCOMES as readonly string[]).includes(outcome))
fail('Persisted audit result_code is unrepresentable');
const instanceId = optionalString(row.instance_id, 'instance_id');
const jobId = optionalString(row.job_id, 'job_id');
return freeze({
id: boundedString(row.id, 'id'),
occurredAt: canonicalTimestamp(row.created_at, 'created_at'),
actorId: boundedString(row.actor, 'actor'),
action: boundedString(row.operation_id, 'operation_id'),
outcome: outcome as ControlPlaneAuditSummaryEvent['outcome'],
requestId: boundedString(row.request_id, 'request_id'),
...(instanceId === undefined ? {} : { instanceId }),
...(jobId === undefined ? {} : { jobId }),
});
});
return freeze({
total: totalRow.total,
succeeded: count('succeeded'),
failed: count('failed'),
partiallySucceeded: count('partially-succeeded'),
denied: count('denied'),
recent: freeze(recent),
});
}
get(id: string): AuditEvent {
inputString(id, 'eventId');
const row = this.db.prepare(`SELECT ${COLUMNS} FROM audit_events WHERE id = ?`).get(id) as
@@ -6,18 +6,43 @@ const SESSION_TTL_MS = 7 * 24 * 60 * 60 * 1000;
const MIN_PASSWORD_LENGTH = 8;
const MAX_PASSWORD_BYTES = 1024;
const SCRYPT_KEY_LENGTH = 32;
// Legacy derivations used N=16384; OWASP guidance for interactive login sits at
// N=2^16/r=8. Rows keep their derivation tag so verification stays correct and
// the upgrade happens transparently on the next successful login.
interface ScryptParams {
readonly N: number;
readonly r: number;
readonly p: number;
}
const SCRYPT_LEGACY: ScryptParams = { N: 16_384, r: 8, p: 1 };
const SCRYPT_CURRENT: ScryptParams = { N: 65_536, r: 8, p: 1 };
const SCRYPT_MAXMEM = 256 * 1024 * 1024;
const scryptTag = (params: ScryptParams): string => `scrypt:${params.N}:${params.r}:${params.p}`;
const parseScryptTag = (value: string | null | undefined): ScryptParams => {
if (!value) return SCRYPT_LEGACY;
const match = /^scrypt:(\d+):(\d+):(\d+)$/.exec(value);
if (!match) return SCRYPT_LEGACY;
const [N, r, p] = match.slice(1).map(Number) as [number, number, number];
return { N, r, p };
};
const IDLE_TIMEOUT_SETTING_KEY = 'console-auth.idle-timeout-minutes';
const DEFAULT_IDLE_TIMEOUT_MINUTES = 30;
const MIN_IDLE_TIMEOUT_MINUTES = 5;
const MAX_IDLE_TIMEOUT_MINUTES = 1440;
interface AuthConfigRow {
readonly protection_enabled: 0 | 1;
readonly password_salt: string | null;
readonly password_hash: string | null;
readonly password_revision: number;
readonly password_kdf: string | null;
}
export interface ConsoleAuthStatus {
readonly configured: boolean;
readonly protectionEnabled: boolean;
readonly authenticated: boolean;
readonly sessionIdleTimeoutMinutes: number;
}
export class ConsoleAuthError extends Error {
@@ -40,6 +65,7 @@ export interface ConsoleAuthServiceOptions {
export class ConsoleAuthService {
private readonly db: Database.Database;
private readonly now: () => Date;
#idleTimeoutMinutes: number = DEFAULT_IDLE_TIMEOUT_MINUTES;
constructor(options: ConsoleAuthServiceOptions) {
this.db = options.db;
@@ -49,17 +75,19 @@ export class ConsoleAuthService {
status(sessionToken?: string): ConsoleAuthStatus {
this.pruneExpired();
const config = this.config();
const sessionIdleTimeoutMinutes = this.idleTimeoutMinutes();
const configured = config?.password_hash !== null && config?.password_hash !== undefined;
const protectionEnabled = config?.protection_enabled === 1;
return {
configured,
protectionEnabled,
authenticated: !protectionEnabled || this.isAuthenticated(sessionToken, config),
};
const authenticated = !protectionEnabled || this.isAuthenticated(sessionToken, config);
return { configured, protectionEnabled, authenticated, sessionIdleTimeoutMinutes };
}
isProtectedPath(pathname: string): boolean {
if (!pathname.startsWith('/api/v1/')) return false;
// Monitoring scrapers cannot hold a browser session. Like the Hub's /health and /metrics,
// this one stays session-free and is meant for the trusted network the console already
// assumes; it carries no device addresses, only opaque node ids and counters.
if (pathname === '/api/v1/metrics') return false;
return !pathname.startsWith('/api/v1/auth/');
}
@@ -67,6 +95,7 @@ export class ConsoleAuthService {
const status = this.status(sessionToken);
if (status.protectionEnabled && !status.authenticated)
throw new ConsoleAuthError('AUTH_REQUIRED');
this.touchSession(sessionToken);
}
async login(password: string): Promise<string> {
@@ -78,19 +107,37 @@ export class ConsoleAuthService {
!config.password_hash
)
throw new ConsoleAuthError('INVALID_CREDENTIALS');
const actual = await this.derive(password, config.password_salt);
const params = parseScryptTag(config.password_kdf);
const actual = await this.derive(password, config.password_salt, params);
const expected = Buffer.from(config.password_hash, 'hex');
if (actual.length !== expected.length || !timingSafeEqual(actual, expected))
throw new ConsoleAuthError('INVALID_CREDENTIALS');
if (params !== SCRYPT_CURRENT) {
// Transparent upgrade: same salt, stronger derivation, same revision so
// live sessions survive.
const hash = (await this.derive(password, config.password_salt, SCRYPT_CURRENT)).toString(
'hex',
);
this.db
.prepare(
'UPDATE console_auth_config SET password_hash=?,password_kdf=?,updated_at=? WHERE singleton=1',
)
.run(hash, scryptTag(SCRYPT_CURRENT), this.now().toISOString());
}
return this.createSession(config.password_revision);
}
async updateSettings(
input: { readonly enabled: boolean; readonly newPassword?: string },
input: {
readonly enabled: boolean;
readonly newPassword?: string;
readonly idleTimeoutMinutes?: number;
},
sessionToken?: string,
): Promise<{ readonly status: ConsoleAuthStatus; readonly sessionToken?: string }> {
const current = this.config();
if (current?.protection_enabled === 1) this.requireSession(sessionToken);
if (input.idleTimeoutMinutes !== undefined) this.saveIdleTimeout(input.idleTimeoutMinutes);
if (input.enabled) {
if (input.newPassword !== undefined) {
@@ -107,14 +154,14 @@ export class ConsoleAuthService {
this.db
.prepare(
`INSERT INTO console_auth_config
(singleton,protection_enabled,password_salt,password_hash,password_revision,updated_at)
VALUES (1,1,?,?,?,?)
(singleton,protection_enabled,password_salt,password_hash,password_revision,password_kdf,updated_at)
VALUES (1,1,?,?,?,?,?)
ON CONFLICT(singleton) DO UPDATE SET
protection_enabled=1,password_salt=excluded.password_salt,
password_hash=excluded.password_hash,password_revision=excluded.password_revision,
updated_at=excluded.updated_at`,
password_kdf=excluded.password_kdf,updated_at=excluded.updated_at`,
)
.run(salt, hash, revision, this.now().toISOString());
.run(salt, hash, revision, scryptTag(SCRYPT_CURRENT), this.now().toISOString());
this.db.prepare('DELETE FROM console_auth_sessions').run();
return this.createSession(revision);
})();
@@ -164,12 +211,37 @@ export class ConsoleAuthService {
private config(): AuthConfigRow | undefined {
return this.db
.prepare(
`SELECT protection_enabled,password_salt,password_hash,password_revision
`SELECT protection_enabled,password_salt,password_hash,password_revision,password_kdf
FROM console_auth_config WHERE singleton=1`,
)
.get() as AuthConfigRow | undefined;
}
private idleTimeoutMinutes(): number {
const row = this.db
.prepare('SELECT value_json FROM app_settings WHERE key = ?')
.get(IDLE_TIMEOUT_SETTING_KEY) as { readonly value_json: unknown } | undefined;
const value = row?.value_json;
return typeof value === 'number' ? this.normalizeIdleTimeout(value) : this.#idleTimeoutMinutes;
}
private saveIdleTimeout(value: number): void {
const minutes = this.normalizeIdleTimeout(value);
const now = this.now().toISOString();
this.db
.prepare(
`INSERT INTO app_settings (key,value_json,created_at,updated_at)
VALUES (?,?,?,?)
ON CONFLICT(key) DO UPDATE SET value_json=excluded.value_json,updated_at=excluded.updated_at`,
)
.run(IDLE_TIMEOUT_SETTING_KEY, Math.round(minutes), now, now);
this.#idleTimeoutMinutes = minutes;
}
private normalizeIdleTimeout(value: number): number {
return Math.min(MAX_IDLE_TIMEOUT_MINUTES, Math.max(MIN_IDLE_TIMEOUT_MINUTES, value));
}
private validatePassword(password: string): void {
const bytes = Buffer.byteLength(password, 'utf8');
if (
@@ -182,13 +254,17 @@ export class ConsoleAuthService {
throw new ConsoleAuthError('PASSWORD_POLICY_FAILED');
}
private derive(password: string, salt: string): Promise<Buffer> {
private derive(
password: string,
salt: string,
params: ScryptParams = SCRYPT_CURRENT,
): Promise<Buffer> {
return new Promise((resolvePromise, reject) => {
deriveScrypt(
password,
Buffer.from(salt, 'hex'),
SCRYPT_KEY_LENGTH,
{ N: 16_384, r: 8, p: 1, maxmem: 64 * 1024 * 1024 },
{ ...params, maxmem: SCRYPT_MAXMEM },
(error, derivedKey) => {
if (error) reject(error);
else resolvePromise(derivedKey);
@@ -204,7 +280,12 @@ export class ConsoleAuthService {
private createSession(passwordRevision: number): string {
const token = randomBytes(32).toString('base64url');
const now = this.now();
const expires = new Date(now.getTime() + SESSION_TTL_MS);
const expires = new Date(
Math.min(
now.getTime() + this.idleTimeoutMinutes() * 60 * 1000,
now.getTime() + SESSION_TTL_MS,
),
);
this.db
.prepare(
`INSERT INTO console_auth_sessions
@@ -230,4 +311,29 @@ export class ConsoleAuthService {
.prepare('DELETE FROM console_auth_sessions WHERE expires_at<=?')
.run(this.now().toISOString());
}
touchSession(token: string | undefined): void {
if (!token) return;
const config = this.config();
if (!config || config.password_revision <= 0) return;
const row = this.db
.prepare('SELECT created_at FROM console_auth_sessions WHERE session_hash=? AND expires_at>?')
.get(this.digest(token), this.now().toISOString()) as
| { readonly created_at: string }
| undefined;
if (!row) return;
const timeoutMs = this.idleTimeoutMinutes() * 60 * 1000;
const now = this.now();
this.db
.prepare(
'UPDATE console_auth_sessions SET expires_at=? WHERE session_hash=? AND expires_at>?',
)
.run(
new Date(
Math.min(now.getTime() + timeoutMs, new Date(row.created_at).getTime() + SESSION_TTL_MS),
).toISOString(),
this.digest(token),
this.now().toISOString(),
);
}
}
@@ -0,0 +1,118 @@
import { describe, expect, it } from 'vitest';
import {
nextOccurrence,
nextOccurrenceFor,
previewCron,
previewTrigger,
reconcileOccurrence,
} from './schedule-time.js';
describe('schedule time', () => {
it('calculates future occurrences in Beijing time across the UTC day boundary', () => {
expect(previewCron('0 9 * * *', 2, new Date('2026-07-30T00:30:00.000Z'))).toEqual([
'2026-07-30T01:00:00.000Z',
'2026-07-31T01:00:00.000Z',
]);
expect(nextOccurrence('30 0 * * *', new Date('2026-07-30T15:59:00.000Z'))).toBe(
'2026-07-30T16:30:00.000Z',
);
});
it('rejects seconds fields and invalid cron grammar', () => {
expect(() => previewCron('* * * * * *', 1)).toThrow(/five-field/i);
expect(() => previewCron('99 99 * * *', 1)).toThrow(/cron/i);
});
it('reconciles a missed due time according to the per-task policy', () => {
const common = {
cronExpression: '*/10 * * * *',
nextDueAt: '2026-07-30T01:00:00.000Z',
};
expect(
reconcileOccurrence(
{ ...common, misfirePolicy: 'skip' },
new Date('2026-07-30T01:25:00.000Z'),
),
).toEqual({
action: 'skip',
dueAt: '2026-07-30T01:20:00.000Z',
nextDueAt: '2026-07-30T01:30:00.000Z',
});
expect(
reconcileOccurrence(
{ ...common, misfirePolicy: 'catch-up-once' },
new Date('2026-07-30T01:25:00.000Z'),
),
).toEqual({
action: 'run',
dueAt: '2026-07-30T01:20:00.000Z',
nextDueAt: '2026-07-30T01:30:00.000Z',
});
});
it('expands Hub fixed schedules over the Beijing weekday and time grid', () => {
const weekdays = { kind: 'fixed' as const, weekdays: [4, 5], times: ['09:00', '21:30'] };
// 2026-07-30 is a Thursday; 01:25Z is 09:25 in Beijing, so the same-day 21:30 slot is next.
expect(previewTrigger(weekdays, 2, new Date('2026-07-30T01:25:00.000Z'))).toEqual([
'2026-07-30T13:30:00.000Z',
'2026-07-31T01:00:00.000Z',
]);
expect(
nextOccurrenceFor(
{ kind: 'fixed', weekdays: [6], times: ['00:30'] },
new Date('2026-07-30T01:25:00.000Z'),
),
).toBe('2026-07-31T16:30:00.000Z');
});
it('keeps interval schedules anchored on the previous occurrence', () => {
const interval = { kind: 'interval' as const, value: 90, unit: 'mins' as const };
expect(nextOccurrenceFor(interval, new Date('2026-07-30T01:00:00.000Z'))).toBe(
'2026-07-30T02:30:00.000Z',
);
expect(
nextOccurrenceFor(
interval,
new Date('2026-07-30T05:00:00.000Z'),
new Date('2026-07-30T01:00:00.000Z'),
),
).toBe('2026-07-30T05:30:00.000Z');
expect(previewTrigger(interval, 3, new Date('2026-07-30T01:00:00.000Z'))).toEqual([
'2026-07-30T02:30:00.000Z',
'2026-07-30T04:00:00.000Z',
'2026-07-30T05:30:00.000Z',
]);
});
it('reconciles fixed and interval misfires against the stored due time', () => {
expect(
reconcileOccurrence(
{
trigger: { kind: 'interval', value: 90, unit: 'mins' },
nextDueAt: '2026-07-30T04:00:00.000Z',
misfirePolicy: 'catch-up-once',
},
new Date('2026-07-30T05:00:00.000Z'),
),
).toEqual({
action: 'run',
dueAt: '2026-07-30T04:00:00.000Z',
nextDueAt: '2026-07-30T05:30:00.000Z',
});
expect(
reconcileOccurrence(
{
trigger: { kind: 'fixed', weekdays: [4, 5], times: ['09:00', '21:30'] },
nextDueAt: '2026-07-30T01:00:00.000Z',
misfirePolicy: 'skip',
},
new Date('2026-07-30T01:25:00.000Z'),
),
).toEqual({
action: 'skip',
dueAt: '2026-07-30T01:00:00.000Z',
nextDueAt: '2026-07-30T13:30:00.000Z',
});
});
});
@@ -0,0 +1,171 @@
import type {
ScheduleMisfirePolicy,
ScheduleTrigger,
ScheduledTask,
} from '@multi-simadmin/contracts';
import { intervalMilliseconds } from '@multi-simadmin/contracts';
import { CronExpressionParser } from 'cron-parser';
const TIMEZONE = 'Asia/Shanghai';
/** Asia/Shanghai has observed UTC+8 without DST since 1991, so fixed-offset arithmetic is exact. */
const ZONE_OFFSET_MS = 8 * 60 * 60 * 1_000;
function validateExpression(expression: string): string {
const clean = expression.trim().replace(/\s+/g, ' ');
if (clean.split(' ').length !== 5)
throw new TypeError('Cron must use standard five-field syntax');
return clean;
}
function parser(expression: string, currentDate: Date) {
try {
return CronExpressionParser.parse(validateExpression(expression), {
currentDate,
tz: TIMEZONE,
});
} catch (error) {
if (error instanceof TypeError && /five-field/.test(error.message)) throw error;
throw new TypeError('Cron expression is invalid');
}
}
export function previewCron(
expression: string,
count: number,
from = new Date(),
): readonly string[] {
if (!Number.isSafeInteger(count) || count < 1 || count > 10)
throw new TypeError('Cron preview count must be between 1 and 10');
const interval = parser(expression, from);
return Array.from({ length: count }, () => interval.next().toDate().toISOString());
}
function shanghaiParts(instant: Date) {
const shifted = new Date(instant.getTime() + ZONE_OFFSET_MS);
const weekday = shifted.getUTCDay();
return {
year: shifted.getUTCFullYear(),
month: shifted.getUTCMonth(),
date: shifted.getUTCDate(),
hour: shifted.getUTCHours(),
minute: shifted.getUTCMinutes(),
// ISO weekday: Monday is 1 and Sunday is 7, matching the Hub weekday vocabulary.
weekday: weekday === 0 ? 7 : weekday,
};
}
function shanghaiInstant(year: number, month: number, date: number, hour: number, minute: number) {
return new Date(Date.UTC(year, month, date, hour, minute) - ZONE_OFFSET_MS);
}
/**
* Minutes since local midnight in Asia/Shanghai. Shared with the notification engine so quiet
* hours and scheduled windows read from one clock.
*/
export function shanghaiMinuteOfDay(instant: Date): number {
const parts = shanghaiParts(instant);
return parts.hour * 60 + parts.minute;
}
function fixedOccurrenceAfter(trigger: Extract<ScheduleTrigger, { kind: 'fixed' }>, after: Date) {
const start = shanghaiParts(after);
for (let offset = 0; offset <= 8; offset += 1) {
const day = shanghaiInstant(start.year, start.month, start.date + offset, 0, 0);
const parts = shanghaiParts(day);
if (!trigger.weekdays.includes(parts.weekday)) continue;
for (const time of trigger.times) {
const [hour, minute] = time.split(':');
const candidate = shanghaiInstant(
parts.year,
parts.month,
parts.date,
Number(hour),
Number(minute),
);
if (candidate.getTime() > after.getTime()) return candidate;
}
}
return undefined;
}
function intervalOccurrenceAfter(
trigger: Extract<ScheduleTrigger, { kind: 'interval' }>,
after: Date,
anchor?: Date,
) {
const period = intervalMilliseconds(trigger);
if (anchor && anchor.getTime() < after.getTime()) {
// Keep the period anchored on the previous occurrence instead of drifting with the clock.
const steps = Math.floor((after.getTime() - anchor.getTime()) / period) + 1;
return new Date(anchor.getTime() + steps * period);
}
return new Date(after.getTime() + period);
}
export function nextOccurrenceFor(trigger: ScheduleTrigger, after: Date, anchor?: Date): string {
if (trigger.kind === 'cron') return nextOccurrence(trigger.expression, after);
if (trigger.kind === 'fixed') {
const next = fixedOccurrenceAfter(trigger, after);
if (!next) throw new TypeError('Fixed schedule has no occurrence within eight days');
return next.toISOString();
}
return intervalOccurrenceAfter(trigger, after, anchor).toISOString();
}
export function previewTrigger(
trigger: ScheduleTrigger,
count: number,
from = new Date(),
): readonly string[] {
if (!Number.isSafeInteger(count) || count < 1 || count > 10)
throw new TypeError('Schedule preview count must be between 1 and 10');
const occurrences: string[] = [];
let cursor = from;
for (let index = 0; index < count; index += 1) {
const next = nextOccurrenceFor(trigger, cursor, from);
occurrences.push(next);
cursor = new Date(next);
}
return occurrences;
}
export function nextOccurrence(expression: string, after: Date): string {
return parser(expression, after).next().toDate().toISOString();
}
export type ReconciledOccurrence =
| { readonly action: 'wait'; readonly dueAt: string; readonly nextDueAt: string }
| { readonly action: 'run' | 'skip'; readonly dueAt: string; readonly nextDueAt: string };
export function reconcileOccurrence(
task: {
readonly trigger?: ScheduledTask['trigger'];
readonly cronExpression?: string;
readonly nextDueAt: string;
readonly misfirePolicy: ScheduleMisfirePolicy;
},
now: Date,
): ReconciledOccurrence {
const nextDue = new Date(task.nextDueAt);
if (!Number.isFinite(nextDue.getTime())) throw new TypeError('nextDueAt is invalid');
if (nextDue.getTime() > now.getTime())
return { action: 'wait', dueAt: task.nextDueAt, nextDueAt: task.nextDueAt };
const trigger = task.trigger ?? {
kind: 'cron' as const,
expression: task.cronExpression ?? task.nextDueAt,
};
let dueAt: string;
if (trigger.kind === 'cron') {
dueAt = parser(trigger.expression, now).prev().toDate().toISOString();
} else {
// For fixed and interval triggers the stored due time is the authoritative occurrence.
dueAt = task.nextDueAt;
}
const nextDueAt = nextOccurrenceFor(trigger, now, nextDue);
return {
action: task.misfirePolicy === 'catch-up-once' ? 'run' : 'skip',
dueAt,
nextDueAt,
};
}
@@ -0,0 +1,252 @@
import Database from 'better-sqlite3';
import { describe, expect, it, vi } from 'vitest';
import { migrateDatabase } from '../../infrastructure/database/migrations.js';
import { ScheduledTaskRepository } from './scheduled-task-repository.js';
import { ScheduledOperationDispatcher } from './scheduled-operation-dispatcher.js';
describe('ScheduledOperationDispatcher', () => {
it('renders SMS time and random macros separately for each target', async () => {
const db = new Database(':memory:');
migrateDatabase(db);
const send = vi.fn(async (_instanceId: string, _input: { content: string }) => ({
sent: true as const,
}));
const dispatcher = new ScheduledOperationDispatcher({
db,
operations: { prepare: vi.fn(), execute: vi.fn() },
messages: { send },
store: {
provider: 'macos-keychain',
set: vi.fn(),
get: vi.fn(async () =>
JSON.stringify({
recipients: ['13800138000'],
content: 'Status ${time} id ${random}',
}),
),
delete: vi.fn(),
},
repository: new ScheduledTaskRepository(db),
now: () => new Date('2026-09-06T04:05:06.000Z'),
});
for (const [id, name, address] of [
['a', 'Alpha', 'http://10.0.0.1'],
['b', 'Beta', 'http://10.0.0.2'],
] as const) {
db.prepare(
'INSERT INTO instances (id,name,base_url,enabled,config_revision,created_at,updated_at) VALUES (?,?,?,?,?,?,?)',
).run(id, name, address, 1, 1, '2026-07-30T00:00:00.000Z', '2026-07-30T00:00:00.000Z');
}
const task = new ScheduledTaskRepository(db).create({
id: 'task-template',
task: {
name: 'Status',
operationType: 'send-sms',
cronExpression: '0 9 * * *',
timezone: 'Asia/Shanghai',
targetSelector: { mode: 'fixed', instanceIds: ['a', 'b'] },
sms: { recipients: ['13800138000'], content: 'Status ${time} id ${random}' },
misfirePolicy: 'skip',
overlapPolicy: 'skip',
retryPolicy: { maxRetries: 0, intervalSeconds: 60 },
enabled: true,
},
smsSecretReference: 'memory://sms',
createdBy: 'operator',
now: '2026-09-06T04:05:06.000Z',
});
const result = await dispatcher.dispatch(
task,
[
{ id: 'a', revision: 1 },
{ id: 'b', revision: 1 },
],
{ actor: 'operator', requestId: 'request-1' },
);
expect(result.outcome).toBe('succeeded');
expect(send).toHaveBeenCalledTimes(2);
for (const call of send.mock.calls) {
expect(call[1]?.content).toMatch(/^Status 2026-09-06 12:05:06 id [A-Za-z0-9]{12}$/u);
}
db.close();
});
it('executes restart targets independently and aggregates partial success', async () => {
const db = new Database(':memory:');
migrateDatabase(db);
const execute = vi
.fn()
.mockResolvedValueOnce({ id: 'job-a', status: 'succeeded' })
.mockResolvedValueOnce({ id: 'job-b', status: 'failed' });
const dispatcher = new ScheduledOperationDispatcher({
db,
operations: {
prepare: vi.fn(async (input) => ({
id: `prep-${input.targets[0]?.instanceId}`,
confirmationToken: 'token',
})),
execute,
},
messages: { send: vi.fn() },
store: { provider: 'macos-keychain', set: vi.fn(), get: vi.fn(), delete: vi.fn() },
repository: new ScheduledTaskRepository(db),
});
const result = await dispatcher.dispatch(
{
id: 'task-1',
name: 'Restart',
operationType: 'restart-service',
cronExpression: '0 9 * * *',
timezone: 'Asia/Shanghai',
targetSelector: { mode: 'fixed', instanceIds: ['a', 'b'] },
misfirePolicy: 'skip',
overlapPolicy: 'skip',
retryPolicy: { maxRetries: 0, intervalSeconds: 60 },
enabled: true,
version: 1,
createdBy: 'operator',
updatedBy: 'operator',
createdAt: '2026-07-30T00:00:00.000Z',
updatedAt: '2026-07-30T00:00:00.000Z',
},
[
{ id: 'a', revision: 1 },
{ id: 'b', revision: 2 },
],
{ actor: 'operator', requestId: 'request-1' },
);
expect(result).toEqual({ outcome: 'partially-succeeded', jobIds: ['job-a', 'job-b'] });
expect(execute).toHaveBeenCalledTimes(2);
db.close();
});
it('retries only failed SMS recipients using the task retry interval', async () => {
const db = new Database(':memory:');
migrateDatabase(db);
db.prepare(
'INSERT INTO instances (id,name,base_url,enabled,config_revision,created_at,updated_at) VALUES (?,?,?,?,?,?,?)',
).run(
'a',
'Alpha',
'http://10.0.0.1',
1,
1,
'2026-07-30T00:00:00.000Z',
'2026-07-30T00:00:00.000Z',
);
db.prepare(
'INSERT INTO instances (id,name,base_url,enabled,config_revision,created_at,updated_at) VALUES (?,?,?,?,?,?,?)',
).run(
'b',
'Beta',
'http://10.0.0.2',
1,
1,
'2026-07-30T00:00:00.000Z',
'2026-07-30T00:00:00.000Z',
);
const repository = new ScheduledTaskRepository(db);
const task = repository.create({
id: 'task-sms',
task: {
name: 'Notify lab',
operationType: 'send-sms',
cronExpression: '0 9 * * *',
timezone: 'Asia/Shanghai',
targetSelector: { mode: 'fixed', instanceIds: ['a'] },
sms: { recipients: ['13800138000', '13900139000'], content: 'Maintenance' },
misfirePolicy: 'skip',
overlapPolicy: 'skip',
retryPolicy: { maxRetries: 1, intervalSeconds: 30 },
enabled: true,
},
smsSecretReference: 'memory://sms',
createdBy: 'operator',
now: '2026-07-30T00:00:00.000Z',
});
const attempts = new Map<string, number>();
const send = vi.fn(async (_instanceId: string, input: { phoneNumber: string }) => {
const count = (attempts.get(input.phoneNumber) ?? 0) + 1;
attempts.set(input.phoneNumber, count);
if (input.phoneNumber === '13800138000' && count === 1) throw new Error('temporary');
return { sent: true as const };
});
const sleep = vi.fn(async () => undefined);
const dispatcher = new ScheduledOperationDispatcher({
db,
operations: { prepare: vi.fn(), execute: vi.fn() },
messages: { send },
store: {
provider: 'macos-keychain',
set: vi.fn(),
get: vi.fn(async () =>
JSON.stringify({ recipients: ['13800138000', '13900139000'], content: 'Maintenance' }),
),
delete: vi.fn(),
},
repository,
sleep,
});
const result = await dispatcher.dispatch(task, [{ id: 'a', revision: 1 }], {
actor: 'operator',
requestId: 'request-1',
});
expect(result.outcome).toBe('succeeded');
expect(send).toHaveBeenCalledTimes(3);
expect(sleep).toHaveBeenCalledTimes(1);
expect(sleep).toHaveBeenCalledWith(30_000);
db.close();
});
it('dispatches a baseband task against the baseband restart operation', async () => {
const db = new Database(':memory:');
migrateDatabase(db);
const prepare = vi.fn(async (input: { operationId: string; parameters: unknown }) => ({
id: 'prep-a',
confirmationToken: 'token',
...input,
}));
const dispatcher = new ScheduledOperationDispatcher({
db,
operations: {
prepare,
execute: vi.fn().mockResolvedValue({ id: 'job-a', status: 'succeeded' }),
},
messages: { send: vi.fn() },
store: { provider: 'macos-keychain', set: vi.fn(), get: vi.fn(), delete: vi.fn() },
repository: new ScheduledTaskRepository(db),
});
const result = await dispatcher.dispatch(
{
id: 'task-baseband',
name: 'Baseband',
operationType: 'restart-baseband',
cronExpression: '0 4 * * *',
timezone: 'Asia/Shanghai',
targetSelector: { mode: 'fixed', instanceIds: ['a'] },
misfirePolicy: 'skip',
overlapPolicy: 'skip',
retryPolicy: { maxRetries: 0, intervalSeconds: 60 },
enabled: true,
version: 1,
createdBy: 'operator',
updatedBy: 'operator',
createdAt: '2026-07-30T00:00:00.000Z',
updatedAt: '2026-07-30T00:00:00.000Z',
},
[{ id: 'a', revision: 1 }],
{ actor: 'operator', requestId: 'request-1' },
);
expect(result.outcome).toBe('succeeded');
expect(prepare.mock.calls[0]?.[0]).toMatchObject({
operationId: 'postBasebandRestart',
parameters: { parameterSchemaId: 'simadmin.58e2204.postBasebandRestart.parameters.v1' },
});
db.close();
});
});
@@ -0,0 +1,331 @@
import { createHash, randomInt, randomUUID } from 'node:crypto';
import type { Job, PrepareOperationRequest, ScheduledTask } from '@multi-simadmin/contracts';
import type { InstanceMessageService } from '../messages/instance-message-service.js';
import type { SecureOperationExecution } from '../operations/secure-operation-execution.js';
import type { SqliteDatabase } from '../../infrastructure/database/database.js';
import type { SecretStore } from '../../infrastructure/secrets/secret-store.js';
import type { DispatchResult } from './scheduler-coordinator.js';
import { ScheduledTaskRepository } from './scheduled-task-repository.js';
import type { ResolvedTarget } from './target-resolver.js';
interface OperationExecutor {
prepare(
input: PrepareOperationRequest,
requestId?: string,
): Promise<{ id: string; confirmationToken: string }>;
execute(
input: { preparationId: string; confirmationToken: string },
actor: string,
requestId: string,
): Promise<Pick<Job, 'id' | 'status'>>;
}
interface MessageExecutor {
send(
instanceId: string,
input: { phoneNumber: string; content: string },
): Promise<{ readonly sent: true }>;
}
interface BackupExecutor {
createBackup(): Promise<{ readonly filename: string; readonly sizeBytes: number }>;
}
export interface ScheduledOperationDispatcherOptions {
readonly db: SqliteDatabase;
readonly operations: Pick<SecureOperationExecution, 'prepare' | 'execute'> | OperationExecutor;
readonly messages: Pick<InstanceMessageService, 'send'> | MessageExecutor;
readonly maintenance?: BackupExecutor;
readonly store: SecretStore;
readonly repository: ScheduledTaskRepository;
readonly now?: () => Date;
readonly id?: () => string;
readonly sleep?: (milliseconds: number) => Promise<void>;
}
const operationSchemas: Record<string, string> = {
'restart-service': 'simadmin.58e2204.postServiceRestart.parameters.v1',
'reboot-system': 'simadmin.58e2204.postSystemReboot.parameters.v1',
'restart-baseband': 'simadmin.58e2204.postBasebandRestart.parameters.v1',
};
const operationNames: Record<string, string> = {
'restart-service': 'postServiceRestart',
'reboot-system': 'postSystemReboot',
'restart-baseband': 'postBasebandRestart',
};
const SMS_RANDOM_LENGTH = 12;
const SMS_RANDOM_ALPHABET = 'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789';
/**
* Hub-compatible SMS macros are filled in just before delivery. Rendering per target keeps
* `${time}` meaningful when a large batch spans seconds and gives every message its own nonce.
*/
export function renderSmsTemplate(
content: string,
now: Date,
random: () => string = () =>
Array.from({ length: SMS_RANDOM_LENGTH }, () =>
SMS_RANDOM_ALPHABET.charAt(randomInt(SMS_RANDOM_ALPHABET.length)),
).join(''),
): string {
const formatter = new Intl.DateTimeFormat('en-CA', {
timeZone: 'Asia/Shanghai',
year: 'numeric',
month: '2-digit',
day: '2-digit',
hour: '2-digit',
minute: '2-digit',
second: '2-digit',
hour12: false,
});
const timestamp = formatter.format(now).replace(', ', ' ');
return content.replaceAll('${time}', timestamp).replaceAll('${random}', random());
}
function aggregate(successes: number, failures: number): DispatchResult['outcome'] {
if (successes > 0 && failures > 0) return 'partially-succeeded';
return failures > 0 ? 'failed' : 'succeeded';
}
export class ScheduledOperationDispatcher {
private readonly now: () => Date;
private readonly id: () => string;
private readonly sleep: (milliseconds: number) => Promise<void>;
constructor(private readonly options: ScheduledOperationDispatcherOptions) {
this.now = options.now ?? (() => new Date());
this.id = options.id ?? randomUUID;
this.sleep =
options.sleep ??
((milliseconds) => new Promise((resolve) => setTimeout(resolve, milliseconds)));
}
async dispatch(
task: ScheduledTask,
targets: readonly ResolvedTarget[],
context: { readonly actor: string; readonly requestId: string },
): Promise<DispatchResult> {
if (task.operationType === 'send-sms') return this.dispatchSms(task, targets, context);
if (task.operationType === 'backup-data') return this.dispatchBackup(task);
const operationId = operationNames[task.operationType];
const schema = operationSchemas[task.operationType];
if (!operationId || !schema)
return { outcome: 'needs-attention', reason: 'Unsupported scheduled operation', jobIds: [] };
// The execution baseline pins delay_seconds to 3; anything else could never
// succeed, so refuse up front instead of exhausting the retry policy.
if (task.operationType === 'reboot-system' && (task.delaySeconds ?? 3) !== 3)
return {
outcome: 'needs-attention',
reason: 'Scheduled reboots only support the pinned 3-second delay',
jobIds: [],
};
const fields =
task.operationType === 'reboot-system'
? [
{
fieldId: 'delay_seconds',
kind: 'number' as const,
value: task.delaySeconds ?? 3,
},
]
: [];
const jobIds: string[] = [];
let successes = 0;
let failures = 0;
for (const target of targets) {
let succeeded = false;
for (let attempt = 0; attempt <= task.retryPolicy.maxRetries; attempt += 1) {
try {
const preparation = await this.options.operations.prepare(
{
operationId,
targets: [{ instanceId: target.id, revision: target.revision }],
parameters: { parameterSchemaId: schema, fields },
},
context.requestId,
);
const job = await this.options.operations.execute(
{ preparationId: preparation.id, confirmationToken: preparation.confirmationToken },
context.actor,
context.requestId,
);
jobIds.push(job.id);
if (job.status === 'succeeded') {
succeeded = true;
break;
}
} catch {
// Continue only within the task's explicit bounded retry policy.
}
if (!succeeded && attempt < task.retryPolicy.maxRetries)
await this.sleep(task.retryPolicy.intervalSeconds * 1_000);
}
if (succeeded) successes += 1;
else failures += 1;
}
return { outcome: aggregate(successes, failures), jobIds };
}
private async dispatchSms(
task: ScheduledTask,
targets: readonly ResolvedTarget[],
context: { readonly actor: string; readonly requestId: string },
): Promise<DispatchResult> {
const reference = this.options.repository.getSmsSecretReference(task.id);
if (!reference)
return { outcome: 'needs-attention', reason: 'SMS secret is missing', jobIds: [] };
const secret = await this.options.store.get(reference);
if (!secret) return { outcome: 'needs-attention', reason: 'SMS secret is missing', jobIds: [] };
let payload: { recipients: string[]; content: string; randomDelaySeconds?: number };
try {
payload = JSON.parse(secret) as typeof payload;
if (
!Array.isArray(payload.recipients) ||
!payload.recipients.length ||
typeof payload.content !== 'string' ||
!payload.content
)
throw new Error('invalid');
} catch {
return { outcome: 'needs-attention', reason: 'SMS secret is invalid', jobIds: [] };
}
const jobIds: string[] = [];
let successes = 0;
let failures = 0;
for (const target of targets) {
const result = await this.recordSmsJob(
target.id,
{ ...payload, content: renderSmsTemplate(payload.content, this.now()) },
task.retryPolicy,
context,
);
jobIds.push(result.id);
if (result.succeeded) successes += 1;
else failures += 1;
}
return { outcome: aggregate(successes, failures), jobIds };
}
private async recordSmsJob(
instanceId: string,
payload: {
readonly recipients: readonly string[];
readonly content: string;
readonly randomDelaySeconds?: number;
},
retryPolicy: ScheduledTask['retryPolicy'],
context: { readonly actor: string; readonly requestId: string },
): Promise<{ readonly id: string; readonly succeeded: boolean }> {
const ids = { job: this.id(), item: this.id(), attempt: this.id(), audit: this.id() };
const startedAt = this.now().toISOString();
const digest = createHash('sha256').update(JSON.stringify(payload)).digest('hex');
this.options.db.transaction(() => {
this.options.db
.prepare(
`INSERT INTO jobs (id,root_job_id,operation_id,risk_level,status,requested_by,request_id,parameters_digest,created_at,started_at,updated_at)
VALUES (?,?,'sendSms','R2','running',?,?,?,?,?,?)`,
)
.run(
ids.job,
ids.job,
context.actor,
context.requestId,
digest,
startedAt,
startedAt,
startedAt,
);
this.options.db
.prepare(
"INSERT INTO job_attempts (id,job_id,status,started_at,created_at) VALUES (?,?,'running',?,?)",
)
.run(ids.attempt, ids.job, startedAt, startedAt);
this.options.db
.prepare(
"INSERT INTO job_items (id,job_id,instance_id,attempt_number,status,created_at,started_at,updated_at) VALUES (?,?,?,1,'running',?,?,?)",
)
.run(ids.item, ids.job, instanceId, startedAt, startedAt, startedAt);
})();
let succeeded = true;
for (const recipient of payload.recipients) {
let delivered = false;
// Hub parity: per-recipient jitter so bulk schedules do not fire in one burst.
const jitter = Math.max(0, Math.floor(payload.randomDelaySeconds ?? 0));
if (jitter > 0) await this.sleep(Math.floor(Math.random() * (jitter + 1)) * 1_000);
for (let attempt = 0; attempt <= retryPolicy.maxRetries; attempt += 1) {
try {
await this.options.messages.send(instanceId, {
phoneNumber: recipient,
content: payload.content,
});
delivered = true;
break;
} catch {
delivered = false;
}
if (!delivered && attempt < retryPolicy.maxRetries)
await this.sleep(retryPolicy.intervalSeconds * 1_000);
}
if (!delivered) succeeded = false;
}
const finishedAt = this.now().toISOString();
const status = succeeded ? 'succeeded' : 'failed';
this.options.db.transaction(() => {
this.options.db
.prepare('UPDATE jobs SET status=?,finished_at=?,updated_at=? WHERE id=?')
.run(status, finishedAt, finishedAt, ids.job);
this.options.db
.prepare(
'UPDATE job_items SET status=?,result_code=?,finished_at=?,updated_at=? WHERE id=?',
)
.run(status, succeeded ? 'SMS_SENT' : 'SMS_FAILED', finishedAt, finishedAt, ids.item);
this.options.db
.prepare('UPDATE job_attempts SET status=?,finished_at=? WHERE id=?')
.run(status, finishedAt, ids.attempt);
this.options.db
.prepare(
`INSERT INTO audit_events
(id,instance_id,job_id,actor,operation_id,risk_level,request_id,parameters_summary_json,body_digest,result_code,duration_ms,created_at)
VALUES (?,?,?,?,'sendSms','R2',?,?,?,?,0,?)`,
)
.run(
ids.audit,
instanceId,
ids.job,
context.actor,
context.requestId,
JSON.stringify([
{ fieldId: 'recipients', displayValue: '[REDACTED]', redacted: true },
{ fieldId: 'content', displayValue: '[REDACTED]', redacted: true },
]),
digest,
succeeded ? 'SMS_SENT' : 'SMS_FAILED',
finishedAt,
);
})();
return { id: ids.job, succeeded };
}
private async dispatchBackup(task: ScheduledTask): Promise<DispatchResult> {
const maintenance = this.options.maintenance;
if (!maintenance)
return {
outcome: 'needs-attention',
reason: 'Backup maintenance service is unavailable',
jobIds: [],
};
try {
const backup = await maintenance.createBackup();
return {
outcome: 'succeeded',
reason: `Backup ${backup.filename} (${backup.sizeBytes} bytes) created for schedule ${task.name}`,
jobIds: [],
};
} catch {
return { outcome: 'failed', reason: 'Control-plane backup failed', jobIds: [] };
}
}
}
@@ -0,0 +1,173 @@
import Database from 'better-sqlite3';
import { describe, expect, it } from 'vitest';
import { migrateDatabase } from '../../infrastructure/database/migrations.js';
import { ScheduledTaskRepository } from './scheduled-task-repository.js';
function fixture() {
const db = new Database(':memory:');
db.pragma('foreign_keys = ON');
migrateDatabase(db);
return { db, repository: new ScheduledTaskRepository(db) };
}
const task = {
name: 'Morning restart',
operationType: 'restart-service' as const,
cronExpression: '0 9 * * *',
timezone: 'Asia/Shanghai' as const,
targetSelector: { mode: 'fixed' as const, instanceIds: ['instance-a', 'instance-b'] },
misfirePolicy: 'skip' as const,
overlapPolicy: 'skip' as const,
retryPolicy: { maxRetries: 0, intervalSeconds: 60 },
enabled: true,
};
describe('ScheduledTaskRepository', () => {
it('round-trips a versioned task without exposing secret payload data', () => {
const { db, repository } = fixture();
const created = repository.create({
id: 'task-1',
task,
createdBy: 'operator',
now: '2026-07-30T00:00:00.000Z',
nextDueAt: '2026-07-30T01:00:00.000Z',
});
expect(created).toMatchObject({
id: 'task-1',
version: 1,
nextDueAt: '2026-07-30T01:00:00.000Z',
});
expect(repository.get('task-1')).toEqual(created);
expect(JSON.stringify(created)).not.toMatch(/secretReference|content|recipient/i);
db.close();
});
it('claims the same scheduled occurrence once and preserves its immutable snapshot', () => {
const { db, repository } = fixture();
repository.create({
id: 'task-1',
task,
createdBy: 'operator',
now: '2026-07-30T00:00:00.000Z',
});
const dueAt = '2026-07-30T01:00:00.000Z';
const first = repository.claimOccurrence({
id: 'run-1',
scheduledTaskId: 'task-1',
scheduleVersion: 1,
dueAt,
claimedAt: '2026-07-30T01:00:01.000Z',
triggerSource: 'scheduled',
targetSnapshot: ['instance-a', 'instance-b'],
taskSnapshot: task,
});
const duplicate = repository.claimOccurrence({
id: 'run-2',
scheduledTaskId: 'task-1',
scheduleVersion: 1,
dueAt,
claimedAt: '2026-07-30T01:00:02.000Z',
triggerSource: 'scheduled',
targetSnapshot: ['changed'],
taskSnapshot: task,
});
expect(first).toMatchObject({ id: 'run-1', targetSnapshot: ['instance-a', 'instance-b'] });
expect(duplicate).toBeNull();
expect(repository.getRun('run-1')?.targetSnapshot).toEqual(['instance-a', 'instance-b']);
db.close();
});
it('rolls back due-time advancement when scheduled claim preparation fails', () => {
const { db, repository } = fixture();
repository.create({
id: 'task-1',
task,
createdBy: 'operator',
now: '2026-07-30T00:00:00.000Z',
nextDueAt: '2026-07-30T01:00:00.000Z',
});
expect(() =>
repository.claimScheduledOccurrence({
id: 'run-1',
scheduledTaskId: 'task-1',
scheduleVersion: 1,
dueAt: '2026-07-30T01:00:00.000Z',
nextDueAt: '2026-07-31T01:00:00.000Z',
claimedAt: '2026-07-30T01:00:00.000Z',
taskSnapshot: task,
overlapPolicy: 'skip',
resolveTargets: () => {
throw new Error('target resolution failed');
},
}),
).toThrow('target resolution failed');
expect(repository.get('task-1')?.nextDueAt).toBe('2026-07-30T01:00:00.000Z');
expect(repository.listRuns()).toHaveLength(0);
db.close();
});
it('soft deletes task configuration while retaining completed run history', () => {
const { db, repository } = fixture();
repository.create({
id: 'task-1',
task,
createdBy: 'operator',
now: '2026-07-30T00:00:00.000Z',
});
repository.claimOccurrence({
id: 'run-1',
scheduledTaskId: 'task-1',
scheduleVersion: 1,
dueAt: '2026-07-30T01:00:00.000Z',
claimedAt: '2026-07-30T01:00:01.000Z',
triggerSource: 'scheduled',
targetSnapshot: [],
taskSnapshot: task,
});
repository.finishRun('run-1', {
outcome: 'no-targets',
reason: 'No enabled instances matched',
jobIds: [],
finishedAt: '2026-07-30T01:00:02.000Z',
});
repository.softDelete('task-1', 1, 'operator', '2026-07-30T02:00:00.000Z');
expect(repository.get('task-1')).toBeNull();
expect(repository.getRun('run-1')).toMatchObject({ outcome: 'no-targets' });
db.close();
});
it('reconciles interrupted started runs to needs-attention', () => {
const { db, repository } = fixture();
repository.create({
id: 'task-1',
task,
createdBy: 'operator',
now: '2026-07-30T00:00:00.000Z',
});
repository.claimOccurrence({
id: 'run-1',
scheduledTaskId: 'task-1',
scheduleVersion: 1,
dueAt: '2026-07-30T01:00:00.000Z',
claimedAt: '2026-07-30T01:00:00.000Z',
triggerSource: 'scheduled',
targetSnapshot: ['instance-a'],
taskSnapshot: task,
});
repository.startRun('run-1', '2026-07-30T01:00:01.000Z');
expect(repository.reconcileInterruptedRuns('2026-07-30T02:00:00.000Z')).toBe(1);
expect(repository.getRun('run-1')).toMatchObject({
outcome: 'needs-attention',
reason: 'Scheduler stopped before the run outcome was known',
finishedAt: '2026-07-30T02:00:00.000Z',
});
expect(repository.hasActiveRun('task-1')).toBe(false);
db.close();
});
});
@@ -0,0 +1,502 @@
import type {
CreateScheduledTaskRequest,
ScheduleTrigger,
ScheduledRun,
ScheduledRunOutcome,
ScheduledRunTriggerSource,
ScheduledTask,
} from '@multi-simadmin/contracts';
import { scheduleTriggerOf } from '@multi-simadmin/contracts';
import type { SqliteDatabase } from '../../infrastructure/database/database.js';
interface ScheduledTaskRow {
id: string;
name: string;
operation_type: ScheduledTask['operationType'];
enabled: number;
version: number;
cron_expression: string;
timezone: ScheduledTask['timezone'];
target_selector_json: string;
sms_secret_reference: string | null;
sms_recipient_count: number | null;
effective_start_at: string | null;
effective_end_at: string | null;
misfire_policy: ScheduledTask['misfirePolicy'];
overlap_policy: ScheduledTask['overlapPolicy'];
retry_policy_json: string;
trigger_json: string | null;
delay_seconds: number | null;
next_due_at: string | null;
last_evaluated_at: string | null;
created_by: string;
updated_by: string;
created_at: string;
updated_at: string;
}
interface ScheduledRunRow {
id: string;
scheduled_task_id: string;
schedule_version: number;
task_snapshot_json: string;
due_at: string;
claimed_at: string;
started_at: string | null;
finished_at: string | null;
target_snapshot_json: string;
outcome: ScheduledRunOutcome | null;
reason: string | null;
job_ids_json: string;
trigger_source: ScheduledRunTriggerSource;
attempt: number;
}
export interface CreateTaskRecord {
readonly id: string;
readonly task: CreateScheduledTaskRequest;
readonly smsSecretReference?: string;
readonly createdBy: string;
readonly now: string;
readonly nextDueAt?: string;
}
export interface UpdateTaskRecord {
readonly id: string;
readonly version: number;
readonly task: CreateScheduledTaskRequest;
readonly smsSecretReference?: string;
readonly updatedBy: string;
readonly now: string;
readonly nextDueAt?: string;
}
export interface ClaimOccurrenceRecord {
readonly id: string;
readonly scheduledTaskId: string;
readonly scheduleVersion: number;
readonly dueAt: string;
readonly claimedAt: string;
readonly triggerSource: ScheduledRunTriggerSource;
readonly targetSnapshot: readonly string[];
readonly taskSnapshot: CreateScheduledTaskRequest;
readonly attempt?: number;
}
export interface ScheduledClaimTarget {
readonly id: string;
readonly revision: number;
}
export interface ClaimScheduledOccurrenceRecord {
readonly id: string;
readonly scheduledTaskId: string;
readonly scheduleVersion: number;
readonly dueAt: string;
readonly nextDueAt: string;
readonly claimedAt: string;
readonly taskSnapshot: CreateScheduledTaskRequest;
readonly overlapPolicy: ScheduledTask['overlapPolicy'];
readonly resolveTargets: () => {
readonly targets: readonly ScheduledClaimTarget[];
readonly targetSnapshot?: readonly string[];
readonly attentionReason?: string;
};
readonly terminal?: {
readonly outcome: ScheduledRunOutcome;
readonly reason: string;
};
}
export interface ScheduledClaimResult {
readonly run: ScheduledRun;
readonly targets: readonly ScheduledClaimTarget[];
readonly disposition: 'execute' | 'queued' | 'finished';
}
export interface FinishRunRecord {
readonly outcome: ScheduledRunOutcome;
readonly reason?: string;
readonly jobIds: readonly string[];
readonly finishedAt: string;
}
function parseJson<T>(value: string, label: string): T {
try {
return JSON.parse(value) as T;
} catch {
throw new Error(`Stored ${label} is invalid`);
}
}
function projectTask(row: ScheduledTaskRow): ScheduledTask {
const trigger: ScheduleTrigger = row.trigger_json
? parseJson<ScheduleTrigger>(row.trigger_json, 'schedule trigger')
: { kind: 'cron', expression: row.cron_expression };
return {
id: row.id,
name: row.name,
operationType: row.operation_type,
trigger,
cronExpression: row.cron_expression,
timezone: row.timezone,
targetSelector: parseJson(row.target_selector_json, 'target selector'),
...(row.sms_secret_reference
? { sms: { configured: true, recipientCount: row.sms_recipient_count ?? 0 } }
: {}),
...(row.delay_seconds === null || row.delay_seconds === undefined
? {}
: { delaySeconds: row.delay_seconds }),
...(row.effective_start_at ? { effectiveStartAt: row.effective_start_at } : {}),
...(row.effective_end_at ? { effectiveEndAt: row.effective_end_at } : {}),
misfirePolicy: row.misfire_policy,
overlapPolicy: row.overlap_policy,
retryPolicy: parseJson(row.retry_policy_json, 'retry policy'),
enabled: row.enabled === 1,
version: row.version,
...(row.next_due_at ? { nextDueAt: row.next_due_at } : {}),
...(row.last_evaluated_at ? { lastEvaluatedAt: row.last_evaluated_at } : {}),
createdBy: row.created_by,
updatedBy: row.updated_by,
createdAt: row.created_at,
updatedAt: row.updated_at,
};
}
function projectRun(row: ScheduledRunRow): ScheduledRun {
const task = parseJson<CreateScheduledTaskRequest>(row.task_snapshot_json, 'task snapshot');
return {
id: row.id,
scheduledTaskId: row.scheduled_task_id,
scheduleVersion: row.schedule_version,
taskName: task.name,
operationType: task.operationType,
dueAt: row.due_at,
claimedAt: row.claimed_at,
...(row.started_at ? { startedAt: row.started_at } : {}),
...(row.finished_at ? { finishedAt: row.finished_at } : {}),
targetSnapshot: parseJson(row.target_snapshot_json, 'target snapshot'),
...(row.outcome ? { outcome: row.outcome } : {}),
...(row.reason ? { reason: row.reason } : {}),
jobIds: parseJson(row.job_ids_json, 'job ids'),
triggerSource: row.trigger_source,
attempt: row.attempt,
};
}
export class ScheduledTaskRepository {
constructor(private readonly db: SqliteDatabase) {}
create(input: CreateTaskRecord): ScheduledTask {
const task = input.task;
this.db
.prepare(
`INSERT INTO scheduled_tasks
(id,name,operation_type,enabled,version,cron_expression,timezone,target_selector_json,
sms_secret_reference,sms_recipient_count,effective_start_at,effective_end_at,misfire_policy,
overlap_policy,retry_policy_json,trigger_json,delay_seconds,next_due_at,created_by,
updated_by,created_at,updated_at)
VALUES (?,?,?,?,1,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)`,
)
.run(
input.id,
task.name,
task.operationType,
task.enabled ? 1 : 0,
task.cronExpression,
task.timezone,
JSON.stringify(task.targetSelector),
input.smsSecretReference ?? null,
task.sms?.recipients.length ?? null,
task.effectiveStartAt ?? null,
task.effectiveEndAt ?? null,
task.misfirePolicy,
task.overlapPolicy,
JSON.stringify(task.retryPolicy),
JSON.stringify(scheduleTriggerOf(task)),
task.delaySeconds ?? null,
input.nextDueAt ?? null,
input.createdBy,
input.createdBy,
input.now,
input.now,
);
const created = this.get(input.id);
if (!created) throw new Error('Scheduled task was not persisted');
return created;
}
update(input: UpdateTaskRecord): ScheduledTask {
const task = input.task;
const result = this.db
.prepare(
`UPDATE scheduled_tasks
SET name = ?, operation_type = ?, enabled = ?, version = version + 1,
cron_expression = ?, timezone = ?, target_selector_json = ?,
sms_secret_reference = ?, sms_recipient_count = ?, effective_start_at = ?,
effective_end_at = ?, misfire_policy = ?, overlap_policy = ?, retry_policy_json = ?,
trigger_json = ?, delay_seconds = ?,
next_due_at = ?, last_evaluated_at = NULL, updated_by = ?, updated_at = ?
WHERE id = ? AND version = ? AND deleted_at IS NULL`,
)
.run(
task.name,
task.operationType,
task.enabled ? 1 : 0,
task.cronExpression,
task.timezone,
JSON.stringify(task.targetSelector),
input.smsSecretReference ?? null,
task.sms?.recipients.length ?? null,
task.effectiveStartAt ?? null,
task.effectiveEndAt ?? null,
task.misfirePolicy,
task.overlapPolicy,
JSON.stringify(task.retryPolicy),
JSON.stringify(scheduleTriggerOf(task)),
task.delaySeconds ?? null,
input.nextDueAt ?? null,
input.updatedBy,
input.now,
input.id,
input.version,
);
if (result.changes !== 1) throw new Error('Scheduled task was not found or version changed');
const updated = this.get(input.id);
if (!updated) throw new Error('Scheduled task was not persisted');
return updated;
}
get(id: string): ScheduledTask | null {
const row = this.db
.prepare('SELECT * FROM scheduled_tasks WHERE id = ? AND deleted_at IS NULL')
.get(id) as ScheduledTaskRow | undefined;
return row ? projectTask(row) : null;
}
list(): readonly ScheduledTask[] {
return (
this.db
.prepare(
'SELECT * FROM scheduled_tasks WHERE deleted_at IS NULL ORDER BY updated_at DESC, id ASC',
)
.all() as ScheduledTaskRow[]
).map(projectTask);
}
getSmsSecretReference(id: string): string | undefined {
const row = this.db
.prepare(
'SELECT sms_secret_reference FROM scheduled_tasks WHERE id = ? AND deleted_at IS NULL',
)
.get(id) as { sms_secret_reference: string | null } | undefined;
return row?.sms_secret_reference ?? undefined;
}
setEnabled(
id: string,
version: number,
enabled: boolean,
actor: string,
now: string,
): ScheduledTask {
const result = this.db
.prepare(
`UPDATE scheduled_tasks
SET enabled = ?, version = version + 1, updated_by = ?, updated_at = ?
WHERE id = ? AND version = ? AND deleted_at IS NULL`,
)
.run(enabled ? 1 : 0, actor, now, id, version);
if (result.changes !== 1) throw new Error('Scheduled task was not found or version changed');
const task = this.get(id);
if (!task) throw new Error('Scheduled task was not persisted');
return task;
}
advanceNextDue(id: string, version: number, nextDueAt: string, evaluatedAt: string): boolean {
return (
this.db
.prepare(
`UPDATE scheduled_tasks SET next_due_at = ?, last_evaluated_at = ?
WHERE id = ? AND version = ? AND enabled = 1 AND deleted_at IS NULL`,
)
.run(nextDueAt, evaluatedAt, id, version).changes === 1
);
}
startRun(id: string, startedAt: string): void {
const result = this.db
.prepare(
'UPDATE scheduled_runs SET started_at = ? WHERE id = ? AND started_at IS NULL AND finished_at IS NULL',
)
.run(startedAt, id);
if (result.changes !== 1) throw new Error('Scheduled run is missing or already started');
}
hasActiveRun(scheduledTaskId: string): boolean {
return !!this.db
.prepare(
'SELECT 1 FROM scheduled_runs WHERE scheduled_task_id = ? AND started_at IS NOT NULL AND finished_at IS NULL LIMIT 1',
)
.get(scheduledTaskId);
}
reconcileInterruptedRuns(finishedAt: string): number {
return this.db
.prepare(
`UPDATE scheduled_runs
SET outcome = 'needs-attention',
reason = 'Scheduler stopped before the run outcome was known',
finished_at = ?
WHERE started_at IS NOT NULL AND finished_at IS NULL`,
)
.run(finishedAt).changes;
}
getQueuedRun(scheduledTaskId: string): ScheduledRun | null {
const row = this.db
.prepare(
`SELECT * FROM scheduled_runs
WHERE scheduled_task_id = ? AND started_at IS NULL AND finished_at IS NULL
ORDER BY claimed_at ASC, id ASC LIMIT 1`,
)
.get(scheduledTaskId) as ScheduledRunRow | undefined;
return row ? projectRun(row) : null;
}
claimOccurrence(input: ClaimOccurrenceRecord): ScheduledRun | null {
const result = this.db
.prepare(
`INSERT OR IGNORE INTO scheduled_runs
(id,scheduled_task_id,schedule_version,task_snapshot_json,due_at,claimed_at,target_snapshot_json,
job_ids_json,trigger_source,attempt)
VALUES (?,?,?,?,?,?,?,?,?,?)`,
)
.run(
input.id,
input.scheduledTaskId,
input.scheduleVersion,
JSON.stringify(input.taskSnapshot),
input.dueAt,
input.claimedAt,
JSON.stringify(input.targetSnapshot),
'[]',
input.triggerSource,
input.attempt ?? 1,
);
return result.changes === 0 ? null : this.getRun(input.id);
}
claimScheduledOccurrence(input: ClaimScheduledOccurrenceRecord): ScheduledClaimResult | null {
return this.db.transaction((): ScheduledClaimResult | null => {
const advanced = this.db
.prepare(
`UPDATE scheduled_tasks SET next_due_at = ?, last_evaluated_at = ?
WHERE id = ? AND version = ? AND enabled = 1 AND deleted_at IS NULL AND next_due_at = ?`,
)
.run(
input.nextDueAt,
input.claimedAt,
input.scheduledTaskId,
input.scheduleVersion,
input.dueAt,
);
if (advanced.changes !== 1) return null;
const resolution = input.terminal
? { targets: [] as readonly ScheduledClaimTarget[] }
: input.resolveTargets();
const targets = [...resolution.targets];
let disposition: ScheduledClaimResult['disposition'] = 'execute';
let terminal =
input.terminal ??
(resolution.attentionReason
? { outcome: 'needs-attention' as const, reason: resolution.attentionReason }
: undefined);
if (!terminal && this.hasActiveRun(input.scheduledTaskId)) {
const queued = this.getQueuedRun(input.scheduledTaskId);
if (input.overlapPolicy === 'queue-once' && !queued) {
disposition = 'queued';
} else {
disposition = 'finished';
terminal = {
outcome: 'skipped',
reason:
input.overlapPolicy === 'queue-once'
? 'Overlap queue is full'
: 'Overlapping occurrence skipped by task policy',
};
}
}
const claimed = this.claimOccurrence({
id: input.id,
scheduledTaskId: input.scheduledTaskId,
scheduleVersion: input.scheduleVersion,
dueAt: input.dueAt,
claimedAt: input.claimedAt,
triggerSource: 'scheduled',
targetSnapshot: resolution.targetSnapshot ?? targets.map((target) => target.id),
taskSnapshot: input.taskSnapshot,
});
if (!claimed) return null;
if (terminal) {
return {
run: this.finishRun(claimed.id, {
outcome: terminal.outcome,
reason: terminal.reason,
jobIds: [],
finishedAt: input.claimedAt,
}),
targets,
disposition: 'finished',
};
}
return { run: claimed, targets, disposition };
})();
}
getRun(id: string): ScheduledRun | null {
const row = this.db.prepare('SELECT * FROM scheduled_runs WHERE id = ?').get(id) as
| ScheduledRunRow
| undefined;
return row ? projectRun(row) : null;
}
listRuns(scheduledTaskId?: string): readonly ScheduledRun[] {
const rows = scheduledTaskId
? (this.db
.prepare(
'SELECT * FROM scheduled_runs WHERE scheduled_task_id = ? ORDER BY due_at DESC, id ASC',
)
.all(scheduledTaskId) as ScheduledRunRow[])
: (this.db
.prepare('SELECT * FROM scheduled_runs ORDER BY due_at DESC, id ASC')
.all() as ScheduledRunRow[]);
return rows.map(projectRun);
}
finishRun(id: string, input: FinishRunRecord): ScheduledRun {
const result = this.db
.prepare(
`UPDATE scheduled_runs SET outcome = ?, reason = ?, job_ids_json = ?, finished_at = ?
WHERE id = ? AND finished_at IS NULL`,
)
.run(input.outcome, input.reason ?? null, JSON.stringify(input.jobIds), input.finishedAt, id);
if (result.changes !== 1) throw new Error('Scheduled run is missing or already finished');
const run = this.getRun(id);
if (!run) throw new Error('Scheduled run was not persisted');
return run;
}
softDelete(id: string, version: number, actor: string, now: string): void {
const result = this.db
.prepare(
`UPDATE scheduled_tasks SET enabled = 0, version = version + 1, updated_by = ?, updated_at = ?, deleted_at = ?
WHERE id = ? AND version = ? AND deleted_at IS NULL`,
)
.run(actor, now, now, id, version);
if (result.changes !== 1) throw new Error('Scheduled task was not found or version changed');
}
}
@@ -0,0 +1,236 @@
import Database from 'better-sqlite3';
import { describe, expect, it } from 'vitest';
import { migrateDatabase } from '../../infrastructure/database/migrations.js';
import type { SecretKey, SecretStore } from '../../infrastructure/secrets/secret-store.js';
import { ScheduledTaskRepository } from './scheduled-task-repository.js';
import { ScheduledTaskService } from './scheduled-task-service.js';
class MemorySecrets implements SecretStore {
readonly provider = 'macos-keychain';
readonly values = new Map<string, string>();
readonly setCalls: Array<{ key: SecretKey; value: string }> = [];
blockRotations = false;
private rotationWaiters: Array<() => void> = [];
async set(key: SecretKey, value: string): Promise<string> {
const reference = `memory://${key.instanceId}/${key.purpose}/${key.slot ?? 'default'}`;
this.setCalls.push({ key, value });
this.values.set(reference, value);
if (this.blockRotations && key.slot) {
await new Promise<void>((resolve) => {
this.rotationWaiters.push(resolve);
if (this.rotationWaiters.length === 2) {
for (const waiter of this.rotationWaiters.splice(0)) waiter();
}
});
}
return reference;
}
async get(reference: string): Promise<string | undefined> {
return this.values.get(reference);
}
async delete(reference: string): Promise<boolean> {
return this.values.delete(reference);
}
}
function fixture() {
const db = new Database(':memory:');
db.pragma('foreign_keys = ON');
migrateDatabase(db);
const store = new MemorySecrets();
const repository = new ScheduledTaskRepository(db);
let id = 0;
const service = new ScheduledTaskService({
repository,
store,
now: () => new Date('2026-07-30T00:00:00.000Z'),
id: () => `task-${++id}`,
});
return { db, repository, service, store };
}
const base = {
name: 'Morning SMS',
operationType: 'send-sms',
cronExpression: '0 9 * * *',
targetSelector: { mode: 'tags', match: 'all', tags: ['lab'] },
sms: { recipients: ['13800138000', '13900139000'], content: 'Maintenance complete' },
};
describe('ScheduledTaskService', () => {
it('stores SMS payload only in the secret store and returns a redacted summary', async () => {
const { db, service, store } = fixture();
const created = await service.create('operator', base);
expect(created.sms).toEqual({ configured: true, recipientCount: 2 });
expect(JSON.stringify(created)).not.toContain('13800138000');
expect(JSON.stringify(created)).not.toContain('Maintenance complete');
expect(store.setCalls[0]?.value).toBe(
JSON.stringify({
recipients: ['13800138000', '13900139000'],
content: 'Maintenance complete',
}),
);
const persisted = db.prepare('SELECT * FROM scheduled_tasks WHERE id = ?').get('task-1');
expect(JSON.stringify(persisted)).not.toContain('13800138000');
expect(JSON.stringify(persisted)).not.toContain('Maintenance complete');
db.close();
});
it('increments the optimistic version when pausing and rejects a stale update', async () => {
const { db, service } = fixture();
const created = await service.create('operator', {
...base,
operationType: 'restart-service',
sms: undefined,
});
expect(service.setEnabled('operator', created.id, created.version, false).version).toBe(2);
expect(() => service.setEnabled('operator', created.id, created.version, true)).toThrow(
/version/i,
);
db.close();
});
it('does not schedule the first occurrence before the effective start', async () => {
const { db, service } = fixture();
const created = await service.create('operator', {
...base,
operationType: 'restart-service',
sms: undefined,
effectiveStartAt: '2026-08-05T00:00:00.000Z',
});
expect(created.nextDueAt).toBe('2026-08-05T01:00:00.000Z');
db.close();
});
it('keeps immutable run history after deleting its schedule', async () => {
const { db, repository, service } = fixture();
const created = await service.create('operator', {
...base,
operationType: 'restart-service',
sms: undefined,
});
repository.claimOccurrence({
id: 'run-1',
scheduledTaskId: created.id,
scheduleVersion: created.version,
dueAt: '2026-07-30T01:00:00.000Z',
claimedAt: '2026-07-30T01:00:00.000Z',
triggerSource: 'manual',
targetSnapshot: [],
taskSnapshot: {
name: base.name,
cronExpression: base.cronExpression,
targetSelector: { mode: 'tags' as const, match: 'all' as const, tags: ['lab'] },
operationType: 'restart-service',
timezone: 'Asia/Shanghai',
misfirePolicy: 'skip',
overlapPolicy: 'skip',
retryPolicy: { maxRetries: 0, intervalSeconds: 60 },
enabled: true,
},
});
await service.remove('operator', created.id, created.version);
expect(service.get(created.id)).toBeNull();
expect(repository.getRun('run-1')).not.toBeNull();
db.close();
});
it('updates the complete task, preserves an unchanged SMS secret, and advances the version', async () => {
const { db, service, store } = fixture();
const created = await service.create('operator', base);
const reference = store.setCalls[0]?.key;
const updated = await service.update('editor', created.id, created.version, {
name: 'Evening SMS',
cronExpression: '30 18 * * *',
effectiveStartAt: '2026-08-01T00:00:00.000Z',
effectiveEndAt: '2026-09-01T00:00:00.000Z',
overlapPolicy: 'queue-once',
});
expect(updated).toMatchObject({
name: 'Evening SMS',
cronExpression: '30 18 * * *',
overlapPolicy: 'queue-once',
version: 2,
updatedBy: 'editor',
sms: { configured: true, recipientCount: 2 },
});
expect(store.setCalls).toHaveLength(1);
expect(store.setCalls[0]?.key).toEqual(reference);
db.close();
});
it('rotates edited SMS secrets and validates retry policy against the effective operation', async () => {
const { db, service, store } = fixture();
const created = await service.create('operator', base);
const oldReference = [...store.values.keys()][0];
const updated = await service.update('editor', created.id, created.version, {
sms: { recipients: ['13700137000'], content: 'Updated content' },
});
expect(updated.sms).toEqual({ configured: true, recipientCount: 1 });
expect(store.setCalls).toHaveLength(2);
expect(oldReference ? store.values.has(oldReference) : true).toBe(false);
const reboot = await service.create('operator', {
...base,
name: 'Reboot',
operationType: 'reboot-system',
sms: undefined,
});
await expect(
service.update('editor', reboot.id, reboot.version, {
retryPolicy: { maxRetries: 1, intervalSeconds: 60 },
}),
).rejects.toThrow(/reboot/i);
db.close();
});
it('does not delete the winning SMS secret when concurrent updates race', async () => {
const { db, repository, service, store } = fixture();
const created = await service.create('operator', base);
store.blockRotations = true;
const results = await Promise.allSettled([
service.update('editor-a', created.id, created.version, {
sms: { recipients: ['13700137000'], content: 'Update A' },
}),
service.update('editor-b', created.id, created.version, {
sms: { recipients: ['13600136000'], content: 'Update B' },
}),
]);
expect(results.filter((result) => result.status === 'fulfilled')).toHaveLength(1);
expect(results.filter((result) => result.status === 'rejected')).toHaveLength(1);
const currentReference = repository.getSmsSecretReference(created.id);
expect(currentReference).toBeDefined();
expect(currentReference ? store.values.has(currentReference) : false).toBe(true);
expect(store.values).toHaveLength(1);
db.close();
});
it('duplicates configuration and SMS secrets into a disabled independent task', async () => {
const { db, service, store } = fixture();
const created = await service.create('operator', base);
const duplicate = await service.duplicate('operator', created.id, created.version);
expect(duplicate).toMatchObject({
id: 'task-2',
name: 'Morning SMS copy',
operationType: 'send-sms',
enabled: false,
version: 1,
sms: { configured: true, recipientCount: 2 },
});
expect(store.setCalls).toHaveLength(2);
expect(store.setCalls[1]?.value).toBe(store.setCalls[0]?.value);
db.close();
});
});
@@ -0,0 +1,241 @@
import { randomUUID } from 'node:crypto';
import {
parseCreateScheduledTaskRequest,
parseScheduleTrigger,
parseUpdateScheduledTaskRequest,
scheduleTriggerOf,
type CreateScheduledTaskRequest,
type ScheduleTrigger,
type ScheduledTask,
type ScheduledSmsInput,
} from '@multi-simadmin/contracts';
import type { SecretStore } from '../../infrastructure/secrets/secret-store.js';
import { nextOccurrenceFor, previewTrigger } from './schedule-time.js';
import { ScheduledTaskRepository } from './scheduled-task-repository.js';
export interface ScheduledTaskServiceOptions {
readonly repository: ScheduledTaskRepository;
readonly store: SecretStore;
readonly now?: () => Date;
readonly id?: () => string;
readonly secretSlot?: () => string;
}
export class ScheduledTaskService {
private readonly repository: ScheduledTaskRepository;
private readonly store: SecretStore;
private readonly now: () => Date;
private readonly id: () => string;
private readonly secretSlot: () => string;
constructor(options: ScheduledTaskServiceOptions) {
this.repository = options.repository;
this.store = options.store;
this.now = options.now ?? (() => new Date());
this.id = options.id ?? randomUUID;
this.secretSlot = options.secretSlot ?? randomUUID;
}
async create(actor: string, value: unknown): Promise<ScheduledTask> {
const task = parseCreateScheduledTaskRequest(value);
const now = this.now();
previewTrigger(scheduleTriggerOf(task), 1, now);
const id = this.id();
let smsSecretReference: string | undefined;
if (task.sms) {
smsSecretReference = await this.store.set(
{ instanceId: id, purpose: 'scheduled-sms' },
JSON.stringify(task.sms),
);
}
try {
return this.repository.create({
id,
task,
...(smsSecretReference ? { smsSecretReference } : {}),
createdBy: actor,
now: now.toISOString(),
nextDueAt: this.nextDueAt(task, now),
});
} catch (error) {
if (smsSecretReference) await this.store.delete(smsSecretReference).catch(() => false);
throw error;
}
}
async update(actor: string, id: string, version: number, value: unknown): Promise<ScheduledTask> {
const source = this.updateObject(value);
const change = parseUpdateScheduledTaskRequest({ ...source, version });
const current = this.repository.get(id);
if (!current || current.version !== version)
throw new Error('Scheduled task was not found or version changed');
const oldReference = this.repository.getSmsSecretReference(id);
const operationType = change.operationType ?? current.operationType;
let smsPayload: ScheduledSmsInput | undefined;
if (operationType === 'send-sms') {
if (change.sms === null) throw new TypeError('SMS configuration is required');
if (change.sms) smsPayload = change.sms;
else smsPayload = await this.readSms(oldReference);
} else if (change.sms) {
throw new TypeError('SMS is only valid for send-sms');
}
const merged = parseCreateScheduledTaskRequest({
name: change.name ?? current.name,
operationType,
trigger:
(change.trigger as ScheduleTrigger | undefined) ??
(change.cronExpression
? ({ kind: 'cron', expression: change.cronExpression } as ScheduleTrigger)
: current.trigger),
cronExpression: change.cronExpression ?? current.cronExpression,
timezone: change.timezone ?? current.timezone,
targetSelector: change.targetSelector ?? current.targetSelector,
...(smsPayload ? { sms: smsPayload } : {}),
...(change.delaySeconds === null
? {}
: change.delaySeconds !== undefined
? { delaySeconds: change.delaySeconds }
: current.delaySeconds !== undefined
? { delaySeconds: current.delaySeconds }
: {}),
...(change.effectiveStartAt === null
? {}
: change.effectiveStartAt
? { effectiveStartAt: change.effectiveStartAt }
: current.effectiveStartAt
? { effectiveStartAt: current.effectiveStartAt }
: {}),
...(change.effectiveEndAt === null
? {}
: change.effectiveEndAt
? { effectiveEndAt: change.effectiveEndAt }
: current.effectiveEndAt
? { effectiveEndAt: current.effectiveEndAt }
: {}),
misfirePolicy: change.misfirePolicy ?? current.misfirePolicy,
overlapPolicy: change.overlapPolicy ?? current.overlapPolicy,
retryPolicy: change.retryPolicy ?? current.retryPolicy,
enabled: change.enabled ?? current.enabled,
});
const now = this.now();
previewTrigger(scheduleTriggerOf(merged), 1, now);
let nextReference = operationType === 'send-sms' ? oldReference : undefined;
let wroteReference: string | undefined;
if (operationType === 'send-sms' && change.sms) {
wroteReference = await this.store.set(
{ instanceId: id, purpose: 'scheduled-sms', slot: this.secretSlot() },
JSON.stringify(change.sms),
);
nextReference = wroteReference;
}
if (operationType === 'send-sms' && !nextReference)
throw new Error('Scheduled SMS secret is missing');
try {
const updated = this.repository.update({
id,
version,
task: merged,
...(nextReference ? { smsSecretReference: nextReference } : {}),
updatedBy: actor,
now: now.toISOString(),
nextDueAt: this.nextDueAt(merged, now),
});
if (oldReference && oldReference !== nextReference)
await this.store.delete(oldReference).catch(() => false);
return updated;
} catch (error) {
if (wroteReference) await this.store.delete(wroteReference).catch(() => false);
throw error;
}
}
async duplicate(actor: string, id: string, version: number): Promise<ScheduledTask> {
const current = this.repository.get(id);
if (!current || current.version !== version)
throw new Error('Scheduled task was not found or version changed');
const smsPayload =
current.operationType === 'send-sms'
? await this.readSms(this.repository.getSmsSecretReference(id))
: undefined;
const suffix = ' copy';
return this.create(actor, {
name: `${current.name.slice(0, 120 - suffix.length)}${suffix}`,
operationType: current.operationType,
trigger: current.trigger,
cronExpression: current.cronExpression,
timezone: current.timezone,
targetSelector: current.targetSelector,
...(smsPayload ? { sms: smsPayload } : {}),
...(current.delaySeconds === undefined ? {} : { delaySeconds: current.delaySeconds }),
...(current.effectiveStartAt ? { effectiveStartAt: current.effectiveStartAt } : {}),
...(current.effectiveEndAt ? { effectiveEndAt: current.effectiveEndAt } : {}),
misfirePolicy: current.misfirePolicy,
overlapPolicy: current.overlapPolicy,
retryPolicy: current.retryPolicy,
enabled: false,
});
}
get(id: string): ScheduledTask | null {
return this.repository.get(id);
}
list(): readonly ScheduledTask[] {
return this.repository.list();
}
preview(value: unknown, count = 5, from = this.now()): readonly string[] {
const trigger =
typeof value === 'string'
? ({ kind: 'cron', expression: value } as ScheduleTrigger)
: parseScheduleTrigger(value);
return previewTrigger(trigger, count, from);
}
setEnabled(actor: string, id: string, version: number, enabled: boolean): ScheduledTask {
return this.repository.setEnabled(id, version, enabled, actor, this.now().toISOString());
}
async remove(actor: string, id: string, version: number): Promise<void> {
const reference = this.repository.getSmsSecretReference(id);
this.repository.softDelete(id, version, actor, this.now().toISOString());
if (reference) await this.store.delete(reference).catch(() => false);
}
private updateObject(value: unknown): Record<string, unknown> {
if (typeof value !== 'object' || value === null || Array.isArray(value))
throw new TypeError('Schedule update must be an object');
return value as Record<string, unknown>;
}
private async readSms(reference: string | undefined): Promise<ScheduledSmsInput> {
if (!reference) throw new Error('Scheduled SMS secret is missing');
const value = await this.store.get(reference);
if (!value) throw new Error('Scheduled SMS secret is missing');
try {
const parsed = JSON.parse(value) as unknown;
return parseCreateScheduledTaskRequest({
name: 'SMS validation',
operationType: 'send-sms',
cronExpression: '0 0 * * *',
timezone: 'Asia/Shanghai',
targetSelector: { mode: 'fixed', instanceIds: ['secret-validation'] },
sms: parsed,
}).sms as ScheduledSmsInput;
} catch {
throw new Error('Scheduled SMS secret is invalid');
}
}
private nextDueAt(task: CreateScheduledTaskRequest, now: Date): string {
const anchor = task.effectiveStartAt
? new Date(Math.max(now.getTime(), Date.parse(task.effectiveStartAt)))
: now;
return nextOccurrenceFor(scheduleTriggerOf(task), anchor);
}
}
@@ -0,0 +1,321 @@
import Database from 'better-sqlite3';
import { describe, expect, it, vi } from 'vitest';
import { migrateDatabase } from '../../infrastructure/database/migrations.js';
import { ScheduledTaskRepository } from './scheduled-task-repository.js';
import { SchedulerCoordinator } from './scheduler-coordinator.js';
import type { ScheduledTask } from '@multi-simadmin/contracts';
import type { ResolvedTarget } from './target-resolver.js';
function fixture() {
const db = new Database(':memory:');
db.pragma('foreign_keys = ON');
migrateDatabase(db);
const now = '2026-07-30T01:00:00.000Z';
db.prepare(
'INSERT INTO instances (id,name,base_url,enabled,config_revision,created_at,updated_at) VALUES (?,?,?,?,?,?,?)',
).run('a', 'Alpha', 'http://10.0.0.1', 1, 2, now, now);
db.prepare(
'INSERT INTO instances (id,name,base_url,enabled,config_revision,created_at,updated_at) VALUES (?,?,?,?,?,?,?)',
).run('b', 'Beta', 'http://10.0.0.2', 1, 3, now, now);
const capability = db.prepare(
"INSERT INTO capabilities (instance_id,operation_id,state,observed_at,created_at,updated_at) VALUES (?,'postServiceRestart','supported',?,?,?)",
);
capability.run('a', now, now, now);
capability.run('b', now, now, now);
const repository = new ScheduledTaskRepository(db);
repository.create({
id: 'task-1',
task: {
name: 'Restart lab',
operationType: 'restart-service',
cronExpression: '0 9 * * *',
timezone: 'Asia/Shanghai',
targetSelector: { mode: 'fixed', instanceIds: ['a', 'b'] },
misfirePolicy: 'skip',
overlapPolicy: 'skip',
retryPolicy: { maxRetries: 0, intervalSeconds: 60 },
enabled: true,
},
createdBy: 'operator',
now,
nextDueAt: now,
});
return { db, repository };
}
describe('SchedulerCoordinator', () => {
it('resolves multiple targets, persists the snapshot, and finishes a manual run', async () => {
const { db, repository } = fixture();
const dispatch = vi.fn<
(
task: ScheduledTask,
targets: readonly ResolvedTarget[],
) => Promise<{ outcome: 'succeeded'; jobIds: string[] }>
>(async () => ({
outcome: 'succeeded',
jobIds: ['job-a', 'job-b'],
}));
const coordinator = new SchedulerCoordinator({
db,
repository,
dispatch,
now: () => new Date('2026-07-30T01:00:00.000Z'),
id: () => 'run-1',
});
const run = await coordinator.runNow('task-1', 'operator', 'request-1');
expect(run).toMatchObject({
id: 'run-1',
triggerSource: 'manual',
targetSnapshot: ['a', 'b'],
outcome: 'succeeded',
jobIds: ['job-a', 'job-b'],
});
expect(dispatch.mock.calls[0]?.[1]).toEqual([
{ id: 'a', revision: 2 },
{ id: 'b', revision: 3 },
]);
db.close();
});
it('does not manually dispatch when a selected target lacks the required capability', async () => {
const { db, repository } = fixture();
db.prepare(
"UPDATE capabilities SET state = 'unsupported' WHERE instance_id = 'b' AND operation_id = 'postServiceRestart'",
).run();
const dispatch = vi.fn();
const coordinator = new SchedulerCoordinator({
db,
repository,
dispatch,
now: () => new Date('2026-07-30T01:00:00.000Z'),
id: () => 'manual-capability-run',
});
const run = await coordinator.runNow('task-1', 'operator', 'request-1');
expect(run).toMatchObject({ outcome: 'needs-attention', targetSnapshot: ['a', 'b'] });
expect(dispatch).not.toHaveBeenCalled();
db.close();
});
it('records no-targets without dispatching and prevents duplicate scheduled claims', async () => {
const { db, repository } = fixture();
db.prepare('UPDATE instances SET enabled = 0').run();
const dispatch = vi.fn();
const coordinator = new SchedulerCoordinator({
db,
repository,
dispatch,
now: () => new Date('2026-07-30T01:00:00.000Z'),
id: () => 'run-1',
});
const first = await coordinator.tick();
const second = await coordinator.tick();
expect(first).toHaveLength(1);
expect(first[0]).toMatchObject({ outcome: 'no-targets' });
expect(second).toHaveLength(0);
expect(dispatch).not.toHaveBeenCalled();
db.close();
});
it('records needs-attention when a matched target loses its required capability', async () => {
const { db, repository } = fixture();
db.prepare(
"UPDATE capabilities SET state = 'unsupported' WHERE instance_id = 'b' AND operation_id = 'postServiceRestart'",
).run();
const dispatch = vi.fn();
const coordinator = new SchedulerCoordinator({
db,
repository,
dispatch,
now: () => new Date('2026-07-30T01:00:00.000Z'),
id: () => 'capability-run',
});
const runs = await coordinator.tick();
expect(runs).toHaveLength(1);
expect(runs[0]).toMatchObject({
outcome: 'needs-attention',
targetSnapshot: ['a', 'b'],
});
expect(runs[0]?.reason).toContain('b');
expect(dispatch).not.toHaveBeenCalled();
db.close();
});
it('queues one overlapping occurrence, skips a second, and dispatches the queued run later', async () => {
const { db, repository } = fixture();
db.prepare("UPDATE scheduled_tasks SET overlap_policy = 'queue-once'").run();
const task = repository.get('task-1')!;
const taskSnapshot = {
name: task.name,
operationType: task.operationType,
cronExpression: task.cronExpression,
timezone: task.timezone,
targetSelector: task.targetSelector,
misfirePolicy: task.misfirePolicy,
overlapPolicy: task.overlapPolicy,
retryPolicy: task.retryPolicy,
enabled: task.enabled,
};
repository.claimOccurrence({
id: 'active-run',
scheduledTaskId: task.id,
scheduleVersion: task.version,
dueAt: '2026-07-30T00:00:00.000Z',
claimedAt: '2026-07-30T00:00:00.000Z',
triggerSource: 'manual',
targetSnapshot: ['a'],
taskSnapshot,
});
repository.startRun('active-run', '2026-07-30T00:00:00.000Z');
let now = new Date('2026-07-30T01:00:00.000Z');
let id = 0;
const dispatch = vi.fn(async () => ({ outcome: 'succeeded' as const, jobIds: ['job-1'] }));
const coordinator = new SchedulerCoordinator({
db,
repository,
dispatch,
now: () => now,
id: () => `queued-${++id}`,
});
expect(await coordinator.tick()).toHaveLength(0);
expect(repository.getQueuedRun('task-1')).toMatchObject({
id: 'queued-1',
targetSnapshot: ['a', 'b'],
});
now = new Date('2026-07-31T01:00:00.000Z');
const queueFull = await coordinator.tick();
expect(queueFull).toHaveLength(1);
expect(queueFull[0]).toMatchObject({ outcome: 'skipped', reason: 'Overlap queue is full' });
expect(dispatch).not.toHaveBeenCalled();
repository.finishRun('active-run', {
outcome: 'succeeded',
jobIds: [],
finishedAt: now.toISOString(),
});
const completed = await coordinator.tick();
expect(completed[0]).toMatchObject({ id: 'queued-1', outcome: 'succeeded' });
expect(dispatch).toHaveBeenCalledTimes(1);
db.close();
});
it('records and advances an occurrence outside the effective window', async () => {
const { db, repository } = fixture();
db.prepare("UPDATE scheduled_tasks SET effective_end_at = '2026-07-29T23:59:00.000Z'").run();
const dispatch = vi.fn();
const coordinator = new SchedulerCoordinator({
db,
repository,
dispatch,
now: () => new Date('2026-07-30T01:00:00.000Z'),
id: () => 'window-skip',
});
const runs = await coordinator.tick();
expect(runs).toHaveLength(1);
expect(runs[0]).toMatchObject({
outcome: 'skipped',
reason: 'Occurrence is outside the effective window',
});
expect(repository.get('task-1')?.nextDueAt).toBe('2026-07-31T01:00:00.000Z');
expect(dispatch).not.toHaveBeenCalled();
db.close();
});
it('does not advance a due occurrence when its claim cannot be persisted', async () => {
const { db, repository } = fixture();
vi.spyOn(repository, 'claimOccurrence').mockImplementationOnce(() => {
throw new Error('claim failed');
});
const coordinator = new SchedulerCoordinator({
db,
repository,
dispatch: vi.fn(),
now: () => new Date('2026-07-30T01:00:00.000Z'),
id: () => 'failed-claim',
});
await expect(coordinator.tick()).rejects.toThrow('claim failed');
expect(repository.get('task-1')?.nextDueAt).toBe('2026-07-30T01:00:00.000Z');
db.close();
});
it('executes at most the queued occurrence for a task during one tick', async () => {
const { db, repository } = fixture();
const task = repository.get('task-1')!;
repository.claimOccurrence({
id: 'queued-run',
scheduledTaskId: task.id,
scheduleVersion: task.version,
dueAt: '2026-07-29T01:00:00.000Z',
claimedAt: '2026-07-29T01:00:00.000Z',
triggerSource: 'scheduled',
targetSnapshot: ['a'],
taskSnapshot: {
name: task.name,
operationType: task.operationType,
cronExpression: task.cronExpression,
timezone: task.timezone,
targetSelector: task.targetSelector,
misfirePolicy: task.misfirePolicy,
overlapPolicy: task.overlapPolicy,
retryPolicy: task.retryPolicy,
enabled: task.enabled,
},
});
let id = 0;
const dispatch = vi.fn(async () => ({ outcome: 'succeeded' as const, jobIds: ['job-1'] }));
const coordinator = new SchedulerCoordinator({
db,
repository,
dispatch,
now: () => new Date('2026-07-30T01:00:00.000Z'),
id: () => `run-${++id}`,
});
const completed = await coordinator.tick();
expect(completed).toHaveLength(1);
expect(completed[0]?.id).toBe('queued-run');
expect(dispatch).toHaveBeenCalledTimes(1);
expect(repository.get('task-1')?.nextDueAt).toBe('2026-07-30T01:00:00.000Z');
db.close();
});
it('does not reschedule after stop while a tick is still running', async () => {
vi.useFakeTimers();
const { db, repository } = fixture();
let finishDispatch!: (value: { outcome: 'succeeded'; jobIds: string[] }) => void;
const dispatch = vi.fn(
() =>
new Promise<{ outcome: 'succeeded'; jobIds: string[] }>((resolve) => {
finishDispatch = resolve;
}),
);
const coordinator = new SchedulerCoordinator({
db,
repository,
dispatch,
now: () => new Date('2026-07-30T01:00:00.000Z'),
id: () => 'in-flight-run',
intervalMs: 1_000,
});
try {
coordinator.start();
expect(dispatch).toHaveBeenCalledTimes(1);
coordinator.stop();
finishDispatch({ outcome: 'succeeded', jobIds: ['job-1'] });
await vi.advanceTimersByTimeAsync(0);
expect(vi.getTimerCount()).toBe(0);
} finally {
coordinator.stop();
vi.useRealTimers();
db.close();
}
});
});
@@ -0,0 +1,329 @@
import { randomUUID } from 'node:crypto';
import type {
CreateScheduledTaskRequest,
ScheduledRun,
ScheduledRunOutcome,
ScheduledTask,
} from '@multi-simadmin/contracts';
import { scheduleTriggerOf } from '@multi-simadmin/contracts';
import type { SqliteDatabase } from '../../infrastructure/database/database.js';
import { nextOccurrenceFor, reconcileOccurrence } from './schedule-time.js';
import { ScheduledTaskRepository } from './scheduled-task-repository.js';
import { resolveOperationTargets, type ResolvedTarget } from './target-resolver.js';
export interface DispatchResult {
readonly outcome: ScheduledRunOutcome;
readonly reason?: string;
readonly jobIds: readonly string[];
}
export type ScheduledDispatch = (
task: ScheduledTask,
targets: readonly ResolvedTarget[],
run: ScheduledRun,
context: { readonly actor: string; readonly requestId: string },
) => Promise<DispatchResult>;
export interface SchedulerCoordinatorOptions {
readonly db: SqliteDatabase;
readonly repository: ScheduledTaskRepository;
readonly dispatch: ScheduledDispatch;
readonly now?: () => Date;
readonly id?: () => string;
readonly intervalMs?: number;
}
function snapshot(task: ScheduledTask): CreateScheduledTaskRequest {
return {
name: task.name,
operationType: task.operationType,
trigger: scheduleTriggerOf(task),
cronExpression: task.cronExpression,
timezone: task.timezone,
targetSelector: task.targetSelector,
...(task.delaySeconds === undefined ? {} : { delaySeconds: task.delaySeconds }),
...(task.effectiveStartAt ? { effectiveStartAt: task.effectiveStartAt } : {}),
...(task.effectiveEndAt ? { effectiveEndAt: task.effectiveEndAt } : {}),
misfirePolicy: task.misfirePolicy,
overlapPolicy: task.overlapPolicy,
retryPolicy: task.retryPolicy,
enabled: task.enabled,
};
}
export class SchedulerCoordinator {
private readonly now: () => Date;
private readonly id: () => string;
private readonly intervalMs: number;
private timer: ReturnType<typeof setTimeout> | undefined;
private started = false;
private lifecycle = 0;
constructor(private readonly options: SchedulerCoordinatorOptions) {
this.now = options.now ?? (() => new Date());
this.id = options.id ?? randomUUID;
this.intervalMs = options.intervalMs ?? 30_000;
}
start(): void {
if (this.started) return;
this.started = true;
const lifecycle = ++this.lifecycle;
const schedule = () => {
if (!this.started || lifecycle !== this.lifecycle) return;
this.timer = setTimeout(() => {
this.timer = undefined;
void this.tick().finally(schedule);
}, this.intervalMs);
this.timer.unref?.();
};
void this.tick().finally(schedule);
}
stop(): void {
this.started = false;
this.lifecycle += 1;
if (this.timer) clearTimeout(this.timer);
this.timer = undefined;
}
async runNow(taskId: string, actor: string, requestId: string): Promise<ScheduledRun> {
const task = this.options.repository.get(taskId);
if (!task) throw new Error('Scheduled task was not found');
return this.runOccurrence(task, this.now().toISOString(), 'manual', actor, requestId);
}
async tick(): Promise<readonly ScheduledRun[]> {
const now = this.now();
const nowIso = now.toISOString();
const completed: ScheduledRun[] = [];
for (const task of this.options.repository.list()) {
const queued = this.options.repository.getQueuedRun(task.id);
if (queued && !this.options.repository.hasActiveRun(task.id)) {
if (queued.scheduleVersion !== task.version) {
completed.push(
this.options.repository.finishRun(queued.id, {
outcome: 'needs-attention',
reason: 'Queued occurrence belongs to an outdated schedule version',
jobIds: [],
finishedAt: nowIso,
}),
);
} else if (!task.enabled) {
completed.push(
this.options.repository.finishRun(queued.id, {
outcome: 'skipped',
reason: 'Schedule was disabled while the occurrence was queued',
jobIds: [],
finishedAt: nowIso,
}),
);
} else {
const resolution = resolveOperationTargets(
this.options.db,
{ mode: 'fixed', instanceIds: queued.targetSnapshot },
task.operationType,
);
if (resolution.unavailableInstanceIds.length > 0) {
completed.push(
this.options.repository.finishRun(queued.id, {
outcome: 'needs-attention',
reason: this.capabilityReason(resolution.unavailableInstanceIds),
jobIds: [],
finishedAt: nowIso,
}),
);
continue;
}
completed.push(
await this.executeClaimed(
task,
queued,
resolution.targets,
'scheduled-automation',
`queue:${queued.id}`,
),
);
}
continue;
}
if (!task.enabled || !task.nextDueAt || task.nextDueAt > nowIso) continue;
const outsideWindow =
(task.effectiveStartAt !== undefined && nowIso < task.effectiveStartAt) ||
(task.effectiveEndAt !== undefined && nowIso > task.effectiveEndAt);
if (outsideWindow) {
const anchor =
task.effectiveStartAt && nowIso < task.effectiveStartAt
? new Date(task.effectiveStartAt)
: now;
const nextDueAt = nextOccurrenceFor(scheduleTriggerOf(task), anchor);
const claimed = this.options.repository.claimScheduledOccurrence({
id: this.id(),
scheduledTaskId: task.id,
scheduleVersion: task.version,
dueAt: task.nextDueAt,
nextDueAt,
claimedAt: nowIso,
taskSnapshot: snapshot(task),
overlapPolicy: task.overlapPolicy,
resolveTargets: () => ({ targets: [] }),
terminal: {
outcome: 'skipped',
reason: 'Occurrence is outside the effective window',
},
});
if (claimed) completed.push(claimed.run);
continue;
}
const exact = task.nextDueAt === nowIso;
const reconciled = exact
? {
action: 'run' as const,
dueAt: task.nextDueAt,
nextDueAt: nextOccurrenceFor(scheduleTriggerOf(task), now, new Date(task.nextDueAt)),
}
: reconcileOccurrence(
{
trigger: task.trigger,
nextDueAt: task.nextDueAt,
misfirePolicy: task.misfirePolicy,
},
now,
);
if (reconciled.action === 'wait') continue;
if (reconciled.action === 'skip') {
const claimed = this.options.repository.claimScheduledOccurrence({
id: this.id(),
scheduledTaskId: task.id,
scheduleVersion: task.version,
dueAt: reconciled.dueAt,
nextDueAt: reconciled.nextDueAt,
claimedAt: nowIso,
taskSnapshot: snapshot(task),
overlapPolicy: task.overlapPolicy,
resolveTargets: () => ({ targets: [] }),
terminal: {
outcome: 'skipped',
reason: 'Missed occurrence skipped by task policy',
},
});
if (claimed) completed.push(claimed.run);
continue;
}
const claimed = this.options.repository.claimScheduledOccurrence({
id: this.id(),
scheduledTaskId: task.id,
scheduleVersion: task.version,
dueAt: reconciled.dueAt,
nextDueAt: reconciled.nextDueAt,
claimedAt: nowIso,
taskSnapshot: snapshot(task),
overlapPolicy: task.overlapPolicy,
resolveTargets: () => {
const resolution = resolveOperationTargets(
this.options.db,
task.targetSelector,
task.operationType,
);
return {
targets: resolution.targets,
targetSnapshot: resolution.targetSnapshot,
...(resolution.unavailableInstanceIds.length > 0
? {
attentionReason: this.capabilityReason(resolution.unavailableInstanceIds),
}
: {}),
};
},
});
if (!claimed || claimed.disposition === 'queued') continue;
if (claimed.disposition === 'finished') {
completed.push(claimed.run);
continue;
}
completed.push(
await this.executeClaimed(
task,
claimed.run,
claimed.targets,
'scheduled-automation',
`schedule:${task.id}:${reconciled.dueAt}`,
),
);
}
return completed;
}
private async runOccurrence(
task: ScheduledTask,
dueAt: string,
triggerSource: 'scheduled' | 'manual',
actor: string,
requestId: string,
): Promise<ScheduledRun> {
const resolution = resolveOperationTargets(
this.options.db,
task.targetSelector,
task.operationType,
);
const claimed = this.options.repository.claimOccurrence({
id: this.id(),
scheduledTaskId: task.id,
scheduleVersion: task.version,
dueAt,
claimedAt: this.now().toISOString(),
triggerSource,
targetSnapshot: resolution.targetSnapshot,
taskSnapshot: snapshot(task),
});
if (!claimed) throw new Error('Scheduled occurrence was already claimed');
if (resolution.unavailableInstanceIds.length > 0)
return this.options.repository.finishRun(claimed.id, {
outcome: 'needs-attention',
reason: this.capabilityReason(resolution.unavailableInstanceIds),
jobIds: [],
finishedAt: this.now().toISOString(),
});
return this.executeClaimed(task, claimed, resolution.targets, actor, requestId);
}
private capabilityReason(instanceIds: readonly string[]): string {
return `Required capability is unavailable for: ${instanceIds.join(', ')}`;
}
private async executeClaimed(
task: ScheduledTask,
claimed: ScheduledRun,
targets: readonly ResolvedTarget[],
actor: string,
requestId: string,
): Promise<ScheduledRun> {
const now = this.now().toISOString();
if (targets.length === 0)
return this.options.repository.finishRun(claimed.id, {
outcome: 'no-targets',
reason: 'No enabled instances matched the current selector',
jobIds: [],
finishedAt: now,
});
this.options.repository.startRun(claimed.id, now);
try {
const result = await this.options.dispatch(task, targets, claimed, { actor, requestId });
return this.options.repository.finishRun(claimed.id, {
outcome: result.outcome,
...(result.reason ? { reason: result.reason } : {}),
jobIds: result.jobIds,
finishedAt: this.now().toISOString(),
});
} catch {
return this.options.repository.finishRun(claimed.id, {
outcome: 'failed',
reason: 'Scheduled dispatch failed',
jobIds: [],
finishedAt: this.now().toISOString(),
});
}
}
}
@@ -0,0 +1,128 @@
import Database from 'better-sqlite3';
import { describe, expect, it } from 'vitest';
import { migrateDatabase } from '../../infrastructure/database/migrations.js';
import { resolveOperationTargets, resolveTargets } from './target-resolver.js';
function fixture() {
const db = new Database(':memory:');
db.pragma('foreign_keys = ON');
migrateDatabase(db);
const now = '2026-07-30T00:00:00.000Z';
const insert = db.prepare(
'INSERT INTO instances (id,name,base_url,enabled,config_revision,created_at,updated_at) VALUES (?,?,?,?,?,?,?)',
);
insert.run('a', 'Alpha', 'http://10.0.0.1', 1, 3, now, now);
insert.run('b', 'Beta', 'http://10.0.0.2', 1, 2, now, now);
insert.run('c', 'Disabled', 'http://10.0.0.3', 0, 1, now, now);
const tag = db.prepare('INSERT INTO instance_tags (instance_id,tag,created_at) VALUES (?,?,?)');
tag.run('a', 'lab', now);
tag.run('a', 'east', now);
tag.run('b', 'lab', now);
tag.run('c', 'east', now);
return db;
}
describe('resolveTargets', () => {
it('resolves enabled fixed targets in selector order with current revisions', () => {
const db = fixture();
expect(resolveTargets(db, { mode: 'fixed', instanceIds: ['b', 'c', 'a'] })).toEqual([
{ id: 'b', revision: 2 },
{ id: 'a', revision: 3 },
]);
db.close();
});
it('resolves dynamic all and any tag matches on every call', () => {
const db = fixture();
expect(resolveTargets(db, { mode: 'tags', match: 'all', tags: ['lab', 'east'] })).toEqual([
{ id: 'a', revision: 3 },
]);
expect(resolveTargets(db, { mode: 'tags', match: 'any', tags: ['lab', 'east'] })).toEqual([
{ id: 'a', revision: 3 },
{ id: 'b', revision: 2 },
]);
db.prepare(
'INSERT INTO instances (id,name,base_url,enabled,config_revision,created_at,updated_at) VALUES (?,?,?,?,?,?,?)',
).run(
'd',
'Dynamic',
'http://10.0.0.4',
1,
1,
'2026-07-30T00:00:00.000Z',
'2026-07-30T00:00:00.000Z',
);
db.prepare('INSERT INTO instance_tags (instance_id,tag,created_at) VALUES (?,?,?)').run(
'd',
'east',
'2026-07-30T00:00:00.000Z',
);
expect(
resolveTargets(db, { mode: 'tags', match: 'any', tags: ['east'] }).map((x) => x.id),
).toEqual(['a', 'd']);
db.close();
});
it('resolves the Hub all-devices and group selectors', () => {
const db = fixture();
const now = '2026-07-30T00:00:00.000Z';
db.prepare(
'INSERT INTO device_groups (id,name,description,created_at,updated_at) VALUES (?,?,?,?,?)',
).run('g-lab', '实验室', '', now, now);
db.prepare('UPDATE instances SET group_id = ? WHERE id IN (?,?)').run('g-lab', 'a', 'c');
expect(resolveTargets(db, { mode: 'all' })).toEqual([
{ id: 'a', revision: 3 },
{ id: 'b', revision: 2 },
]);
expect(resolveTargets(db, { mode: 'group', groupId: 'g-lab' })).toEqual([
{ id: 'a', revision: 3 },
]);
expect(resolveTargets(db, { mode: 'groups', groupIds: ['g-lab', 'g-office'] })).toEqual([
{ id: 'a', revision: 3 },
]);
db.close();
});
it('dispatches control-plane backups without resolving device targets', () => {
const db = fixture();
expect(resolveOperationTargets(db, { mode: 'all' }, 'backup-data')).toEqual({
targets: [{ id: 'control-plane', revision: 1 }],
targetSnapshot: [],
unavailableInstanceIds: [],
});
db.close();
});
it('dispatches devices whose capability was never probed', () => {
const db = fixture();
expect(resolveOperationTargets(db, { mode: 'all' }, 'restart-baseband')).toEqual({
targets: [
{ id: 'a', revision: 3 },
{ id: 'b', revision: 2 },
],
targetSnapshot: ['a', 'b'],
unavailableInstanceIds: [],
});
db.close();
});
it('separates targets whose required operation capability is unavailable', () => {
const db = fixture();
const now = '2026-07-30T00:00:00.000Z';
db.prepare(
'INSERT INTO capabilities (instance_id,operation_id,state,observed_at,created_at,updated_at) VALUES (?,?,?,?,?,?)',
).run('a', 'postServiceRestart', 'supported', now, now, now);
db.prepare(
'INSERT INTO capabilities (instance_id,operation_id,state,observed_at,created_at,updated_at) VALUES (?,?,?,?,?,?)',
).run('b', 'postServiceRestart', 'unsupported', now, now, now);
expect(
resolveOperationTargets(db, { mode: 'fixed', instanceIds: ['a', 'b'] }, 'restart-service'),
).toEqual({
targets: [{ id: 'a', revision: 3 }],
targetSnapshot: ['a', 'b'],
unavailableInstanceIds: ['b'],
});
db.close();
});
});
@@ -0,0 +1,129 @@
import type { ScheduledOperationType, ScheduleTargetSelector } from '@multi-simadmin/contracts';
import type { SqliteDatabase } from '../../infrastructure/database/database.js';
export interface ResolvedTarget {
readonly id: string;
readonly revision: number;
}
interface TargetRow {
id: string;
config_revision: number;
}
export interface OperationTargetResolution {
readonly targets: readonly ResolvedTarget[];
readonly targetSnapshot: readonly string[];
readonly unavailableInstanceIds: readonly string[];
}
/** Sentinel target for control-plane-local actions; it is never an instance id. */
export const LOCAL_TARGET: ResolvedTarget = { id: 'control-plane', revision: 1 };
const operationIds: Record<ScheduledOperationType, string> = {
'restart-service': 'postServiceRestart',
'reboot-system': 'postSystemReboot',
'restart-baseband': 'postBasebandRestart',
'send-sms': 'postSmsSend',
'backup-data': 'localBackup',
};
export function resolveTargets(
db: SqliteDatabase,
selector: ScheduleTargetSelector,
): readonly ResolvedTarget[] {
if (selector.mode === 'fixed') {
const placeholders = selector.instanceIds.map(() => '?').join(',');
const rows = db
.prepare(
`SELECT id, config_revision FROM instances
WHERE enabled = 1 AND id IN (${placeholders})`,
)
.all(...selector.instanceIds) as TargetRow[];
const byId = new Map(rows.map((row) => [row.id, row]));
return selector.instanceIds.flatMap((id) => {
const row = byId.get(id);
return row ? [{ id: row.id, revision: row.config_revision }] : [];
});
}
if (selector.mode === 'all') {
const rows = db
.prepare('SELECT id, config_revision FROM instances WHERE enabled = 1 ORDER BY id ASC')
.all() as TargetRow[];
return rows.map((row) => ({ id: row.id, revision: row.config_revision }));
}
if (selector.mode === 'group') {
const rows = db
.prepare(
'SELECT id, config_revision FROM instances WHERE enabled = 1 AND group_id = ? ORDER BY id ASC',
)
.all(selector.groupId) as TargetRow[];
return rows.map((row) => ({ id: row.id, revision: row.config_revision }));
}
if (selector.mode === 'groups') {
const placeholders = selector.groupIds.map(() => '?').join(',');
const rows = db
.prepare(
`SELECT id, config_revision FROM instances
WHERE enabled = 1 AND group_id IN (${placeholders})
ORDER BY id ASC`,
)
.all(...selector.groupIds) as TargetRow[];
return rows.map((row) => ({ id: row.id, revision: row.config_revision }));
}
const placeholders = selector.tags.map(() => '?').join(',');
const comparison = selector.match === 'all' ? '= ?' : '> 0';
const parameters: Array<string | number> = [...selector.tags];
if (selector.match === 'all') parameters.push(selector.tags.length);
const rows = db
.prepare(
`SELECT i.id, i.config_revision
FROM instances i
JOIN instance_tags t ON t.instance_id = i.id
WHERE i.enabled = 1 AND t.tag IN (${placeholders})
GROUP BY i.id, i.config_revision
HAVING COUNT(DISTINCT t.tag) ${comparison}
ORDER BY i.id ASC`,
)
.all(...parameters) as TargetRow[];
return rows.map((row) => ({ id: row.id, revision: row.config_revision }));
}
export function resolveOperationTargets(
db: SqliteDatabase,
selector: ScheduleTargetSelector,
operationType: ScheduledOperationType,
): OperationTargetResolution {
if (operationType === 'backup-data')
return { targets: [LOCAL_TARGET], targetSnapshot: [], unavailableInstanceIds: [] };
const candidates = resolveTargets(db, selector);
if (candidates.length === 0)
return { targets: [], targetSnapshot: [], unavailableInstanceIds: [] };
const placeholders = candidates.map(() => '?').join(',');
const rows = db
.prepare(
`SELECT instance_id, state FROM capabilities
WHERE operation_id = ? AND instance_id IN (${placeholders})`,
)
.all(operationIds[operationType], ...candidates.map((target) => target.id)) as Array<{
instance_id: string;
state: string;
}>;
const states = new Map(rows.map((row) => [row.instance_id, row.state]));
// Only a probed "unsupported" verdict blocks a device. Capability rows are written by active
// probes, so an absent row means "never observed", which must dispatch optimistically and let
// the job result report the truth; otherwise every scheduled task stalls on missing evidence.
const isBlocked = (target: ResolvedTarget): boolean => states.get(target.id) === 'unsupported';
return {
targets: candidates.filter((target) => !isBlocked(target)),
targetSnapshot: candidates.map((target) => target.id),
unavailableInstanceIds: candidates
.filter((target) => isBlocked(target))
.map((target) => target.id),
};
}
@@ -5,8 +5,10 @@ import { migrateDatabase } from '../../infrastructure/database/migrations.js';
import type { SecretStore } from '../../infrastructure/secrets/secret-store.js';
import type { TransportResponse } from '../../infrastructure/transport/safe-instance-transport.js';
import { ConnectionProbe } from './connection-probe.js';
import { ConnectionLogService } from '../system/connection-log-service.js';
class Store implements SecretStore {
readonly provider = 'macos-keychain';
async set(key: { instanceId: string; purpose: string; slot?: string }) {
return `keychain://multi-simadmin/${Buffer.from(JSON.stringify([key.instanceId, key.purpose, key.slot])).toString('base64url')}`;
}
@@ -21,7 +23,7 @@ const dbs: Database.Database[] = [];
afterEach(() => {
for (const db of dbs.splice(0)) db.close();
});
const fixture = (response: TransportResponse) => {
const fixture = (response: TransportResponse, options: { logs?: boolean } = {}) => {
const db = new Database(':memory:');
db.pragma('foreign_keys=ON');
migrateDatabase(db);
@@ -32,6 +34,7 @@ const fixture = (response: TransportResponse) => {
idFactory: () => 'instance-1',
now: () => new Date('2026-07-16T12:00:00.000Z'),
});
const connectionLogs = options.logs ? new ConnectionLogService({ db }) : undefined;
const transport = {
get: async (url: string) => {
expect(url).toBe('http://192.168.1.20:3000/api/health');
@@ -40,12 +43,14 @@ const fixture = (response: TransportResponse) => {
};
return {
db,
connectionLogs,
instances,
probe: new ConnectionProbe({
db,
instances,
transport,
now: () => new Date('2026-07-16T12:00:00.000Z'),
...(connectionLogs ? { connectionLogs } : {}),
}),
};
};
@@ -92,4 +97,116 @@ describe('ConnectionProbe', () => {
const { probe } = fixture({ status: 200, headers: {}, body: '{}' });
await expect(probe.test('missing')).rejects.toMatchObject({ code: 'NOT_FOUND' });
});
it('journals a successful probe and marks an unauthenticated one as stale', async () => {
const { db, instances, probe } = fixture(
{ status: 401, headers: {}, body: '{}' },
{ logs: true },
);
await instances.create({ name: 'LAN', origin: 'http://192.168.1.20:3000' });
await probe.test('instance-1');
const rows = db
.prepare('SELECT outcome, state, error_code, http_status, duration_ms FROM connection_logs')
.all() as {
outcome: string;
state: string;
error_code: string | null;
http_status: number;
duration_ms: number;
}[];
expect(rows).toEqual([
{
outcome: 'stale',
state: 'stale',
error_code: 'HTTP_401',
http_status: 401,
duration_ms: 0,
},
]);
});
it('journals a transport failure before rethrowing it', async () => {
const db = new Database(':memory:');
db.pragma('foreign_keys=ON');
migrateDatabase(db);
dbs.push(db);
const instances = new InstanceService({
db,
store: new Store(),
idFactory: () => 'instance-1',
now: () => new Date('2026-07-16T12:00:00.000Z'),
});
await instances.create({ name: 'LAN', origin: 'http://192.168.1.20:3000' });
const connectionLogs = new ConnectionLogService({ db });
const probe = new ConnectionProbe({
db,
instances,
connectionLogs,
transport: {
get: async () => {
throw Object.assign(new Error('boom'), { code: 'ECONNREFUSED' });
},
},
});
await expect(probe.test('instance-1')).rejects.toThrow('boom');
expect(connectionLogs.list().items[0]).toMatchObject({
outcome: 'failed',
state: 'unknown',
errorCode: 'ECONNREFUSED',
httpStatus: null,
});
});
it('honours a configured snapshot lifetime so the offline window drives expiry', async () => {
const db = new Database(':memory:');
db.pragma('foreign_keys=ON');
migrateDatabase(db);
dbs.push(db);
const instances = new InstanceService({
db,
store: new Store(),
idFactory: () => 'instance-1',
now: () => new Date('2026-07-16T12:00:00.000Z'),
});
await instances.create({ name: 'LAN', origin: 'http://192.168.1.20:3000' });
let clock = new Date('2026-07-16T12:00:00.000Z');
const probe = new ConnectionProbe({
db,
instances,
transport: { get: async () => ({ status: 200, headers: {}, body: '{}' }) },
now: () => clock,
snapshotTtlMs: () => 120_000,
});
await probe.test('instance-1');
expect(probe.reachability().get('instance-1')).toEqual({
reachable: true,
authenticated: true,
checkedAt: '2026-07-16T12:00:00.000Z',
});
clock = new Date('2026-07-16T12:01:30.000Z');
expect(probe.reachability().get('instance-1')).toMatchObject({ reachable: true });
clock = new Date('2026-07-16T12:02:01.000Z');
expect(probe.reachability().get('instance-1')).toEqual({
reachable: false,
authenticated: false,
checkedAt: '2026-07-16T12:00:00.000Z',
});
});
it('reports an expired authenticated snapshot as offline and never as authenticated', async () => {
const { db, instances, probe } = fixture({ status: 200, headers: {}, body: '{}' });
await instances.create({ name: 'LAN', origin: 'http://192.168.1.20:3000' });
expect(probe.reachability().size).toBe(0);
await probe.test('instance-1');
db.prepare(
`UPDATE status_snapshots SET expires_at = '2020-01-01T00:00:00.000Z' WHERE instance_id = 'instance-1'`,
).run();
expect(probe.reachability().get('instance-1')).toEqual({
reachable: false,
authenticated: false,
checkedAt: '2026-07-16T12:00:00.000Z',
});
});
});
@@ -3,6 +3,7 @@ import type Database from 'better-sqlite3';
import type { SessionMetadata } from '@multi-simadmin/contracts';
import { InstanceService, InstanceServiceError } from '../instances/instance-service.js';
import type { TransportResponse } from '../../infrastructure/transport/safe-instance-transport.js';
import type { ConnectionLogService } from '../system/connection-log-service.js';
export interface ConnectionTransport {
readonly get: (url: string) => Promise<TransportResponse>;
@@ -12,21 +13,59 @@ export interface ConnectionProbeOptions {
readonly instances: InstanceService;
readonly transport: ConnectionTransport;
readonly now?: () => Date;
/** Optional journal of every reachability attempt, used by the log centre. */
readonly connectionLogs?: ConnectionLogService;
/**
* How long a probe stays authoritative. Defaults to one heartbeat period; the fleet heartbeat
* passes the configured offline window so a lost probe is tolerated exactly as long as the
* operator asked for.
*/
readonly snapshotTtlMs?: number | (() => number);
}
const CONNECTION_SNAPSHOT_TTL_MS = 30_000;
/** Last known reachability for one instance, read from the snapshot journal without probing. */
export interface ConnectionState {
readonly reachable: boolean;
readonly authenticated: boolean;
readonly checkedAt: string | null;
}
interface SnapshotRow {
readonly instance_id: string;
readonly payload_json: string | null;
readonly observed_at: string;
readonly expires_at: string | null;
}
export class ConnectionProbe {
private readonly now: () => Date;
constructor(private readonly options: ConnectionProbeOptions) {
this.now = options.now ?? (() => new Date());
}
#ttlMs(): number {
const value = this.options.snapshotTtlMs;
const resolved = typeof value === 'function' ? value() : value;
return Number.isFinite(resolved) && (resolved ?? 0) > 0
? (resolved as number)
: CONNECTION_SNAPSHOT_TTL_MS;
}
async test(instanceId: string): Promise<SessionMetadata> {
const instance = await this.options.instances.get(instanceId);
if (!instance) throw new InstanceServiceError('NOT_FOUND', 'Instance was not found');
const response = await this.options.transport.get(`${instance.origin}/api/health`);
const startedAt = this.now().getTime();
let response: TransportResponse;
try {
response = await this.options.transport.get(`${instance.origin}/api/health`);
} catch (error) {
this.#record(instanceId, null, 'failed', startedAt, errorCode(error));
throw error;
}
const observed = this.now();
const observedAt = observed.toISOString();
const expiresAt = new Date(observed.getTime() + CONNECTION_SNAPSHOT_TTL_MS).toISOString();
const expiresAt = new Date(observed.getTime() + this.#ttlMs()).toISOString();
const authenticated = response.status >= 200 && response.status < 300;
this.options.db
.prepare(
@@ -48,6 +87,68 @@ export class ConnectionProbe {
expiresAt,
observedAt,
);
this.#record(
instanceId,
response.status,
authenticated ? 'success' : 'stale',
startedAt,
authenticated ? null : `HTTP_${response.status}`,
);
return { instanceId, authenticated, checkedAt: observedAt };
}
/**
* Reachability for every instance that has ever been probed. Anything missing, or whose
* snapshot outlived the offline window, is reported as unreachable.
*/
reachability(): ReadonlyMap<string, ConnectionState> {
const rows = this.options.db
.prepare(
`SELECT instance_id, payload_json, observed_at, expires_at
FROM status_snapshots WHERE category = 'connection'`,
)
.all() as unknown as readonly SnapshotRow[];
const now = this.now().toISOString();
const states = new Map<string, ConnectionState>();
for (const row of rows) {
let authenticated = false;
try {
const payload = JSON.parse(row.payload_json ?? '{}') as { authenticated?: unknown };
authenticated = payload.authenticated === true;
} catch {
authenticated = false;
}
const reachable = Boolean(row.expires_at) && (row.expires_at as string) > now;
states.set(row.instance_id, {
reachable,
authenticated: reachable ? authenticated : false,
checkedAt: row.observed_at,
});
}
return states;
}
/** Best-effort: the journal is diagnostics, never a reason to fail a probe. */
#record(
instanceId: string,
httpStatus: number | null,
outcome: 'success' | 'stale' | 'failed',
startedAt: number,
errorCode: string | null,
): void {
this.options.connectionLogs?.record({
instanceId,
outcome,
state: outcome === 'success' ? 'fresh' : outcome === 'stale' ? 'stale' : 'unknown',
errorCode,
httpStatus,
durationMs: Math.max(0, this.now().getTime() - startedAt),
observedAt: this.now().toISOString(),
});
}
}
function errorCode(error: unknown): string {
const code = (error as { code?: unknown })?.code;
return typeof code === 'string' ? code.slice(0, 64) : 'UPSTREAM_UNAVAILABLE';
}
@@ -0,0 +1,160 @@
import Database from 'better-sqlite3';
import { afterEach, describe, expect, it } from 'vitest';
import { migrateDatabase } from '../../infrastructure/database/migrations.js';
import {
ConnectionSettingsService,
DEFAULT_CONNECTION_SETTINGS,
validateConnectionSettings,
} from './connection-settings-service.js';
const dbs: Database.Database[] = [];
afterEach(() => {
for (const db of dbs.splice(0)) db.close();
});
function service() {
const db = new Database(':memory:');
db.pragma('foreign_keys = ON');
migrateDatabase(db);
dbs.push(db);
return {
db,
settings: new ConnectionSettingsService({
db,
now: () => new Date('2026-09-05T00:00:00.000Z'),
}),
};
}
function stored(db: Database.Database): string | undefined {
const row = db
.prepare('SELECT value_json FROM app_settings WHERE key = ?')
.get('connection.settings') as { value_json: string } | undefined;
return row?.value_json;
}
describe('validateConnectionSettings', () => {
it('fills the missing field from the defaults', () => {
expect(validateConnectionSettings({ heartbeatSeconds: 60 })).toEqual({
heartbeatSeconds: 60,
offlineSeconds: 120,
authorizationMode: 'auto',
});
expect(validateConnectionSettings({ offlineSeconds: 300 })).toEqual({
heartbeatSeconds: 30,
offlineSeconds: 300,
authorizationMode: 'auto',
});
});
it('rejects payloads that are not objects or carry unknown keys', () => {
expect(() => validateConnectionSettings(null)).toThrow(TypeError);
expect(() => validateConnectionSettings([])).toThrow(TypeError);
expect(() => validateConnectionSettings('30')).toThrow(TypeError);
expect(() => validateConnectionSettings({ heartbeat_interval: 30 })).toThrow(TypeError);
});
it('rejects non-integer or out-of-range cadence', () => {
expect(() => validateConnectionSettings({ heartbeatSeconds: 30.5 })).toThrow(RangeError);
expect(() => validateConnectionSettings({ heartbeatSeconds: 4 })).toThrow(RangeError);
expect(() => validateConnectionSettings({ heartbeatSeconds: 301 })).toThrow(RangeError);
expect(() => validateConnectionSettings({ offlineSeconds: 1_801 })).toThrow(RangeError);
});
it('requires the offline window to tolerate at least two missed beats', () => {
expect(() => validateConnectionSettings({ heartbeatSeconds: 30, offlineSeconds: 59 })).toThrow(
RangeError,
);
expect(validateConnectionSettings({ heartbeatSeconds: 30, offlineSeconds: 60 })).toEqual({
heartbeatSeconds: 30,
offlineSeconds: 60,
authorizationMode: 'auto',
});
});
it('requires a known authorization mode and keeps the mode omitted on update', () => {
expect(() => validateConnectionSettings({ authorizationMode: 'operator' })).toThrow(TypeError);
expect(validateConnectionSettings({ authorizationMode: 'manual' })).toEqual({
heartbeatSeconds: 30,
offlineSeconds: 90,
authorizationMode: 'manual',
});
});
});
describe('ConnectionSettingsService', () => {
it('starts from the shipped defaults', () => {
const { settings } = service();
expect(settings.get()).toEqual({
heartbeatSeconds: 30,
offlineSeconds: 90,
authorizationMode: 'auto',
});
expect(settings.heartbeatMs).toBe(30_000);
expect(settings.snapshotTtlMs).toBe(90_000);
});
it('persists an update and reads it back', () => {
const { db, settings } = service();
expect(settings.update({ heartbeatSeconds: 45, offlineSeconds: 180 })).toEqual({
heartbeatSeconds: 45,
offlineSeconds: 180,
authorizationMode: 'auto',
});
expect(stored(db)).toBe(
'{"heartbeatSeconds":45,"offlineSeconds":180,"authorizationMode":"auto"}',
);
expect(settings.get()).toEqual({
heartbeatSeconds: 45,
offlineSeconds: 180,
authorizationMode: 'auto',
});
expect(settings.heartbeatMs).toBe(45_000);
expect(settings.snapshotTtlMs).toBe(180_000);
});
it('rewrites the same row instead of inserting a second one', () => {
const { db, settings } = service();
settings.update({ heartbeatSeconds: 10, offlineSeconds: 20 });
settings.update({ heartbeatSeconds: 20, offlineSeconds: 40, authorizationMode: 'manual' });
const rows = db.prepare('SELECT COUNT(*) AS count FROM app_settings').get() as {
count: number;
};
expect(rows.count).toBe(1);
expect(settings.get()).toEqual({
heartbeatSeconds: 20,
offlineSeconds: 40,
authorizationMode: 'manual',
});
});
it('falls back to the defaults when the stored row is corrupt', () => {
const { db, settings } = service();
const now = new Date('2026-09-05T00:00:00.000Z').toISOString();
db.prepare(
'INSERT INTO app_settings (key,value_json,created_at,updated_at) VALUES (?,?,?,?)',
).run('connection.settings', 'not json', now, now);
expect(settings.get()).toEqual(DEFAULT_CONNECTION_SETTINGS);
db.prepare('UPDATE app_settings SET value_json = ? WHERE key = ?').run(
'{"heartbeatSeconds":9999,"offlineSeconds":10}',
'connection.settings',
);
expect(settings.get()).toEqual(DEFAULT_CONNECTION_SETTINGS);
});
it('keeps a rejected update from touching the stored row', () => {
const { db, settings } = service();
settings.update({ heartbeatSeconds: 60, offlineSeconds: 120 });
expect(() => settings.update({ heartbeatSeconds: 60, offlineSeconds: 61 })).toThrow(RangeError);
expect(settings.get()).toEqual({
heartbeatSeconds: 60,
offlineSeconds: 120,
authorizationMode: 'auto',
});
expect(stored(db)).toBe(
'{"heartbeatSeconds":60,"offlineSeconds":120,"authorizationMode":"auto"}',
);
});
});
@@ -0,0 +1,126 @@
import type Database from 'better-sqlite3';
/**
* Reachability cadence, the Hub "connection settings" panel rebuilt on the control plane's own
* snapshot journal. The heartbeat decides how often every instance is probed; the offline window
* decides how long a lost probe is tolerated before the fleet view calls the device offline.
*/
export interface ConnectionSettings {
readonly heartbeatSeconds: number;
readonly offlineSeconds: number;
readonly authorizationMode: 'auto' | 'manual';
}
export const MIN_HEARTBEAT_SECONDS = 5;
export const MAX_HEARTBEAT_SECONDS = 300;
export const MAX_OFFLINE_SECONDS = 1_800;
export const DEFAULT_CONNECTION_SETTINGS: Readonly<ConnectionSettings> = Object.freeze({
heartbeatSeconds: 30,
offlineSeconds: 90,
authorizationMode: 'auto',
});
const SETTING_KEY = 'connection.settings';
const FIELDS: readonly (keyof ConnectionSettings)[] = [
'heartbeatSeconds',
'offlineSeconds',
'authorizationMode',
];
function safeInteger(value: unknown, name: string): number {
if (!Number.isSafeInteger(value))
throw new RangeError(`${name} must be a whole number of seconds`);
return value as number;
}
export function validateConnectionSettings(value: unknown): ConnectionSettings {
const source =
value !== null && typeof value === 'object' && !Array.isArray(value)
? (value as Record<string, unknown>)
: null;
if (!source) throw new TypeError('The connection settings payload is invalid');
const unknown = Object.keys(source).filter(
(key) => !FIELDS.includes(key as keyof ConnectionSettings),
);
if (unknown.length > 0) throw new TypeError(`Unknown connection setting: ${unknown[0]}`);
const authorizationMode =
source.authorizationMode === undefined
? DEFAULT_CONNECTION_SETTINGS.authorizationMode
: source.authorizationMode;
if (authorizationMode !== 'auto' && authorizationMode !== 'manual')
throw new TypeError('authorizationMode must be auto or manual');
const heartbeatSeconds = safeInteger(
source.heartbeatSeconds ?? DEFAULT_CONNECTION_SETTINGS.heartbeatSeconds,
'heartbeatSeconds',
);
if (heartbeatSeconds < MIN_HEARTBEAT_SECONDS || heartbeatSeconds > MAX_HEARTBEAT_SECONDS)
throw new RangeError('heartbeatSeconds must be between 5 and 300');
// An omitted window scales with the heartbeat so a partial payload stays valid; an explicit one
// is checked strictly below.
const fallbackOffline = Math.min(
MAX_OFFLINE_SECONDS,
Math.max(DEFAULT_CONNECTION_SETTINGS.offlineSeconds, heartbeatSeconds * 2),
);
const offlineSeconds = safeInteger(source.offlineSeconds ?? fallbackOffline, 'offlineSeconds');
if (offlineSeconds < heartbeatSeconds * 2)
throw new RangeError('offlineSeconds must be at least twice the heartbeat interval');
if (offlineSeconds > MAX_OFFLINE_SECONDS)
throw new RangeError(`offlineSeconds must not exceed ${MAX_OFFLINE_SECONDS}`);
return { heartbeatSeconds, offlineSeconds, authorizationMode };
}
interface SettingsRow {
readonly value_json?: string | null;
}
export class ConnectionSettingsService {
readonly #db: Database.Database;
readonly #now: () => Date;
constructor(options: { readonly db: Database.Database; readonly now?: () => Date }) {
this.#db = options.db;
this.#now = options.now ?? (() => new Date());
}
get(): ConnectionSettings {
const row = this.#db
.prepare('SELECT value_json FROM app_settings WHERE key = ?')
.get(SETTING_KEY) as SettingsRow | undefined;
if (!row?.value_json) return { ...DEFAULT_CONNECTION_SETTINGS };
try {
return validateConnectionSettings(JSON.parse(row.value_json));
} catch {
// A hand-edited row must never take the heartbeat loop down.
return { ...DEFAULT_CONNECTION_SETTINGS };
}
}
update(value: unknown): ConnectionSettings {
const settings = validateConnectionSettings(value);
const now = this.#now().toISOString();
this.#db
.prepare(
`INSERT INTO app_settings (key,value_json,created_at,updated_at)
VALUES (?,?,?,?)
ON CONFLICT(key) DO UPDATE SET value_json = excluded.value_json,
updated_at = excluded.updated_at`,
)
.run(SETTING_KEY, JSON.stringify(settings), now, now);
return settings;
}
get heartbeatMs(): number {
return this.get().heartbeatSeconds * 1_000;
}
/** Snapshot lifetime: a probe stays authoritative for one full offline window. */
get snapshotTtlMs(): number {
return this.get().offlineSeconds * 1_000;
}
}
@@ -0,0 +1,260 @@
import { afterEach, describe, expect, it, vi } from 'vitest';
import type { Instance, InstancePage } from '@multi-simadmin/contracts';
import type { InstanceService } from '../instances/instance-service.js';
import type { ConnectionState, ConnectionProbe } from './connection-probe.js';
import type { ConnectionSettingsService } from './connection-settings-service.js';
import { FleetHeartbeatCoordinator, type FleetStateTransition } from './fleet-heartbeat.js';
function instance(id: string): Instance {
return {
id,
name: id,
origin: `http://10.0.0.1:3000`,
authMode: 'none',
configRevision: 1,
createdAt: '2026-09-05T00:00:00.000Z',
updatedAt: '2026-09-05T00:00:00.000Z',
} as unknown as Instance;
}
function page(items: readonly Instance[], pageSize: number): InstancePage {
return { items: [...items], page: { page: 1, pageSize, total: items.length } };
}
function coordinator(options: {
readonly pages: ReadonlyMap<number, readonly Instance[]>;
readonly pageSize: number;
readonly probe?: (id: string) => Promise<void>;
readonly heartbeatMs?: number;
readonly concurrency?: number;
/** Reachability the snapshot journal already holds when the first beat starts. */
readonly before?: ReadonlyMap<string, ConnectionState>;
/** Lets a test flip a node to reachable-but-unauthenticated after it answers. */
readonly authenticated?: (id: string) => boolean;
readonly onTransition?: (transitions: readonly FleetStateTransition[]) => Promise<void>;
}) {
const listed: string[] = [];
const instances = {
async list(query: { page?: number; pageSize?: number }) {
const pageSize = options.pageSize;
const current = query.page ?? 1;
listed.push(String(current));
return page(options.pages.get(current) ?? [], pageSize);
},
} as unknown as InstanceService;
const probed: string[] = [];
const journal = new Map<string, ConnectionState>(options.before ?? []);
const probe = {
async test(id: string) {
probed.push(id);
try {
await options.probe?.(id);
} catch {
// A failed probe leaves the previous snapshot in place and lets it expire on its own,
// which is exactly what the offline window is for.
throw new Error('probe failed');
}
const authenticated = options.authenticated?.(id) ?? true;
journal.set(id, {
reachable: true,
authenticated,
checkedAt: '2026-09-05T00:00:00.000Z',
});
return {
instanceId: id,
authenticated,
checkedAt: '2026-09-05T00:00:00.000Z',
};
},
reachability: () => journal,
} as unknown as ConnectionProbe;
const settings = {
heartbeatMs: options.heartbeatMs ?? 30_000,
} as unknown as ConnectionSettingsService;
const beat = new FleetHeartbeatCoordinator({
instances,
probe,
settings,
pageSize: options.pageSize,
now: () => new Date('2026-09-05T00:00:00.000Z'),
...(options.concurrency === undefined ? {} : { concurrency: options.concurrency }),
...(options.onTransition === undefined ? {} : { onTransition: options.onTransition }),
});
return { beat, listed, probed, journal };
}
afterEach(() => {
vi.useRealTimers();
});
describe('FleetHeartbeatCoordinator', () => {
it('probes every instance once across pages', async () => {
const { beat, listed, probed } = coordinator({
pageSize: 2,
pages: new Map([
[1, [instance('a'), instance('b')]],
[2, [instance('c')]],
]),
});
await expect(beat.runOnce()).resolves.toEqual({
probed: 3,
failed: 0,
transitions: [],
startedAt: '2026-09-05T00:00:00.000Z',
finishedAt: '2026-09-05T00:00:00.000Z',
});
expect(listed).toEqual(['1', '2']);
expect([...probed].sort()).toEqual(['a', 'b', 'c']);
});
it('records a baseline on the first beat instead of announcing every node as recovered', async () => {
const announced: (readonly FleetStateTransition[])[] = [];
const { beat } = coordinator({
pageSize: 10,
pages: new Map([[1, [instance('a'), instance('b')]]]),
onTransition: async (transitions) => {
announced.push(transitions);
},
});
await expect(beat.runOnce()).resolves.toMatchObject({ transitions: [] });
await expect(beat.runOnce()).resolves.toMatchObject({ transitions: [] });
expect(announced).toEqual([]);
});
it('announces a node only once its snapshot has actually expired', async () => {
const down: ConnectionState = {
reachable: false,
authenticated: false,
checkedAt: '2026-09-04T00:00:00.000Z',
};
const up: ConnectionState = {
reachable: true,
authenticated: true,
checkedAt: '2026-09-04T00:00:00.000Z',
};
const announced: (readonly FleetStateTransition[])[] = [];
const { beat, journal } = coordinator({
pageSize: 10,
concurrency: 1,
pages: new Map([[1, [instance('a'), instance('b')]]]),
before: new Map([
['a', up],
['b', down],
]),
probe: async (id) => {
// b never answers; a answers on the first beat and goes quiet afterwards.
if (id === 'b' || !journal.get('a')?.reachable) throw new Error('ECONNREFUSED');
},
onTransition: async (transitions) => {
announced.push(transitions);
},
});
// a is reachable and b is still down, so nothing moved against the baseline.
await expect(beat.runOnce()).resolves.toMatchObject({
probed: 1,
failed: 1,
transitions: [],
});
// a's snapshot ages out. A single lost probe would not be enough on its own; the beat
// reports offline only once the offline window the operator configured has passed.
journal.set('a', down);
await expect(beat.runOnce()).resolves.toMatchObject({
transitions: [{ instanceId: 'a', from: 'online', to: 'offline' }],
});
expect(announced).toHaveLength(1);
});
it('announces an expired device session as a state change', async () => {
const announced: (readonly FleetStateTransition[])[] = [];
let sessionValid = true;
const { beat } = coordinator({
pageSize: 10,
concurrency: 1,
pages: new Map([[1, [instance('a')]]]),
authenticated: () => sessionValid,
onTransition: async (transitions) => {
announced.push(transitions);
},
});
// The first beat only records that the node is healthy.
await expect(beat.runOnce()).resolves.toMatchObject({ transitions: [] });
sessionValid = false;
await expect(beat.runOnce()).resolves.toMatchObject({
transitions: [{ instanceId: 'a', from: 'online', to: 'auth-required' }],
});
expect(announced).toHaveLength(1);
});
it('stops paging on a short page and probes a duplicated id once', async () => {
const { beat, listed, probed } = coordinator({
pageSize: 2,
pages: new Map([[1, [instance('a'), instance('a')]]]),
});
await beat.runOnce();
expect(listed).toEqual(['1', '2']);
expect(probed).toEqual(['a']);
});
it('counts a failing device without abandoning the rest of the fleet', async () => {
const { beat } = coordinator({
pageSize: 10,
concurrency: 1,
pages: new Map([[1, [instance('a'), instance('b'), instance('c')]]]),
probe: async (id) => {
if (id === 'b') throw new Error('ECONNREFUSED');
},
});
await expect(beat.runOnce()).resolves.toMatchObject({ probed: 2, failed: 1 });
});
it('shares one pass between concurrent refresh requests', async () => {
let release: (() => void) | undefined;
const gate = new Promise<void>((resolve) => {
release = resolve;
});
const { beat, probed } = coordinator({
pageSize: 10,
concurrency: 1,
pages: new Map([[1, [instance('a')]]]),
probe: () => gate,
});
const first = beat.runOnce();
const second = beat.runOnce();
release?.();
await Promise.all([first, second]);
expect(probed).toEqual(['a']);
});
it('runs a beat on the configured cadence and stops cleanly', async () => {
vi.useFakeTimers();
const { beat, probed } = coordinator({
pageSize: 10,
pages: new Map([[1, [instance('a')]]]),
heartbeatMs: 5_000,
});
beat.start();
beat.start();
await vi.advanceTimersByTimeAsync(5_000);
expect(probed).toEqual(['a']);
await vi.advanceTimersByTimeAsync(5_000);
expect(probed).toEqual(['a', 'a']);
beat.stop();
await vi.advanceTimersByTimeAsync(60_000);
expect(probed).toHaveLength(2);
});
it('refuses an unusable worker or page size', () => {
const base = {
instances: {} as unknown as InstanceService,
probe: {} as unknown as ConnectionProbe,
settings: {} as unknown as ConnectionSettingsService,
};
expect(() => new FleetHeartbeatCoordinator({ ...base, concurrency: 0 })).toThrow(RangeError);
expect(() => new FleetHeartbeatCoordinator({ ...base, concurrency: 17 })).toThrow(RangeError);
expect(() => new FleetHeartbeatCoordinator({ ...base, pageSize: 0 })).toThrow(RangeError);
expect(() => new FleetHeartbeatCoordinator({ ...base, pageSize: 101 })).toThrow(RangeError);
});
});
@@ -0,0 +1,203 @@
import type { ConnectionProbe, ConnectionState } from './connection-probe.js';
import type { ConnectionSettingsService } from './connection-settings-service.js';
import type { InstanceService } from '../instances/instance-service.js';
/** Reachability class the console and the notification rules both speak in. */
export type FleetObservedState = 'online' | 'auth-required' | 'offline';
/**
* Operator-facing copy shared by the notification templates so every channel reads alike.
* `FLEET_STATE_NAMES` describes a steady state, `FLEET_STATE_EVENTS` describes arriving at it.
*/
export const FLEET_STATE_NAMES: Readonly<Record<FleetObservedState, string>> = Object.freeze({
online: '在线',
'auth-required': '需要认证',
offline: '离线',
});
export const FLEET_STATE_EVENTS: Readonly<Record<FleetObservedState, string>> = Object.freeze({
online: '恢复在线',
'auth-required': '设备会话已失效',
offline: '已离线',
});
export const FLEET_STATE_CODES: Readonly<Record<FleetObservedState, string>> = Object.freeze({
online: 'online',
'auth-required': 'auth_required',
offline: 'offline',
});
export interface FleetStateTransition {
readonly instanceId: string;
readonly from: FleetObservedState;
readonly to: FleetObservedState;
}
export interface FleetHeartbeatSummary {
readonly probed: number;
readonly failed: number;
readonly transitions: readonly FleetStateTransition[];
readonly startedAt: string;
readonly finishedAt: string;
}
export interface FleetHeartbeatOptions {
readonly instances: InstanceService;
readonly probe: ConnectionProbe;
readonly settings: ConnectionSettingsService;
readonly now?: () => Date;
/** Devices touched at the same time; the control plane talks to LAN hosts, not the internet. */
readonly concurrency?: number;
readonly pageSize?: number;
/**
* Called once per beat with every state change the beat revealed. Reachability is derived
* from snapshot expiry, so a single lost probe never fires anything; a node has to stay
* down for the whole offline window before it counts as a transition.
*/
readonly onTransition?: (transitions: readonly FleetStateTransition[]) => Promise<void>;
}
const DEFAULT_CONCURRENCY = 6;
const DEFAULT_PAGE_SIZE = 100;
const MAX_PAGES = 100;
function observedState(state: ConnectionState | undefined): FleetObservedState {
if (!state?.reachable) return 'offline';
return state.authenticated ? 'online' : 'auth-required';
}
/**
* Keeps device online state honest without waiting for an operator to open a page: every beat
* probes each instance once and lets the snapshot journal carry the result until it expires.
*/
export class FleetHeartbeatCoordinator {
readonly #instances: InstanceService;
readonly #probe: ConnectionProbe;
readonly #settings: ConnectionSettingsService;
readonly #now: () => Date;
readonly #concurrency: number;
readonly #pageSize: number;
readonly #onTransition:
| ((transitions: readonly FleetStateTransition[]) => Promise<void>)
| undefined;
#timer: ReturnType<typeof setTimeout> | undefined;
#running: Promise<FleetHeartbeatSummary> | undefined;
#stopped = false;
/** States seen by the previous beat; undefined until the first beat establishes a baseline. */
#previous: ReadonlyMap<string, FleetObservedState> | undefined;
constructor(options: FleetHeartbeatOptions) {
this.#instances = options.instances;
this.#probe = options.probe;
this.#settings = options.settings;
this.#now = options.now ?? (() => new Date());
const concurrency = options.concurrency ?? DEFAULT_CONCURRENCY;
if (!Number.isSafeInteger(concurrency) || concurrency < 1 || concurrency > 16)
throw new RangeError('concurrency must be between 1 and 16');
const pageSize = options.pageSize ?? DEFAULT_PAGE_SIZE;
if (!Number.isSafeInteger(pageSize) || pageSize < 1 || pageSize > 100)
throw new RangeError('pageSize must be between 1 and 100');
this.#concurrency = concurrency;
this.#pageSize = pageSize;
this.#onTransition = options.onTransition;
}
start(): void {
this.#stopped = false;
if (this.#timer) return;
this.#schedule();
}
stop(): void {
this.#stopped = true;
if (this.#timer) clearTimeout(this.#timer);
this.#timer = undefined;
}
/** Runs a beat on demand; concurrent callers share the same pass. */
runOnce(): Promise<FleetHeartbeatSummary> {
if (this.#running) return this.#running;
const pending = this.#beat().finally(() => {
if (this.#running === pending) this.#running = undefined;
});
this.#running = pending;
return pending;
}
#schedule(): void {
if (this.#stopped) return;
const delay = this.#settings.heartbeatMs;
this.#timer = setTimeout(() => {
this.#timer = undefined;
void this.runOnce()
.catch(() => undefined)
.finally(() => this.#schedule());
}, delay);
this.#timer.unref?.();
}
async #beat(): Promise<FleetHeartbeatSummary> {
const startedAt = this.#now().toISOString();
const ids: string[] = [];
for (let page = 1; page <= MAX_PAGES; page += 1) {
const current = await this.#instances.list({ page, pageSize: this.#pageSize });
for (const instance of current.items) if (!ids.includes(instance.id)) ids.push(instance.id);
if (current.items.length < this.#pageSize) break;
}
const before = this.#previous;
let probed = 0;
let failed = 0;
let cursor = 0;
const workers = Array.from(
{ length: Math.min(this.#concurrency, ids.length) },
async (): Promise<void> => {
for (;;) {
const index = cursor;
cursor += 1;
if (index >= ids.length) return;
try {
await this.#probe.test(ids[index] as string);
probed += 1;
} catch {
failed += 1;
}
}
},
);
await Promise.all(workers);
const transitions = this.#diff(before, ids);
if (transitions.length > 0) await this.#onTransition?.(transitions);
return { probed, failed, transitions, startedAt, finishedAt: this.#now().toISOString() };
}
/** Snapshot of the given instances, including ones that have never been probed. */
#states(ids: readonly string[]): ReadonlyMap<string, FleetObservedState> {
const reachability = this.#probe.reachability();
const states = new Map<string, FleetObservedState>();
for (const id of ids) {
const state = reachability.get(id);
states.set(id, observedState(state));
}
return states;
}
/**
* The first beat only records a baseline. Without one, every instance would start as
* "offline" and then announce itself as recovered, so a restart would flood the queue.
*/
#diff(
before: ReadonlyMap<string, FleetObservedState> | undefined,
ids: readonly string[],
): readonly FleetStateTransition[] {
const after = this.#states(ids);
this.#previous = after;
if (before === undefined) return [];
const transitions: FleetStateTransition[] = [];
for (const [id, to] of after) {
const from = before.get(id);
if (from === undefined || from === to) continue;
transitions.push({ instanceId: id, from, to });
}
return transitions;
}
}
@@ -27,6 +27,7 @@ describe('InstanceCredentialResolver', () => {
it('resolves only the canonical password reference for the requested existing instance', async () => {
let asked: string | undefined;
const { db, resolver } = fixture({
provider: 'macos-keychain',
set: async () => '',
get: async (reference) => {
asked = reference;
@@ -44,6 +45,7 @@ describe('InstanceCredentialResolver', () => {
it('does not call SecretStore when no saved credential exists', async () => {
let calls = 0;
const { resolver } = fixture({
provider: 'macos-keychain',
set: async () => '',
get: async () => {
calls += 1;
@@ -57,6 +59,7 @@ describe('InstanceCredentialResolver', () => {
it('rejects a reference bound to another instance before SecretStore access', async () => {
let calls = 0;
const { db, resolver } = fixture({
provider: 'macos-keychain',
set: async () => '',
get: async () => {
calls += 1;
@@ -1,9 +1,8 @@
import type Database from 'better-sqlite3';
import { parseKeychainReference } from '../../infrastructure/secrets/keychain-secret-store.js';
import { parseSecretReference } from '../../infrastructure/secrets/secret-reference.js';
import type { SecretStore } from '../../infrastructure/secrets/secret-store.js';
const PURPOSE = 'instance-password';
const PROVIDER = 'macos-keychain';
export class CredentialResolverError extends Error {
constructor(readonly code: 'NOT_FOUND' | 'CREDENTIAL_UNAVAILABLE') {
super(code);
@@ -24,10 +23,12 @@ export class InstanceCredentialResolver {
.prepare(
'SELECT external_reference FROM secret_references WHERE instance_id=? AND purpose=? AND provider=?',
)
.get(instanceId, PURPOSE, PROVIDER) as { external_reference: string } | undefined;
.get(instanceId, PURPOSE, this.options.store.provider) as
| { external_reference: string }
| undefined;
if (!row) throw new CredentialResolverError('CREDENTIAL_UNAVAILABLE');
try {
const parsed = parseKeychainReference(row.external_reference);
const parsed = parseSecretReference(row.external_reference);
if (parsed.instanceId !== instanceId || parsed.purpose !== PURPOSE || !parsed.slot)
throw new Error('invalid binding');
const secret = await this.options.store.get(row.external_reference);
@@ -1,6 +1,6 @@
export interface UpstreamRequest {
readonly url: string;
readonly method: 'GET' | 'POST';
readonly method: 'GET' | 'POST' | 'DELETE';
readonly headers: Readonly<Record<string, string>>;
/** One-shot secret. Request implementations must not log or persist this field. */
readonly secret?: string;
@@ -8,6 +8,13 @@ export interface UpstreamRequest {
readonly body?: '[REDACTED]';
/** Explicit one-shot SMS action payload. Implementations must not log or persist it. */
readonly sms?: { readonly phoneNumber: string; readonly content: string };
/** Explicit one-shot SMS batch-delete payload. Implementations must not log or persist it. */
readonly smsBatchDelete?: { readonly ids: readonly number[] };
/**
* Explicit one-shot device action payload for an allowlisted device mutation. The gateway
* re-validates and canonically serializes it; implementations must not log or persist it.
*/
readonly deviceAction?: { readonly body: Readonly<Record<string, unknown>> | undefined };
}
export interface UpstreamResponse {
readonly status: number;
@@ -4,7 +4,8 @@ import { describe, expect, it, vi } from 'vitest';
import { EventJournal, formatServerSentEvent, type EventEnvelope } from './event-journal.js';
import { migrateDatabase } from '../../infrastructure/database/migrations.js';
const at = '2026-07-17T12:34:56.789Z';
// Journal retention prunes envelopes older than a day, so fixtures stay clock-relative.
const at = new Date(Date.now() - 60_000).toISOString();
function event(kind: EventEnvelope['kind'], id = `${kind}-1`): EventEnvelope {
const common = {
@@ -185,7 +185,6 @@ export class EventJournal {
const sequence = Number(result.lastInsertRowid);
if (!Number.isSafeInteger(sequence))
throw new Error('event journal sequence is not a safe integer');
this.#prune();
const appended = { sequence, envelope };
for (const subscriber of [...this.#subscribers]) subscriber(appended);
return appended;
@@ -236,7 +235,11 @@ export class EventJournal {
};
}
#prune(): void {
/**
* Maintenance hook for a periodic timer. Pruning per append turned every
* insert into an unindexable full-table json_extract scan.
*/
prune(): void {
const cutoff = new Date(Date.now() - this.#retentionMs).toISOString();
this.#database
.prepare("DELETE FROM event_journal WHERE json_extract(envelope_json, '$.occurredAt') < ?")
@@ -0,0 +1,245 @@
import Database from 'better-sqlite3';
import { describe, expect, it } from 'vitest';
import { migrateDatabase } from '../../infrastructure/database/migrations.js';
import { DeviceIdentityError, DeviceIdentityService } from './device-identity-service.js';
interface Fixture {
readonly db: Database.Database;
readonly service: DeviceIdentityService;
readonly advance: (minutes: number) => void;
}
function fixture(): Fixture {
const db = new Database(':memory:');
db.pragma('foreign_keys=ON');
migrateDatabase(db);
let clock = new Date('2026-09-05T02:00:00.000Z');
const service = new DeviceIdentityService({ db, now: () => clock });
return {
db,
service,
advance: (minutes: number) => {
clock = new Date(clock.getTime() + minutes * 60_000);
},
};
}
function manualFixture() {
const db = new Database(':memory:');
db.pragma('foreign_keys=ON');
migrateDatabase(db);
let clock = new Date('2026-09-05T02:00:00.000Z');
const service = new DeviceIdentityService({
db,
now: () => clock,
authorizationMode: () => 'manual',
});
return {
db,
service,
advance: (minutes: number) => {
clock = new Date(clock.getTime() + minutes * 60_000);
},
};
}
function register(db: Database.Database, id: string): void {
db.prepare(
`INSERT INTO instances
(id,name,base_url,auth_mode,enabled,config_revision,created_at,updated_at)
VALUES (?,?,?,'password',1,1,?,?)`,
).run(id, id, `http://${id}.lan:8080`, '2026-09-05T02:00:00.000Z', '2026-09-05T02:00:00.000Z');
}
const device = (overrides: Record<string, unknown> = {}) => ({
imei: '860000000000001',
manufacturer: 'Quectel',
model: 'RM500Q',
revision: 'RM500QEAAAR13',
...overrides,
});
describe('DeviceIdentityService', () => {
it('baselines the first report instead of calling it a conflict', () => {
const { db, service } = fixture();
register(db, 'a');
const identity = service.observe('a', { ...device(), origin: 'http://a.lan:8080' });
expect(identity).toMatchObject({
instanceId: 'a',
status: 'confirmed',
reasons: [],
imei: '860000000000001',
model: 'RM500Q',
origin: 'http://a.lan:8080',
});
expect(service.isBlocked('a')).toBe(false);
});
it('keeps a device that reports no identity out of the guard', () => {
const { db, service } = fixture();
register(db, 'a');
expect(service.observe('a', { agent: 'simadmin-agent 1.9.6' })).toBeUndefined();
expect(service.observe('a', { imei: ' ' })).toBeUndefined();
expect(service.list()).toEqual([]);
});
it('holds a first report until an operator confirms it in manual authorization mode', () => {
const { db, service } = manualFixture();
register(db, 'a');
const identity = service.observe('a', device());
expect(identity).toMatchObject({
instanceId: 'a',
status: 'pending',
reasons: [],
imei: '860000000000001',
});
expect(service.isBlocked('a')).toBe(true);
expect(service.confirm('a')).toMatchObject({ status: 'confirmed', reasons: [] });
expect(service.isBlocked('a')).toBe(false);
});
it('holds a node whose hardware no longer matches the record', () => {
const { db, service } = fixture();
register(db, 'a');
service.observe('a', device());
const swapped = service.observe('a', device({ model: 'RG500Q', imei: '860000000000009' }));
expect(swapped?.status).toBe('pending');
expect(swapped?.reasons).toEqual(['hardware_swapped']);
expect(swapped?.changes).toEqual([
{ field: 'imei', from: '860000000000001', to: '860000000000009' },
{ field: 'model', from: 'RM500Q', to: 'RG500Q' },
]);
expect(service.isBlocked('a')).toBe(true);
});
it('compares only the fields a partial report actually carries', () => {
const { db, service } = fixture();
register(db, 'a');
service.observe('a', device());
const partial = service.observe('a', { model: 'RM500Q' });
expect(partial?.status).toBe('confirmed');
expect(partial?.reasons).toEqual([]);
// The dropped fields stay on the row: a silent read must not erase what we already knew.
expect(partial?.imei).toBe('860000000000001');
});
it('asks both records when one identity is claimed twice', () => {
const { db, service } = fixture();
register(db, 'a');
register(db, 'b');
service.observe('a', device());
const newcomer = service.observe('b', device());
expect(newcomer?.status).toBe('pending');
expect(newcomer?.reasons).toEqual(['imei_claimed']);
expect(newcomer?.conflicts).toEqual([{ reason: 'imei_claimed', instanceIds: ['a'] }]);
expect(service.get('a')?.status).toBe('pending');
expect(service.get('a')?.conflicts).toEqual([{ reason: 'imei_claimed', instanceIds: ['b'] }]);
expect(service.summarize()).toEqual({ tracked: 2, pending: ['a', 'b'] });
});
it('accepts the twins that were on screen when the operator confirmed', () => {
const { db, service } = fixture();
register(db, 'a');
register(db, 'b');
service.observe('a', device());
service.observe('b', device());
expect(service.confirm('a')).toMatchObject({ status: 'confirmed', reasons: [] });
service.observe('a', device());
expect(service.get('a')?.status).toBe('confirmed');
// A third claimant is new information, so the guard comes back down.
register(db, 'c');
service.observe('c', device());
expect(service.get('a')?.status).toBe('pending');
// 'b' was on screen when the operator confirmed; only the newcomer is still a question.
expect(service.get('a')?.conflicts).toEqual([{ reason: 'imei_claimed', instanceIds: ['c'] }]);
});
it('re-baselines swapped hardware on confirmation', () => {
const { db, service } = fixture();
register(db, 'a');
service.observe('a', device());
service.observe('a', device({ model: 'RG500Q' }));
const confirmed = service.confirm('a');
expect(confirmed).toMatchObject({ status: 'confirmed', reasons: [], changes: [] });
expect(service.isBlocked('a')).toBe(false);
service.observe('a', device({ model: 'RG500Q' }));
expect(service.get('a')?.status).toBe('confirmed');
service.observe('a', device({ model: 'EM120' }));
expect(service.get('a')?.reasons).toEqual(['hardware_swapped']);
});
it('sanitizes device-controlled evidence before it is stored', () => {
const { db, service } = fixture();
register(db, 'a');
const identity = service.observe('a', {
imei: ' 860000000000001\u0000\u001b ',
model: 'x'.repeat(200),
revision: 42,
manufacturer: null,
});
expect(identity?.imei).toBe('860000000000001');
expect(identity?.model).toHaveLength(64);
expect(identity?.revision).toBe('42');
expect(identity?.manufacturer).toBe('');
});
it('reports a missing identity rather than inventing one on confirm', () => {
const { db, service } = fixture();
register(db, 'a');
expect(() => service.confirm('a')).toThrow(DeviceIdentityError);
try {
service.confirm('a');
} catch (error) {
expect(error).toBeInstanceOf(DeviceIdentityError);
expect((error as DeviceIdentityError).code).toBe('NOT_FOUND');
}
});
it('forgets the identity with the node it belonged to', () => {
const { db, service } = fixture();
register(db, 'a');
register(db, 'b');
service.observe('a', device());
service.observe('b', device());
db.prepare('DELETE FROM instances WHERE id=?').run('b');
// The row goes with the node, and the survivor is released the next time anything reads it.
expect(service.list().map((identity) => identity.instanceId)).toEqual(['a']);
expect(service.get('a')).toMatchObject({ status: 'pending' });
expect(service.observe('a', device())).toMatchObject({ status: 'confirmed', reasons: [] });
});
it('advances the observation time without touching the confirmation time', () => {
const { db, service, advance } = fixture();
register(db, 'a');
const first = service.observe('a', device());
expect(first?.observedAt).toBe('2026-09-05T02:00:00.000Z');
advance(5);
const second = service.observe('a', device());
expect(second?.observedAt).toBe('2026-09-05T02:05:00.000Z');
expect(second?.confirmedAt).toBe('2026-09-05T02:00:00.000Z');
expect(second?.fingerprint).toBe(first?.fingerprint);
});
});
@@ -0,0 +1,461 @@
import { createHash } from 'node:crypto';
import type Database from 'better-sqlite3';
export type IdentityStatus = 'confirmed' | 'pending';
/**
* `hardware_swapped` is the record that used to answer at this address reporting different
* hardware now; `imei_claimed` is a second record answering with the same identity. Neither can
* be resolved from here, which is why both stop control traffic until an operator says which
* device is the real one.
*/
export type IdentityReason = 'hardware_swapped' | 'imei_claimed';
const IDENTITY_FIELDS = ['imei', 'manufacturer', 'model', 'revision'] as const;
type IdentityField = (typeof IDENTITY_FIELDS)[number];
type IdentityValues = Partial<Record<IdentityField, string>>;
const MAX_FIELD_LENGTH: Readonly<Record<IdentityField | 'agent' | 'origin', number>> = {
imei: 32,
manufacturer: 64,
model: 64,
revision: 64,
agent: 64,
origin: 256,
};
const CONTROL_CHARACTERS = /[\u0000-\u0008\u000b\u000c\u000e-\u001f\u007f]/gu;
export interface DeviceIdentityEvidence {
readonly imei?: unknown;
readonly manufacturer?: unknown;
readonly model?: unknown;
readonly revision?: unknown;
readonly agent?: unknown;
readonly origin?: unknown;
}
export interface DeviceIdentityConflict {
readonly reason: IdentityReason;
readonly instanceIds: readonly string[];
}
export interface DeviceIdentityFieldChange {
readonly field: IdentityField;
readonly from: string;
readonly to: string;
}
export interface DeviceIdentity {
readonly instanceId: string;
readonly status: IdentityStatus;
readonly reasons: readonly IdentityReason[];
readonly conflicts: readonly DeviceIdentityConflict[];
readonly imei: string;
readonly manufacturer: string;
readonly model: string;
readonly revision: string;
readonly agent: string;
readonly origin: string;
readonly fingerprint: string;
readonly confirmedFingerprint: string;
readonly changes: readonly DeviceIdentityFieldChange[];
readonly observedAt: string;
readonly confirmedAt: string;
}
export type DeviceIdentityErrorCode = 'NOT_FOUND' | 'VALIDATION_FAILED';
export class DeviceIdentityError extends Error {
constructor(
readonly code: DeviceIdentityErrorCode,
message: string,
) {
super(message);
this.name = 'DeviceIdentityError';
}
}
export interface DeviceIdentityOptions {
readonly db: Database.Database;
readonly now?: () => Date;
/** Follows the connection setting: manual authorization makes a first report wait. */
readonly authorizationMode?: () => 'auto' | 'manual';
}
interface IdentityRow {
instance_id: string;
status: string;
reasons: string;
imei: string;
manufacturer: string;
model: string;
revision: string;
agent: string;
origin: string;
fingerprint: string;
confirmed_fingerprint: string;
confirmed_values_json: string;
confirmed_peers_json: string;
detail_json: string;
observed_at: string;
confirmed_at: string;
}
/** Device payloads are untrusted: only a trimmed, control-free, length-capped string is kept. */
function text(value: unknown, limit: number): string {
if (typeof value !== 'string' && typeof value !== 'number') return '';
const raw = typeof value === 'number' ? String(value) : value;
return raw.replace(CONTROL_CHARACTERS, '').trim().slice(0, limit);
}
function identityValues(evidence: DeviceIdentityEvidence): IdentityValues {
const values: IdentityValues = {};
for (const field of IDENTITY_FIELDS) {
const value = text(evidence[field], MAX_FIELD_LENGTH[field]);
if (value) values[field] = value;
}
return values;
}
function digest(values: IdentityValues): string {
const canonical = IDENTITY_FIELDS.filter((field) => values[field] !== undefined)
.map((field) => `${field}=${values[field]}`)
.join('\n');
return createHash('sha256').update(canonical, 'utf8').digest('hex').slice(0, 16);
}
function stringArray(value: unknown): string[] {
return Array.isArray(value)
? value.filter((item): item is string => typeof item === 'string')
: [];
}
function parseValues(value: string): IdentityValues {
try {
const parsed: unknown = JSON.parse(value);
if (parsed === null || typeof parsed !== 'object' || Array.isArray(parsed)) return {};
const record = parsed as Record<string, unknown>;
const values: IdentityValues = {};
for (const field of IDENTITY_FIELDS) {
const entry = text(record[field], MAX_FIELD_LENGTH[field]);
if (entry) values[field] = entry;
}
return values;
} catch {
return {};
}
}
/** The evidence the console shows: who else claims this identity and what changed. */
interface IdentityDetail {
readonly peers: readonly string[];
readonly changes: readonly DeviceIdentityFieldChange[];
}
function parseDetail(value: string): IdentityDetail {
let parsed: unknown = {};
try {
parsed = JSON.parse(value);
} catch {
return { peers: [], changes: [] };
}
if (parsed === null || typeof parsed !== 'object' || Array.isArray(parsed))
return { peers: [], changes: [] };
const record = parsed as Record<string, unknown>;
const entries = Array.isArray(record.changes) ? record.changes : [];
const changes = entries.filter(
(entry): entry is DeviceIdentityFieldChange =>
entry !== null &&
typeof entry === 'object' &&
!Array.isArray(entry) &&
IDENTITY_FIELDS.includes((entry as Record<string, unknown>).field as IdentityField) &&
typeof (entry as Record<string, unknown>).from === 'string' &&
typeof (entry as Record<string, unknown>).to === 'string',
);
return { peers: stringArray(record.peers), changes };
}
function parseReasons(value: string): IdentityReason[] {
const allowed: IdentityReason[] = ['hardware_swapped', 'imei_claimed'];
try {
const parsed: unknown = JSON.parse(value);
if (!Array.isArray(parsed)) return [];
return parsed.filter((item): item is IdentityReason =>
allowed.includes(item as IdentityReason),
);
} catch {
return [];
}
}
/**
* Tracks the hardware behind each registered node and holds the control plane still when that
* hardware stops matching the record. Observations come from device reads the console is making
* anyway, so a node only gains an identity row once something has actually looked at it.
*/
export class DeviceIdentityService {
readonly #db: Database.Database;
readonly #clock: () => Date;
readonly #authorizationMode: () => 'auto' | 'manual';
readonly #select: Database.Statement;
readonly #selectPeers: Database.Statement;
readonly #insert: Database.Statement;
readonly #update: Database.Statement;
constructor(options: DeviceIdentityOptions) {
this.#db = options.db;
this.#clock = options.now ?? (() => new Date());
this.#authorizationMode = options.authorizationMode ?? (() => 'auto');
this.#select = options.db.prepare('SELECT * FROM device_identities WHERE instance_id=?');
this.#selectPeers = options.db.prepare(
"SELECT instance_id FROM device_identities WHERE imei=? AND instance_id<>? AND imei<>'' ORDER BY instance_id",
);
this.#insert = options.db.prepare(
`INSERT INTO device_identities (instance_id,status,reasons,imei,manufacturer,model,revision,
agent,origin,fingerprint,confirmed_fingerprint,confirmed_values_json,confirmed_peers_json,
detail_json,observed_at,confirmed_at,created_at,updated_at)
VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)`,
);
this.#update = options.db.prepare(
`UPDATE device_identities SET status=?,reasons=?,imei=?,manufacturer=?,model=?,revision=?,
agent=?,origin=?,fingerprint=?,confirmed_fingerprint=?,confirmed_values_json=?,
confirmed_peers_json=?,detail_json=?,observed_at=?,confirmed_at=?,updated_at=?
WHERE instance_id=?`,
);
}
/**
* Records what a device just said about itself. Returns undefined when the payload carried no
* identity at all, which keeps a device that does not report its IMEI out of the guard rather
* than inventing a fingerprint from nothing.
*/
observe(instanceId: string, evidence: DeviceIdentityEvidence): DeviceIdentity | undefined {
if (typeof instanceId !== 'string' || !instanceId) return undefined;
const values = identityValues(evidence);
const agent = text(evidence.agent, MAX_FIELD_LENGTH.agent);
const origin = text(evidence.origin, MAX_FIELD_LENGTH.origin);
if (Object.keys(values).length === 0) return undefined;
const now = this.#clock().toISOString();
const fingerprint = digest(values);
const manualFirstReport = this.#authorizationMode() === 'manual';
this.#db.transaction(() => {
const existing = this.#select.get(instanceId) as IdentityRow | undefined;
if (!existing) {
this.#insert.run(
instanceId,
manualFirstReport ? 'pending' : 'confirmed',
'[]',
values.imei ?? '',
values.manufacturer ?? '',
values.model ?? '',
values.revision ?? '',
agent,
origin,
fingerprint,
fingerprint,
JSON.stringify(values),
'[]',
'{"peers":[]}',
now,
now,
now,
now,
);
// A first report is a baseline, but it can still walk into a twin that is already
// registered, so the new row goes through the same evaluation as every other read.
const inserted = this.#select.get(instanceId) as IdentityRow;
this.#write(inserted, values, agent, origin, fingerprint, now, now, manualFirstReport);
} else {
this.#write(existing, values, agent, origin, fingerprint, now, now);
}
this.#syncClaims(instanceId, values.imei ?? '', now);
})();
return this.get(instanceId);
}
list(): readonly DeviceIdentity[] {
const rows = this.#db
.prepare('SELECT * FROM device_identities ORDER BY status DESC, instance_id')
.all() as IdentityRow[];
return rows.map((row) => this.#present(row));
}
get(instanceId: string): DeviceIdentity | undefined {
const row = this.#select.get(instanceId) as IdentityRow | undefined;
return row ? this.#present(row) : undefined;
}
/** Control operations only; reading a device is always allowed, unbinding is the way out. */
isBlocked(instanceId: string): boolean {
const row = this.#select.get(instanceId) as IdentityRow | undefined;
return row?.status === 'pending';
}
summarize(): { readonly tracked: number; readonly pending: readonly string[] } {
const rows = this.#db
.prepare('SELECT instance_id,status FROM device_identities ORDER BY instance_id')
.all() as Array<{ instance_id: string; status: string }>;
return {
tracked: rows.length,
pending: rows.filter((row) => row.status === 'pending').map((row) => row.instance_id),
};
}
/**
* The operator says "this is the device I mean". The current report becomes the baseline and
* the other claimants that were visible at that moment are accepted, so a known twin does not
* reopen the question on the next read while a third one still does.
*/
confirm(instanceId: string): DeviceIdentity {
const row = this.#select.get(instanceId) as IdentityRow | undefined;
if (!row) throw new DeviceIdentityError('NOT_FOUND', 'Device identity was not found');
const now = this.#clock().toISOString();
const values = identityValues({
imei: row.imei,
manufacturer: row.manufacturer,
model: row.model,
revision: row.revision,
});
const fingerprint = row.fingerprint || digest(values);
this.#db.transaction(() => {
this.#update.run(
'confirmed',
'[]',
row.imei,
row.manufacturer,
row.model,
row.revision,
row.agent,
row.origin,
fingerprint,
fingerprint,
JSON.stringify(values),
JSON.stringify(this.#peers(instanceId, row.imei)),
'{"peers":[]}',
row.observed_at,
now,
now,
instanceId,
);
this.#syncClaims(instanceId, row.imei, now);
})();
const updated = this.#select.get(instanceId) as IdentityRow | undefined;
if (!updated) throw new DeviceIdentityError('NOT_FOUND', 'Device identity was not found');
return this.#present(updated);
}
#peers(instanceId: string, imei: string): string[] {
if (!imei) return [];
return (this.#selectPeers.all(imei, instanceId) as Array<{ instance_id: string }>).map(
(row) => row.instance_id,
);
}
/** Re-evaluated on every read, so a conflict that walks away stops holding the node. */
#write(
existing: IdentityRow,
values: IdentityValues,
agent: string,
origin: string,
fingerprint: string,
observedAt: string,
updatedAt: string,
forcePending = false,
): void {
const confirmed = parseValues(existing.confirmed_values_json);
const accepted = parseStringArray(existing.confirmed_peers_json);
const changes: DeviceIdentityFieldChange[] = [];
for (const field of IDENTITY_FIELDS) {
const before = confirmed[field];
const after = values[field];
if (before !== undefined && after !== undefined && before !== after)
changes.push({ field, from: before, to: after });
}
const peers = this.#peers(existing.instance_id, values.imei ?? '');
const unaccepted = peers.filter((peer) => !accepted.includes(peer));
const reasons: IdentityReason[] = [];
if (changes.length > 0) reasons.push('hardware_swapped');
if (unaccepted.length > 0) reasons.push('imei_claimed');
this.#update.run(
reasons.length > 0 || forcePending ? 'pending' : 'confirmed',
JSON.stringify(reasons),
values.imei ?? existing.imei,
values.manufacturer ?? existing.manufacturer,
values.model ?? existing.model,
values.revision ?? existing.revision,
agent || existing.agent,
origin || existing.origin,
fingerprint,
existing.confirmed_fingerprint,
existing.confirmed_values_json,
existing.confirmed_peers_json,
JSON.stringify({ peers: unaccepted, changes }),
observedAt,
existing.confirmed_at,
updatedAt,
existing.instance_id,
);
}
/**
* A twin is told about itself: both records wait for the operator, not just the newcomer. Each
* peer is re-read from its own row, so a claim that walks away releases the peer it was holding
* and a peer that also drifted keeps its hardware_swapped reason.
*/
#syncClaims(instanceId: string, imei: string, now: string): void {
if (!imei) return;
for (const { instance_id: peerId } of this.#selectPeers.all(imei, instanceId) as Array<{
instance_id: string;
}>) {
const peer = this.#select.get(peerId) as IdentityRow | undefined;
if (!peer) continue;
this.#write(
peer,
identityValues(peer),
peer.agent,
peer.origin,
peer.fingerprint || digest(identityValues(peer)),
peer.observed_at,
now,
);
}
}
#present(row: IdentityRow): DeviceIdentity {
const reasons = parseReasons(row.reasons);
const detail = parseDetail(row.detail_json);
const conflicts: DeviceIdentityConflict[] = [];
if (reasons.includes('hardware_swapped'))
conflicts.push({ reason: 'hardware_swapped', instanceIds: [] });
if (reasons.includes('imei_claimed'))
conflicts.push({ reason: 'imei_claimed', instanceIds: detail.peers });
return {
instanceId: row.instance_id,
status: row.status === 'pending' ? 'pending' : 'confirmed',
reasons,
conflicts,
imei: row.imei,
manufacturer: row.manufacturer,
model: row.model,
revision: row.revision,
agent: row.agent,
origin: row.origin,
fingerprint: row.fingerprint,
confirmedFingerprint: row.confirmed_fingerprint,
changes: detail.changes,
observedAt: row.observed_at,
confirmedAt: row.confirmed_at,
};
}
}
function parseStringArray(value: string): string[] {
try {
const parsed: unknown = JSON.parse(value);
return Array.isArray(parsed) ? stringArray(parsed) : [];
} catch {
return [];
}
}
@@ -0,0 +1,107 @@
import Database from 'better-sqlite3';
import { describe, expect, it } from 'vitest';
import { migrateDatabase } from '../../infrastructure/database/migrations.js';
import { SafeUpstreamGateway } from '../../infrastructure/transport/safe-upstream-gateway.js';
import {
InstanceSessionStore,
type UpstreamRequest,
} from '../connections/upstream-session-client.js';
import type { InstanceService } from '../instances/instance-service.js';
import {
DEVICE_ACTIONS,
type DeviceActionDefinition,
type DeviceActionField,
} from './device-action-catalog.js';
import { DeviceActionService } from './device-action-service.js';
const ORIGIN = 'http://192.168.1.20:8080';
/** A value that satisfies the catalog's own validator for every declared field kind. */
function sampleValue(field: DeviceActionField): unknown {
if (field.id === 'phone_number' || field.id === 'sms_center') return '+15550001111';
if (field.id === 'mccmnc') return '00101';
if (field.id === 'iccid') return '89882020202220963176';
switch (field.kind) {
case 'boolean':
return true;
case 'number':
return field.min ?? 1;
case 'string-list':
return ['1'];
case 'choice':
return field.choices?.[0]?.value ?? 'auto';
default:
return 'sample';
}
}
function paramsFor(action: DeviceActionDefinition): Record<string, unknown> {
const params: Record<string, unknown> = {};
for (const field of action.fields ?? []) params[field.id] = sampleValue(field);
return params;
}
function harness() {
const db = new Database(':memory:');
db.pragma('foreign_keys = ON');
migrateDatabase(db);
db.prepare(
`INSERT INTO instances (id,name,base_url,config_revision,created_at,updated_at)
VALUES ('node-a','Node A',?,1,?,?)`,
).run(ORIGIN, '2026-09-04T00:00:00.000Z', '2026-09-04T00:00:00.000Z');
const sessions = new InstanceSessionStore();
sessions.set('node-a', ORIGIN, 'simadmin_session=opaque');
const dispatched: { url: string; method: string; body: string }[] = [];
const gateway = new SafeUpstreamGateway({
transport: {
get: async (url, headers) => {
void headers;
dispatched.push({ url, method: 'GET', body: '' });
return { status: 200, headers: {}, body: '{}' };
},
post: async (url, headers, body) => {
void headers;
dispatched.push({ url, method: 'POST', body });
return { status: 200, headers: {}, body: '{"status":"success"}' };
},
delete: async (url, headers) => {
void headers;
dispatched.push({ url, method: 'DELETE', body: '' });
return { status: 200, headers: {}, body: '{"status":"success"}' };
},
},
});
const instances = {
get: async (id: string) =>
id === 'node-a' ? { id: 'node-a', name: 'Node A', origin: ORIGIN } : undefined,
} as unknown as InstanceService;
const service = new DeviceActionService({
instances,
sessions,
db,
request: (request: UpstreamRequest) => gateway.request(request),
now: () => new Date('2026-09-04T00:00:00.000Z'),
id: () => 'audit-row-1',
});
return { service, db, dispatched };
}
const context = { actor: 'loopback-control-plane', requestId: 'req-1', confirm: true };
describe('device action catalog transport parity', () => {
it('covers every module the console can render', () => {
expect(DEVICE_ACTIONS.length).toBeGreaterThan(40);
});
for (const action of DEVICE_ACTIONS) {
it(`dispatches ${action.id} through the pinned allowlist`, async () => {
const { service, dispatched } = harness();
const result = await service.execute('node-a', action.id, paramsFor(action), context);
expect(result.ok).toBe(true);
expect(dispatched).toHaveLength(1);
expect(dispatched[0]?.url.startsWith(`${ORIGIN}/api/`)).toBe(true);
});
}
});
@@ -0,0 +1,946 @@
import type { InstanceModuleKey } from './instance-module-catalog.js';
export type DeviceActionRisk = 'R1' | 'R2' | 'R3';
export type DeviceActionFieldKind =
| 'boolean'
| 'number'
| 'string'
| 'secret'
| 'choice'
| 'string-list';
export interface DeviceActionChoice {
readonly value: string;
readonly label: string;
}
export interface DeviceActionField {
/** Device payload key, or the name of a `{placeholder}` in the action path. */
readonly id: string;
readonly label: string;
readonly kind: DeviceActionFieldKind;
readonly in?: 'body' | 'path' | 'query';
readonly required?: boolean;
readonly choices?: readonly DeviceActionChoice[];
readonly min?: number;
readonly max?: number;
readonly maxLength?: number;
readonly pattern?: string;
readonly hint?: string;
}
export interface DeviceActionDefinition {
readonly id: string;
readonly module: InstanceModuleKey | 'messages';
readonly title: string;
readonly description: string;
readonly risk: DeviceActionRisk;
readonly method: 'POST' | 'DELETE';
/** Device path appended to `/api`; may contain `{field}` placeholders filled from path fields. */
readonly path: string;
/** Constant payload merged before any user-supplied body fields. */
readonly fixed?: Readonly<Record<string, unknown>>;
/** Sends a literal `{}` document when no field value survives, matching endpoints that still want JSON. */
readonly emptyJsonObject?: boolean;
readonly fields?: readonly DeviceActionField[];
readonly timeoutMs?: number;
}
const BOOL = (id: string, label: string, hint?: string): DeviceActionField => ({
id,
label,
kind: 'boolean',
required: true,
...(hint ? { hint } : {}),
});
const TEXT = (
id: string,
label: string,
options: Partial<DeviceActionField> = {},
): DeviceActionField => ({ id, label, kind: 'string', required: true, maxLength: 128, ...options });
const RADIO_MODES: readonly DeviceActionChoice[] = [
{ value: 'auto', label: '自动' },
{ value: 'lte', label: '仅 LTE' },
{ value: 'nr', label: '5G NR' },
];
const WORK_MODES: readonly DeviceActionChoice[] = [
{ value: 'sim', label: '本机 SIM 管理' },
{ value: 'sim_overseas', label: '海外卡模式' },
{ value: 'esim', label: 'eSIM 模式' },
];
const APN_PROTOCOLS: readonly DeviceActionChoice[] = [
{ value: 'ipv4', label: 'IPv4' },
{ value: 'ipv6', label: 'IPv6' },
{ value: 'ipv4v6', label: 'IPv4/IPv6' },
];
const APN_AUTH: readonly DeviceActionChoice[] = [
{ value: 'none', label: '不认证' },
{ value: 'pap', label: 'PAP' },
{ value: 'chap', label: 'CHAP' },
];
const CELL_RATS: readonly DeviceActionChoice[] = [
{ value: 'lte', label: 'LTE' },
{ value: 'nr', label: 'NR' },
{ value: 'wcdma', label: 'WCDMA' },
{ value: 'gsm', label: 'GSM' },
];
const EMPTY_BODY = {};
/**
* Every device mutation the console is allowed to perform. Paths and payload keys mirror the
* SimAdmin device API one for one; anything not listed here cannot be reached from the UI.
*/
export const DEVICE_ACTIONS: readonly DeviceActionDefinition[] = Object.freeze([
{
id: 'sim.refresh-details',
module: 'sim',
title: '刷新 SIM 详情',
description: '要求设备重新读取 SIM 卡详细信息并刷新缓存。',
risk: 'R1',
method: 'POST',
path: '/sim/details/refresh',
fixed: EMPTY_BODY,
timeoutMs: 8_000,
},
{
id: 'sim.cache-phone',
module: 'sim',
title: '登记本机号码',
description: '把本机号码写入设备缓存,供短信与通话界面显示。',
risk: 'R1',
method: 'POST',
path: '/sim/cache',
fields: [TEXT('phone_number', '本机号码', { maxLength: 32, pattern: '^\\+?[0-9 ()-]{3,32}$' })],
},
{
id: 'sim.cache-sms-center',
module: 'sim',
title: '登记短信中心号码',
description: '写入短信中心(SMSC)号码。',
risk: 'R1',
method: 'POST',
path: '/sim/cache',
fields: [
TEXT('sms_center', '短信中心号码', { maxLength: 32, pattern: '^\\+?[0-9 ()-]{3,32}$' }),
],
},
{
id: 'apn.save',
module: 'sim',
title: '保存 APN 配置',
description: '修改指定 PDN 上下文的 APN、协议与认证方式。',
risk: 'R2',
method: 'POST',
path: '/apn',
fields: [
TEXT('context_path', 'PDN 上下文路径', { maxLength: 128 }),
{ id: 'apn', label: 'APN 名称', kind: 'string', maxLength: 128 },
{ id: 'protocol', label: '协议', kind: 'choice', choices: APN_PROTOCOLS },
{ id: 'username', label: '用户名', kind: 'string', maxLength: 128 },
{ id: 'password', label: '密码', kind: 'secret', maxLength: 128 },
{ id: 'auth_method', label: '认证方式', kind: 'choice', choices: APN_AUTH },
],
},
{
id: 'band-lock.apply',
module: 'sim',
title: '锁定频段',
description: '按制式提交允许使用的频段列表,设备只会驻留这些频段。',
risk: 'R2',
method: 'POST',
path: '/band-lock',
fields: [
{ id: 'lte_fdd_bands', label: 'LTE FDD 频段', kind: 'string-list' },
{ id: 'lte_tdd_bands', label: 'LTE TDD 频段', kind: 'string-list' },
{ id: 'nr_fdd_bands', label: 'NR FDD 频段', kind: 'string-list' },
{ id: 'nr_tdd_bands', label: 'NR TDD 频段', kind: 'string-list' },
],
},
{
id: 'band-lock.clear',
module: 'sim',
title: '解除频段锁定',
description: '提交空频段列表,恢复设备默认选网。',
risk: 'R2',
method: 'POST',
path: '/band-lock',
fixed: { lte_fdd_bands: [], lte_tdd_bands: [], nr_fdd_bands: [], nr_tdd_bands: [] },
},
{
id: 'cell-lock.apply',
module: 'sim',
title: '锁定小区',
description: '按频点与 PCI 锁定到指定小区,用于信号排查。',
risk: 'R2',
method: 'POST',
path: '/cell-lock',
fields: [
{ id: 'rat', label: '制式', kind: 'choice', required: true, choices: CELL_RATS },
BOOL('enable', '启用小区锁定'),
{
id: 'arfcn',
label: 'ARFCN 频点',
kind: 'number',
required: true,
min: 0,
max: 2_684_354_555,
},
{ id: 'pci', label: 'PCI 物理小区号', kind: 'number', required: true, min: 0, max: 1_007 },
],
},
{
id: 'cell-lock.unlock-all',
module: 'sim',
title: '解除小区锁定',
description: '释放全部已锁定的小区。',
risk: 'R2',
method: 'POST',
path: '/cell-lock/unlock-all',
fixed: EMPTY_BODY,
},
{
id: 'data.set',
module: 'cellular',
title: '数据开关',
description: '开启或关闭设备的蜂窝数据连接。',
risk: 'R1',
method: 'POST',
path: '/data',
fields: [BOOL('active', '启用蜂窝数据')],
},
{
id: 'network.scan',
module: 'cellular',
title: '扫描运营商',
description: '让设备重新扫描可见运营商列表。',
risk: 'R1',
method: 'POST',
path: '/network/operators/scan',
fixed: EMPTY_BODY,
timeoutMs: 30_000,
},
{
id: 'network.register-manual',
module: 'cellular',
title: '手动注册网络',
description: '按 MCCMNC 强制注册到指定运营商。',
risk: 'R2',
method: 'POST',
path: '/network/register-manual',
fields: [TEXT('mccmnc', '运营商 MCCMNC', { maxLength: 6, pattern: '^[0-9]{5,6}$' })],
timeoutMs: 30_000,
},
{
id: 'radio-mode.set',
module: 'cellular',
title: '切换网络模式',
description: '切换首选无线制式,切换期间会短暂断网。',
risk: 'R2',
method: 'POST',
path: '/radio-mode',
fields: [
{ id: 'mode', label: '网络模式', kind: 'choice', required: true, choices: RADIO_MODES },
],
timeoutMs: 20_000,
},
{
id: 'roaming.set',
module: 'cellular',
title: '数据漫游开关',
description: '允许或禁止在漫游网络上注册。',
risk: 'R2',
method: 'POST',
path: '/roaming',
fields: [BOOL('allowed', '允许漫游')],
},
{
id: 'airplane-mode.set',
module: 'cellular',
title: '飞行模式',
description: '开启飞行模式会立即切断全部无线连接。',
risk: 'R3',
method: 'POST',
path: '/airplane-mode',
fields: [BOOL('enabled', '开启飞行模式')],
},
{
id: 'cell-monitor.start',
module: 'cellular',
title: '启动小区监视',
description: '开始持续采集服务小区与邻区数据。',
risk: 'R1',
method: 'POST',
path: '/cell-monitor/start',
fixed: EMPTY_BODY,
},
{
id: 'cell-monitor.stop',
module: 'cellular',
title: '停止小区监视',
description: '停止后台小区采样,降低设备负载。',
risk: 'R1',
method: 'POST',
path: '/cell-monitor/stop',
fixed: EMPTY_BODY,
},
{
id: 'baseband.restart',
module: 'cellular',
title: '重启基带',
description: '重启调制解调器,期间设备会完全离线约一分钟。',
risk: 'R3',
method: 'POST',
path: '/baseband/restart',
fixed: EMPTY_BODY,
timeoutMs: 60_000,
},
{
id: 'wlan.enabled',
module: 'device-network',
title: 'WLAN 开关',
description: '开启或关闭设备无线网卡。',
risk: 'R1',
method: 'POST',
path: '/device-network/wlan/enabled',
fields: [BOOL('enabled', '启用 WLAN')],
},
{
id: 'wlan.scan',
module: 'device-network',
title: '扫描 WLAN',
description: '触发一次无线网络扫描并刷新热点列表。',
risk: 'R1',
method: 'POST',
path: '/device-network/wlan/scan',
fixed: EMPTY_BODY,
timeoutMs: 30_000,
},
{
id: 'wlan.connect',
module: 'device-network',
title: '连接 WLAN',
description: '加入指定热点,密码留空表示使用已保存的凭据。',
risk: 'R2',
method: 'POST',
path: '/device-network/wlan/connect',
fields: [
TEXT('ssid', '网络名称(SSID', { maxLength: 64 }),
{ id: 'password', label: '密码', kind: 'secret', maxLength: 128 },
{ id: 'auto_join', label: '自动加入', kind: 'boolean' },
],
timeoutMs: 30_000,
},
{
id: 'wlan.disconnect',
module: 'device-network',
title: '断开 WLAN',
description: '断开当前无线连接,不会删除已保存配置。',
risk: 'R1',
method: 'POST',
path: '/device-network/wlan/disconnect',
fixed: EMPTY_BODY,
},
{
id: 'wlan.forget',
module: 'device-network',
title: '忽略 WLAN 网络',
description: '删除已保存的热点配置。',
risk: 'R2',
method: 'POST',
path: '/device-network/wlan/forget',
fields: [
TEXT('uuid', '配置 UUID', { maxLength: 64 }),
TEXT('connection_id', '连接标识', { maxLength: 64 }),
],
},
{
id: 'ddns.sync',
module: 'device-network',
title: '立即同步 DDNS',
description: '强制把当前公网地址推送到 DDNS 服务商。',
risk: 'R1',
method: 'POST',
path: '/device-network/ddns/sync',
fixed: EMPTY_BODY,
timeoutMs: 30_000,
},
{
id: 'ddns.logs-clear',
module: 'device-network',
title: '清空 DDNS 日志',
description: '删除设备上的 DDNS 更新记录。',
risk: 'R2',
method: 'POST',
path: '/device-network/ddns/logs/clear',
fixed: EMPTY_BODY,
emptyJsonObject: true,
},
{
id: 'esim.enable-profile',
module: 'esim',
title: '切换 eSIM 配置',
description: '启用指定 ICCID 的 Profile,设备会重新注网。',
risk: 'R2',
method: 'POST',
path: '/esim/profiles/{iccid}/enable',
fields: [
TEXT('iccid', 'ICCID', { in: 'path', maxLength: 32, pattern: '^[A-Za-z0-9_.\\-]{1,32}$' }),
],
timeoutMs: 60_000,
},
{
id: 'esim.rename-profile',
module: 'esim',
title: '重命名 eSIM 配置',
description: '修改 Profile 的显示名称。',
risk: 'R1',
method: 'POST',
path: '/esim/profiles/{iccid}/rename',
fields: [
TEXT('iccid', 'ICCID', { in: 'path', maxLength: 32, pattern: '^[A-Za-z0-9_.\\-]{1,32}$' }),
TEXT('name', '新名称', { maxLength: 64 }),
],
timeoutMs: 60_000,
},
{
id: 'esim.delete-profile',
module: 'esim',
title: '删除 eSIM 配置',
description: '从 eUICC 移除 Profile,操作不可撤销。',
risk: 'R3',
method: 'DELETE',
path: '/esim/profiles/{iccid}',
fields: [
TEXT('iccid', 'ICCID', { in: 'path', maxLength: 32, pattern: '^[A-Za-z0-9_.\\-]{1,32}$' }),
],
timeoutMs: 60_000,
},
{
id: 'esim.download-profile',
module: 'esim',
title: '下载 eSIM 配置',
description: '通过 SM-DP+ 服务器向 eUICC 写入新的 Profile。',
risk: 'R3',
method: 'POST',
path: '/esim/profiles',
fields: [
TEXT('smdp', 'SM-DP+ 地址', { maxLength: 128 }),
TEXT('matching_id', 'Matching ID', { maxLength: 64 }),
{ id: 'confirmation_code', label: '确认码', kind: 'secret', maxLength: 64 },
{ id: 'imei', label: 'IMEI', kind: 'string', maxLength: 32 },
],
timeoutMs: 180_000,
},
{
id: 'esim.lpac-repair',
module: 'esim',
title: '修复 lpac',
description: '重新初始化设备的 lpac 组件。',
risk: 'R2',
method: 'POST',
path: '/esim/lpac/repair',
fields: [{ id: 'proxy_prefix', label: '代理前缀', kind: 'string', maxLength: 128 }],
timeoutMs: 120_000,
},
{
id: 'call.dial',
module: 'calls',
title: '拨号',
description: '让设备拨打指定号码。',
risk: 'R2',
method: 'POST',
path: '/call/dial',
fields: [TEXT('phone_number', '被叫号码', { maxLength: 32, pattern: '^\\+?[0-9 ()-]{3,32}$' })],
},
{
id: 'call.answer',
module: 'calls',
title: '接听通话',
description: '接听指定通话对象。',
risk: 'R1',
method: 'POST',
path: '/call/answer',
fields: [TEXT('path', '通话对象路径', { maxLength: 128 })],
},
{
id: 'call.hangup',
module: 'calls',
title: '挂断通话',
description: '挂断指定通话对象。',
risk: 'R1',
method: 'POST',
path: '/call/hangup',
fields: [TEXT('path', '通话对象路径', { maxLength: 128 })],
},
{
id: 'call.hangup-all',
module: 'calls',
title: '挂断全部通话',
description: '结束设备上所有进行中的通话。',
risk: 'R2',
method: 'POST',
path: '/call/hangup-all',
fixed: EMPTY_BODY,
},
{
id: 'call.waiting',
module: 'calls',
title: '呼叫等待',
description: '开启或关闭网络侧呼叫等待业务。',
risk: 'R2',
method: 'POST',
path: '/call/settings',
fixed: { property: 'VoiceCallWaiting' },
fields: [
{
id: 'value',
label: '呼叫等待',
kind: 'choice',
required: true,
choices: [
{ value: 'enabled', label: '开启' },
{ value: 'disabled', label: '关闭' },
],
},
],
},
{
id: 'call.history-clear',
module: 'calls',
title: '清空通话记录',
description: '删除设备上的全部通话历史。',
risk: 'R2',
method: 'POST',
path: '/call/history/clear',
fixed: EMPTY_BODY,
},
{
id: 'sms.clear',
module: 'messages',
title: '清空全部短信',
description: '删除设备上保存的所有短信记录。',
risk: 'R3',
method: 'POST',
path: '/sms/clear',
fixed: EMPTY_BODY,
},
{
id: 'work-mode.set',
module: 'configuration',
title: '切换工作模式',
description: '切换设备的管理模式,设备会重启相关服务。',
risk: 'R3',
method: 'POST',
path: '/work-mode',
fixed: { confirm: true },
fields: [
{ id: 'mode', label: '工作模式', kind: 'choice', required: true, choices: WORK_MODES },
],
timeoutMs: 20_000,
},
{
id: 'hub.configure',
module: 'configuration',
title: '配置设备回连',
description: '设置设备回连中心平台的地址与本地兜底策略。',
risk: 'R2',
method: 'POST',
path: '/hub',
fields: [
BOOL('enabled', '启用回连'),
{ id: 'url', label: '回连地址', kind: 'string', maxLength: 256 },
{ id: 'local_fallback_enabled', label: '启用本地兜底', kind: 'boolean' },
{
id: 'local_fallback_timeout_seconds',
label: '本地兜底超时(秒)',
kind: 'number',
min: 10,
max: 86_400,
},
],
},
{
id: 'hub.unbind',
module: 'configuration',
title: '解除回连绑定',
description: '清除设备的中心平台绑定,改由本机管理。',
risk: 'R3',
method: 'POST',
path: '/hub/unbind',
fixed: EMPTY_BODY,
},
{
id: 'notifications.test-channel',
module: 'notifications',
title: '发送测试通知',
description: '通过指定渠道发送一条测试消息。',
risk: 'R1',
method: 'POST',
path: '/notifications/test/{channel}',
fields: [TEXT('channel', '渠道标识', { in: 'path', maxLength: 64 })],
timeoutMs: 30_000,
},
{
id: 'notifications.logs-clear',
module: 'notifications',
title: '清空通知日志',
description: '删除设备上的通知发送记录。',
risk: 'R2',
method: 'POST',
path: '/notifications/logs/clear',
fixed: EMPTY_BODY,
emptyJsonObject: true,
},
{
id: 'notifications.queue-retry-all',
module: 'notifications',
title: '重投全部待办通知',
description: '让设备立即重试通知队列中所有待发送与失败的任务。',
risk: 'R2',
method: 'POST',
path: '/notifications/queue/retry-all',
fixed: EMPTY_BODY,
timeoutMs: 30_000,
},
{
id: 'notifications.queue-clear',
module: 'notifications',
title: '清空通知队列',
description: '丢弃设备上尚未发送与已处理的通知队列条目。',
risk: 'R2',
method: 'POST',
path: '/notifications/queue/clear',
fixed: EMPTY_BODY,
},
{
id: 'automation.test-task',
module: 'automation',
title: '试运行自动化任务',
description: '立即执行一次指定的设备自动化任务。',
risk: 'R1',
method: 'POST',
path: '/automation/test/{taskId}',
fields: [TEXT('taskId', '任务标识', { in: 'path', maxLength: 64 })],
timeoutMs: 60_000,
},
{
id: 'automation.logs-clear',
module: 'automation',
title: '清空自动化日志',
description: '删除设备上的自动化执行记录。',
risk: 'R2',
method: 'POST',
path: '/automation/logs/clear',
fixed: EMPTY_BODY,
emptyJsonObject: true,
},
{
id: 'ota.check-release',
module: 'ota',
title: '检查最新版本',
description: '向发布源查询最新固件版本信息。',
risk: 'R1',
method: 'POST',
path: '/ota/latest-release',
fixed: { include_variants: true },
fields: [{ id: 'proxy_prefix', label: '加速节点前缀', kind: 'string', maxLength: 256 }],
timeoutMs: 60_000,
},
{
id: 'ota.online-prepare',
module: 'ota',
title: '在线下载更新包',
description: '让设备下载并暂存指定的升级包。',
risk: 'R2',
method: 'POST',
path: '/ota/online-prepare',
fields: [
TEXT('asset_name', '升级包名称', { maxLength: 128 }),
{ id: 'proxy_prefix', label: '加速节点前缀', kind: 'string', maxLength: 256 },
],
timeoutMs: 300_000,
},
{
id: 'ota.apply',
module: 'ota',
title: '应用更新',
description: '安装已暂存的升级包,设备可能自动重启。',
risk: 'R3',
method: 'POST',
path: '/ota/apply',
fields: [BOOL('restart_now', '升级后立即重启')],
timeoutMs: 300_000,
},
{
id: 'ota.cancel',
module: 'ota',
title: '取消更新',
description: '丢弃已暂存的升级包。',
risk: 'R1',
method: 'POST',
path: '/ota/cancel',
fixed: EMPTY_BODY,
},
{
id: 'vowifi.feature',
module: 'vowifi',
title: 'VoWiFi 功能开关',
description: '启用或停用设备的 VoWiFi 能力。',
risk: 'R2',
method: 'POST',
path: '/vowifi/feature',
fields: [BOOL('enabled', '启用 VoWiFi')],
timeoutMs: 20_000,
},
{
id: 'vowifi.connection',
module: 'vowifi',
title: 'VoWiFi 连接开关',
description: '建立或断开 VoWiFi IMS 连接。',
risk: 'R2',
method: 'POST',
path: '/vowifi/connection',
fields: [BOOL('enabled', '保持连接')],
timeoutMs: 120_000,
},
{
id: 'vowifi.connect',
module: 'vowifi',
title: '发起 VoWiFi 注册',
description: '立即尝试一次 IMS 注册。',
risk: 'R2',
method: 'POST',
path: '/vowifi/connect',
fixed: EMPTY_BODY,
timeoutMs: 120_000,
},
{
id: 'backup.export-local',
module: 'device-backup',
title: '本机生成备份',
description: '在设备上生成所选组件的备份文件。',
risk: 'R2',
method: 'POST',
path: '/backup/export-local',
fields: [
{
id: 'components',
label: '备份组件',
kind: 'string-list',
required: true,
hint: '留空表示全部',
},
],
timeoutMs: 120_000,
},
{
id: 'backup.data-clear',
module: 'device-backup',
title: '清除设备数据',
description: '按组件清除设备本地数据,操作不可撤销。',
risk: 'R3',
method: 'POST',
path: '/backup/data/clear',
fields: [{ id: 'components', label: '要清除的组件', kind: 'string-list', required: true }],
timeoutMs: 120_000,
},
{
id: 'backup.delete-file',
module: 'device-backup',
title: '删除备份文件',
description: '删除设备上的一个备份文件。',
risk: 'R3',
method: 'DELETE',
path: '/backup/files/{name}',
fields: [TEXT('name', '备份文件名', { in: 'path', maxLength: 128 })],
},
{
id: 'backup.apply-file',
module: 'device-backup',
title: '恢复备份文件',
description: '用设备上的备份文件恢复所选组件。',
risk: 'R3',
method: 'POST',
path: '/backup/files/{name}/apply',
fields: [
TEXT('name', '备份文件名', { in: 'path', maxLength: 128 }),
{
id: 'mode',
label: '恢复模式',
kind: 'choice',
in: 'query',
required: true,
choices: [
{ value: 'replace', label: '整体替换' },
{ value: 'merge', label: '合并' },
],
},
{ id: 'components', label: '恢复组件', kind: 'string-list', in: 'query', required: true },
],
timeoutMs: 120_000,
},
{
id: 'network.register-auto',
module: 'cellular',
title: '自动注册网络',
description: '让设备重新向网络发起自动注册,通常用于掉网后恢复驻网。',
risk: 'R2',
method: 'POST',
path: '/network/register-auto',
fixed: EMPTY_BODY,
timeoutMs: 30_000,
},
{
id: 'call.volume.set',
module: 'calls',
title: '调整通话音量',
description: '设置通话的扬声器与麦克风音量,部分固件未开放该能力。',
risk: 'R1',
method: 'POST',
path: '/call/volume',
fields: [
{ id: 'speaker_volume', label: '扬声器音量', kind: 'number', min: 0, max: 100 },
{ id: 'microphone_volume', label: '麦克风音量', kind: 'number', min: 0, max: 100 },
{ id: 'muted', label: '静音', kind: 'boolean' },
],
},
{
id: 'call.forwarding.set',
module: 'calls',
title: '设置呼叫转移',
description: '按转移类型登记或清除呼转号码,部分固件未开放该能力。',
risk: 'R2',
method: 'POST',
path: '/call/forwarding',
fields: [
{
id: 'forward_type',
label: '转移类型',
kind: 'choice',
required: true,
choices: [
{ value: 'unconditional', label: '无条件转移' },
{ value: 'busy', label: '遇忙转移' },
{ value: 'no_reply', label: '无应答转移' },
{ value: 'not_reachable', label: '不可达转移' },
],
},
TEXT('number', '转移目标号码', { maxLength: 32 }),
{ id: 'timeout', label: '无应答等待(秒)', kind: 'number', min: 0, max: 300 },
],
},
{
id: 'esim.config.save',
module: 'esim',
title: '保存 eSIM 配置',
description: '设置设备侧 lpac 可执行文件路径与自定义 eUICC 可用内存。',
risk: 'R2',
method: 'POST',
path: '/esim/config',
fields: [
{ id: 'lpac_path', label: 'lpac 路径', kind: 'string', maxLength: 256 },
{
id: 'custom_memory_total_kb',
label: '自定义内存总量(KB',
kind: 'number',
min: 1,
max: 1_000_000,
},
],
},
{
id: 'wlan.profile.save',
module: 'device-network',
title: '保存 WLAN 配置',
description: '修改已保存热点的自动加入与 IPv4 获取方式。',
risk: 'R2',
method: 'POST',
path: '/device-network/wlan/profile',
fields: [
TEXT('connection_id', '连接标识', { maxLength: 64 }),
{ id: 'auto_join', label: '自动加入', kind: 'boolean' },
{
id: 'ipv4_mode',
label: 'IPv4 模式',
kind: 'choice',
choices: [
{ value: 'dhcp', label: 'DHCP 自动获取' },
{ value: 'manual', label: '手动指定' },
],
},
{ id: 'ipv4_address', label: 'IPv4 地址', kind: 'string', maxLength: 45 },
{ id: 'ipv4_prefix', label: 'IPv4 前缀长度', kind: 'number', min: 0, max: 128 },
{ id: 'ipv4_gateway', label: 'IPv4 网关', kind: 'string', maxLength: 45 },
],
},
{
id: 'auth.settings.save',
module: 'configuration',
title: '保存安全设置',
description: '调整设备后台的密码策略与会话有效期。',
risk: 'R2',
method: 'POST',
path: '/auth/settings',
fields: [
{ id: 'password_protection_enabled', label: '启用密码保护', kind: 'boolean' },
{ id: 'password_min_length', label: '密码最小长度', kind: 'number', min: 1, max: 32 },
{ id: 'password_require_letters', label: '密码须含字母', kind: 'boolean' },
{ id: 'password_require_digits', label: '密码须含数字', kind: 'boolean' },
{ id: 'password_require_symbols', label: '密码须含符号', kind: 'boolean' },
{
id: 'session_ttl_seconds',
label: '会话有效期(秒)',
kind: 'number',
min: 60,
max: 2_592_000,
},
{
id: 'idle_timeout_seconds',
label: '空闲超时(秒)',
kind: 'number',
min: 60,
max: 2_592_000,
},
],
},
{
id: 'auth.password.set',
module: 'configuration',
title: '修改设备管理密码',
description: '设置设备后台的登录密码,修改后需要重新登录设备。',
risk: 'R3',
method: 'POST',
path: '/auth/password',
fields: [
{ id: 'new_password', label: '新密码', kind: 'secret', required: true, maxLength: 128 },
],
},
]);
export const DEVICE_ACTION_MODULES: readonly (InstanceModuleKey | 'messages')[] = [
'overview',
'sim',
'cellular',
'device-network',
'esim',
'calls',
'messages',
'configuration',
'device-backup',
'notifications',
'automation',
'ota',
'vowifi',
];
export function findDeviceAction(id: string): DeviceActionDefinition | undefined {
return DEVICE_ACTIONS.find((action) => action.id === id);
}
export function deviceActionsFor(
module: InstanceModuleKey | 'messages',
): readonly DeviceActionDefinition[] {
return DEVICE_ACTIONS.filter((action) => action.module === module);
}
@@ -0,0 +1,221 @@
import Database from 'better-sqlite3';
import { describe, expect, it } from 'vitest';
import { migrateDatabase } from '../../infrastructure/database/migrations.js';
import {
InstanceSessionStore,
type UpstreamRequest,
} from '../connections/upstream-session-client.js';
import type { InstanceService } from '../instances/instance-service.js';
import { DeviceActionError, DeviceActionService } from './device-action-service.js';
interface Reply {
readonly status: number;
readonly body?: unknown;
}
function fixture(
replies: readonly Reply[] = [{ status: 200, body: { status: 'success' } }],
extra: {
readonly ensureSession?: (id: string, origin: string, force?: boolean) => Promise<void>;
readonly identities?: { isBlocked(instanceId: string): boolean };
} = {},
) {
const db = new Database(':memory:');
db.pragma('foreign_keys = ON');
migrateDatabase(db);
db.prepare(
`INSERT INTO instances (id,name,base_url,config_revision,created_at,updated_at)
VALUES ('node-a','Node A','http://node-a.local',1,?,?)`,
).run('2026-09-04T00:00:00.000Z', '2026-09-04T00:00:00.000Z');
const sessions = new InstanceSessionStore();
sessions.set('node-a', 'http://node-a.local', 'simadmin_session=opaque');
const calls: UpstreamRequest[] = [];
let cursor = 0;
const instances = {
get: async (id: string) =>
id === 'node-a' ? { id: 'node-a', name: 'Node A', origin: 'http://node-a.local' } : undefined,
} as unknown as InstanceService;
const service = new DeviceActionService({
instances,
sessions,
db,
request: async (request) => {
calls.push(request);
const reply = replies[Math.min(cursor, replies.length - 1)] as Reply;
cursor += 1;
return {
status: reply.status,
headers: {},
body: reply.body === undefined ? '' : JSON.stringify(reply.body),
};
},
...(extra.ensureSession ? { ensureSession: extra.ensureSession } : {}),
...(extra.identities ? { identities: extra.identities } : {}),
now: () => new Date('2026-09-04T00:00:00.000Z'),
id: () => 'audit-row-1',
});
return { service, db, calls, sessions };
}
const context = { actor: 'loopback-control-plane', requestId: 'req-1', confirm: true };
describe('DeviceActionService', () => {
it('exposes the catalog so the console can render controls without knowing device paths', () => {
const { service } = fixture();
const actions = service.list();
expect(actions.length).toBeGreaterThan(40);
expect(actions.every((action) => !('path' in action))).toBe(true);
expect(actions.find((action) => action.id === 'band-lock.apply')?.fields).toHaveLength(4);
});
it('dispatches a zero-body action with only an accept header', async () => {
const { service, calls } = fixture();
const result = await service.execute('node-a', 'cell-lock.unlock-all', {}, context);
expect(result.ok).toBe(true);
expect(calls[0]).toMatchObject({
url: 'http://node-a.local/api/cell-lock/unlock-all',
method: 'POST',
headers: { accept: 'application/json', cookie: 'simadmin_session=opaque' },
deviceAction: { body: undefined },
});
});
it('merges fixed payload keys with validated body fields', async () => {
const { service, calls } = fixture();
await service.execute('node-a', 'call.waiting', { value: 'enabled' }, context);
expect(calls[0]?.deviceAction?.body).toEqual({
property: 'VoiceCallWaiting',
value: 'enabled',
});
});
it('substitutes path parameters and builds the restore query in order', async () => {
const { service, calls } = fixture();
await service.execute(
'node-a',
'backup.apply-file',
{ name: 'backup-2026.tar.gz', mode: 'merge', components: ['instances', 'jobs'] },
context,
);
expect(calls[0]).toMatchObject({
url: 'http://node-a.local/api/backup/files/backup-2026.tar.gz/apply?mode=merge&components=instances,jobs',
method: 'POST',
});
});
it('rejects a risky action without an explicit confirmation', async () => {
const { service, calls } = fixture();
await expect(
service.execute(
'node-a',
'airplane-mode.set',
{ enabled: true },
{ actor: 'loopback-control-plane', requestId: 'req-1' },
),
).rejects.toThrow(DeviceActionError);
expect(calls).toHaveLength(0);
});
it('rejects unknown params, out-of-range numbers and values outside a choice set', async () => {
const { service, calls } = fixture();
await expect(
service.execute('node-a', 'data.set', { active: true, extra: 1 }, context),
).rejects.toMatchObject({ code: 'VALIDATION_FAILED', fieldId: 'extra' });
await expect(
service.execute(
'node-a',
'cell-lock.apply',
{ rat: 'lte', enable: true, arfcn: -1, pci: 5 },
context,
),
).rejects.toMatchObject({ code: 'VALIDATION_FAILED', fieldId: 'arfcn' });
await expect(
service.execute('node-a', 'radio-mode.set', { mode: 'cdma' }, context),
).rejects.toMatchObject({ code: 'VALIDATION_FAILED', fieldId: 'mode' });
await expect(
service.execute('node-a', 'sim.cache-phone', { phone_number: 'abc' }, context),
).rejects.toMatchObject({ code: 'VALIDATION_FAILED', fieldId: 'phone_number' });
expect(calls).toHaveLength(0);
});
it('refuses a path parameter that could escape the device path', async () => {
const { service, calls } = fixture();
await expect(
service.execute('node-a', 'esim.delete-profile', { iccid: '../../admin' }, context),
).rejects.toMatchObject({ code: 'VALIDATION_FAILED', fieldId: 'iccid' });
expect(calls).toHaveLength(0);
});
it('refreshes an expired device session once and retries', async () => {
const { service, calls, sessions } = fixture(
[
{ status: 401, body: { status: 'error', msg: 'unauthorized' } },
{ status: 200, body: { status: 'success', data: { applied: true } } },
],
{
ensureSession: async () => {
sessions.set('node-a', 'http://node-a.local', 'simadmin_session=rotated');
},
},
);
const result = await service.execute('node-a', 'ota.cancel', {}, context);
expect(calls).toHaveLength(2);
expect(result.ok).toBe(true);
expect(calls[0]?.headers.cookie).toBe('simadmin_session=opaque');
expect(calls.at(-1)?.headers.cookie).toBe('simadmin_session=rotated');
});
it('reports a device failure without pretending it succeeded', async () => {
const { service } = fixture([{ status: 200, body: { status: 'error', msg: 'SIM 卡未就绪' } }]);
const result = await service.execute('node-a', 'sim.refresh-details', {}, context);
expect(result.ok).toBe(false);
expect(result.message).toBe('SIM 卡未就绪');
});
it('writes a fully redacted audit record for every dispatch', async () => {
const { service, db } = fixture();
await service.execute(
'node-a',
'wlan.connect',
{ ssid: 'office', password: 'sup3r-secret', auto_join: true },
context,
);
const row = db
.prepare(
'SELECT actor,operation_id,risk_level,result_code,parameters_summary_json,duration_ms FROM audit_events WHERE id=?',
)
.get('audit-row-1') as Record<string, unknown>;
expect(row).toMatchObject({
actor: 'loopback-control-plane',
operation_id: 'wlan.connect',
risk_level: 'R2',
result_code: 'succeeded',
});
const summary = JSON.parse(String(row.parameters_summary_json)) as Record<string, unknown>[];
expect(summary.map((item) => item.fieldId)).toEqual(['ssid', 'password', 'auto_join']);
expect(JSON.stringify(summary)).not.toContain('sup3r-secret');
expect(JSON.stringify(summary)).not.toContain('office');
expect(summary.every((item) => item.redacted === true)).toBe(true);
});
it('holds every control while the device identity is disputed', async () => {
const { service, calls } = fixture(undefined, {
identities: { isBlocked: () => true },
});
await expect(
service.execute('node-a', 'cell-lock.unlock-all', {}, context),
).rejects.toMatchObject({ code: 'IDENTITY_UNCONFIRMED' });
expect(calls).toHaveLength(0);
});
it('still lets an operator release the binding on a disputed device', async () => {
const { service, calls } = fixture(undefined, {
identities: { isBlocked: () => true },
});
const result = await service.execute('node-a', 'hub.unbind', {}, context);
expect(result.ok).toBe(true);
expect(calls).toHaveLength(1);
});
});
@@ -0,0 +1,458 @@
import { randomUUID } from 'node:crypto';
import type { InstanceService } from './instance-service.js';
import {
DEVICE_ACTIONS,
findDeviceAction,
type DeviceActionDefinition,
type DeviceActionField,
type DeviceActionRisk,
} from './device-action-catalog.js';
import type { SqliteDatabase } from '../../infrastructure/database/database.js';
import type {
InstanceSessionStore,
UpstreamResponse,
UpstreamSessionClientOptions,
} from '../connections/upstream-session-client.js';
export type DeviceActionErrorCode =
| 'NOT_FOUND'
| 'VALIDATION_FAILED'
| 'SESSION_INVALID'
| 'UPSTREAM_FAILED'
| 'NOT_DISPATCHED'
| 'IDENTITY_UNCONFIRMED';
export class DeviceActionError extends Error {
constructor(
readonly code: DeviceActionErrorCode,
readonly fieldId?: string,
) {
super(code);
this.name = 'DeviceActionError';
}
}
export interface DeviceActionDescriptor {
readonly id: string;
readonly module: DeviceActionDefinition['module'];
readonly title: string;
readonly description: string;
readonly risk: DeviceActionRisk;
readonly method: DeviceActionDefinition['method'];
readonly fields: readonly DeviceActionField[];
}
export interface DeviceActionResult {
readonly actionId: string;
readonly title: string;
readonly status: number;
readonly ok: boolean;
readonly data: unknown;
readonly message: string | null;
readonly durationMs: number;
}
export interface DeviceActionContext {
readonly actor: string;
readonly requestId: string;
/** R2 and R3 actions only dispatch when the caller repeats an explicit confirmation. */
readonly confirm?: boolean;
}
const CONTROL_CHARACTERS = /[\u0000-\u0008\u000b\u000c\u000e-\u001f\u007f]/u;
const SECRET_KEY =
/(password|passwd|secret|token|cookie|authorization|apikey|api_key|privatekey|private_key|session|confirmation_code)/iu;
const MAX_RESPONSE_BYTES = 32_768;
const MAX_DEPTH = 5;
const MAX_ARRAY_ENTRIES = 64;
const MAX_STRING_LENGTH = 512;
const DEFAULT_TIMEOUT_MS = 15_000;
const TIMEOUT_SENTINEL: UpstreamResponse = { status: 0, headers: {}, body: '' };
function isRecord(value: unknown): value is Record<string, unknown> {
return value !== null && typeof value === 'object' && !Array.isArray(value);
}
function sanitize(value: unknown, depth = 0): unknown {
if (value === null) return null;
switch (typeof value) {
case 'string':
return value.length > MAX_STRING_LENGTH
? `${value.slice(0, MAX_STRING_LENGTH)}...`
: CONTROL_CHARACTERS.test(value)
? value.replace(CONTROL_CHARACTERS, '')
: value;
case 'number':
return Number.isFinite(value) ? value : null;
case 'boolean':
return value;
default:
break;
}
if (depth >= MAX_DEPTH) return null;
if (Array.isArray(value))
return value.slice(0, MAX_ARRAY_ENTRIES).map((item) => sanitize(item, depth + 1));
if (isRecord(value)) {
const output: Record<string, unknown> = {};
for (const [key, entry] of Object.entries(value)) {
if (SECRET_KEY.test(key)) continue;
output[key] = sanitize(entry, depth + 1);
}
return output;
}
return null;
}
async function withTimeout(
operation: Promise<UpstreamResponse>,
timeoutMs: number,
): Promise<UpstreamResponse> {
let timer: ReturnType<typeof setTimeout> | undefined;
const guard = new Promise<UpstreamResponse>((resolve) => {
timer = setTimeout(() => resolve(TIMEOUT_SENTINEL), timeoutMs);
});
try {
return await Promise.race([operation, guard]);
} finally {
if (timer) clearTimeout(timer);
}
}
function validateBoolean(field: DeviceActionField, raw: unknown): boolean {
if (typeof raw !== 'boolean') throw new DeviceActionError('VALIDATION_FAILED', field.id);
return raw;
}
function validateNumber(field: DeviceActionField, raw: unknown): number {
if (typeof raw !== 'number' || !Number.isSafeInteger(raw))
throw new DeviceActionError('VALIDATION_FAILED', field.id);
if (field.min !== undefined && raw < field.min)
throw new DeviceActionError('VALIDATION_FAILED', field.id);
if (field.max !== undefined && raw > field.max)
throw new DeviceActionError('VALIDATION_FAILED', field.id);
return raw;
}
function validateText(field: DeviceActionField, raw: unknown): string {
if (typeof raw !== 'string') throw new DeviceActionError('VALIDATION_FAILED', field.id);
const value = raw.trim();
if (value.length === 0) {
if (field.required) throw new DeviceActionError('VALIDATION_FAILED', field.id);
return '';
}
if (value.length > (field.maxLength ?? 128))
throw new DeviceActionError('VALIDATION_FAILED', field.id);
if (CONTROL_CHARACTERS.test(value)) throw new DeviceActionError('VALIDATION_FAILED', field.id);
if (field.pattern !== undefined && !new RegExp(field.pattern, 'u').test(value))
throw new DeviceActionError('VALIDATION_FAILED', field.id);
return value;
}
function validateChoice(field: DeviceActionField, raw: unknown): string {
const value = validateText(field, raw);
if (!value) return '';
if (!(field.choices ?? []).some((choice) => choice.value === value))
throw new DeviceActionError('VALIDATION_FAILED', field.id);
return value;
}
function validateList(field: DeviceActionField, raw: unknown): string[] {
if (!Array.isArray(raw) || raw.length > 128)
throw new DeviceActionError('VALIDATION_FAILED', field.id);
const output: string[] = [];
for (const item of raw) {
if (typeof item !== 'string' || item.length > 64 || CONTROL_CHARACTERS.test(item))
throw new DeviceActionError('VALIDATION_FAILED', field.id);
const value = item.trim();
if (value) output.push(value);
}
if (field.required && output.length === 0)
throw new DeviceActionError('VALIDATION_FAILED', field.id);
return output;
}
/**
* The audit reader only accepts fully redacted summaries, so a submitted value is never stored:
* the record says which fields were supplied and leaves the content out.
*/
function summarize(
action: DeviceActionDefinition,
values: Readonly<Record<string, unknown>>,
): readonly { fieldId: string; displayValue: string; redacted: true }[] {
return (action.fields ?? [])
.filter((field) => values[field.id] !== undefined)
.map((field) => ({ fieldId: field.id, displayValue: '[REDACTED]', redacted: true as const }));
}
export interface DeviceActionServiceOptions {
readonly instances: InstanceService;
readonly sessions: InstanceSessionStore;
readonly request: UpstreamSessionClientOptions['request'];
readonly db: SqliteDatabase;
readonly ensureSession?: (instanceId: string, origin: string, force?: boolean) => Promise<void>;
readonly now?: () => Date;
readonly id?: () => string;
/**
* The identity guard. A node whose reported hardware stopped matching its record, or whose
* identity is claimed by another record, is not safe to control: the command would land on a
* device the operator is not looking at. Unbinding stays available because it is the way out.
*/
readonly identities?: { isBlocked(instanceId: string): boolean };
}
/**
* Runs an allowlisted device mutation on behalf of the console. Every dispatch is validated
* against the catalog, audited before the response leaves the process, and re-authorized once
* when the device session has expired.
*/
export class DeviceActionService {
private readonly now: () => Date;
private readonly id: () => string;
constructor(private readonly options: DeviceActionServiceOptions) {
this.now = options.now ?? (() => new Date());
this.id = options.id ?? randomUUID;
}
list(): readonly DeviceActionDescriptor[] {
return DEVICE_ACTIONS.map((action) =>
Object.freeze({
id: action.id,
module: action.module,
title: action.title,
description: action.description,
risk: action.risk,
method: action.method,
fields: Object.freeze(
[...(action.fields ?? [])].map((field) => Object.freeze({ ...field })),
),
}),
);
}
async execute(
instanceId: string,
actionId: string,
params: Readonly<Record<string, unknown>>,
context: DeviceActionContext,
): Promise<DeviceActionResult> {
const action = findDeviceAction(actionId);
if (!action) throw new DeviceActionError('NOT_FOUND');
if (action.risk !== 'R1' && context.confirm !== true)
throw new DeviceActionError('VALIDATION_FAILED', 'confirm');
const instance = await this.options.instances.get(instanceId);
if (!instance) throw new DeviceActionError('NOT_FOUND');
const session = this.options.sessions.sessionFor(instanceId);
if (session && session.origin !== instance.origin)
throw new DeviceActionError('SESSION_INVALID');
if (action.id !== 'hub.unbind' && this.options.identities?.isBlocked(instanceId))
throw new DeviceActionError('IDENTITY_UNCONFIRMED');
const values = this.#validate(action, params);
const { path, query, body } = this.#assemble(action, values);
if (!session && this.options.ensureSession) {
try {
await this.options.ensureSession(instanceId, instance.origin);
} catch {
// Passwordless devices accept anonymous mutations; otherwise the probe reports below.
}
}
const timeoutMs = action.timeoutMs ?? DEFAULT_TIMEOUT_MS;
const target = `${instance.origin}/api${path}${query}`;
const send = async (cookie: string | undefined): Promise<UpstreamResponse> => {
const headers: Record<string, string> = {};
if (body === undefined) headers.accept = 'application/json';
else {
headers.accept = 'application/json';
headers['content-type'] = 'application/json';
}
if (cookie) headers.cookie = cookie;
return withTimeout(
this.options.request({
url: target,
method: action.method,
headers,
deviceAction: { body },
}),
timeoutMs,
);
};
const startedAt = this.now().getTime();
let token = this.options.sessions.sessionFor(instanceId)?.cookie;
let response: UpstreamResponse;
try {
response = await send(token);
if ((response.status === 401 || response.status === 403) && this.options.ensureSession) {
try {
await this.options.ensureSession(instanceId, instance.origin, true);
} catch {
// Keep the device answer; the result below reports the failure.
}
const refreshed = this.options.sessions.sessionFor(instanceId)?.cookie;
if (refreshed && refreshed !== token) {
token = refreshed;
response = await send(token);
}
}
} catch {
// The pinned transport refused or could not reach the device: nothing left this process.
const durationMs = Math.max(0, this.now().getTime() - startedAt);
this.#audit(action, instanceId, context, values, 0, false, durationMs);
throw new DeviceActionError('UPSTREAM_FAILED');
}
const durationMs = Math.max(0, this.now().getTime() - startedAt);
const parsed = this.#interpret(response);
this.#audit(action, instanceId, context, values, response.status, parsed.ok, durationMs);
return {
actionId: action.id,
title: action.title,
status: response.status,
ok: parsed.ok,
data: parsed.data,
message: parsed.message,
durationMs,
};
}
#validate(
action: DeviceActionDefinition,
params: Readonly<Record<string, unknown>>,
): Record<string, unknown> {
const values: Record<string, unknown> = {};
for (const field of action.fields ?? []) {
const raw = params[field.id];
if (raw === undefined || raw === null) {
if (field.required) throw new DeviceActionError('VALIDATION_FAILED', field.id);
continue;
}
switch (field.kind) {
case 'boolean':
values[field.id] = validateBoolean(field, raw);
break;
case 'number':
values[field.id] = validateNumber(field, raw);
break;
case 'string':
case 'secret': {
const text = validateText(field, raw);
if (text) values[field.id] = text;
break;
}
case 'choice': {
const choice = validateChoice(field, raw);
if (choice) values[field.id] = choice;
break;
}
case 'string-list': {
const list = validateList(field, raw);
if (list.length > 0 || field.required) values[field.id] = list;
break;
}
default:
throw new DeviceActionError('VALIDATION_FAILED', field.id);
}
}
const declared = new Set((action.fields ?? []).map((field) => field.id));
for (const key of Object.keys(params))
if (!declared.has(key)) throw new DeviceActionError('VALIDATION_FAILED', key);
return values;
}
#assemble(
action: DeviceActionDefinition,
values: Record<string, unknown>,
): { path: string; query: string; body: Readonly<Record<string, unknown>> | undefined } {
let path = action.path;
const queryParts: string[] = [];
const body: Record<string, unknown> = { ...(action.fixed ?? {}) };
for (const field of action.fields ?? []) {
const value = values[field.id];
if (value === undefined) continue;
const place = field.in ?? 'body';
if (place === 'path') {
if (!/^[A-Za-z0-9_.\-]{1,128}$/u.test(String(value)))
throw new DeviceActionError('VALIDATION_FAILED', field.id);
const token = `{${field.id}}`;
if (!path.includes(token)) throw new DeviceActionError('VALIDATION_FAILED', field.id);
path = path.replace(token, String(value));
} else if (place === 'query') {
const rendered = Array.isArray(value)
? value.map((item) => encodeURIComponent(item)).join(',')
: encodeURIComponent(String(value));
queryParts.push(`${field.id}=${rendered}`);
} else body[field.id] = value;
}
if (path.includes('{')) throw new DeviceActionError('VALIDATION_FAILED');
const query = queryParts.length ? `?${queryParts.join('&')}` : '';
const empty = action.emptyJsonObject === true;
return {
path,
query,
body: Object.keys(body).length || empty ? body : undefined,
};
}
#interpret(response: UpstreamResponse): {
ok: boolean;
data: unknown;
message: string | null;
} {
if (response.status === 0) return { ok: false, data: null, message: '设备未在限定时间内响应' };
if (Buffer.byteLength(response.body, 'utf8') > MAX_RESPONSE_BYTES)
return { ok: false, data: null, message: '设备响应超出长度限制' };
let root: unknown = null;
if (response.body.trim().length > 0) {
try {
root = JSON.parse(response.body);
} catch {
root = null;
}
}
const record = isRecord(root) ? root : undefined;
const status = typeof record?.status === 'string' ? record.status : undefined;
const ok =
response.status >= 200 &&
response.status < 300 &&
(status === undefined || status === 'success' || status === 'ok');
const rawMessage =
(typeof record?.message === 'string' && record.message) ||
(typeof record?.msg === 'string' && record.msg) ||
null;
const message = rawMessage ? rawMessage.slice(0, MAX_STRING_LENGTH) : null;
return { ok, data: sanitize(record?.data ?? record ?? null), message };
}
#audit(
action: DeviceActionDefinition,
instanceId: string,
context: DeviceActionContext,
values: Record<string, unknown>,
status: number,
ok: boolean,
durationMs: number,
): void {
const created_at = this.now().toISOString();
this.options.db
.prepare(
`INSERT INTO audit_events
(id,instance_id,job_id,actor,operation_id,risk_level,request_id,parameters_summary_json,body_digest,result_code,duration_ms,created_at)
VALUES (?,?,NULL,?,?,?,?,?,?,?,?,?)`,
)
.run(
this.id(),
instanceId,
context.actor,
action.id,
action.risk,
context.requestId,
JSON.stringify(summarize(action, values)),
null,
ok ? 'succeeded' : 'failed',
durationMs,
created_at,
);
}
}
@@ -0,0 +1,242 @@
import { describe, expect, it } from 'vitest';
import type { networkInterfaces } from 'node:os';
import type { InstancePage } from '@multi-simadmin/contracts';
import {
DeviceDiscoveryService,
DiscoveryError,
localRanges,
type DiscoveryTransport,
} from './device-discovery-service.js';
const INTERFACES: ReturnType<typeof networkInterfaces> = {
en0: [
{
address: '192.168.1.23',
family: 'IPv4',
internal: false,
netmask: '255.255.255.0',
cidr: '192.168.1.23/24',
mac: '',
},
{
address: '127.0.0.1',
family: 'IPv4',
internal: true,
netmask: '255.0.0.0',
cidr: '127.0.0.1/8',
mac: '',
},
{
address: 'fe80::1',
family: 'IPv6',
internal: false,
netmask: '',
scopeid: 0,
cidr: 'fe80::1/64',
mac: '',
},
],
en1: [
{
address: '10.20.30.40',
family: 'IPv4',
internal: false,
netmask: '255.255.255.0',
cidr: '10.20.30.40/24',
mac: '',
},
],
en2: [
{
address: '203.0.113.9',
family: 'IPv4',
internal: false,
netmask: '255.255.255.0',
cidr: '203.0.113.9/24',
mac: '',
},
],
};
function page(items: InstancePage['items'], pageSize = 200): InstancePage {
return { items, page: { page: 1, pageSize, total: items.length } };
}
const instances = {
list: async () =>
page([
{
id: 'inst-known',
name: '已知设备',
origin: 'http://192.168.1.50:3000',
tags: [],
groupId: null,
revision: 1,
capabilityStatus: 'unknown',
freshness: 'unknown',
credentialConfigured: false,
},
]),
};
function transportFor(
responses: Readonly<Record<string, { status: number; body: string }>>,
): DiscoveryTransport & { urls: string[] } {
const urls: string[] = [];
return {
urls,
async get(url) {
urls.push(url);
const hit = responses[url];
if (hit) return hit;
throw new Error('ECONNREFUSED');
},
};
}
const DEVICE_BODY = JSON.stringify({
model: 'UFI-003',
manufacturer: 'Comtrade',
firmware_version: '2.4.1',
imei: '490154203237518',
phone_number: '13800000000',
oversized: 'x'.repeat(200),
});
function service(options: {
readonly transport: DiscoveryTransport;
readonly now?: () => Date;
readonly maxSessions?: number;
}) {
return new DeviceDiscoveryService({
transport: options.transport,
instances,
interfaces: () => INTERFACES,
ports: [3000],
concurrency: 8,
maxTargets: 600,
...(options.now ? { now: options.now } : {}),
...(options.maxSessions === undefined ? {} : { maxSessions: options.maxSessions }),
});
}
async function settled(
discovery: DeviceDiscoveryService,
sessionId: string,
): Promise<Awaited<ReturnType<DeviceDiscoveryService['status']>>> {
for (let attempt = 0; attempt < 200; attempt += 1) {
const state = await discovery.status(sessionId);
if (state.status === 'completed') return state;
await new Promise((resolve) => setTimeout(resolve, 5));
}
throw new Error('scan did not settle');
}
describe('localRanges', () => {
it('keeps only non-internal private IPv4 subnets', () => {
expect(localRanges(INTERFACES)).toEqual(['10.20.30.0/24', '192.168.1.0/24']);
});
});
describe('DeviceDiscoveryService', () => {
it('reports reachable devices with whitelisted identity and known instances', async () => {
const transport = transportFor({
'http://192.168.1.50:3000/api/health': { status: 200, body: '{"ok":true}' },
'http://192.168.1.50:3000/api/device': { status: 200, body: DEVICE_BODY },
'http://10.20.30.77:3000/api/health': { status: 401, body: '' },
});
const discovery = service({ transport });
const started = await discovery.start();
expect(started.ranges).toEqual(['10.20.30.0/24', '192.168.1.0/24']);
expect(started.total).toBe(508);
const state = await settled(discovery, started.sessionId);
expect(state.scanned).toBe(508);
expect(state.devices.map((device) => device.origin)).toEqual([
'http://10.20.30.77:3000',
'http://192.168.1.50:3000',
]);
const known = state.devices.find((device) => device.origin === 'http://192.168.1.50:3000');
expect(known?.knownInstanceId).toBe('inst-known');
expect(known?.identity).toEqual({
model: 'UFI-003',
manufacturer: 'Comtrade',
firmware_version: '2.4.1',
});
const anonymous = state.devices.find((device) => device.origin === 'http://10.20.30.77:3000');
expect(anonymous?.httpStatus).toBe(401);
expect(anonymous?.knownInstanceId).toBeNull();
expect(anonymous?.identity).toEqual({});
expect(transport.urls.every((url) => /\/api\/(health|device)$/u.test(url))).toBe(true);
});
it('drops an expired lease for good and closes a live session on demand', async () => {
let now = new Date('2026-09-04T10:00:00.000Z');
const transport = transportFor({});
const discovery = service({ transport, now: () => now });
const started = await discovery.start();
now = new Date('2026-09-04T11:00:00.000Z');
await expect(discovery.renew(started.sessionId)).rejects.toBeInstanceOf(DiscoveryError);
// A lease that lapsed is never resurrected, even if the clock comes back.
now = new Date('2026-09-04T10:00:30.000Z');
await expect(discovery.status(started.sessionId)).rejects.toMatchObject({
code: 'NOT_FOUND',
});
const live = await discovery.start();
const renewed = await discovery.renew(live.sessionId);
expect(renewed.expiresAt).toBe('2026-09-04T10:01:30.000Z');
expect((await discovery.status(live.sessionId)).sessionId).toBe(live.sessionId);
await discovery.stop(live.sessionId);
await expect(discovery.status(live.sessionId)).rejects.toMatchObject({ code: 'NOT_FOUND' });
await expect(discovery.stop(live.sessionId)).resolves.toBeUndefined();
});
it('refuses more sessions than the configured ceiling', async () => {
const discovery = service({ transport: transportFor({}), maxSessions: 1 });
await discovery.start();
await expect(discovery.start()).rejects.toMatchObject({ code: 'TOO_MANY_SESSIONS' });
});
it('probes a manually entered address and normalises the origin', async () => {
const transport = transportFor({
'http://192.168.68.1:3000/api/health': { status: 200, body: '{}' },
'http://192.168.68.1:3000/api/device': { status: 200, body: DEVICE_BODY },
});
const discovery = service({ transport });
const probe = await discovery.probe('192.168.68.1:3000');
expect(probe).toMatchObject({
origin: 'http://192.168.68.1:3000',
reachable: true,
httpStatus: 200,
knownInstanceId: null,
});
expect(probe.identity.model).toBe('UFI-003');
expect('imei' in probe.identity).toBe(false);
expect('phone_number' in probe.identity).toBe(false);
expect('oversized' in probe.identity).toBe(false);
});
it('marks an unreachable address without failing the request', async () => {
const probe = await service({ transport: transportFor({}) }).probe('http://10.0.0.9:3000');
expect(probe).toMatchObject({ reachable: false, httpStatus: null, identity: {} });
});
it('rejects addresses that are not a private LAN root URL', async () => {
const discovery = service({ transport: transportFor({}) });
const rejected = [
undefined,
'https://example.com',
'http://8.8.8.8:3000',
'http://admin:p@ss@192.168.1.5:3000',
'http://192.168.1.5:3000/api/device',
'http://192.168.1.5:3000/?next=x',
'ftp://192.168.1.5:3000',
'http://192.168.1.5:99999',
];
for (const value of rejected) {
await expect(discovery.probe(value)).rejects.toMatchObject({ code: 'VALIDATION_FAILED' });
}
});
});
@@ -0,0 +1,392 @@
/**
* LAN discovery for SimAdmin devices, fused from the Hub "局域网发现 + 地址接入" onboarding flow.
*
* The scan is deliberately narrow: it only ever dials literal private IPv4 addresses taken from
* the host's own network interfaces, on a fixed port set, with one read-only probe path. Nothing
* here accepts a caller-supplied host, so the endpoint cannot be turned into an SSRF oracle.
*/
import { networkInterfaces } from 'node:os';
import { randomUUID } from 'node:crypto';
import type { InstancePage, InstancePageQuery } from '@multi-simadmin/contracts';
export interface DiscoveryTransport {
get(url: string): Promise<{ readonly status: number; readonly body: string }>;
}
export interface KnownInstanceOrigin {
list(query: InstancePageQuery): Promise<InstancePage>;
}
export type DiscoveredDevice = Readonly<{
origin: string;
address: string;
port: number;
secure: boolean;
httpStatus: number;
identity: Readonly<Record<string, string>>;
knownInstanceId: string | null;
}>;
export type DiscoverySession = Readonly<{
sessionId: string;
status: 'scanning' | 'completed';
createdAt: string;
expiresAt: string;
scanned: number;
total: number;
ranges: readonly string[];
devices: readonly DiscoveredDevice[];
}>;
export type DeviceProbe = Readonly<{
origin: string;
reachable: boolean;
httpStatus: number | null;
identity: Readonly<Record<string, string>>;
knownInstanceId: string | null;
}>;
export type DiscoveryErrorCode = 'VALIDATION_FAILED' | 'NOT_FOUND' | 'TOO_MANY_SESSIONS';
export class DiscoveryError extends Error {
constructor(
readonly code: DiscoveryErrorCode,
message: string,
) {
super(message);
this.name = 'DiscoveryError';
}
}
export interface DeviceDiscoveryServiceOptions {
readonly transport: DiscoveryTransport;
readonly instances: KnownInstanceOrigin;
readonly interfaces?: () => ReturnType<typeof networkInterfaces>;
readonly now?: () => Date;
readonly idFactory?: () => string;
readonly ports?: readonly number[];
readonly leaseMs?: number;
readonly concurrency?: number;
readonly maxTargets?: number;
readonly maxSessions?: number;
}
/** SimAdmin serves its console on 3000; 8080/8443 cover the usual reverse-proxy setups. */
export const DEFAULT_DISCOVERY_PORTS: readonly number[] = [3000, 8080, 8443];
const DEFAULT_LEASE_MS = 60_000;
const DEFAULT_CONCURRENCY = 48;
const DEFAULT_MAX_TARGETS = 1_024;
const DEFAULT_MAX_SESSIONS = 4;
const HEALTH_PATH = '/api/health';
const DEVICE_PATH = '/api/device';
/** Only non-identifying fields cross the boundary; IMEI, ICCI and phone numbers never do. */
const IDENTITY_FIELDS: readonly string[] = [
'model',
'manufacturer',
'brand',
'firmware_version',
'os_version',
'version',
];
const MAX_IDENTITY_VALUE_LENGTH = 80;
const MAX_IDENTITY_FIELDS = 6;
type Target = Readonly<{ origin: string; address: string; port: number; secure: boolean }>;
interface SessionRecord {
readonly sessionId: string;
readonly createdAt: Date;
expiresAt: Date;
status: 'scanning' | 'completed';
closed: boolean;
scanned: number;
readonly total: number;
readonly ranges: readonly string[];
readonly devices: DiscoveredDevice[];
}
function privateV4(address: string): boolean {
const parts = address.split('.');
if (parts.length !== 4 || parts.some((part) => !/^\d{1,3}$/.test(part))) return false;
const octets = parts.map(Number);
if (octets.some((part) => part > 255)) return false;
const [a, b] = octets as [number, number];
return a === 10 || (a === 172 && b >= 16 && b <= 31) || (a === 192 && b === 168);
}
function securePort(port: number): boolean {
return port === 443 || port === 8443;
}
function originFor(address: string, port: number): string {
return `${securePort(port) ? 'https' : 'http'}://${address}:${port}`;
}
/** Every /24 the host itself sits on, plus an optional caller-supplied /24 in dotted form. */
export function localRanges(interfaces: ReturnType<typeof networkInterfaces>): readonly string[] {
const ranges = new Set<string>();
for (const entries of Object.values(interfaces)) {
for (const entry of entries ?? []) {
if (entry.internal || entry.family !== 'IPv4') continue;
if (!privateV4(entry.address)) continue;
const [a, b, c] = entry.address.split('.');
if (!a || !b || !c) continue;
ranges.add(`${a}.${b}.${c}.0/24`);
}
}
return [...ranges].sort();
}
function expand(ranges: readonly string[], ports: readonly number[], limit: number): Target[] {
const targets: Target[] = [];
const seen = new Set<string>();
for (const range of ranges) {
const prefix = range.replace(/\.0\/24$/u, '');
for (let host = 1; host <= 254 && targets.length < limit; host += 1) {
const address = `${prefix}.${host}`;
for (const port of ports) {
if (targets.length >= limit) break;
const key = `${address}:${port}`;
if (seen.has(key)) continue;
seen.add(key);
targets.push({ origin: originFor(address, port), address, port, secure: securePort(port) });
}
}
}
return targets;
}
function identity(payload: unknown): Readonly<Record<string, string>> {
if (typeof payload !== 'object' || payload === null) return {};
const source = payload as Record<string, unknown>;
const result: Record<string, string> = {};
for (const field of IDENTITY_FIELDS) {
if (Object.keys(result).length >= MAX_IDENTITY_FIELDS) break;
const value = source[field];
if (typeof value !== 'string' && typeof value !== 'number') continue;
const text = String(value).trim();
if (!text || text.length > MAX_IDENTITY_VALUE_LENGTH || /[\x00-\x1F\x7F]/u.test(text)) continue;
result[field] = text;
}
return result;
}
function parseJson(body: string): unknown {
try {
return JSON.parse(body) as unknown;
} catch {
return null;
}
}
function normalizeDeviceUrl(raw: unknown): Target {
if (typeof raw !== 'string')
throw new DiscoveryError('VALIDATION_FAILED', '设备地址必须是字符串');
const candidate = raw.trim();
let url: URL;
try {
url = new URL(candidate.includes('://') ? candidate : `http://${candidate}`);
} catch {
throw new DiscoveryError('VALIDATION_FAILED', '设备地址格式不正确');
}
if (url.protocol !== 'http:' && url.protocol !== 'https:')
throw new DiscoveryError('VALIDATION_FAILED', '设备地址必须使用 HTTP 或 HTTPS');
if (url.username || url.password || url.search || url.hash)
throw new DiscoveryError('VALIDATION_FAILED', '设备地址不能包含凭据、查询或片段');
if (url.pathname !== '/' && url.pathname !== '')
throw new DiscoveryError('VALIDATION_FAILED', '设备地址只能是根地址');
const address = url.hostname.replace(/^\[|\]$/g, '');
if (!privateV4(address))
throw new DiscoveryError('VALIDATION_FAILED', '设备地址必须是私有局域网 IPv4 地址');
const port = url.port ? Number(url.port) : url.protocol === 'https:' ? 443 : 80;
if (!Number.isInteger(port) || port < 1 || port > 65_535)
throw new DiscoveryError('VALIDATION_FAILED', '设备地址端口不正确');
return { origin: url.origin, address, port, secure: url.protocol === 'https:' };
}
export class DeviceDiscoveryService {
private readonly sessions = new Map<string, SessionRecord>();
private readonly ranges: () => ReturnType<typeof networkInterfaces>;
private readonly clock: () => Date;
private readonly id: () => string;
private readonly ports: readonly number[];
private readonly leaseMs: number;
private readonly concurrency: number;
private readonly maxTargets: number;
private readonly maxSessions: number;
constructor(private readonly options: DeviceDiscoveryServiceOptions) {
this.ranges = options.interfaces ?? networkInterfaces;
this.clock = options.now ?? (() => new Date());
this.id = options.idFactory ?? randomUUID;
this.ports = options.ports ?? DEFAULT_DISCOVERY_PORTS;
this.leaseMs = options.leaseMs ?? DEFAULT_LEASE_MS;
this.concurrency = options.concurrency ?? DEFAULT_CONCURRENCY;
this.maxTargets = options.maxTargets ?? DEFAULT_MAX_TARGETS;
this.maxSessions = options.maxSessions ?? DEFAULT_MAX_SESSIONS;
}
/** Opens a short-lease scan; the caller renews it while the wizard stays open. */
async start(): Promise<DiscoverySession> {
this.#evict();
if (this.sessions.size >= this.maxSessions)
throw new DiscoveryError('TOO_MANY_SESSIONS', '设备发现会话数量已达上限');
const ranges = localRanges(this.ranges());
const targets = expand(ranges, this.ports, this.maxTargets);
const now = this.clock();
const record: SessionRecord = {
sessionId: this.id(),
createdAt: now,
expiresAt: new Date(now.getTime() + this.leaseMs),
status: 'scanning',
closed: false,
scanned: 0,
total: targets.length,
ranges,
devices: [],
};
this.sessions.set(record.sessionId, record);
void this.#sweep(record, targets);
return this.#view(record);
}
async renew(sessionId: string): Promise<DiscoverySession> {
const record = this.#require(sessionId);
record.expiresAt = new Date(this.clock().getTime() + this.leaseMs);
return this.#view(record);
}
async status(sessionId: string): Promise<DiscoverySession> {
return this.#view(this.#require(sessionId));
}
async stop(sessionId: string): Promise<void> {
const record = this.#sessions().get(sessionId);
if (!record) return;
record.closed = true;
this.sessions.delete(sessionId);
}
/** Single-address check used by the manual "输入设备地址" step of the wizard. */
async probe(rawUrl: unknown): Promise<DeviceProbe> {
const target = normalizeDeviceUrl(rawUrl);
const known = await this.#knownOrigins();
const knownInstanceId = known.get(target.origin) ?? null;
const response = await this.options.transport
.get(`${target.origin}${HEALTH_PATH}`)
.catch(() => null);
if (!response)
return {
origin: target.origin,
reachable: false,
httpStatus: null,
identity: {},
knownInstanceId,
};
return {
origin: target.origin,
reachable: true,
httpStatus: response.status,
identity: await this.#identify(target),
knownInstanceId,
};
}
async #sweep(record: SessionRecord, targets: readonly Target[]): Promise<void> {
const known = await this.#knownOrigins().catch(() => new Map<string, string>());
let cursor = 0;
const worker = async (): Promise<void> => {
for (;;) {
if (record.closed || this.clock() >= record.expiresAt) return;
const index = cursor;
cursor += 1;
if (index >= targets.length) return;
const target = targets[index]!;
const device = await this.#inspect(target, known);
record.scanned += 1;
if (device && !record.closed) record.devices.push(device);
}
};
const lanes = Math.max(1, Math.min(this.concurrency, targets.length || 1));
await Promise.all(Array.from({ length: lanes }, () => worker()));
if (!record.closed) record.status = 'completed';
}
async #inspect(
target: Target,
known: ReadonlyMap<string, string>,
): Promise<DiscoveredDevice | null> {
const response = await this.options.transport
.get(`${target.origin}${HEALTH_PATH}`)
.catch(() => null);
if (!response) return null;
return {
origin: target.origin,
address: target.address,
port: target.port,
secure: target.secure,
httpStatus: response.status,
identity: await this.#identify(target),
knownInstanceId: known.get(target.origin) ?? null,
};
}
async #identify(target: Target): Promise<Readonly<Record<string, string>>> {
const response = await this.options.transport
.get(`${target.origin}${DEVICE_PATH}`)
.catch(() => null);
if (!response || response.status >= 400) return {};
return identity(parseJson(response.body));
}
async #knownOrigins(): Promise<Map<string, string>> {
const map = new Map<string, string>();
let page = 1;
while (page <= 10) {
const result = await this.options.instances.list({ page, pageSize: 200 });
for (const instance of result.items) map.set(instance.origin, instance.id);
const seen = page * result.page.pageSize;
if (seen >= result.page.total || result.items.length === 0) break;
page += 1;
}
return map;
}
#require(sessionId: string): SessionRecord {
const record = this.#sessions().get(sessionId);
if (!record) throw new DiscoveryError('NOT_FOUND', '设备发现会话不存在或已过期');
return record;
}
/**
* Leases are dropped lazily: a session lives exactly as long as its lease, so a wizard that
* stops renewing simply loses its scan and has to start a new one. Nothing outlives the lease.
*/
#sessions(): Map<string, SessionRecord> {
this.#evict();
return this.sessions;
}
#evict(): void {
const now = this.clock();
for (const [key, record] of this.sessions) {
if (record.expiresAt.getTime() <= now.getTime()) this.sessions.delete(key);
}
}
#view(record: SessionRecord): DiscoverySession {
return {
sessionId: record.sessionId,
status: record.closed ? 'completed' : record.status,
createdAt: record.createdAt.toISOString(),
expiresAt: record.expiresAt.toISOString(),
scanned: record.scanned,
total: record.total,
ranges: record.ranges,
devices: [...record.devices].sort((left, right) =>
left.address.localeCompare(right.address, 'en', { numeric: true }),
),
};
}
}
@@ -0,0 +1,134 @@
/**
* Read-only catalog of SimAdmin device endpoints, grouped by the console module that renders
* them. Every entry is a GET-safe probe: the control plane never mutates a device through this
* catalog, so a module read is always safe to retry and safe to run on a schedule.
*/
export type InstanceModuleKey =
| 'overview'
| 'sim'
| 'cellular'
| 'device-network'
| 'esim'
| 'calls'
| 'configuration'
| 'device-backup'
| 'notifications'
| 'automation'
| 'ota'
| 'vowifi';
export interface ModuleProbe {
/** Stable identifier the console renders; never a device-controlled value. */
readonly key: string;
/** Path appended to the instance origin, always rooted at the device /api namespace. */
readonly path: string;
readonly timeoutMs?: number;
}
export const INSTANCE_MODULE_KEYS: readonly InstanceModuleKey[] = [
'overview',
'sim',
'cellular',
'device-network',
'esim',
'calls',
'configuration',
'device-backup',
'notifications',
'automation',
'ota',
'vowifi',
];
export const INSTANCE_MODULE_PROBES: Readonly<Record<InstanceModuleKey, readonly ModuleProbe[]>> = {
overview: [
{ key: 'device', path: '/device' },
{ key: 'sim', path: '/sim' },
{ key: 'network', path: '/network' },
{ key: 'stats', path: '/stats' },
{ key: 'connectivity', path: '/connectivity' },
{ key: 'data', path: '/data' },
{ key: 'cpu', path: '/stats/cpu' },
{ key: 'smsStats', path: '/sms/stats' },
{ key: 'networkSpeed', path: '/network/speed' },
],
sim: [
{ key: 'sim', path: '/sim' },
{ key: 'apn', path: '/apn' },
{ key: 'bandLock', path: '/band-lock' },
{ key: 'cellLock', path: '/cell-lock' },
],
cellular: [
{ key: 'network', path: '/network' },
{ key: 'signalStrength', path: '/network/signal-strength' },
{ key: 'cells', path: '/cells' },
{ key: 'cellLocation', path: '/location/cell-info' },
{ key: 'operators', path: '/network/operators' },
{ key: 'roaming', path: '/roaming' },
{ key: 'radioMode', path: '/radio-mode' },
{ key: 'airplaneMode', path: '/airplane-mode' },
{ key: 'basebandRestart', path: '/baseband/restart/status' },
{ key: 'cellMonitor', path: '/cell-monitor/status' },
],
'device-network': [
{ key: 'interfaces', path: '/network/interfaces' },
{ key: 'addresses', path: '/network/connection-addresses' },
{ key: 'wlanStatus', path: '/device-network/wlan/status' },
{ key: 'wlanProfiles', path: '/device-network/wlan/profiles' },
{ key: 'ddnsStatus', path: '/device-network/ddns/status' },
{ key: 'ddnsConfig', path: '/device-network/ddns/config' },
{ key: 'ddnsLogs', path: '/device-network/ddns/logs' },
],
esim: [
{ key: 'euicc', path: '/esim/euicc', timeoutMs: 30_000 },
{ key: 'profiles', path: '/esim/profiles?cached=1' },
{ key: 'config', path: '/esim/config' },
{ key: 'lpacStatus', path: '/esim/lpac/status' },
],
calls: [
{ key: 'calls', path: '/calls' },
{ key: 'history', path: '/call/history?limit=50' },
{ key: 'settings', path: '/call/settings' },
{ key: 'forwarding', path: '/call/forwarding' },
{ key: 'volume', path: '/call/volume' },
{ key: 'voicemail', path: '/voicemail/status' },
{ key: 'ims', path: '/ims/status' },
],
configuration: [
{ key: 'workMode', path: '/work-mode' },
{ key: 'authSettings', path: '/auth/settings' },
{ key: 'authStatus', path: '/auth/status' },
{ key: 'hub', path: '/hub' },
],
'device-backup': [
{ key: 'files', path: '/backup/files' },
{ key: 'config', path: '/backup/config' },
{ key: 'options', path: '/backup/options' },
],
notifications: [
{ key: 'config', path: '/notifications/config' },
{ key: 'queue', path: '/notifications/queue?limit=50' },
{ key: 'logs', path: '/notifications/logs?limit=50' },
],
automation: [
{ key: 'config', path: '/automation/config' },
{ key: 'logs', path: '/automation/logs' },
],
ota: [{ key: 'status', path: '/ota/status' }],
vowifi: [
{ key: 'status', path: '/vowifi/status' },
{ key: 'control', path: '/vowifi/control' },
{ key: 'profile', path: '/vowifi/profile' },
{ key: 'profiles', path: '/vowifi/profiles' },
// The aggregate diagnostics feed carries the registration timeline the Hub shows.
{ key: 'diagnostics', path: '/vowifi/diagnostics?limit=50', timeoutMs: 30_000 },
{ key: 'events', path: '/vowifi/events?limit=50', timeoutMs: 10_000 },
{ key: 'smsDeliveries', path: '/vowifi/sms/delivery?limit=20', timeoutMs: 10_000 },
{ key: 'soakRuns', path: '/vowifi/soak?limit=20', timeoutMs: 10_000 },
{ key: 'restore', path: '/vowifi/esim-restore/status', timeoutMs: 10_000 },
],
};
export function isInstanceModuleKey(value: unknown): value is InstanceModuleKey {
return typeof value === 'string' && (INSTANCE_MODULE_KEYS as readonly string[]).includes(value);
}
@@ -0,0 +1,232 @@
import { describe, expect, it } from 'vitest';
import {
InstanceSessionStore,
type UpstreamRequest,
} from '../connections/upstream-session-client.js';
import type { InstanceService } from '../instances/instance-service.js';
import { InstanceModuleService } from './instance-module-service.js';
interface Reply {
readonly status: number;
readonly body?: unknown;
}
function fixture(
probes: Readonly<Record<string, Reply>>,
extra: {
readonly onIdentity?: (
instanceId: string,
evidence: Readonly<Record<string, unknown>>,
origin: string,
) => void;
} = {},
) {
const sessions = new InstanceSessionStore();
const calls: UpstreamRequest[] = [];
const instances = {
get: async (id: string) =>
id === 'node-a'
? {
id: 'node-a',
name: 'Node A',
origin: 'http://node-a.local',
tags: [],
groupId: null,
revision: 1,
capabilityStatus: 'unknown',
freshness: 'unknown',
credentialConfigured: false,
}
: undefined,
} as unknown as InstanceService;
const service = new InstanceModuleService({
instances,
sessions,
request: async (request) => {
calls.push(request);
const path = request.url.slice('http://node-a.local/api'.length);
const reply = probes[path] ?? { status: 404 };
return {
status: reply.status,
headers: {},
body: reply.body === undefined ? '' : JSON.stringify(reply.body),
};
},
now: () => new Date('2026-09-04T00:00:00.000Z'),
...(extra.onIdentity ? { onIdentity: extra.onIdentity } : {}),
});
return { service, sessions, calls };
}
describe('InstanceModuleService', () => {
it('reads every probe for a module and classifies each section', async () => {
const { service } = fixture({
'/device': { status: 200, body: { model: 'LPAX', android_version: '13' } },
'/stats': { status: 200, body: { cpu_percent: 12 } },
'/connectivity': { status: 200, body: {} },
'/network/speed': {
status: 200,
body: { interfaces: [{ interface: 'wwan0', rx_bytes_per_sec: 1536 }] },
},
});
const snapshot = await service.read('node-a', 'overview');
expect(snapshot.observedAt).toBe('2026-09-04T00:00:00.000Z');
const byKey = Object.fromEntries(snapshot.sections.map((section) => [section.key, section]));
expect(byKey.device?.state).toBe('ok');
expect(byKey.device?.data).toMatchObject({ model: 'LPAX' });
expect(byKey.stats?.state).toBe('ok');
expect(byKey.connectivity?.state).toBe('empty');
expect(byKey.networkSpeed?.state).toBe('ok');
expect(byKey.networkSpeed?.data).toMatchObject({
interfaces: [{ interface: 'wwan0', rx_bytes_per_sec: 1536 }],
});
expect(byKey.data?.state).toBe('unsupported');
});
it('reports auth-required when the device rejects an anonymous read', async () => {
const { service } = fixture({
'/sim': { status: 401 },
'/apn': { status: 200, body: { apns: [] } },
'/band-lock': { status: 200, body: { locked: false } },
'/cell-lock': { status: 200, body: { locked: false } },
});
const snapshot = await service.read('node-a', 'sim');
const sim = snapshot.sections.find((section) => section.key === 'sim');
expect(sim?.state).toBe('auth-required');
expect(sim?.status).toBe(401);
});
it('retries once with a refreshed session and keeps the authenticated answer', async () => {
const sessions = new InstanceSessionStore();
let attempts = 0;
const service = new InstanceModuleService({
instances: {
get: async () => ({ id: 'node-a', origin: 'http://node-a.local' }),
} as unknown as InstanceService,
sessions,
request: async () => {
attempts += 1;
return attempts === 1
? { status: 401, headers: {}, body: '' }
: { status: 200, headers: {}, body: JSON.stringify({ imei: '123' }) };
},
ensureSession: async () => {
sessions.set('node-a', 'http://node-a.local', 'simadmin_session=abc');
},
});
const snapshot = await service.read('node-a', 'sim');
expect(snapshot.authenticated).toBe(true);
expect(snapshot.sections.find((section) => section.key === 'sim')?.state).toBe('ok');
expect(attempts).toBeGreaterThan(1);
});
it('redacts credential-looking fields and bounds oversized payloads', async () => {
const { service } = fixture({
'/ota/status': {
status: 200,
body: {
version: '1.2.3',
admin_password: 'hunter2',
session_token: 'nope',
nested: { log: 'x'.repeat(5_000) },
},
},
});
const snapshot = await service.read('node-a', 'ota');
const raw = JSON.stringify(snapshot);
expect(raw).not.toContain('hunter2');
expect(raw).not.toContain('nope');
const section = snapshot.sections[0];
const data = section?.data as { nested?: { log?: string } };
expect((data.nested?.log ?? '').length).toBeLessThanOrEqual(2_004);
});
it('turns a hanging probe into a failed section instead of blocking the module', async () => {
const hanging = new InstanceModuleService({
instances: {
get: async () => ({ id: 'node-a', origin: 'http://node-a.local' }),
} as unknown as InstanceService,
sessions: new InstanceSessionStore(),
request: async (request) => {
if (request.url.endsWith('/device')) return new Promise<never>(() => {});
return { status: 200, headers: {}, body: '{}' };
},
defaultTimeoutMs: 100,
});
const snapshot = await hanging.read('node-a', 'overview');
expect(snapshot.sections.find((section) => section.key === 'device')?.state).toBe('failed');
expect(snapshot.sections.find((section) => section.key === 'stats')?.state).toBe('empty');
});
it('unwraps the device response envelope so panels see the real payload', async () => {
const { service } = fixture({
'/device': {
status: 200,
body: { status: 'ok', message: 'Success', data: { model: 'LPAX', imei: '123' } },
},
'/stats': { status: 200, body: { status: 'ok', message: 'Success', data: {} } },
'/connectivity': {
status: 200,
body: { status: 'error', message: 'Connectivity is not exposed by this backend' },
},
});
const snapshot = await service.read('node-a', 'overview');
const byKey = Object.fromEntries(snapshot.sections.map((section) => [section.key, section]));
expect(byKey.device?.state).toBe('ok');
expect(byKey.device?.data).toEqual({ model: 'LPAX', imei: '123' });
expect(byKey.stats?.state).toBe('empty');
expect(byKey.connectivity?.state).toBe('unsupported');
});
it('rejects unknown modules and unknown instances', async () => {
const { service } = fixture({});
await expect(service.read('node-a', 'nope' as 'overview')).rejects.toMatchObject({
code: 'VALIDATION_FAILED',
});
await expect(service.read('missing', 'overview')).rejects.toMatchObject({ code: 'NOT_FOUND' });
});
it('hands the hardware report to the identity guard while reading the overview', async () => {
const seen: [string, Record<string, unknown>, string][] = [];
const { service } = fixture(
{
'/device': { status: 200, body: { imei: '860000000000001', model: 'LPAX' } },
'/stats': { status: 200, body: {} },
'/connectivity': { status: 200, body: {} },
},
{
onIdentity: (instanceId, evidence, origin) => {
seen.push([instanceId, evidence as Record<string, unknown>, origin]);
},
},
);
await service.read('node-a', 'overview');
expect(seen).toEqual([
['node-a', { imei: '860000000000001', model: 'LPAX' }, 'http://node-a.local'],
]);
});
it('stays quiet when a module other than the overview is read', async () => {
const seen: string[] = [];
const { service } = fixture(
{ '/sim': { status: 200, body: { imei: '860000000000001' } } },
{ onIdentity: (instanceId) => void seen.push(instanceId) },
);
await service.read('node-a', 'sim');
expect(seen).toEqual([]);
});
it('never loses an overview the operator asked for because bookkeeping threw', async () => {
const { service } = fixture(
{ '/device': { status: 200, body: { model: 'LPAX' } } },
{
onIdentity: () => {
throw new Error('guard unavailable');
},
},
);
const snapshot = await service.read('node-a', 'overview');
expect(snapshot.sections.find((section) => section.key === 'device')?.state).toBe('ok');
});
});
@@ -0,0 +1,349 @@
import type { InstanceService } from './instance-service.js';
import {
INSTANCE_MODULE_PROBES,
type InstanceModuleKey,
type ModuleProbe,
} from './instance-module-catalog.js';
import type {
InstanceSessionStore,
UpstreamResponse,
UpstreamSessionClientOptions,
} from '../connections/upstream-session-client.js';
export type InstanceModuleErrorCode = 'NOT_FOUND' | 'VALIDATION_FAILED';
export class InstanceModuleError extends Error {
constructor(
readonly code: InstanceModuleErrorCode,
message: string = code,
) {
super(message);
this.name = 'InstanceModuleError';
}
}
export type ModuleSectionState = 'ok' | 'empty' | 'auth-required' | 'unsupported' | 'failed';
export interface ModuleSection {
readonly key: string;
readonly path: string;
readonly state: ModuleSectionState;
readonly status: number | null;
readonly data: unknown;
}
export interface ModuleSnapshot {
readonly instanceId: string;
readonly module: InstanceModuleKey;
readonly observedAt: string;
readonly authenticated: boolean;
readonly sections: readonly ModuleSection[];
}
const MAX_SECTION_BYTES = 64_000;
const MAX_SNAPSHOT_BYTES = 220_000;
const MAX_DEPTH = 6;
const MAX_ARRAY_ENTRIES = 200;
const MAX_STRING_LENGTH = 2_000;
const CONCURRENCY = 4;
const DEFAULT_PROBE_TIMEOUT_MS = 6_000;
const CONTROL_CHARACTERS = /[\u0000-\u0008\u000b\u000c\u000e-\u001f\u007f]/u;
const SECRET_KEY =
/(password|passwd|secret|token|cookie|authorization|apikey|api_key|privatekey|private_key|session)/iu;
function isRecord(value: unknown): value is Record<string, unknown> {
return value !== null && typeof value === 'object' && !Array.isArray(value);
}
/**
* Device payloads are untrusted. Depth, breadth and string length are bounded and anything that
* looks like a credential is dropped before the snapshot leaves the control plane.
*/
function sanitize(value: unknown, depth = 0): unknown {
if (value === null) return null;
switch (typeof value) {
case 'string':
return value.length > MAX_STRING_LENGTH
? `${value.slice(0, MAX_STRING_LENGTH)}...`
: CONTROL_CHARACTERS.test(value)
? value.replace(CONTROL_CHARACTERS, '')
: value;
case 'number':
return Number.isFinite(value) ? value : null;
case 'boolean':
return value;
default:
break;
}
if (depth >= MAX_DEPTH) return null;
if (Array.isArray(value))
return value.slice(0, MAX_ARRAY_ENTRIES).map((item) => sanitize(item, depth + 1));
if (isRecord(value)) {
const output: Record<string, unknown> = {};
for (const [key, entry] of Object.entries(value)) {
if (SECRET_KEY.test(key)) continue;
output[key] = sanitize(entry, depth + 1);
}
return output;
}
return null;
}
function isEmpty(value: unknown): boolean {
if (value === null || value === undefined) return true;
if (Array.isArray(value)) return value.length === 0;
if (isRecord(value)) return Object.keys(value).length === 0;
return false;
}
interface DecodedBody {
readonly data: unknown;
readonly failed: boolean;
readonly envelope: ModuleSectionState | undefined;
}
/**
* Device routes answer inside a `{status, message, data}` envelope. The payload the console renders
* lives one level deeper, so the envelope is inspected for a capability verdict and then removed.
* Error bodies are left whole because the message is the only useful part of them.
*/
function decodeBody(response: UpstreamResponse): DecodedBody {
if (Buffer.byteLength(response.body, 'utf8') > MAX_SECTION_BYTES)
return { data: null, failed: true, envelope: undefined };
try {
const decoded = sanitize(JSON.parse(response.body));
const envelope = envelopeState(decoded);
if (envelope) return { data: decoded, failed: false, envelope };
return { data: unwrapEnvelope(decoded), failed: false, envelope };
} catch {
return { data: null, failed: true, envelope: undefined };
}
}
function unwrapEnvelope(value: unknown): unknown {
if (!isRecord(value) || typeof value.status !== 'string' || value.status !== 'ok') return value;
return 'data' in value ? value.data : value;
}
/**
* Devices answer 200 with an error envelope when a feature is compiled out of the firmware, for
* example "Call forwarding is not exposed by ModemManager on this backend". That is a capability
* verdict, not a read failure, so it must not render as data or count against the device.
*/
function envelopeState(data: unknown): ModuleSectionState | undefined {
if (!isRecord(data) || data.status !== 'error') return undefined;
const message = typeof data.message === 'string' ? data.message.toLocaleLowerCase() : '';
if (/not exposed|not supported|unsupported|not implemented|no such|disabled by/u.test(message))
return 'unsupported';
return 'failed';
}
function classify(status: number, body: DecodedBody): ModuleSectionState {
if (status === 401 || status === 403) return 'auth-required';
if (status === 404 || status === 405 || status === 501) return 'unsupported';
if (body.failed || status < 200 || status >= 300) return 'failed';
return body.envelope ?? (isEmpty(body.data) ? 'empty' : 'ok');
}
async function withTimeout<T>(
operation: Promise<T>,
timeoutMs: number,
onTimeout: () => T,
): Promise<T> {
let timer: ReturnType<typeof setTimeout> | undefined;
const guard = new Promise<T>((resolve) => {
timer = setTimeout(() => resolve(onTimeout()), timeoutMs);
});
try {
return await Promise.race([operation, guard]);
} finally {
if (timer) clearTimeout(timer);
}
}
const TIMEOUT_SENTINEL: UpstreamResponse = { status: 0, headers: {}, body: '' };
export interface InstanceModuleServiceOptions {
readonly instances: InstanceService;
readonly sessions: InstanceSessionStore;
readonly request: UpstreamSessionClientOptions['request'];
readonly ensureSession?: (instanceId: string, origin: string, force?: boolean) => Promise<void>;
readonly now?: () => Date;
/** Fallback for catalog entries without an explicit budget; keeps a dead device from stalling. */
readonly defaultTimeoutMs?: number;
/**
* Hears what the device just said about its own hardware. The overview module already reads
* /device, so identity is observed by the pages the operator opens rather than by a poller
* that costs the LAN extra traffic.
*/
readonly onIdentity?: (
instanceId: string,
evidence: Readonly<Record<string, unknown>>,
origin: string,
) => void;
}
/**
* Reads a whole console module from a device in one call. Individual probes fail independently so
* a device that only implements part of the API still renders the sections it supports.
*/
/** Shared across the probes of one read so a device that needs a login is logged in once. */
interface ProbeContext {
cookie: string | undefined;
refreshAttempted: boolean;
}
export class InstanceModuleService {
private readonly now: () => Date;
private readonly defaultTimeoutMs: number;
constructor(private readonly options: InstanceModuleServiceOptions) {
this.now = options.now ?? (() => new Date());
this.defaultTimeoutMs = options.defaultTimeoutMs ?? DEFAULT_PROBE_TIMEOUT_MS;
}
async read(instanceId: string, module: InstanceModuleKey): Promise<ModuleSnapshot> {
const probes = INSTANCE_MODULE_PROBES[module];
if (!probes) throw new InstanceModuleError('VALIDATION_FAILED', 'Unknown instance module');
const instance = await this.options.instances.get(instanceId);
if (!instance) throw new InstanceModuleError('NOT_FOUND', 'Instance was not found');
const session = this.options.sessions.sessionFor(instanceId);
if (session && session.origin !== instance.origin)
throw new InstanceModuleError('VALIDATION_FAILED', 'Instance session does not match origin');
if (!session && this.options.ensureSession) {
try {
await this.options.ensureSession(instanceId, instance.origin);
} catch {
// Passwordless devices are readable anonymously; probes report auth-required otherwise.
}
}
const context: ProbeContext = {
cookie: this.options.sessions.sessionFor(instanceId)?.cookie,
refreshAttempted: false,
};
const sections = await this.#probeAll(instanceId, instance.origin, probes, context);
this.#reportIdentity(module, instanceId, instance.origin, sections);
return {
instanceId,
module,
observedAt: this.now().toISOString(),
authenticated: Boolean(context.cookie),
sections: trimToBudget(sections),
};
}
#reportIdentity(
module: InstanceModuleKey,
instanceId: string,
origin: string,
sections: readonly ModuleSection[],
): void {
const report = this.options.onIdentity;
if (!report || module !== 'overview') return;
const device = sections.find((section) => section.key === 'device' && section.state === 'ok');
if (!device || !isRecord(device.data)) return;
try {
report(instanceId, device.data, origin);
} catch {
// Bookkeeping for the identity guard never breaks a read the operator asked for.
}
}
async #probeAll(
instanceId: string,
origin: string,
probes: readonly ModuleProbe[],
context: ProbeContext,
): Promise<readonly ModuleSection[]> {
const results: ModuleSection[] = new Array<ModuleSection>(probes.length);
const queue = probes.map((probe, index) => ({ probe, index }));
const workers = Array.from({ length: Math.min(CONCURRENCY, queue.length) }, async () => {
for (;;) {
const next = queue.shift();
if (!next) return;
results[next.index] = await this.#probeOne(instanceId, origin, next.probe, context);
}
});
await Promise.all(workers);
return results;
}
async #probeOne(
instanceId: string,
origin: string,
probe: ModuleProbe,
context: ProbeContext,
): Promise<ModuleSection> {
const timeoutMs = probe.timeoutMs ?? this.defaultTimeoutMs;
const send = (token: string | undefined): Promise<UpstreamResponse> =>
this.options.request({
url: `${origin}/api${probe.path}`,
method: 'GET',
headers: { accept: 'application/json', ...(token ? { cookie: token } : {}) },
});
let response: UpstreamResponse;
try {
response = await withTimeout(send(context.cookie), timeoutMs, () => TIMEOUT_SENTINEL);
const rejected = response.status === 401 || response.status === 403;
if (rejected && this.options.ensureSession && !context.refreshAttempted) {
context.refreshAttempted = true;
try {
await this.options.ensureSession(instanceId, origin, true);
} catch {
// Keep the unauthenticated answer; the section reports auth-required below.
}
const refreshed = this.options.sessions.sessionFor(instanceId)?.cookie;
if (refreshed) {
context.cookie = refreshed;
response = await withTimeout(send(refreshed), timeoutMs, () => TIMEOUT_SENTINEL);
}
}
} catch {
return {
key: probe.key,
path: probe.path,
state: 'failed',
status: null,
data: null,
};
}
const body = decodeBody(response);
return {
key: probe.key,
path: probe.path,
state: classify(response.status, body),
status: response.status === 0 ? null : response.status,
data: body.data,
};
}
}
/** Drops the tail of the largest payloads until the whole snapshot fits the response budget. */
function trimToBudget(sections: readonly ModuleSection[]): readonly ModuleSection[] {
const sizes = sections.map((section) =>
section.state === 'ok' || section.state === 'empty'
? Buffer.byteLength(JSON.stringify(section.data ?? null), 'utf8')
: 0,
);
let total = sizes.reduce((sum, size) => sum + size, 0);
if (total <= MAX_SNAPSHOT_BYTES) return sections;
const trimmed = sections.map((section, index) => ({ section, size: sizes[index] ?? 0 }));
for (;;) {
const largest = trimmed.reduce(
(best, item) => (item.size > (best?.size ?? 0) ? item : best),
trimmed[0] as { section: ModuleSection; size: number } | undefined,
);
if (!largest || largest.size === 0 || total <= MAX_SNAPSHOT_BYTES) break;
const cut = Math.min(largest.size, total - MAX_SNAPSHOT_BYTES) + 1_024;
largest.section = {
...largest.section,
state: 'ok',
data: { truncated: true, note: 'payload exceeded the response budget' },
};
total -= cut;
largest.size = 0;
}
return trimmed.map((item) => item.section);
}
@@ -10,6 +10,7 @@ import type { Instance } from '@multi-simadmin/contracts';
import { InstanceService, InstanceServiceError } from './instance-service.js';
class MemorySecrets implements SecretStore {
readonly provider = 'macos-keychain';
readonly values = new Map<string, string>();
readonly deletes: string[] = [];
readonly sets: Array<{
@@ -109,6 +110,7 @@ describe('InstanceService', () => {
name: 'Alpha',
origin: 'http://192.168.1.10:8080',
tags: ['a', 'z'],
groupId: null,
revision: 1,
capabilityStatus: 'unknown',
freshness: 'unknown',
@@ -10,10 +10,9 @@ import type {
SnapshotFreshness,
} from '@multi-simadmin/contracts';
import type { SecretStore } from '../../infrastructure/secrets/secret-store.js';
import { parseKeychainReference } from '../../infrastructure/secrets/keychain-secret-store.js';
import { parseSecretReference } from '../../infrastructure/secrets/secret-reference.js';
const PURPOSE = 'instance-password';
const PROVIDER = 'macos-keychain';
const MAX_TAGS = 50;
const MAX_TAG_LENGTH = 100;
@@ -53,6 +52,7 @@ interface InstanceRow {
base_url: string;
config_revision: number;
updated_at: string;
group_id: string | null;
}
interface SecretRow {
id: string;
@@ -161,6 +161,7 @@ export class InstanceService {
const name = normalizeName(input.name);
const origin = normalizeOrigin(input.origin);
const tags = normalizeTags(input.tags);
const groupId = this.normalizeGroupId(input.groupId);
validatePassword(input.password);
const instanceId = this.id();
let newSecret: { id: string; external: string } | undefined;
@@ -171,9 +172,9 @@ export class InstanceService {
this.db.transaction(() => {
this.db
.prepare(
'INSERT INTO instances (id,name,base_url,auth_mode,enabled,config_revision,created_at,updated_at) VALUES (?,?,?, ?,1,1,?,?)',
'INSERT INTO instances (id,name,base_url,auth_mode,enabled,config_revision,created_at,updated_at,group_id) VALUES (?,?,?, ?,1,1,?,?,?)',
)
.run(instanceId, name, origin, newSecret ? 'password' : 'none', now, now);
.run(instanceId, name, origin, newSecret ? 'password' : 'none', now, now, groupId);
this.replaceTags(instanceId, tags, now);
if (newSecret) this.insertReference(newSecret.id, instanceId, newSecret.external, now);
})();
@@ -217,6 +218,8 @@ export class InstanceService {
);
if (query.tag !== undefined)
values = values.filter(({ value }) => value.tags.includes(query.tag!.trim()));
if (query.groupId !== undefined)
values = values.filter(({ value }) => value.groupId === query.groupId!.trim());
if (query.credentialConfigured !== undefined)
values = values.filter(
({ value }) => value.credentialConfigured === query.credentialConfigured,
@@ -257,6 +260,8 @@ export class InstanceService {
const name = patch.name === undefined ? current.name : normalizeName(patch.name);
const origin = patch.origin === undefined ? current.base_url : normalizeOrigin(patch.origin);
const tags = patch.tags === undefined ? undefined : normalizeTags(patch.tags);
const groupId =
patch.groupId === undefined ? current.group_id : this.normalizeGroupId(patch.groupId);
let committedOldSecret: SecretRow | undefined;
let newSecret: { id: string; external: string } | undefined;
if (patch.password?.action === 'set')
@@ -272,7 +277,7 @@ export class InstanceService {
const transactionOldSecret = this.secret(instanceId);
const changed = this.db
.prepare(
'UPDATE instances SET name=?,base_url=?,auth_mode=?,config_revision=config_revision+1,updated_at=? WHERE id=? AND config_revision=?',
'UPDATE instances SET name=?,base_url=?,auth_mode=?,config_revision=config_revision+1,updated_at=?,group_id=? WHERE id=? AND config_revision=?',
)
.run(
name,
@@ -281,6 +286,7 @@ export class InstanceService {
? 'password'
: 'none',
now,
groupId,
instanceId,
revision,
);
@@ -401,7 +407,7 @@ export class InstanceService {
.prepare(
'INSERT INTO secret_references (id,instance_id,purpose,provider,external_reference,created_at,updated_at) VALUES (?,?,?,?,?,?,?)',
)
.run(id, instanceId, PURPOSE, PROVIDER, external, now, now);
.run(id, instanceId, PURPOSE, this.store.provider, external, now, now);
}
private async storeSecret(
instanceId: string,
@@ -415,7 +421,7 @@ export class InstanceService {
throw new InstanceServiceError('SECRET_STORE_FAILED', 'Could not store instance credential');
}
try {
const parsed = parseKeychainReference(external);
const parsed = parseSecretReference(external);
if (parsed.instanceId !== instanceId || parsed.purpose !== PURPOSE || parsed.slot !== slot)
throw new Error('secret store returned a reference with an invalid binding');
} catch {
@@ -497,7 +503,7 @@ export class InstanceService {
AND secret_cleanup_tasks.purpose=excluded.purpose
AND secret_cleanup_tasks.provider=excluded.provider`,
)
.run(reference, instanceId, PURPOSE, PROVIDER, now, now);
.run(reference, instanceId, PURPOSE, this.store.provider, now, now);
if (queued.changes !== 1)
throw new InstanceServiceError('DATABASE_FAILED', 'Cleanup task could not be persisted');
}
@@ -522,13 +528,13 @@ export class InstanceService {
);
}
private validateCleanupEntry(entry: CleanupEntry): void {
if (entry.purpose !== PURPOSE || entry.provider !== PROVIDER)
if (entry.purpose !== PURPOSE || entry.provider !== this.store.provider)
throw new InstanceServiceError('DATABASE_FAILED', 'Cleanup task is invalid');
this.validateCleanupReference(entry.instanceId, entry.reference);
}
private validateCleanupReference(instanceId: string, reference: string): void {
try {
const parsed = parseKeychainReference(reference);
const parsed = parseSecretReference(reference);
if (
parsed.instanceId !== instanceId ||
parsed.purpose !== PURPOSE ||
@@ -542,13 +548,23 @@ export class InstanceService {
private loadRows(where: string, parameters: unknown[]): AggregateRow[] {
return this.db
.prepare(
`SELECT i.id,i.name,i.base_url,i.config_revision,i.updated_at,
`SELECT i.id,i.name,i.base_url,i.config_revision,i.updated_at,i.group_id,
group_concat(t.tag, char(31)) tags,
CASE WHEN EXISTS(SELECT 1 FROM secret_references r WHERE r.instance_id=i.id AND r.purpose=?) THEN 1 ELSE 0 END credential
FROM instances i LEFT JOIN instance_tags t ON t.instance_id=i.id ${where} GROUP BY i.id`,
)
.all(PURPOSE, ...parameters) as AggregateRow[];
}
private normalizeGroupId(value: string | null | undefined): string | null {
if (value === undefined || value === null) return null;
if (typeof value !== 'string') validation('groupId must be a string or null');
const trimmed = value.trim();
if (!trimmed) return null;
const exists = this.db.prepare('SELECT 1 FROM device_groups WHERE id=?').get(trimmed);
if (!exists) throw new InstanceServiceError('NOT_FOUND', 'Device group was not found');
return trimmed;
}
private toInstance(row: AggregateRow): Instance {
const capabilityRows = this.db
.prepare('SELECT state FROM capabilities WHERE instance_id=?')
@@ -577,6 +593,7 @@ export class InstanceService {
name: row.name,
origin: row.base_url,
tags: row.tags ? row.tags.split(String.fromCharCode(31)).sort(codePointCompare) : [],
groupId: row.group_id ?? null,
revision: row.config_revision,
capabilityStatus,
freshness,
@@ -23,6 +23,22 @@ export class JobQueryError extends Error {
}
}
export interface ControlPlaneJobSummary {
readonly total: number;
readonly active: number;
readonly succeeded: number;
readonly failed: number;
readonly attention: number;
readonly recent: readonly ControlPlaneJobSummaryItem[];
}
export interface ControlPlaneJobSummaryItem {
readonly id: string;
readonly operationId: string;
readonly status: Job['status'];
readonly createdAt: string;
}
interface JobRow {
id: unknown;
operation_id: unknown;
@@ -121,6 +137,55 @@ function safeError(serialized: unknown): ProblemDetails | undefined {
export class JobQueryService {
constructor(private readonly db: Database.Database) {}
summary(limit = 5): ControlPlaneJobSummary {
if (!Number.isSafeInteger(limit) || limit < 1 || limit > 20) validation('limit is invalid');
const totalRow = this.db.prepare('SELECT COUNT(*) AS total FROM jobs').get() as
| {
total: unknown;
}
| undefined;
if (!totalRow || typeof totalRow.total !== 'number' || !Number.isSafeInteger(totalRow.total))
fail('Persisted job count is invalid');
const statusRows = this.db
.prepare('SELECT status, COUNT(*) AS count FROM jobs GROUP BY status')
.all() as Array<{ status: unknown; count: unknown }>;
const counts = new Map<string, number>();
for (const row of statusRows) {
const status = requiredString(row.status, 'job status');
if (!(JOB_STATUSES as readonly string[]).includes(status))
fail('Persisted job status is invalid');
if (!Number.isSafeInteger(row.count) || (row.count as number) < 0)
fail('Persisted job status count is invalid');
counts.set(status, row.count as number);
}
const count = (status: Job['status']): number => counts.get(status) ?? 0;
const active = count('queued') + count('running') + count('cancelling');
const failed = count('failed');
const attention = failed + count('partially-succeeded') + count('unknown-result') + active;
const recentRows = this.db
.prepare(`SELECT ${JOB_COLUMNS} FROM jobs ORDER BY created_at DESC, id ASC LIMIT ?`)
.all(limit) as JobRow[];
const recent = recentRows.map((row) => {
const jobStatus = requiredString(row.status, 'status');
if (!(JOB_STATUSES as readonly string[]).includes(jobStatus))
fail('Persisted job status is invalid');
return freeze({
id: requiredString(row.id, 'id'),
operationId: requiredString(row.operation_id, 'operation_id'),
status: jobStatus as Job['status'],
createdAt: timestamp(row.created_at, 'created_at'),
});
});
return freeze({
total: totalRow.total,
active,
succeeded: count('succeeded'),
failed,
attention,
recent: freeze(recent),
});
}
get(id: string): Job {
if (typeof id !== 'string' || id.length === 0) validation('id must be a non-empty string');
const row = this.db.prepare(`SELECT ${JOB_COLUMNS} FROM jobs WHERE id = ?`).get(id) as
@@ -0,0 +1,108 @@
import Database from 'better-sqlite3';
import { describe, expect, it } from 'vitest';
import { migrateDatabase } from '../../infrastructure/database/migrations.js';
import { DEFAULT_RECONCILE_GRACE_MS, JobReconcileService } from './job-reconcile-service.js';
const NOW = new Date('2026-07-20T12:00:00.000Z');
const minutesAgo = (minutes: number): string =>
new Date(NOW.getTime() - minutes * 60_000).toISOString();
function setup(): { db: Database.Database; subject: JobReconcileService } {
const db = new Database(':memory:');
migrateDatabase(db);
return {
db,
subject: new JobReconcileService({ db, now: () => NOW }),
};
}
function insertJob(db: Database.Database, id: string, status: string, createdAt: string): void {
db.prepare(
`INSERT INTO jobs
(id, root_job_id, operation_id, risk_level, status, requested_by, request_id,
parameters_digest, created_at, updated_at)
VALUES (?, ?, 'op.one', 'R2', ?, 'actor', 'request-1', 'digest', ?, ?)`,
).run(id, id, status, createdAt, createdAt);
db.prepare(
`INSERT INTO job_items
(id, job_id, instance_id, attempt_number, status, created_at, updated_at)
VALUES (?, ?, 'instance-1', 1, ?, ?, ?)`,
).run(`${id}-item`, id, status === 'queued' ? 'queued' : 'running', createdAt, createdAt);
if (status !== 'queued') {
db.prepare(
`INSERT INTO job_attempts (id, job_id, status, started_at, created_at)
VALUES (?, ?, 'running', ?, ?)`,
).run(`${id}-attempt`, id, createdAt, createdAt);
}
}
const rows = (db: Database.Database, table: string): Array<Record<string, unknown>> =>
db.prepare(`SELECT * FROM ${table}`).all() as Array<Record<string, unknown>>;
describe('JobReconcileService', () => {
it('closes dispatched jobs that can no longer finish', () => {
const { db, subject } = setup();
insertJob(db, 'job-running', 'running', minutesAgo(60));
expect(subject.summary().pending).toBe(1);
expect(subject.reconcile()).toEqual({ interrupted: 1, pending: 0 });
expect(rows(db, 'jobs')[0]).toMatchObject({
id: 'job-running',
status: 'unknown-result',
finished_at: NOW.toISOString(),
});
expect(rows(db, 'job_items')[0]).toMatchObject({
status: 'unknown-result',
result_code: 'INTERRUPTED',
});
expect(rows(db, 'job_attempts')[0]).toMatchObject({
status: 'unknown-result',
finished_at: NOW.toISOString(),
});
});
it('cancels queued jobs that never reached a transport', () => {
const { db, subject } = setup();
insertJob(db, 'job-queued', 'queued', minutesAgo(30));
expect(subject.reconcile()).toEqual({ interrupted: 1, pending: 0 });
expect(rows(db, 'jobs')[0]).toMatchObject({ status: 'cancelled' });
expect(rows(db, 'job_items')[0]).toMatchObject({ result_code: 'NEVER_DISPATCHED' });
});
it('leaves jobs inside the grace window alone', () => {
const { db, subject } = setup();
insertJob(db, 'job-fresh', 'running', minutesAgo(2));
insertJob(db, 'job-old', 'cancelling', minutesAgo(DEFAULT_RECONCILE_GRACE_MS + 1));
expect(subject.reconcile()).toEqual({ interrupted: 1, pending: 1 });
const byId = new Map(rows(db, 'jobs').map((row) => [row.id, row.status]));
expect(byId.get('job-fresh')).toBe('running');
expect(byId.get('job-old')).toBe('unknown-result');
});
it('reports the oldest pending job so the console can warn about it', () => {
const { db, subject } = setup();
insertJob(db, 'job-newer', 'running', minutesAgo(20));
insertJob(db, 'job-older', 'running', minutesAgo(40));
insertJob(db, 'job-done', 'succeeded', minutesAgo(90));
expect(subject.summary()).toEqual({
pending: 2,
dispatched: 2,
queued: 0,
oldestCreatedAt: minutesAgo(40),
});
});
it('is idempotent and rejects a negative window', () => {
const { db, subject } = setup();
insertJob(db, 'job-once', 'running', minutesAgo(60));
expect(subject.reconcile(0).interrupted).toBe(1);
expect(subject.reconcile(0)).toEqual({ interrupted: 0, pending: 0 });
expect(() => subject.reconcile(-1)).toThrow(RangeError);
});
});
@@ -0,0 +1,178 @@
import type { SqliteDatabase } from '../../infrastructure/database/database.js';
/**
* Reconciles jobs that the console still believes are in flight.
*
* Every job is executed inside the request that created it, so a row left in `running` after a
* crash can never finish on its own. The official Hub exposes the same capability on its command
* ledger; here it closes the loop against our own job table instead of a remote queue.
*
* `queued` rows never reached a transport, so they are cancelled. `running` and `cancelling` rows
* may or may not have reached the device, so they become `unknown-result`, which is the only
* honest terminal state for an interrupted write.
*
* The service also owns operator cancellation, which the published contract promises on
* `POST /api/v1/jobs/{jobId}/cancel`. Both paths share one rule: a job only ever moves from an
* active state to a terminal one, and every write is guarded by the active state, so a cancel
* racing an in-flight executor loses cleanly instead of overwriting a real result.
*/
const DISPATCHED_STATUSES = new Set(['running', 'cancelling']);
const QUEUED_STATUSES = new Set(['queued']);
const ACTIVE_STATUSES = new Set([...DISPATCHED_STATUSES, ...QUEUED_STATUSES]);
const ACTIVE_PLACEHOLDERS = '?, ?, ?';
/** On-demand sweeps leave a margin so a slow batch in another request is never clobbered. */
export const DEFAULT_RECONCILE_GRACE_MS = 15 * 60 * 1000;
export interface JobReconcileResult {
/** Jobs closed by this sweep. */
readonly interrupted: number;
/** Jobs still considered in flight once the sweep finished. */
readonly pending: number;
}
export interface JobReconcileSummary {
readonly pending: number;
readonly dispatched: number;
readonly queued: number;
readonly oldestCreatedAt: string | null;
}
export interface JobReconcileOptions {
readonly db: SqliteDatabase;
readonly now?: () => Date;
}
export type JobCancelErrorCode = 'NOT_FOUND' | 'NOT_CANCELLABLE';
export class JobCancelError extends Error {
constructor(
readonly code: JobCancelErrorCode,
message: string,
) {
super(message);
this.name = 'JobCancelError';
}
}
const CANCEL_RESULT_CODE = 'CANCELLED_BY_OPERATOR';
const isActiveStatus = (status: unknown): status is string =>
typeof status === 'string' && ACTIVE_STATUSES.has(status);
export class JobReconcileService {
readonly #db: SqliteDatabase;
readonly #now: () => Date;
constructor(options: JobReconcileOptions) {
this.#db = options.db;
this.#now = options.now ?? (() => new Date());
}
summary(olderThanMs = 0): JobReconcileSummary {
const cutoff = this.#cutoff(olderThanMs);
const rows = this.#db
.prepare(
'SELECT status, created_at FROM jobs WHERE status IN (?,?,?) AND created_at <= ? ORDER BY created_at ASC, id ASC',
)
.all('queued', 'running', 'cancelling', cutoff) as Array<{
status: unknown;
created_at: unknown;
}>;
let dispatched = 0;
let queued = 0;
for (const row of rows) {
if (typeof row.status === 'string' && DISPATCHED_STATUSES.has(row.status)) dispatched += 1;
else if (typeof row.status === 'string' && QUEUED_STATUSES.has(row.status)) queued += 1;
}
const oldest = rows[0]?.created_at;
return {
pending: rows.length,
dispatched,
queued,
oldestCreatedAt: typeof oldest === 'string' ? oldest : null,
};
}
reconcile(olderThanMs = DEFAULT_RECONCILE_GRACE_MS): JobReconcileResult {
const cutoff = this.#cutoff(olderThanMs);
const closedAt = this.#now().toISOString();
const interrupted = this.#db.transaction((): number => {
const rows = this.#db
.prepare(
'SELECT id, status FROM jobs WHERE status IN (?,?,?) AND created_at <= ? ORDER BY created_at ASC, id ASC',
)
.all('queued', 'running', 'cancelling', cutoff) as Array<{
id: unknown;
status: unknown;
}>;
const closeItems = this.#db.prepare(
`UPDATE job_items SET status = ?, result_code = ?, finished_at = ?, updated_at = ?
WHERE job_id = ? AND status IN ('queued','running','cancelling')`,
);
const closeAttempts = this.#db.prepare(
"UPDATE job_attempts SET status = ?, finished_at = ? WHERE job_id = ? AND status = 'running'",
);
const closeJob = this.#db.prepare(
'UPDATE jobs SET status = ?, finished_at = ?, updated_at = ? WHERE id = ? AND status IN (?,?,?)',
);
let count = 0;
for (const row of rows) {
if (typeof row.id !== 'string' || !isActiveStatus(row.status)) continue;
const dispatched = DISPATCHED_STATUSES.has(row.status);
const finalStatus = dispatched ? 'unknown-result' : 'cancelled';
const resultCode = dispatched ? 'INTERRUPTED' : 'NEVER_DISPATCHED';
closeItems.run(finalStatus, resultCode, closedAt, closedAt, row.id);
closeAttempts.run(finalStatus, closedAt, row.id);
closeJob.run(finalStatus, closedAt, closedAt, row.id, 'queued', 'running', 'cancelling');
count += 1;
}
return count;
})();
return { interrupted, pending: this.summary().pending };
}
#cutoff(olderThanMs: number): string {
if (!Number.isFinite(olderThanMs) || olderThanMs < 0) {
throw new RangeError('olderThanMs must be a non-negative finite number');
}
return new Date(this.#now().getTime() - olderThanMs).toISOString();
}
/** Requests cancellation of a job that has not reached a terminal state yet. */
cancel(jobId: string): void {
if (typeof jobId !== 'string' || jobId.length === 0 || jobId.length > 128) {
throw new JobCancelError('NOT_FOUND', 'Job was not found');
}
const closedAt = this.#now().toISOString();
const changed = this.#db.transaction((): boolean => {
const row = this.#db.prepare('SELECT status FROM jobs WHERE id = ?').get(jobId) as
| { status: unknown }
| undefined;
if (!row) throw new JobCancelError('NOT_FOUND', 'Job was not found');
if (!isActiveStatus(row.status))
throw new JobCancelError('NOT_CANCELLABLE', 'Job already reached a terminal state');
this.#db
.prepare(
`UPDATE job_items SET status = 'cancelled', result_code = ?, finished_at = ?, updated_at = ?
WHERE job_id = ? AND status IN ('queued', 'running', 'cancelling')`,
)
.run(CANCEL_RESULT_CODE, closedAt, closedAt, jobId);
this.#db
.prepare(
"UPDATE job_attempts SET status = 'cancelled', finished_at = ? WHERE job_id = ? AND status = 'running'",
)
.run(closedAt, jobId);
const updated = this.#db
.prepare(
`UPDATE jobs SET status = 'cancelled', finished_at = ?, updated_at = ?
WHERE id = ? AND status IN (${ACTIVE_PLACEHOLDERS})`,
)
.run(closedAt, closedAt, jobId, 'queued', 'running', 'cancelling');
return updated.changes === 1;
})();
if (!changed)
throw new JobCancelError('NOT_CANCELLABLE', 'Job already reached a terminal state');
}
}
@@ -20,16 +20,9 @@ function pendingDatabase(): Database.Database {
db.pragma('foreign_keys = ON');
migrateDatabase(db);
const now = '2026-07-16T00:00:00.000Z';
db.prepare('INSERT INTO instances VALUES (?,?,?,?,?,?,?,?)').run(
'alpha',
'Alpha',
'https://203.0.113.8',
'none',
1,
3,
now,
now,
);
db.prepare(
'INSERT INTO instances (id,name,base_url,auth_mode,enabled,config_revision,created_at,updated_at) VALUES (?,?,?,?,?,?,?,?)',
).run('alpha', 'Alpha', 'https://203.0.113.8', 'none', 1, 3, now, now);
db.prepare('INSERT INTO app_settings VALUES (?,?,?,?)').run(
'legacy-import.instance.alpha',
JSON.stringify({
@@ -46,6 +39,7 @@ function pendingDatabase(): Database.Database {
}
class FakeStore implements SecretStore {
readonly provider = 'macos-keychain';
readonly sets: { key: SecretKey; value: string }[] = [];
readonly deletes: string[] = [];
setError?: Error;
@@ -233,7 +227,9 @@ describe('activatePendingSecret', () => {
migrateDatabase(first);
const now = '2026-07-16T00:00:00.000Z';
first
.prepare('INSERT INTO instances VALUES (?,?,?,?,?,?,?,?)')
.prepare(
'INSERT INTO instances (id,name,base_url,auth_mode,enabled,config_revision,created_at,updated_at) VALUES (?,?,?,?,?,?,?,?)',
)
.run('alpha', 'Alpha', 'https://203.0.113.8', 'none', 1, 3, now, now);
first
.prepare('INSERT INTO app_settings VALUES (?,?,?,?)')
@@ -4,7 +4,6 @@ import { getManagedDatabaseIdentity } from '../../infrastructure/database/databa
import type { SecretStore } from '../../infrastructure/secrets/secret-store.js';
const PURPOSE = 'instance-password';
const PROVIDER = 'macos-keychain';
const metadataKey = (instanceId: string): string => `legacy-import.instance.${instanceId}`;
const referenceId = (instanceId: string): string => `legacy-secret:${instanceId}:${PURPOSE}`;
@@ -163,7 +162,7 @@ async function activatePendingSecretExclusive({
throw new ActivationError('DATABASE_FAILED', 'Instance changed during secret activation');
db.prepare(
'INSERT INTO secret_references (id,instance_id,purpose,provider,external_reference,created_at,updated_at) VALUES (?,?,?,?,?,?,?)',
).run(id, instanceId, PURPOSE, PROVIDER, externalReference, now, now);
).run(id, instanceId, PURPOSE, store.provider, externalReference, now, now);
const updated = db
.prepare(
"UPDATE instances SET auth_mode='password',config_revision=config_revision+1,updated_at=? WHERE id=? AND auth_mode='none' AND config_revision=?",
@@ -274,26 +274,12 @@ describe('legacy import', () => {
const path = await legacyFile(valid);
const db = database();
const now = new Date().toISOString();
db.prepare('INSERT INTO instances VALUES (?,?,?,?,?,?,?,?)').run(
'alpha',
'Different',
'https://203.0.113.99',
'none',
1,
7,
now,
now,
);
db.prepare('INSERT INTO instances VALUES (?,?,?,?,?,?,?,?)').run(
'owner',
'Owner',
'http://[2001:4860:4860::8888]:8080',
'none',
1,
1,
now,
now,
);
db.prepare(
'INSERT INTO instances (id,name,base_url,auth_mode,enabled,config_revision,created_at,updated_at) VALUES (?,?,?,?,?,?,?,?)',
).run('alpha', 'Different', 'https://203.0.113.99', 'none', 1, 7, now, now);
db.prepare(
'INSERT INTO instances (id,name,base_url,auth_mode,enabled,config_revision,created_at,updated_at) VALUES (?,?,?,?,?,?,?,?)',
).run('owner', 'Owner', 'http://[2001:4860:4860::8888]:8080', 'none', 1, 1, now, now);
const preview = await previewLegacyImport(path, db);
expect(preview.instances.map((item) => item.status)).toEqual(['conflict', 'conflict']);
expect(preview.counts.conflict).toBe(2);
@@ -0,0 +1,258 @@
import Database from 'better-sqlite3';
import { afterEach, beforeEach, describe, expect, it } from 'vitest';
import { migrateDatabase } from '../../infrastructure/database/migrations.js';
import { HubMessageService } from './hub-message-service.js';
import type { InstanceMessage } from './instance-message-service.js';
const now = new Date('2026-09-03T10:00:00.000Z');
const instanceMessages = new Map<string, readonly InstanceMessage[]>([
[
'device-1',
[
{
id: '101',
direction: 'incoming',
phoneNumber: '10086',
content: '余额提醒',
timestamp: '2026-09-03T09:59:00.000Z',
status: 'received',
transport: 'modem',
},
],
],
]);
const messageService = {
list: async (instanceId: string, query: { limit: number; offset: number }) => ({
messages:
query.offset >= (instanceMessages.get(instanceId)?.length ?? 0)
? []
: (instanceMessages.get(instanceId) ?? []).slice(query.offset, query.offset + query.limit),
}),
send: async () => {
const current = [...(instanceMessages.get('device-1') ?? [])];
const message: InstanceMessage = {
id: '102',
direction: 'outgoing',
phoneNumber: '10086',
content: '已发送',
timestamp: now.toISOString(),
status: 'sent',
transport: 'modem',
};
instanceMessages.set('device-1', [message, ...current]);
return { sent: true as const };
},
deleteMany: async (items: readonly { instanceId: string; id: string }[]) => ({
requested: items.length,
deleted: items.length,
failed: 0,
failures: [],
}),
};
const initialInstanceMessages = new Map(instanceMessages);
describe('HubMessageService', () => {
let db: Database.Database;
beforeEach(() => {
db = new Database(':memory:');
db.pragma('foreign_keys = ON');
migrateDatabase(db);
db.prepare(
`INSERT INTO instances (id,name,base_url,auth_mode,enabled,config_revision,created_at,updated_at)
VALUES ('device-1','Modem A','http://device-1.invalid','password',1,1,?,?)`,
).run(now.toISOString(), now.toISOString());
});
afterEach(() => {
instanceMessages.clear();
for (const [instanceId, messages] of initialInstanceMessages) {
instanceMessages.set(instanceId, [...messages]);
}
db.close();
});
function service() {
return new HubMessageService(db, {
instances: {
list: async () => ({
items: [
{
id: 'device-1',
name: 'Modem A',
origin: 'http://device-1.invalid',
tags: [],
revision: 1,
capabilityStatus: 'unknown',
freshness: 'unknown',
credentialConfigured: true,
},
],
page: { page: 1, pageSize: 100, total: 1 },
}),
},
messages: messageService,
now: () => now,
});
}
it('persists a full SMS sync and serves subsequent reads from SQLite', async () => {
const hub = service();
const first = await hub.syncAll();
expect(first).toMatchObject({ scanned: 1, synced: 1, available: 1 });
instanceMessages.set('device-1', []);
const page = await hub.list({ limit: 10, offset: 0 });
expect(page.total).toBe(1);
expect(page.items[0]).toMatchObject({
instanceId: 'device-1',
instanceName: 'Modem A',
phoneNumber: '10086',
content: '余额提醒',
syncedAt: now.toISOString(),
});
});
it('records outbound SMS in the central message store after upstream delivery', async () => {
const hub = service();
await hub.syncAll();
await hub.send('device-1', { phoneNumber: '10086', content: '已发送' });
const page = await hub.list({ limit: 10, offset: 0 });
expect(page).toMatchObject({ total: 2 });
});
it('deletes the central copy without requiring another upstream read', async () => {
const hub = service();
await hub.syncAll();
const result = await hub.deleteMany([{ instanceId: 'device-1', id: '101' }]);
expect(result).toMatchObject({ requested: 1, deleted: 1, failed: 0 });
expect(await hub.list({ limit: 10, offset: 0 })).toMatchObject({ total: 0 });
});
describe('conversations', () => {
const at = (secondsAgo: number): string =>
new Date(now.getTime() - secondsAgo * 1_000).toISOString();
function message(
id: string,
direction: InstanceMessage['direction'],
phoneNumber: string,
content: string,
timestamp: string,
): InstanceMessage {
return {
id,
direction,
phoneNumber,
content,
timestamp,
status: direction,
transport: 'modem',
};
}
function seed(hub: HubMessageService): void {
hub.upsert('device-1', message('a', 'incoming', '10086', '余额提醒', at(300)), now);
hub.upsert('device-1', message('b', 'outgoing', '10086', '已查询', at(200)), now);
hub.upsert('device-1', message('c', 'incoming', '13900139000', '验证码 1234', at(100)), now);
hub.upsert('device-1', message('d', 'incoming', '10086', '流量提醒', at(50)), now);
}
it('groups the archive per node and phone with counts and a newest preview', () => {
const hub = service();
seed(hub);
const page = hub.conversations({ limit: 10 });
expect(page.totalCount).toBe(2);
expect(page.stats).toEqual({ incoming: 3, outgoing: 1, total: 4 });
expect(page.items.map((item) => item.phoneNumber)).toEqual(['10086', '13900139000']);
expect(page.items[0]).toMatchObject({
instanceId: 'device-1',
instanceName: 'Modem A',
messageCount: 3,
incomingCount: 2,
lastMessage: { content: '流量提醒', timestamp: at(50) },
});
expect(page.items[1]).toMatchObject({
messageCount: 1,
incomingCount: 1,
lastMessage: { content: '验证码 1234' },
});
});
it('filters conversations by node, phone and search text', () => {
const hub = service();
seed(hub);
expect(hub.conversations({ instanceId: 'device-2' }).totalCount).toBe(0);
expect(hub.conversations({ phoneNumber: '10086' }).items).toHaveLength(1);
const searched = hub.conversations({ search: '验证码' });
expect(searched.items.map((item) => item.phoneNumber)).toEqual(['13900139000']);
expect(searched.totalCount).toBe(1);
});
it('keeps a thread when only one message matches the direction filter', () => {
const hub = service();
seed(hub);
const outgoing = hub.conversations({ direction: 'outgoing' });
expect(outgoing.totalCount).toBe(1);
expect(outgoing.items[0]).toMatchObject({
phoneNumber: '10086',
// The thread is still reported at its full size even though only one row matched.
messageCount: 3,
incomingCount: 2,
lastMessage: { direction: 'outgoing', content: '已查询' },
});
const incoming = hub.conversations({ direction: 'incoming' });
expect(incoming.totalCount).toBe(2);
expect(incoming.items.map((item) => item.phoneNumber)).toEqual(['10086', '13900139000']);
expect(incoming.items[0]?.lastMessage).toMatchObject({
direction: 'incoming',
content: '流量提醒',
});
expect(incoming.items[1]?.lastMessage).toMatchObject({
direction: 'incoming',
content: '验证码 1234',
});
});
it('reports direction totals for the whole filtered archive, not just the page', () => {
const hub = service();
seed(hub);
const page = hub.conversations({ limit: 1 });
expect(page.items).toHaveLength(1);
expect(page.stats).toEqual({ incoming: 3, outgoing: 1, total: 4 });
expect(hub.conversations({ instanceId: 'device-1', search: '10086' }).stats).toEqual({
incoming: 2,
outgoing: 1,
total: 3,
});
expect(hub.conversations({ instanceId: 'device-2' }).stats).toEqual({
incoming: 0,
outgoing: 0,
total: 0,
});
});
it('pages conversations by newest activity', () => {
const hub = service();
seed(hub);
const first = hub.conversations({ limit: 1, offset: 0 });
const second = hub.conversations({ limit: 1, offset: 1 });
expect(first.totalCount).toBe(2);
expect(first.items[0]?.phoneNumber).toBe('10086');
expect(second.items[0]?.phoneNumber).toBe('13900139000');
});
});
});
@@ -0,0 +1,505 @@
import { randomUUID } from 'node:crypto';
import type Database from 'better-sqlite3';
import type { InstancePageQuery } from '@multi-simadmin/contracts';
import type {
DeleteSmsMessageRequest,
InstanceMessage,
InstanceMessageService,
MessageDeleteResult,
} from './instance-message-service.js';
export interface HubMessagePage {
readonly items: readonly HubMessage[];
readonly total: number;
}
/** SQL literal sets; upstream labels the same direction two different ways. */
const INCOMING_DIRECTIONS = "'incoming','received'";
const OUTGOING_DIRECTIONS = "'outgoing','sent'";
export interface HubMessage {
readonly id: string;
readonly instanceId: string;
readonly instanceName: string;
readonly direction: string;
readonly phoneNumber: string;
readonly content: string;
readonly timestamp: string;
readonly status: string;
readonly transport: string;
readonly syncedAt: string;
}
export interface HubMessageDevice {
readonly id: string;
readonly name: string;
readonly availability: 'online' | 'unavailable';
}
export interface HubMessageSnapshot {
readonly messages: readonly HubMessage[];
readonly devices: readonly HubMessageDevice[];
readonly total: number;
}
/**
* One phone-thread per node, mirroring the Hub conversation model: the archive is
* grouped by (instance, phone number) and carries the newest message as a preview.
*/
export interface HubMessageConversation {
readonly instanceId: string;
readonly instanceName: string;
readonly phoneNumber: string;
readonly messageCount: number;
readonly incomingCount: number;
readonly lastMessage: HubMessage;
}
/** Direction totals for the whole filtered archive, mirroring the Hub SMS counters. */
export interface HubMessageDirectionStats {
readonly incoming: number;
readonly outgoing: number;
readonly total: number;
}
export interface HubMessageConversationPage {
readonly items: readonly HubMessageConversation[];
readonly totalCount: number;
readonly stats: HubMessageDirectionStats;
}
export interface HubSyncSummary {
readonly scanned: number;
readonly available: number;
readonly synced: number;
readonly unavailable: number;
}
interface HubMessageServiceOptions {
readonly instances: {
readonly list: (query?: InstancePageQuery) => Promise<{
readonly items: readonly { readonly id: string; readonly name: string }[];
}>;
};
readonly messages: Pick<InstanceMessageService, 'list' | 'send' | 'deleteMany'>;
readonly onIncomingMessage?: (instanceId: string, message: InstanceMessage) => Promise<void>;
readonly now?: () => Date;
readonly id?: () => string;
readonly maximumMessagesPerDevice?: number;
readonly refreshTtlMs?: number;
}
const PAGE_SIZE = 100;
const DEFAULT_MAXIMUM = 5_000;
const MAX_DEVICES = 5_000;
const DEFAULT_REFRESH_TTL_MS = 5_000;
const isoTimestamp = (value: string, fallback: Date): string => {
const time = Date.parse(value);
return Number.isFinite(time) ? new Date(time).toISOString() : fallback.toISOString();
};
export class HubMessageService {
readonly #db: Database.Database;
readonly #options: HubMessageServiceOptions;
readonly #availability = new Map<string, boolean>();
#lastRefreshAt = Number.NEGATIVE_INFINITY;
#inFlight: Promise<HubSyncSummary> | undefined;
#lastSummary: HubSyncSummary = { scanned: 0, available: 0, synced: 0, unavailable: 0 };
constructor(db: Database.Database, options: HubMessageServiceOptions) {
this.#db = db;
this.#options = options;
}
async #allInstances(): Promise<readonly { readonly id: string; readonly name: string }[]> {
const items: { readonly id: string; readonly name: string }[] = [];
let page = 1;
while (items.length < MAX_DEVICES) {
const current = await this.#options.instances.list({ page, pageSize: PAGE_SIZE });
items.push(...current.items.slice(0, MAX_DEVICES - items.length));
if (current.items.length < PAGE_SIZE) break;
page += 1;
}
return items;
}
async syncAll(): Promise<HubSyncSummary> {
const instances = await this.#allInstances();
let available = 0;
let synced = 0;
for (const instance of instances) {
try {
synced += await this.syncDevice(instance.id);
available += 1;
} catch {
// Preserve the central archive when a device is temporarily unavailable.
}
}
return {
scanned: instances.length,
available,
synced,
unavailable: instances.length - available,
};
}
async syncDevice(instanceId: string): Promise<number> {
try {
const stored = await this.#pullDevice(instanceId);
this.#availability.set(instanceId, true);
return stored;
} catch (error) {
this.#availability.set(instanceId, false);
throw error;
}
}
async #pullDevice(instanceId: string): Promise<number> {
const observedAt = this.#nowDate();
let stored = 0;
for (let offset = 0; offset < this.#maximumPerDevice(); offset += PAGE_SIZE) {
const result = await this.#options.messages.list(instanceId, {
limit: PAGE_SIZE,
offset,
});
for (const message of result.messages) {
const inserted = this.upsert(instanceId, message, observedAt);
stored += inserted;
if (
inserted === 1 &&
(message.direction === 'incoming' || message.direction === 'received')
)
await this.#options.onIncomingMessage?.(instanceId, message);
}
if (result.messages.length < PAGE_SIZE) break;
}
return stored;
}
upsert(instanceId: string, message: InstanceMessage, observedAt: Date = this.#nowDate()): number {
const exists = this.#db
.prepare('SELECT 1 FROM sms_messages WHERE instance_id=? AND upstream_id=?')
.get(instanceId, message.id);
const result = this.#db
.prepare(
`INSERT INTO sms_messages
(id,instance_id,upstream_id,direction,phone_number,content,timestamp,status,transport,synced_at)
VALUES (?,?,?,?,?,?,?,?,?,?)
ON CONFLICT(instance_id, upstream_id) DO UPDATE SET
direction=excluded.direction,
phone_number=excluded.phone_number,
content=excluded.content,
timestamp=excluded.timestamp,
status=excluded.status,
transport=excluded.transport,
synced_at=excluded.synced_at`,
)
.run(
this.#id(),
instanceId,
message.id,
message.direction,
message.phoneNumber,
message.content,
isoTimestamp(message.timestamp, observedAt),
message.status,
message.transport,
observedAt.toISOString(),
);
return exists ? 0 : result.changes;
}
async list(
query: {
readonly limit?: number;
readonly offset?: number;
readonly search?: string;
readonly instanceId?: string;
readonly phoneNumber?: string;
} = {},
): Promise<HubMessagePage> {
const limit = Math.min(Math.max(query.limit ?? 24, 1), 100);
const offset = Math.max(query.offset ?? 0, 0);
const filter = this.#filter(query);
const where = filter.where;
const parameters = [...filter.parameters];
const total = Number(
(
this.#db
.prepare(`SELECT COUNT(*) AS count FROM sms_messages${where}`)
.get(...parameters) as { count?: number } | undefined
)?.count ?? 0,
);
const rows = this.#db
.prepare(
`SELECT s.*, COALESCE(i.name, '已移除节点') AS instance_name
FROM sms_messages s LEFT JOIN instances i ON i.id = s.instance_id${where}
ORDER BY s.timestamp DESC, s.id DESC LIMIT ? OFFSET ?`,
)
.all(...parameters, limit, offset) as Array<Record<string, unknown>>;
return {
items: Object.freeze(rows.map((row) => this.#message(row))),
total,
};
}
/**
* Groups the central archive into phone threads. The newest message of each group
* is resolved by the same `timestamp, id` ordering used by `list`, so the preview
* always matches the first row a reader sees when opening the conversation.
*/
conversations(
query: {
readonly limit?: number;
readonly offset?: number;
readonly search?: string;
readonly instanceId?: string;
readonly phoneNumber?: string;
readonly direction?: 'incoming' | 'outgoing';
} = {},
): HubMessageConversationPage {
const limit = Math.min(Math.max(query.limit ?? 50, 1), 200);
const offset = Math.max(query.offset ?? 0, 0);
const filter = this.#filter(query);
const scoped = `SELECT id, instance_id, direction, phone_number, content, timestamp,
status, transport, synced_at FROM sms_messages${filter.where}`;
// A thread only survives a direction filter when it holds at least one matching message,
// and the preview resolves inside that subset, while `message_count` stays the full size.
const matched =
query.direction === 'incoming'
? `direction IN (${INCOMING_DIRECTIONS})`
: query.direction === 'outgoing'
? `direction IN (${OUTGOING_DIRECTIONS})`
: '1';
const counters = this.#db
.prepare(
`SELECT COUNT(*) AS total,
SUM(CASE WHEN direction IN (${INCOMING_DIRECTIONS}) THEN 1 ELSE 0 END) AS incoming,
SUM(CASE WHEN direction IN (${OUTGOING_DIRECTIONS}) THEN 1 ELSE 0 END) AS outgoing
FROM (${scoped})`,
)
.get(...filter.parameters) as Record<string, number | null> | undefined;
const stats = Object.freeze({
incoming: Number(counters?.incoming ?? 0),
outgoing: Number(counters?.outgoing ?? 0),
total: Number(counters?.total ?? 0),
});
// One row per surviving thread, so the outer COUNT is the thread total.
const groupedSql = `SELECT instance_id FROM (
SELECT *, CASE WHEN ${matched} THEN 1 ELSE 0 END AS matched FROM (${scoped})
) GROUP BY instance_id, phone_number HAVING SUM(matched) > 0`;
const totalCount = Number(
(
this.#db
.prepare(`SELECT COUNT(*) AS count FROM (${groupedSql})`)
.get(...filter.parameters) as { count?: number } | undefined
)?.count ?? 0,
);
const rows = this.#db
.prepare(
`WITH flagged AS (
SELECT *, CASE WHEN ${matched} THEN 1 ELSE 0 END AS matched
FROM (${scoped})
),
grouped AS (
SELECT instance_id, phone_number,
COUNT(*) AS message_count,
SUM(CASE WHEN direction IN (${INCOMING_DIRECTIONS}) THEN 1 ELSE 0 END) AS incoming_count,
MAX(CASE WHEN matched = 1 THEN timestamp || '#' || id END) AS latest_key
FROM flagged GROUP BY instance_id, phone_number
HAVING SUM(matched) > 0
)
SELECT s.id, s.direction, s.phone_number, s.content, s.timestamp, s.status,
s.transport, s.synced_at, g.instance_id, g.message_count, g.incoming_count,
COALESCE(i.name, '已移除节点') AS instance_name
FROM grouped g
JOIN flagged s
ON s.instance_id = g.instance_id AND s.phone_number = g.phone_number
AND (s.timestamp || '#' || s.id) = g.latest_key
LEFT JOIN instances i ON i.id = g.instance_id
ORDER BY g.latest_key DESC LIMIT ? OFFSET ?`,
)
.all(...filter.parameters, limit, offset) as Array<Record<string, unknown>>;
return {
items: Object.freeze(
rows.flatMap((row) => {
const messageCount = Number(row.message_count ?? 0);
if (!Number.isSafeInteger(messageCount) || messageCount < 1) return [];
return [
Object.freeze({
instanceId: String(row.instance_id),
instanceName: String(row.instance_name),
phoneNumber: String(row.phone_number),
messageCount,
incomingCount: Number(row.incoming_count ?? 0),
lastMessage: this.#message(row),
}),
];
}),
),
totalCount,
stats,
};
}
#filter(
query: Readonly<{ search?: string; instanceId?: string; phoneNumber?: string }>,
): Readonly<{ where: string; parameters: readonly string[] }> {
const conditions: string[] = [];
const parameters: string[] = [];
if (query.instanceId) {
conditions.push('instance_id = ?');
parameters.push(query.instanceId);
}
if (query.phoneNumber) {
conditions.push('phone_number = ?');
parameters.push(query.phoneNumber);
}
const search = query.search?.trim();
if (search) {
conditions.push(
`(content LIKE ? ESCAPE '\\' OR phone_number LIKE ? ESCAPE '\\' OR instance_id IN (
SELECT id || '|' || name FROM instances WHERE name LIKE ? ESCAPE '\\'
))`,
);
const pattern = `%${search.replaceAll('\\', '\\\\').replaceAll('%', '\\%').replaceAll('_', '\\_')}%`;
parameters.push(pattern, pattern, pattern);
}
return {
where: conditions.length > 0 ? ` WHERE ${conditions.join(' AND ')}` : '',
parameters,
};
}
async snapshot(
query: {
readonly limit?: number;
readonly offset?: number;
readonly search?: string;
readonly instanceId?: string;
readonly phoneNumber?: string;
} = {},
): Promise<HubMessageSnapshot> {
await this.refresh();
const [page, instances] = await Promise.all([this.list(query), this.#allInstances()]);
return {
messages: page.items,
devices: instances.map((instance) => ({
id: instance.id,
name: instance.name,
availability: this.#availability.get(instance.id) ? 'online' : 'unavailable',
})),
total: page.total,
};
}
/**
* Pulls every node into the central archive, but only when the last pull is older
* than the refresh window; concurrent readers share a single in-flight sync.
*/
async refresh(force = false): Promise<HubSyncSummary> {
const ttl = Math.max(0, this.#options.refreshTtlMs ?? DEFAULT_REFRESH_TTL_MS);
const age = this.#nowDate().getTime() - this.#lastRefreshAt;
if (!force && this.#lastRefreshAt > Number.NEGATIVE_INFINITY && age < ttl) {
return this.#lastSummary;
}
if (!this.#inFlight) {
this.#inFlight = this.syncAll().then((summary) => {
this.#lastSummary = summary;
this.#lastRefreshAt = this.#nowDate().getTime();
return summary;
});
void this.#inFlight.catch(() => undefined);
}
try {
return await this.#inFlight;
} finally {
this.#inFlight = undefined;
}
}
async send(
instanceId: string,
input: { readonly phoneNumber: string; readonly content: string },
): Promise<{ readonly sent: true }> {
const result = await this.#options.messages.send(instanceId, input);
try {
await this.syncDevice(instanceId);
} catch {
// Upstream accepted the send; the next sync can repair the central copy.
}
return result;
}
async deleteMany(input: readonly DeleteSmsMessageRequest[]): Promise<MessageDeleteResult> {
const lookup = this.#db.prepare(
'SELECT id,upstream_id FROM sms_messages WHERE instance_id=? AND (id=? OR upstream_id=?)',
);
const resolved = input.map((item) => {
const row = lookup.get(item.instanceId, item.id, item.id) as
| { id: string; upstream_id: string }
| undefined;
return {
instanceId: item.instanceId,
upstreamId: row?.upstream_id ?? item.id,
centralId: row?.id ?? item.id,
};
});
const result = await this.#options.messages.deleteMany(
resolved.map(({ instanceId, upstreamId }) => ({ instanceId, id: upstreamId })),
);
const byUpstream = new Map(
resolved.map((item) => [`${item.instanceId}|${item.upstreamId}`, item]),
);
const failed = new Set(result.failures.map((failure) => `${failure.instanceId}|${failure.id}`));
const remove = this.#db.prepare(
'DELETE FROM sms_messages WHERE instance_id=? AND (id=? OR upstream_id=?)',
);
this.#db.transaction(() => {
for (const item of resolved) {
if (failed.has(`${item.instanceId}|${item.upstreamId}`)) continue;
remove.run(item.instanceId, item.centralId, item.upstreamId);
}
})();
return Object.freeze({
...result,
failures: Object.freeze(
result.failures.map((failure) => {
const item = byUpstream.get(`${failure.instanceId}|${failure.id}`);
return Object.freeze({ ...failure, id: item?.centralId ?? failure.id });
}),
),
});
}
#message(row: Record<string, unknown>): HubMessage {
return Object.freeze({
id: String(row.id),
instanceId: String(row.instance_id),
instanceName: String(row.instance_name),
direction: String(row.direction),
phoneNumber: String(row.phone_number),
content: String(row.content),
timestamp: String(row.timestamp),
status: String(row.status),
transport: String(row.transport),
syncedAt: String(row.synced_at),
});
}
#maximumPerDevice(): number {
return Math.max(PAGE_SIZE, this.#options.maximumMessagesPerDevice ?? DEFAULT_MAXIMUM);
}
#nowDate(): Date {
return this.#options.now?.() ?? new Date();
}
#id(): string {
return this.#options.id?.() ?? randomUUID();
}
}
@@ -4,12 +4,19 @@ import {
InstanceMessageService,
MessageServiceError,
parseMessageList,
validMessageContent,
validMessageId,
} from './instance-message-service.js';
const instance = { id: 'alpha', origin: 'http://192.168.1.10:8080' };
const instances = { get: vi.fn(async (id: string) => (id === 'alpha' ? instance : undefined)) };
describe('InstanceMessageService', () => {
it('accepts the automation contract maximum of 2,000 SMS characters', () => {
expect(validMessageContent('字'.repeat(2_000))).toBe(true);
expect(validMessageContent('字'.repeat(2_001))).toBe(false);
});
it('parses a bounded explicit message allowlist and never exposes pdu or excess fields', () => {
const messages = parseMessageList(
{
@@ -51,6 +58,43 @@ describe('InstanceMessageService', () => {
expect(JSON.stringify(messages)).not.toContain('secret');
});
it('accepts the current SimAdmin SMS shape without the removed transport field', () => {
expect(
parseMessageList(
{
status: 200,
headers: {},
body: JSON.stringify({
status: 'success',
data: {
messages: [
{
id: 42,
direction: 'incoming',
phone_number: '10086',
content: 'current payload',
timestamp: '2026-07-29 12:30:00',
status: 'received',
},
],
},
}),
},
10,
),
).toEqual([
{
id: '42',
direction: 'incoming',
phoneNumber: '10086',
content: 'current payload',
timestamp: '2026-07-29 12:30:00',
status: 'received',
transport: 'modem',
},
]);
});
it('uses exact owner, bounded query and matching optional session cookie', async () => {
const sessions = new InstanceSessionStore();
sessions.set('alpha', instance.origin, 'simadmin_session=opaque');
@@ -97,6 +141,88 @@ describe('InstanceMessageService', () => {
});
});
it('deletes a bounded batch with explicit IDs and reports per-instance failures', async () => {
expect(validMessageId('1')).toBe(true);
expect(validMessageId('9007199254740991')).toBe(true);
expect(validMessageId('9007199254740992')).toBe(false);
const request = vi.fn(async (value: unknown) => {
const body = value as { readonly smsBatchDelete?: { readonly ids: readonly number[] } };
const deleted = body.smsBatchDelete?.ids.length ?? 0;
return {
status: 200,
headers: {},
body: JSON.stringify({
status: 'success',
data: { deleted },
}),
};
});
const service = new InstanceMessageService({
instances: instances as never,
sessions: new InstanceSessionStore(),
request,
});
const result = await service.deleteMany([
{ instanceId: 'alpha', id: '1' },
{ instanceId: 'alpha', id: '2' },
{ instanceId: 'missing', id: '3' },
]);
expect(result).toEqual({
requested: 3,
deleted: 2,
failed: 1,
failures: [
{
instanceId: 'missing',
instanceName: '',
id: '3',
code: 'NOT_FOUND',
},
],
});
expect(request).toHaveBeenCalledWith({
url: 'http://192.168.1.10:8080/api/sms/batch-delete',
method: 'POST',
headers: {
accept: 'application/json',
'content-type': 'application/json',
},
smsBatchDelete: { ids: [1, 2] },
});
});
it('fails closed for invalid batch payloads and malformed upstream delete replies', async () => {
const request = vi.fn(async () => ({
status: 200,
headers: {},
body: '{"status":"success","data":{"deleted":2}}',
}));
const service = new InstanceMessageService({
instances: instances as never,
sessions: new InstanceSessionStore(),
request,
});
await expect(service.deleteMany([])).rejects.toMatchObject({
code: 'VALIDATION_FAILED',
});
await expect(
service.deleteMany([{ instanceId: 'alpha', id: 'not-a-number' }]),
).rejects.toMatchObject({ code: 'VALIDATION_FAILED' });
request.mockResolvedValueOnce({
status: 200,
headers: {},
body: '{"status":"success","data":{"deleted":-1}}',
});
await expect(service.deleteMany([{ instanceId: 'alpha', id: '1' }])).resolves.toMatchObject({
requested: 1,
deleted: 0,
failed: 1,
failures: [{ code: 'UPSTREAM_FAILED' }],
});
});
it('fails closed for missing owners, stale sessions and upstream status:error', async () => {
const sessions = new InstanceSessionStore();
sessions.set('alpha', 'http://192.168.1.99', 'simadmin_session=opaque');
@@ -24,6 +24,22 @@ export interface SendMessageInput {
readonly phoneNumber: string;
readonly content: string;
}
export interface DeleteSmsMessageRequest {
readonly instanceId: string;
readonly id: string;
}
export interface MessageDeleteFailure {
readonly instanceId: string;
readonly instanceName: string;
readonly id: string;
readonly code: MessageServiceErrorCode;
}
export interface MessageDeleteResult {
readonly requested: number;
readonly deleted: number;
readonly failed: number;
readonly failures: readonly MessageDeleteFailure[];
}
export type MessageServiceErrorCode =
| 'NOT_FOUND'
| 'VALIDATION_FAILED'
@@ -37,6 +53,7 @@ export class MessageServiceError extends Error {
}
const MAX_RESPONSE_BYTES = 262_144;
const MAX_BATCH_DELETE = 500;
const DIRECTIONS = new Set<MessageDirection>(['received', 'sent', 'incoming', 'outgoing']);
const record = (value: unknown): Record<string, unknown> | undefined =>
value !== null && typeof value === 'object' && !Array.isArray(value)
@@ -48,6 +65,10 @@ const bounded = (value: unknown, maximum: number): string | undefined =>
!/[\u0000-\u0008\u000b\u000c\u000e-\u001f\u007f]/u.test(String(value))
? String(value)
: undefined;
const safeCount = (value: unknown, maximum: number): number | undefined =>
typeof value === 'number' && Number.isSafeInteger(value) && value >= 0 && value <= maximum
? value
: undefined;
export function validPhoneNumber(value: string): boolean {
return value.length >= 3 && value.length <= 32 && /^\+?[0-9][0-9 ()-]*$/u.test(value);
@@ -55,12 +76,18 @@ export function validPhoneNumber(value: string): boolean {
export function validMessageContent(value: string): boolean {
return (
value.length >= 1 &&
value.length <= 1600 &&
Buffer.byteLength(value, 'utf8') <= 6400 &&
value.length <= 2000 &&
Buffer.byteLength(value, 'utf8') <= 8000 &&
!/[\u0000-\u0008\u000b\u000c\u000e-\u001f\u007f]/u.test(value)
);
}
export function validMessageId(value: string): boolean {
if (!/^\d{1,18}$/u.test(value)) return false;
const parsed = Number(value);
return Number.isSafeInteger(parsed) && parsed >= 0;
}
export function parseMessageList(
response: UpstreamResponse,
limit: number,
@@ -87,10 +114,12 @@ export function parseMessageList(
const value = record(item);
const id = bounded(value?.id, 128);
const phoneNumber = bounded(value?.phone_number, 32);
const content = bounded(value?.content, 1600);
const content = bounded(value?.content, 2000);
const timestamp = bounded(value?.timestamp, 64);
const status = bounded(value?.status, 32);
const transport = bounded(value?.transport, 32);
// Current SimAdmin SmsMessage has no transport field; older captures sometimes did.
// Keep the aggregate contract stable without dropping every current message.
const transport = bounded(value?.transport, 32) ?? 'modem';
const rawDirection = bounded(value?.direction, 16);
if (
!id ||
@@ -98,8 +127,7 @@ export function parseMessageList(
!validPhoneNumber(phoneNumber) ||
content === undefined ||
timestamp === undefined ||
status === undefined ||
transport === undefined
status === undefined
)
continue;
const direction: MessageDirection =
@@ -127,12 +155,37 @@ function parseSendSuccess(response: UpstreamResponse): void {
}
}
function parseBatchDeleteSuccess(response: UpstreamResponse, expected: number): number {
if (
response.status < 200 ||
response.status >= 300 ||
Buffer.byteLength(response.body, 'utf8') > 32_768
)
throw new MessageServiceError('UPSTREAM_FAILED');
let root: Record<string, unknown> | undefined;
try {
root = record(JSON.parse(response.body));
} catch {
throw new MessageServiceError('UPSTREAM_FAILED');
}
if (root?.status !== 'success' && root?.status !== 'ok')
throw new MessageServiceError('UPSTREAM_FAILED');
const deleted = safeCount(record(root.data)?.deleted, expected);
if (deleted === undefined) throw new MessageServiceError('UPSTREAM_FAILED');
return deleted;
}
export class InstanceMessageService {
constructor(
private readonly options: {
readonly instances: InstanceService;
readonly sessions: InstanceSessionStore;
readonly request: UpstreamSessionClientOptions['request'];
readonly ensureSession?: (
instanceId: string,
origin: string,
force?: boolean,
) => Promise<void>;
},
) {}
@@ -142,7 +195,14 @@ export class InstanceMessageService {
const session = this.options.sessions.sessionFor(instanceId);
if (session && session.origin !== instance.origin)
throw new MessageServiceError('SESSION_INVALID');
return { instance, session };
if (!session && this.options.ensureSession) {
try {
await this.options.ensureSession(instanceId, instance.origin);
} catch {
// No saved credential means this may be a passwordless instance; read anonymously.
}
}
return { instance, session: this.options.sessions.sessionFor(instanceId) };
}
async list(
@@ -163,11 +223,17 @@ export class InstanceMessageService {
throw new MessageServiceError('VALIDATION_FAILED');
const { instance, session } = await this.owner(instanceId);
const direction = query.direction ? `&direction=${query.direction}` : '';
const response = await this.options.request({
url: `${instance.origin}/api/sms/list?limit=${query.limit}&offset=${query.offset}${direction}`,
method: 'GET',
headers: { accept: 'application/json', ...(session ? { cookie: session.cookie } : {}) },
});
const request = (cookie?: string) =>
this.options.request({
url: `${instance.origin}/api/sms/list?limit=${query.limit}&offset=${query.offset}${direction}`,
method: 'GET',
headers: { accept: 'application/json', ...(cookie ? { cookie } : {}) },
});
let response = await request(session?.cookie);
if ([401, 403].includes(response.status) && this.options.ensureSession) {
await this.options.ensureSession(instanceId, instance.origin, true);
response = await request(this.options.sessions.sessionFor(instanceId)?.cookie);
}
return { messages: parseMessageList(response, query.limit) };
}
@@ -175,17 +241,97 @@ export class InstanceMessageService {
if (!validPhoneNumber(input.phoneNumber) || !validMessageContent(input.content))
throw new MessageServiceError('VALIDATION_FAILED');
const { instance, session } = await this.owner(instanceId);
const response = await this.options.request({
url: `${instance.origin}/api/sms/send`,
method: 'POST',
headers: {
accept: 'application/json',
'content-type': 'application/json',
...(session ? { cookie: session.cookie } : {}),
},
sms: { phoneNumber: input.phoneNumber, content: input.content },
});
const request = (cookie?: string) =>
this.options.request({
url: `${instance.origin}/api/sms/send`,
method: 'POST',
headers: {
accept: 'application/json',
'content-type': 'application/json',
...(cookie ? { cookie } : {}),
},
sms: { phoneNumber: input.phoneNumber, content: input.content },
});
let response = await request(session?.cookie);
if ([401, 403].includes(response.status) && this.options.ensureSession) {
await this.options.ensureSession(instanceId, instance.origin, true);
response = await request(this.options.sessions.sessionFor(instanceId)?.cookie);
}
parseSendSuccess(response);
return { sent: true };
}
async deleteMany(input: readonly DeleteSmsMessageRequest[]): Promise<MessageDeleteResult> {
if (!Array.isArray(input) || input.length < 1 || input.length > MAX_BATCH_DELETE)
throw new MessageServiceError('VALIDATION_FAILED');
const grouped = new Map<string, Array<{ readonly id: string; readonly numericId: number }>>();
const requested = input.length;
for (const item of input) {
const instanceId = typeof item?.instanceId === 'string' ? item.instanceId : '';
const id = typeof item?.id === 'string' ? item.id : '';
if (!instanceId || !validMessageId(id)) throw new MessageServiceError('VALIDATION_FAILED');
const numericId = Number(id);
const current = grouped.get(instanceId);
if (current === undefined) grouped.set(instanceId, [{ id, numericId }]);
else current.push({ id, numericId });
}
let deleted = 0;
const failures: MessageDeleteFailure[] = [];
for (const [instanceId, entries] of grouped) {
const ids = entries.map((entry) => entry.numericId);
let instanceName = '';
try {
const instance = await this.options.instances.get(instanceId);
if (!instance) throw new MessageServiceError('NOT_FOUND');
instanceName = instance.name;
const session = this.options.sessions.sessionFor(instanceId);
if (session && session.origin !== instance.origin)
throw new MessageServiceError('SESSION_INVALID');
if (!session && this.options.ensureSession) {
try {
await this.options.ensureSession(instanceId, instance.origin);
} catch {
// Passwordless instances may still accept anonymous batch deletion.
}
}
const request = (cookie?: string) =>
this.options.request({
url: `${instance.origin}/api/sms/batch-delete`,
method: 'POST',
headers: {
accept: 'application/json',
'content-type': 'application/json',
...(cookie ? { cookie } : {}),
},
smsBatchDelete: { ids },
});
let response = await request(this.options.sessions.sessionFor(instanceId)?.cookie);
if ([401, 403].includes(response.status) && this.options.ensureSession) {
await this.options.ensureSession(instanceId, instance.origin, true);
response = await request(this.options.sessions.sessionFor(instanceId)?.cookie);
}
deleted += parseBatchDeleteSuccess(response, entries.length);
} catch (error) {
const code =
error instanceof MessageServiceError ? error.code : ('UPSTREAM_FAILED' as const);
for (const entry of entries) {
failures.push(
Object.freeze({
instanceId,
instanceName,
id: entry.id,
code,
}),
);
}
}
}
return {
requested,
deleted,
failed: failures.length,
failures: Object.freeze(failures),
};
}
}
@@ -0,0 +1,190 @@
import Database from 'better-sqlite3';
import { afterEach, beforeEach, describe, expect, it } from 'vitest';
import { migrateDatabase } from '../../infrastructure/database/migrations.js';
import { MessageServiceError } from './instance-message-service.js';
import { SmsOutboxService } from './sms-outbox-service.js';
const START = new Date('2026-09-05T02:00:00.000Z');
interface Clock {
current: Date;
advance: (ms: number) => void;
}
function clock(): Clock {
const state: Clock = {
current: new Date(START),
advance: (ms: number) => {
state.current = new Date(state.current.getTime() + ms);
},
};
return state;
}
describe('SmsOutboxService', () => {
let db: Database.Database;
let time: Clock;
let offline: Set<string>;
let delivered: Array<{ readonly instanceId: string; readonly content: string }>;
let sendError: (() => Error) | undefined;
beforeEach(() => {
db = new Database(':memory:');
db.pragma('foreign_keys = ON');
migrateDatabase(db);
for (const [id, name] of [
['device-1', 'Modem A'],
['device-2', 'Modem B'],
]) {
db.prepare(
`INSERT INTO instances (id,name,base_url,auth_mode,enabled,config_revision,created_at,updated_at)
VALUES (?,?,?,'password',1,1,?,?)`,
).run(id, name, `http://${id}.invalid`, START.toISOString(), START.toISOString());
}
time = clock();
offline = new Set<string>();
delivered = [];
sendError = undefined;
});
afterEach(() => {
db.close();
});
function service(overrides: { readonly maxAttempts?: number } = {}): SmsOutboxService {
let sequence = 0;
return new SmsOutboxService(db, {
send: async (instanceId, input) => {
if (sendError) throw sendError();
delivered.push({ instanceId, content: input.content });
return { sent: true };
},
offlineInstances: () => offline,
now: () => time.current,
id: () => {
sequence += 1;
return `queue-${sequence}`;
},
maxAttempts: overrides.maxAttempts ?? 3,
backoffMs: 15_000,
});
}
it('delivers straight through while the device is reachable', async () => {
const result = await service().submit('device-1', {
phoneNumber: '13800138000',
content: '在线直发',
});
expect(result.status).toBe('sent');
expect(result.item).toBeNull();
expect(delivered).toEqual([{ instanceId: 'device-1', content: '在线直发' }]);
expect(service().list().total).toBe(0);
});
it('queues for an offline device and delivers once it returns', async () => {
offline.add('device-1');
const outbox = service();
const result = await outbox.submit('device-1', {
phoneNumber: '13800138000',
content: '离线排队',
});
expect(result.status).toBe('queued');
expect(result.item?.status).toBe('queued');
expect(result.item?.attempts).toBe(0);
expect(delivered).toEqual([]);
expect(await outbox.flush()).toMatchObject({ attempted: 1, delivered: 0, deferred: 1 });
expect(outbox.get(result.item?.id ?? '')?.status).toBe('queued');
offline.delete('device-1');
expect(await outbox.flush()).toMatchObject({ attempted: 1, delivered: 1, remaining: 0 });
expect(delivered).toEqual([{ instanceId: 'device-1', content: '离线排队' }]);
const settled = outbox.get(result.item?.id ?? '');
expect(settled?.status).toBe('sent');
expect(settled?.sentAt).toBe(time.current.toISOString());
});
it('falls back to the queue when a live send fails', async () => {
sendError = () => new MessageServiceError('UPSTREAM_FAILED');
const outbox = service();
const result = await outbox.submit('device-1', {
phoneNumber: '13800138000',
content: '节点抖动',
});
expect(result.status).toBe('queued');
expect(result.item?.lastError).toBe('UPSTREAM_FAILED');
});
it('backs off between attempts and gives up at the limit', async () => {
sendError = () => new MessageServiceError('UPSTREAM_FAILED');
const outbox = service();
await outbox.submit('device-1', { phoneNumber: '13800138000', content: '重试退避' });
expect(await outbox.flush()).toMatchObject({ attempted: 1, deferred: 1 });
expect(outbox.list().items[0]).toMatchObject({ attempts: 1, status: 'queued' });
// The next attempt is parked behind the backoff window.
expect(await outbox.flush()).toMatchObject({ attempted: 0 });
time.advance(15_000);
expect(await outbox.flush()).toMatchObject({ attempted: 1, deferred: 1 });
expect(outbox.list().items[0]).toMatchObject({ attempts: 2, status: 'queued' });
time.advance(30_000);
expect(await outbox.flush()).toMatchObject({ attempted: 1, failed: 1, remaining: 0 });
const exhausted = outbox.list({ status: 'all' }).items[0];
expect(exhausted).toMatchObject({ status: 'failed', attempts: 3 });
const revived = outbox.retry(exhausted?.id ?? '');
expect(revived.status).toBe('queued');
expect(revived.maxAttempts).toBeGreaterThanOrEqual(revived.attempts + 1);
});
it('cancels pending work, prunes history, and reconciles an interrupted process', async () => {
offline.add('device-1');
const outbox = service();
const first = await outbox.submit('device-1', {
phoneNumber: '13800138000',
content: '待取消',
});
const second = await outbox.submit('device-1', {
phoneNumber: '13800138001',
content: '待删除',
});
expect(outbox.summary()).toMatchObject({ queued: 2 });
expect(outbox.cancel(first.item?.id ?? '').status).toBe('cancelled');
expect(() => outbox.cancel(first.item?.id ?? '')).toThrow(MessageServiceError);
outbox.remove(second.item?.id ?? '');
expect(outbox.list({ status: 'all' }).total).toBe(1);
time.advance(3_600_000);
expect(outbox.prune(0)).toBe(1);
expect(outbox.list({ status: 'all' }).total).toBe(0);
db.prepare(
`INSERT INTO sms_outbox (id,instance_id,phone_number,content,status,attempts,max_attempts,
available_at,created_at,updated_at)
VALUES ('stale','device-1','13800138000','中断','sending',1,3,?,?,?)`,
).run(START.toISOString(), START.toISOString(), START.toISOString());
expect(outbox.reconcileInterrupted()).toBe(1);
expect(outbox.get('stale')).toMatchObject({ status: 'queued', lastError: 'INTERRUPTED' });
});
it('rejects malformed submissions and unknown queue entries', async () => {
const outbox = service();
await expect(outbox.submit('', { phoneNumber: '13800138000', content: 'x' })).rejects.toThrow(
MessageServiceError,
);
await expect(
outbox.submit('device-1', { phoneNumber: 'not-a-number', content: 'x' }),
).rejects.toThrow(MessageServiceError);
await expect(
outbox.submit('device-1', { phoneNumber: '13800138000', content: ' ' }),
).rejects.toThrow(MessageServiceError);
expect(outbox.get('missing')).toBeUndefined();
expect(() => outbox.cancel('missing')).toThrow(MessageServiceError);
expect(outbox.list({ status: 'all', limit: 10, offset: 0 }).items).toEqual([]);
});
});
@@ -0,0 +1,483 @@
import { randomUUID } from 'node:crypto';
import type Database from 'better-sqlite3';
import { MessageServiceError } from './instance-message-service.js';
export type SmsOutboxStatus = 'queued' | 'sending' | 'sent' | 'failed' | 'cancelled';
export interface SmsOutboxItem {
readonly id: string;
readonly instanceId: string;
readonly phoneNumber: string;
readonly content: string;
readonly status: SmsOutboxStatus;
readonly attempts: number;
readonly maxAttempts: number;
readonly lastError: string | null;
readonly availableAt: string;
readonly sentAt: string | null;
readonly createdAt: string;
readonly updatedAt: string;
}
export interface SmsOutboxPage {
readonly items: readonly SmsOutboxItem[];
readonly total: number;
}
export interface SmsOutboxSummary {
readonly queued: number;
readonly sending: number;
readonly failed: number;
readonly sent: number;
}
export interface SmsOutboxFlushResult {
readonly attempted: number;
readonly delivered: number;
readonly deferred: number;
readonly failed: number;
readonly remaining: number;
}
export interface SmsOutboxSubmitResult {
readonly status: 'sent' | 'queued';
readonly item: SmsOutboxItem | null;
}
export interface SmsOutboxServiceOptions {
readonly send: (
instanceId: string,
input: { readonly phoneNumber: string; readonly content: string },
) => Promise<unknown>;
/**
* Devices the heartbeat positively marked unreachable. Never-probed devices stay optimistic so
* a send still goes straight to the node; only known-dead devices queue.
*/
readonly offlineInstances: () => ReadonlySet<string>;
readonly now?: () => Date;
readonly id?: () => string;
readonly maxAttempts?: number;
readonly backoffMs?: number;
readonly maximumBackoffMs?: number;
readonly pageSize?: number;
}
const DEFAULT_MAX_ATTEMPTS = 24;
const DEFAULT_BACKOFF_MS = 15_000;
const DEFAULT_MAXIMUM_BACKOFF_MS = 600_000;
const DEFAULT_PAGE_SIZE = 50;
const MAX_PAGE_SIZE = 200;
const MAX_PHONE_LENGTH = 32;
const MAX_CONTENT_LENGTH = 2_000;
const MAX_ERROR_LENGTH = 200;
const OPEN_STATUSES: readonly SmsOutboxStatus[] = ['queued', 'sending'];
interface OutboxRow {
readonly id: string;
readonly instance_id: string;
readonly phone_number: string;
readonly content: string;
readonly status: string;
readonly attempts: number;
readonly max_attempts: number;
readonly last_error: string | null;
readonly available_at: string;
readonly sent_at: string | null;
readonly created_at: string;
readonly updated_at: string;
}
function truncate(value: string, maximum: number): string {
return value.length <= maximum ? value : `${value.slice(0, maximum - 1)}`;
}
function describeError(error: unknown): string {
if (error instanceof MessageServiceError) return error.code;
if (error instanceof Error && error.message.trim() !== '')
return truncate(error.message.replace(/\s+/gu, ' '), MAX_ERROR_LENGTH);
return 'DELIVERY_FAILED';
}
function validPhoneNumber(value: unknown): value is string {
return (
typeof value === 'string' &&
value.trim().length >= 3 &&
value.trim().length <= MAX_PHONE_LENGTH &&
/^[+0-9][0-9 ()-]*$/u.test(value.trim())
);
}
function validContent(value: unknown): value is string {
return (
typeof value === 'string' &&
value.trim().length > 0 &&
value.trim().length <= MAX_CONTENT_LENGTH
);
}
/**
* Durable outbound SMS queue. The Hub keeps accepting sends while a device is offline and hands
* them over once the device returns, so this service owns the wait rather than failing the request.
*/
export class SmsOutboxService {
readonly #db: Database.Database;
readonly #options: SmsOutboxServiceOptions;
readonly #now: () => Date;
readonly #id: () => string;
readonly #maxAttempts: number;
readonly #backoffMs: number;
readonly #maximumBackoffMs: number;
readonly #pageSize: number;
#flushing: Promise<SmsOutboxFlushResult> | undefined;
constructor(db: Database.Database, options: SmsOutboxServiceOptions) {
const maxAttempts = options.maxAttempts ?? DEFAULT_MAX_ATTEMPTS;
if (!Number.isSafeInteger(maxAttempts) || maxAttempts < 1 || maxAttempts > 1_000)
throw new RangeError('maxAttempts must be between 1 and 1000');
const backoffMs = options.backoffMs ?? DEFAULT_BACKOFF_MS;
if (!Number.isSafeInteger(backoffMs) || backoffMs < 1_000 || backoffMs > 600_000)
throw new RangeError('backoffMs must be between 1000 and 600000');
const maximumBackoffMs = options.maximumBackoffMs ?? DEFAULT_MAXIMUM_BACKOFF_MS;
if (!Number.isSafeInteger(maximumBackoffMs) || maximumBackoffMs < backoffMs)
throw new RangeError('maximumBackoffMs must be at least backoffMs');
const pageSize = options.pageSize ?? DEFAULT_PAGE_SIZE;
if (!Number.isSafeInteger(pageSize) || pageSize < 1 || pageSize > MAX_PAGE_SIZE)
throw new RangeError(`pageSize must be between 1 and ${MAX_PAGE_SIZE}`);
this.#db = db;
this.#options = options;
this.#now = options.now ?? (() => new Date());
this.#id = options.id ?? (() => randomUUID());
this.#maxAttempts = maxAttempts;
this.#backoffMs = backoffMs;
this.#maximumBackoffMs = maximumBackoffMs;
this.#pageSize = pageSize;
}
/**
* Delivers immediately when the device is reachable and otherwise parks the message in the
* queue. A failed live attempt also falls back to the queue so a flapping device never loses
* an operator send.
*/
async submit(
instanceId: string,
input: { readonly phoneNumber: unknown; readonly content: unknown },
): Promise<SmsOutboxSubmitResult> {
if (typeof instanceId !== 'string' || instanceId.trim() === '')
throw new MessageServiceError('VALIDATION_FAILED');
if (!validPhoneNumber(input.phoneNumber) || !validContent(input.content))
throw new MessageServiceError('VALIDATION_FAILED');
const phoneNumber = input.phoneNumber.trim();
const content = input.content.trim();
if (!this.#offline().has(instanceId)) {
try {
await this.#options.send(instanceId, { phoneNumber, content });
return { status: 'sent', item: null };
} catch (error) {
if (error instanceof MessageServiceError && error.code === 'VALIDATION_FAILED') throw error;
const item = this.#enqueue(instanceId, phoneNumber, content, describeError(error));
return { status: 'queued', item };
}
}
const item = this.#enqueue(instanceId, phoneNumber, content, null);
return { status: 'queued', item };
}
list(
query: {
readonly status?: SmsOutboxStatus | 'open' | 'all';
readonly instanceId?: string;
readonly limit?: number;
readonly offset?: number;
} = {},
): SmsOutboxPage {
const limit = Math.min(Math.max(Math.trunc(query.limit ?? this.#pageSize), 1), MAX_PAGE_SIZE);
const offset = Math.max(Math.trunc(query.offset ?? 0), 0);
const conditions: string[] = [];
const parameters: unknown[] = [];
const status = query.status ?? 'open';
if (status === 'open') {
conditions.push(`status IN (${OPEN_STATUSES.map(() => '?').join(',')})`);
parameters.push(...OPEN_STATUSES);
} else if (status !== 'all') {
conditions.push('status = ?');
parameters.push(status);
}
if (typeof query.instanceId === 'string' && query.instanceId.trim() !== '') {
conditions.push('instance_id = ?');
parameters.push(query.instanceId.trim());
}
const where = conditions.length > 0 ? ` WHERE ${conditions.join(' AND ')}` : '';
const total = (this.#db
.prepare(`SELECT COUNT(*) AS count FROM sms_outbox${where}`)
.get(...parameters) ?? { count: 0 }) as { count: number };
const rows = this.#db
.prepare(
`SELECT * FROM sms_outbox${where}
ORDER BY CASE status WHEN 'sending' THEN 0 WHEN 'queued' THEN 1 ELSE 2 END,
available_at, created_at
LIMIT ? OFFSET ?`,
)
.all(...parameters, limit, offset) as readonly OutboxRow[];
return Object.freeze({
items: Object.freeze(rows.map((row) => this.#item(row))),
total: Number.isSafeInteger(total.count) ? total.count : 0,
});
}
summary(): SmsOutboxSummary {
const rows = this.#db
.prepare('SELECT status, COUNT(*) AS count FROM sms_outbox GROUP BY status')
.all() as readonly { readonly status: string; readonly count: number }[];
const counts = new Map(rows.map((row) => [row.status, Number(row.count) || 0]));
return Object.freeze({
queued: counts.get('queued') ?? 0,
sending: counts.get('sending') ?? 0,
failed: counts.get('failed') ?? 0,
sent: counts.get('sent') ?? 0,
});
}
get(id: string): SmsOutboxItem | undefined {
const row = this.#db.prepare('SELECT * FROM sms_outbox WHERE id=?').get(id) as
| OutboxRow
| undefined;
return row ? this.#item(row) : undefined;
}
cancel(id: string): SmsOutboxItem {
const current = this.#require(id);
if (!OPEN_STATUSES.includes(current.status)) throw new MessageServiceError('VALIDATION_FAILED');
const timestamp = this.#now().toISOString();
this.#db
.prepare("UPDATE sms_outbox SET status='cancelled',last_error=?,updated_at=? WHERE id=?")
.run('CANCELLED_BY_OPERATOR', timestamp, id);
return this.#require(id);
}
/** Re-arms a cancelled or exhausted message so the next flush can try it again. */
retry(id: string): SmsOutboxItem {
const current = this.#require(id);
if (current.status === 'sending' || current.status === 'sent')
throw new MessageServiceError('VALIDATION_FAILED');
const timestamp = this.#now().toISOString();
this.#db
.prepare(
`UPDATE sms_outbox
SET status='queued',available_at=?,last_error=NULL,max_attempts=MAX(max_attempts,attempts+1),
updated_at=?
WHERE id=?`,
)
.run(timestamp, timestamp, id);
return this.#require(id);
}
remove(id: string): void {
this.#require(id);
this.#db.prepare('DELETE FROM sms_outbox WHERE id=?').run(id);
}
/** Drops delivered and abandoned entries; queued work is always preserved. */
prune(olderThanMs: number): number {
if (!Number.isSafeInteger(olderThanMs) || olderThanMs < 0)
throw new RangeError('olderThanMs must be a non-negative integer');
const cutoff = new Date(this.#now().getTime() - olderThanMs).toISOString();
const info = this.#db
.prepare(
`DELETE FROM sms_outbox
WHERE status IN ('sent','failed','cancelled') AND updated_at <= ?`,
)
.run(cutoff);
return Number(info.changes ?? 0);
}
/** Startup sweep: nothing can be mid-flight before the first request of a fresh process. */
reconcileInterrupted(): number {
const timestamp = this.#now().toISOString();
const info = this.#db
.prepare(
`UPDATE sms_outbox
SET status='queued',last_error=?,available_at=?,updated_at=? WHERE status='sending'`,
)
.run('INTERRUPTED', timestamp, timestamp);
return Number(info.changes ?? 0);
}
/** Concurrent callers share one pass, the same way the heartbeat collapses overlapping beats. */
flush(): Promise<SmsOutboxFlushResult> {
if (this.#flushing) return this.#flushing;
const pending = this.#flush().finally(() => {
if (this.#flushing === pending) this.#flushing = undefined;
});
this.#flushing = pending;
return pending;
}
async #flush(): Promise<SmsOutboxFlushResult> {
const claimed = this.#claim();
let delivered = 0;
let deferred = 0;
let failed = 0;
const offline = this.#offline();
for (const item of claimed) {
if (offline.has(item.instanceId)) {
// Waiting for the device is not a failed attempt, so keep the item due on the next tick.
this.#defer(item.id, 'DEVICE_OFFLINE');
deferred += 1;
continue;
}
try {
await this.#options.send(item.instanceId, {
phoneNumber: item.phoneNumber,
content: item.content,
});
this.#settle(item.id, 'sent', null);
delivered += 1;
} catch (error) {
const reason = describeError(error);
if (error instanceof MessageServiceError && error.code === 'VALIDATION_FAILED') {
this.#settle(item.id, 'failed', reason);
failed += 1;
continue;
}
if (this.#release(item.id, reason) === 'failed') failed += 1;
else deferred += 1;
}
}
const summary = this.summary();
return Object.freeze({
attempted: claimed.length,
delivered,
deferred,
failed,
remaining: summary.queued + summary.sending,
});
}
#offline(): ReadonlySet<string> {
try {
return this.#options.offlineInstances();
} catch {
// A broken reachability read must not lose queued work; treat every device as reachable.
return new Set();
}
}
#claim(): readonly SmsOutboxItem[] {
const now = this.#now();
const timestamp = now.toISOString();
return this.#db.transaction((): readonly SmsOutboxItem[] => {
const rows = this.#db
.prepare(
`SELECT * FROM sms_outbox
WHERE status='queued' AND available_at <= ?
ORDER BY available_at, created_at LIMIT ?`,
)
.all(timestamp, this.#pageSize) as readonly OutboxRow[];
const claim = this.#db.prepare(
`UPDATE sms_outbox
SET status='sending',attempts=attempts+1,last_error=NULL,updated_at=?
WHERE id=? AND status='queued'`,
);
const claimed: SmsOutboxItem[] = [];
for (const row of rows) {
const info = claim.run(timestamp, row.id);
if (Number(info.changes ?? 0) > 0) claimed.push(this.#item(row, 'sending'));
}
return claimed;
})();
}
/** Puts a claimed item back in line; returns 'failed' once the attempt budget is spent. */
#release(id: string, reason: string): 'queued' | 'failed' {
const current = this.#require(id);
const backoff = Math.min(
this.#backoffMs * 2 ** Math.max(current.attempts - 1, 0),
this.#maximumBackoffMs,
);
const timestamp = this.#now().toISOString();
const availableAt = new Date(this.#now().getTime() + backoff).toISOString();
const exhausted = current.attempts >= current.maxAttempts;
this.#db
.prepare(
`UPDATE sms_outbox
SET status=?,last_error=?,available_at=?,updated_at=? WHERE id=?`,
)
.run(exhausted ? 'failed' : 'queued', reason, availableAt, timestamp, id);
return exhausted ? 'failed' : 'queued';
}
#defer(id: string, reason: string): void {
const timestamp = this.#now().toISOString();
this.#db
.prepare(
"UPDATE sms_outbox SET status='queued',last_error=?,available_at=?,updated_at=? WHERE id=?",
)
.run(reason, timestamp, timestamp, id);
}
#settle(id: string, status: 'sent' | 'failed', reason: string | null): void {
const timestamp = this.#now().toISOString();
this.#db
.prepare(
`UPDATE sms_outbox
SET status=?,last_error=?,sent_at=?,updated_at=? WHERE id=?`,
)
.run(status, reason, status === 'sent' ? timestamp : null, timestamp, id);
}
#enqueue(instanceId: string, phoneNumber: string, content: string, reason: string | null) {
const timestamp = this.#now().toISOString();
const id = this.#id();
this.#db
.prepare(
`INSERT INTO sms_outbox (id,instance_id,phone_number,content,status,attempts,max_attempts,
last_error,available_at,created_at,updated_at)
VALUES (?,?,?,?,'queued',0,?,?,?,?,?)`,
)
.run(
id,
instanceId,
phoneNumber,
content,
this.#maxAttempts,
reason,
timestamp,
timestamp,
timestamp,
);
return this.#require(id);
}
#require(id: string): SmsOutboxItem {
const item = this.get(id);
if (!item) throw new MessageServiceError('NOT_FOUND');
return item;
}
#item(row: OutboxRow, status?: SmsOutboxStatus): SmsOutboxItem {
return Object.freeze({
id: String(row.id),
instanceId: String(row.instance_id),
phoneNumber: String(row.phone_number),
content: String(row.content),
status: status ?? this.#status(row.status),
attempts: Number.isSafeInteger(row.attempts) ? row.attempts : Number(row.attempts) || 0,
maxAttempts: Number.isSafeInteger(row.max_attempts)
? row.max_attempts
: Number(row.max_attempts) || this.#maxAttempts,
lastError: typeof row.last_error === 'string' ? row.last_error : null,
availableAt: String(row.available_at),
sentAt: typeof row.sent_at === 'string' ? row.sent_at : null,
createdAt: String(row.created_at),
updatedAt: String(row.updated_at),
});
}
#status(value: unknown): SmsOutboxStatus {
return value === 'sending' || value === 'sent' || value === 'failed' || value === 'cancelled'
? value
: 'queued';
}
}
@@ -0,0 +1,489 @@
import Database from 'better-sqlite3';
import { afterEach, describe, expect, it } from 'vitest';
import { migrateDatabase } from '../../infrastructure/database/migrations.js';
import type { SecretStore } from '../../infrastructure/secrets/secret-store.js';
import {
CentralNotificationService,
type NotificationDeliverRequest,
} from './central-notification-service.js';
class MemorySecrets implements SecretStore {
readonly provider = 'macos-keychain';
readonly values = new Map<string, string>();
async set(key: { instanceId: string; purpose: string; slot?: string }, value: string) {
const reference = `memory://notification/${key.instanceId}/${key.purpose}`;
this.values.set(reference, value);
return reference;
}
async get(reference: string) {
return this.values.get(reference);
}
async delete(reference: string) {
return this.values.delete(reference);
}
}
const directories: Database.Database[] = [];
afterEach(() => {
for (const db of directories.splice(0)) db.close();
});
function fixture(clock?: () => Date) {
const db = new Database(':memory:');
db.pragma('foreign_keys = ON');
migrateDatabase(db);
directories.push(db);
const store = new MemorySecrets();
let sequence = 0;
const deliveries: NotificationDeliverRequest[] = [];
const service = new CentralNotificationService(db, {
store,
now: clock ?? (() => new Date('2026-09-03T08:00:00.000Z')),
id: () => `id-${++sequence}`,
deliver: async (request) => {
deliveries.push(request);
const endpoint = String(request.config['url'] ?? request.config['server_url'] ?? '');
if (endpoint.includes('unavailable')) throw new Error('channel unreachable');
return { ok: true };
},
});
return { db, service, store, deliveries };
}
const bark = {
name: 'Bark',
type: 'bark' as const,
enabled: true,
config: { server_url: 'https://api.day.app', group: 'Island', device_key: 'device-key' },
};
describe('CentralNotificationService', () => {
it('stores channel credentials in the secret store and returns redacted channels', async () => {
const { db, service, store } = fixture();
const channel = await service.createChannel(bark);
expect(channel).toMatchObject({
name: 'Bark',
type: 'bark',
enabled: true,
config: { server_url: 'https://api.day.app', group: 'Island' },
hasSecret: true,
secretFields: ['device_key'],
});
expect(JSON.stringify(channel)).not.toContain('device-key');
expect(store.values.get(channel.secretReference ?? '')).toBe(
JSON.stringify({ device_key: 'device-key' }),
);
expect(
db
.prepare(
'SELECT config_json, secret_reference, secret_fields FROM notification_channels WHERE id = ?',
)
.get(channel.id),
).toEqual({
config_json: JSON.stringify({
server_url: 'https://api.day.app',
group: 'Island',
sound: '',
level: '',
icon: '',
auto_copy: true,
save_history: true,
}),
secret_reference: channel.secretReference,
secret_fields: 'device_key',
});
});
it('creates, matches, and updates device-scoped rules without exposing credentials', async () => {
const { service } = fixture();
const channel = await service.createChannel(bark);
const rule = await service.createRule({
name: '短信转发',
eventType: 'sms',
enabled: true,
condition: { field: 'content', mode: 'contains', value: '验证码' },
scope: { mode: 'tags', tags: ['lab', 'east'], match: 'any' },
channelIds: [channel.id],
templates: { title: '来自 {{sender}}', body: '{{content}}' },
});
expect(rule).toMatchObject({
name: '短信转发',
eventType: 'sms',
channels: [{ id: channel.id, name: 'Bark', enabled: true }],
});
expect(
service.matchRules('sms', {
instanceId: 'instance-1',
instanceTags: ['east'],
fields: { content: '您的验证码是 1234' },
}),
).toEqual([rule]);
const updated = await service.updateRule(rule.id, {
name: '验证码转发',
scope: { mode: 'devices', instanceIds: ['instance-2'] },
});
expect(updated.name).toBe('验证码转发');
expect(updated.scope).toEqual({ mode: 'devices', instanceIds: ['instance-2'] });
});
it('matches rules for any selected device group', async () => {
const { service } = fixture();
const channel = await service.createChannel(bark);
const rule = await service.createRule({
name: '外场转发',
eventType: 'system',
enabled: true,
condition: { field: 'status', mode: 'all' },
scope: { mode: 'groups', groups: ['group-field', 'group-lab'] },
channelIds: [channel.id],
templates: { title: '{{title}}', body: '{{content}}' },
});
expect(rule.scope).toEqual({ mode: 'groups', groups: ['group-field', 'group-lab'] });
expect(
service.matchRules('system', {
instanceId: 'instance-1',
instanceGroupIds: ['group-field'],
fields: { status: 'offline' },
}),
).toEqual([rule]);
expect(
service.matchRules('system', {
instanceId: 'instance-2',
instanceGroupIds: ['group-office'],
fields: { status: 'offline' },
}),
).toEqual([]);
});
it('tests an enabled channel, records delivery, and never logs the secret', async () => {
const { db, service, deliveries } = fixture();
const channel = await service.createChannel(bark);
const result = await service.testChannel(channel.id, {
title: '测试通知',
body: '融合控制台通知链路正常',
});
expect(result).toMatchObject({ ok: true, status: 'success' });
expect(deliveries).toEqual([
{
channelId: channel.id,
type: 'bark',
config: {
server_url: 'https://api.day.app',
group: 'Island',
sound: '',
level: '',
icon: '',
auto_copy: true,
save_history: true,
device_key: 'device-key',
},
eventType: 'test',
occurredAt: '2026-09-03T08:00:00.000Z',
title: '测试通知',
body: '融合控制台通知链路正常',
},
]);
const delivery = db.prepare('SELECT * FROM notification_deliveries').get() as Record<
string,
unknown
>;
expect(delivery).toMatchObject({
event_type: 'test',
status: 'success',
channel_id: channel.id,
});
expect(JSON.stringify(delivery)).not.toContain('device-key');
expect(service.listLogs().items[0]).toMatchObject({
eventType: 'test',
status: 'success',
channelName: 'Bark',
});
});
it('records failed channel tests with a safe delivery detail', async () => {
const { service } = fixture();
const channel = await service.createChannel({
...bark,
config: { ...bark.config, server_url: 'https://unavailable.invalid' },
});
await expect(
service.testChannel(channel.id, { title: '测试通知', body: 'hello' }),
).resolves.toMatchObject({ ok: false, status: 'failed', detail: 'channel unreachable' });
expect(service.listLogs().items[0]).toMatchObject({
status: 'failed',
detail: 'channel unreachable',
});
});
describe('delivery log retention', () => {
function insertLog(
db: Database.Database,
channelId: string,
id: string,
createdAt: string,
status: 'success' | 'failed' = 'success',
eventType = 'sms',
): void {
db.prepare(
`INSERT INTO notification_deliveries
(id,rule_id,channel_id,instance_id,event_type,status,detail,created_at,sent_at)
VALUES (?,NULL,?,NULL,?,?,NULL,?,?)`,
).run(id, channelId, eventType, status, createdAt, createdAt);
}
it('keeps the Hub retention defaults until they are stored', () => {
const { db, service } = fixture();
expect(service.logCleanup()).toEqual({
retentionDaysEnabled: true,
retentionDays: 180,
maxEntriesEnabled: false,
maxEntries: 10_000,
});
service.updateLogCleanup({ retentionDays: 30, maxEntriesEnabled: true, maxEntries: 500 });
expect(service.logCleanup()).toEqual({
retentionDaysEnabled: true,
retentionDays: 30,
maxEntriesEnabled: true,
maxEntries: 500,
});
const stored = db
.prepare('SELECT value_json FROM app_settings WHERE key = ?')
.get('notifications.logs.cleanup') as { value_json: string };
expect(JSON.parse(stored.value_json)).toMatchObject({ retentionDays: 30, maxEntries: 500 });
});
it('rejects out-of-range and unknown retention fields', () => {
const { service } = fixture();
expect(() => service.updateLogCleanup({ retentionDays: 0 })).toThrow();
expect(() => service.updateLogCleanup({ maxEntries: 0 })).toThrow();
expect(() => service.updateLogCleanup({ retentionDays: 30, extra: true })).toThrow();
expect(() => service.updateLogCleanup('180')).toThrow();
});
it('prunes by age and then by the entry ceiling', async () => {
const { db, service } = fixture();
const channel = await service.createChannel(bark);
insertLog(db, channel.id, 'old', '2020-01-01T00:00:00.000Z');
insertLog(db, channel.id, 'recent', '2026-09-03T07:00:00.000Z');
insertLog(db, channel.id, 'newest', '2026-09-03T07:30:00.000Z');
service.updateLogCleanup({ retentionDays: 30, maxEntriesEnabled: false });
expect(service.pruneLogs()).toBe(1);
expect(service.listLogs().items.map((item) => item.id)).toEqual(['newest', 'recent']);
service.updateLogCleanup({
retentionDaysEnabled: false,
maxEntriesEnabled: true,
maxEntries: 1,
});
expect(service.pruneLogs()).toBe(1);
expect(service.listLogs().items.map((item) => item.id)).toEqual(['newest']);
});
it('filters and clears logs by status, event type and date range', async () => {
const { db, service } = fixture();
const channel = await service.createChannel(bark);
insertLog(db, channel.id, 'a', '2026-09-01T00:00:00.000Z', 'success', 'sms');
insertLog(db, channel.id, 'b', '2026-09-02T00:00:00.000Z', 'failed', 'sms');
insertLog(db, channel.id, 'c', '2026-09-03T00:00:00.000Z', 'failed', 'system');
expect(service.listLogs(1, 50, { status: 'failed' }).items.map((item) => item.id)).toEqual([
'c',
'b',
]);
expect(service.listLogs(1, 50, { eventType: 'system' }).page.total).toBe(1);
expect(
service
.listLogs(1, 50, { from: '2026-09-02T00:00:00.000Z', to: '2026-09-02T12:00:00.000Z' })
.items.map((item) => item.id),
).toEqual(['b']);
expect(service.clearLogs({ status: 'failed', eventType: 'sms' })).toBe(1);
expect(service.listLogs().items.map((item) => item.id)).toEqual(['c', 'a']);
expect(service.clearLogs('2026-09-01T12:00:00.000Z')).toBe(1);
expect(service.listLogs().page.total).toBe(1);
expect(() => service.clearLogs({ before: 'not-a-date' })).toThrow();
});
});
describe('rate limit and quiet hours', () => {
const sms = { fields: { content: '您的验证码是 1234' } };
async function rule(
service: CentralNotificationService,
channelId: string,
suppression: Record<string, unknown>,
) {
return service.createRule({
name: '抑制规则',
eventType: 'sms',
enabled: true,
condition: { field: 'content', mode: 'all' },
scope: { mode: 'all' },
channelIds: [channelId],
templates: { title: '标题', body: '{{content}}' },
...suppression,
});
}
it('stores the Hub defaults when a rule omits both suppression blocks', async () => {
const { service } = fixture();
const channel = await service.createChannel(bark);
const created = await rule(service, channel.id, {});
expect(created.rateLimit).toEqual({ enabled: false, maxMessages: 20, windowSeconds: 60 });
expect(created.quietHours).toEqual([]);
});
it('drops events inside a quiet window and records one suppressed log', async () => {
// 08:30 Shanghai time sits inside 08:00-09:00.
const { service } = fixture(() => new Date('2026-09-03T00:30:00.000Z'));
const channel = await service.createChannel(bark);
const created = await rule(service, channel.id, {
quietHours: [{ start: '08:00', end: '09:00' }],
});
expect(await service.enqueueEvent('sms', sms)).toEqual([]);
const log = service.listLogs().items[0];
expect(log).toMatchObject({
status: 'quiet_hours',
eventType: 'sms',
ruleId: created.id,
channelName: '—',
detail: '免打扰时段:08:00-09:00',
});
expect(log?.channelId).toBeUndefined();
expect(service.logSummary().suppressed).toBe(1);
expect(service.listLogs(1, 50, { status: 'quiet_hours' }).page.total).toBe(1);
expect(service.listLogs(1, 50, { status: 'success' }).page.total).toBe(0);
});
it('treats a window that ends before it starts as crossing midnight', async () => {
let instant = new Date('2026-09-02T23:30:00.000Z');
const { service } = fixture(() => instant);
const channel = await service.createChannel(bark);
await rule(service, channel.id, { quietHours: [{ start: '22:00', end: '08:00' }] });
// 07:30 Shanghai, inside the window.
expect(await service.enqueueEvent('sms', sms)).toEqual([]);
// 09:00 Shanghai, outside the window.
instant = new Date('2026-09-03T01:00:00.000Z');
expect(await service.enqueueEvent('sms', sms)).toHaveLength(1);
});
it('enforces the rate limit inside the window and releases it afterwards', async () => {
let instant = new Date('2026-09-03T00:00:00.000Z');
const { service } = fixture(() => instant);
const channel = await service.createChannel(bark);
await rule(service, channel.id, {
rateLimit: { enabled: true, maxMessages: 2, windowSeconds: 60 },
});
expect(await service.enqueueEvent('sms', sms)).toHaveLength(1);
expect(await service.enqueueEvent('sms', sms)).toHaveLength(1);
expect(await service.enqueueEvent('sms', sms)).toEqual([]);
expect(service.logSummary()).toMatchObject({ suppressed: 1, total: 1 });
instant = new Date('2026-09-03T00:01:01.000Z');
expect(await service.enqueueEvent('sms', sms)).toHaveLength(1);
});
it('prefers the quiet window over the rate limit', async () => {
const { service } = fixture(() => new Date('2026-09-03T00:30:00.000Z'));
const channel = await service.createChannel(bark);
await rule(service, channel.id, {
rateLimit: { enabled: true, maxMessages: 5, windowSeconds: 60 },
quietHours: [{ start: '08:00', end: '09:00' }],
});
expect(await service.enqueueEvent('sms', sms)).toEqual([]);
expect(service.listLogs().items[0]?.status).toBe('quiet_hours');
});
it('leaves suppression settings untouched when a patch omits them', async () => {
const { service } = fixture();
const channel = await service.createChannel(bark);
const created = await rule(service, channel.id, {
rateLimit: { enabled: true, maxMessages: 3, windowSeconds: 120 },
quietHours: [{ start: '23:00', end: '06:00' }],
});
const patched = await service.updateRule(created.id, { name: '改名后的规则' });
expect(patched.name).toBe('改名后的规则');
expect(patched.rateLimit).toEqual({ enabled: true, maxMessages: 3, windowSeconds: 120 });
expect(patched.quietHours).toEqual([{ start: '23:00', end: '06:00' }]);
const disabled = await service.updateRule(created.id, {
rateLimit: { enabled: false, maxMessages: 3, windowSeconds: 120 },
quietHours: [],
});
expect(disabled.rateLimit.enabled).toBe(false);
expect(disabled.quietHours).toEqual([]);
});
it('rejects malformed suppression settings', async () => {
const { service } = fixture();
const channel = await service.createChannel(bark);
const base = {
name: '抑制规则',
eventType: 'sms',
enabled: true,
condition: { field: 'content', mode: 'all' },
scope: { mode: 'all' },
channelIds: [channel.id],
templates: { title: '标题', body: '正文' },
};
await expect(
service.createRule({
...base,
rateLimit: { enabled: true, maxMessages: 0, windowSeconds: 60 },
}),
).rejects.toThrow();
await expect(
service.createRule({
...base,
rateLimit: { enabled: true, maxMessages: 20, windowSeconds: 86_401 },
}),
).rejects.toThrow();
await expect(
service.createRule({
...base,
rateLimit: { enabled: true, maxMessages: 20, windowSeconds: 60, extra: 1 },
}),
).rejects.toThrow();
await expect(
service.createRule({ ...base, quietHours: [{ start: '08:00', end: '08:00' }] }),
).rejects.toThrow();
await expect(
service.createRule({ ...base, quietHours: [{ start: '8:00', end: '09:00' }] }),
).rejects.toThrow();
await expect(
service.createRule({ ...base, quietHours: [{ start: '08:00', end: '09:00', extra: 1 }] }),
).rejects.toThrow();
await expect(
service.createRule({
...base,
quietHours: Array.from({ length: 9 }, (_, index) => ({
start: '00:00',
end: index === 8 ? '23:59' : String(index).padStart(2, '0') + ':30',
})),
}),
).rejects.toThrow();
});
});
});
File diff suppressed because it is too large Load Diff
@@ -0,0 +1,305 @@
import { createHmac } from 'node:crypto';
import { describe, expect, it } from 'vitest';
import { deliverThroughChannel, type HttpRequest } from './channel-delivery.js';
const NOW = new Date('2026-09-03T08:00:00.000Z');
function recorder(response: { status: number; body: string }) {
const calls: HttpRequest[] = [];
return {
calls,
request: async (request: HttpRequest) => {
calls.push(request);
return response;
},
};
}
const base = {
title: '设备离线',
body: 'SIM 卡所在设备已断开',
eventType: 'device',
occurredAt: NOW.toISOString(),
instanceName: 'lab-01',
};
describe('deliverThroughChannel', () => {
it('signs the generic webhook with the Hub timestamp scheme', async () => {
const sink = recorder({ status: 204, body: '' });
const result = await deliverThroughChannel(
{
...base,
type: 'webhook',
config: {
url: 'https://hook.example.test/notify',
http_method: 'post',
secret: 'topsecret',
headers: 'X-Tenant: acme\nbad header: skip',
},
},
{ request: sink.request, now: () => NOW },
);
expect(result).toEqual({ ok: true });
const [request] = sink.calls;
expect(request?.url).toBe('https://hook.example.test/notify');
expect(request?.method).toBe('POST');
const payload = JSON.parse(String(request?.body)) as Record<string, unknown>;
expect(payload).toEqual({
event: 'device',
title: '设备离线',
body: 'SIM 卡所在设备已断开',
instance: 'lab-01',
occurred_at: NOW.toISOString(),
});
const timestamp = String(request?.headers['x-hub-timestamp']);
expect(timestamp).toBe(String(NOW.getTime()));
expect(request?.headers['x-hub-signature']).toBe(
`sha256=${createHmac('sha256', 'topsecret')
.update(`${timestamp}.${String(request?.body)}`)
.digest('base64')}`,
);
expect(request?.headers['x-tenant']).toBe('acme');
expect(Object.keys(request?.headers ?? {})).not.toContain('bad header: skip');
});
it('posts Bark options to the device-key path', async () => {
const sink = recorder({ status: 200, body: '{"code":200}' });
const result = await deliverThroughChannel(
{
...base,
type: 'bark',
config: {
server_url: 'https://bark.example.test/',
device_key: 'abc def',
group: 'Island',
sound: 'bell',
level: 'timeSensitive',
icon: 'https://bark.example.test/icon.png',
auto_copy: true,
save_history: false,
},
},
{ request: sink.request, now: () => NOW },
);
expect(result).toEqual({ ok: true });
const [request] = sink.calls;
expect(request?.url).toBe('https://bark.example.test/abc%20def');
expect(JSON.parse(String(request?.body))).toEqual({
title: '设备离线',
body: 'SIM 卡所在设备已断开',
group: 'Island',
autocopy: 1,
save: 0,
sound: 'bell',
level: 'timeSensitive',
icon: 'https://bark.example.test/icon.png',
});
});
it('treats a Bark error code in a 200 response as a failure', async () => {
const sink = recorder({ status: 200, body: '{"code":400,"message":"bad device key"}' });
const result = await deliverThroughChannel(
{
...base,
type: 'bark',
config: { server_url: 'https://bark.example.test', device_key: 'k' },
},
{ request: sink.request, now: () => NOW },
);
expect(result).toEqual({ ok: false, detail: 'code=400 bad device key' });
});
it('signs the DingTalk robot webhook', async () => {
const sink = recorder({ status: 200, body: '{"errcode":0}' });
await deliverThroughChannel(
{
...base,
type: 'dingtalk_robot',
config: {
webhook_url: 'https://oapi.dingtalk.com/robot/send',
access_token: 'token-1',
secret: 'SEC-abc',
at_mobiles: '13800000000, 13900000000',
at_all: true,
},
},
{ request: sink.request, now: () => NOW },
);
const [request] = sink.calls;
const url = new URL(String(request?.url));
expect(url.searchParams.get('access_token')).toBe('token-1');
const timestamp = String(url.searchParams.get('timestamp'));
expect(url.searchParams.get('sign')).toBe(
createHmac('sha256', 'SEC-abc').update(`${timestamp}\nSEC-abc`).digest('base64'),
);
const body = JSON.parse(String(request?.body)) as Record<string, unknown>;
expect(body['msgtype']).toBe('text');
expect(body['at']).toEqual({ atMobiles: ['13800000000', '13900000000'], isAtAll: true });
});
it('signs the Feishu robot webhook with the key-derived signature', async () => {
const sink = recorder({ status: 200, body: '{"code":0}' });
await deliverThroughChannel(
{
...base,
type: 'feishu_robot',
config: { webhook_url: 'https://open.feishu.cn/open-apis/bot/v2/hook/x', secret: 'fs' },
},
{ request: sink.request, now: () => NOW },
);
const body = JSON.parse(String(sink.calls[0]?.body)) as Record<string, unknown>;
const timestamp = String(Math.floor(NOW.getTime() / 1000));
expect(body['timestamp']).toBe(timestamp);
expect(body['sign']).toBe(createHmac('sha256', `${timestamp}\nfs`).update('').digest('base64'));
});
it('exchanges credentials before sending a WeCom application message', async () => {
const calls: HttpRequest[] = [];
const result = await deliverThroughChannel(
{
...base,
type: 'wecom_app',
config: {
corp_id: 'wx-corp',
agent_id: 1000002,
secret: 'app-secret',
to_user: 'alice|bob',
to_party: '2',
safe: true,
},
},
{
now: () => NOW,
request: async (request) => {
calls.push(request);
if (request.url.includes('gettoken'))
return { status: 200, body: '{"errcode":0,"access_token":"token-9"}' };
return { status: 200, body: '{"errcode":0,"errmsg":"ok"}' };
},
},
);
expect(result).toEqual({ ok: true });
expect(calls).toHaveLength(2);
expect(new URL(String(calls[0]?.url)).searchParams.get('corpsecret')).toBe('app-secret');
const send = calls[1];
expect(send?.url).toBe('https://qyapi.weixin.qq.com/cgi-bin/message/send?access_token=token-9');
expect(JSON.parse(String(send?.body))).toEqual({
touser: 'alice|bob',
msgtype: 'text',
agentid: 1000002,
text: { content: '设备离线\nSIM 卡所在设备已断开\n来源:lab-01\n' + NOW.toISOString() },
safe: 1,
toparty: '2',
});
});
it('reports a WeCom token failure without attempting the send', async () => {
const calls: HttpRequest[] = [];
const result = await deliverThroughChannel(
{
...base,
type: 'wecom_app',
config: { corp_id: 'c', agent_id: 1, secret: 's' },
},
{
now: () => NOW,
request: async (request) => {
calls.push(request);
return { status: 200, body: '{"errcode":40013,"errmsg":"invalid corp"}' };
},
},
);
expect(calls).toHaveLength(1);
expect(result).toEqual({ ok: false, detail: 'errcode=40013 invalid corp' });
});
it('uses the DingTalk v2 token header for application messages', async () => {
const calls: HttpRequest[] = [];
const result = await deliverThroughChannel(
{
...base,
type: 'dingtalk_app',
config: {
app_key: 'key',
app_secret: 'shh',
robot_code: 'robot',
open_conversation_id: 'cid',
msg_key: 'sampleMarkdown',
},
},
{
now: () => NOW,
request: async (request) => {
calls.push(request);
if (request.url.endsWith('accessToken'))
return { status: 200, body: '{"accessToken":"at-1"}' };
return { status: 200, body: '{"processQueryKey":"pk"}' };
},
},
);
expect(result).toEqual({ ok: true });
const send = calls[1];
expect(send?.headers['x-acs-dingtalk-access-token']).toBe('at-1');
const body = JSON.parse(String(send?.body)) as Record<string, unknown>;
expect(body['msgKey']).toBe('sampleMarkdown');
expect(JSON.parse(String(body['msgParam']))).toEqual({
title: '设备离线',
text: '**设备离线**\n\nSIM 卡所在设备已断开\n\n来源:lab-01\n\n' + NOW.toISOString(),
});
});
it('routes Server 酱 v3 keys to the ft07 endpoint as a form body', async () => {
const sink = recorder({ status: 200, body: '{"code":0}' });
await deliverThroughChannel(
{ ...base, type: 'serverchan', config: { send_key: 'sct12345@9876', uid: '42' } },
{ request: sink.request, now: () => NOW },
);
const [request] = sink.calls;
expect(request?.url).toBe('https://push.ft07.com/send/sct12345%409876.send');
expect(request?.headers['content-type']).toBe('application/x-www-form-urlencoded');
expect(request?.body).toContain('title=' + encodeURIComponent('设备离线'));
expect(request?.body).toContain('uid=42');
});
it('falls back to the legacy Server 酱 host for plain keys', async () => {
const sink = recorder({ status: 200, body: '' });
await deliverThroughChannel(
{ ...base, type: 'serverchan', config: { send_key: 'SCT999' } },
{ request: sink.request, now: () => NOW },
);
expect(sink.calls[0]?.url).toBe('https://sctapi.ftqq.com/SCT999.send');
});
it('rejects non-HTTP endpoints and missing credentials with a readable detail', async () => {
await expect(
deliverThroughChannel(
{ ...base, type: 'webhook', config: { url: 'file:///etc/passwd' } },
{ now: () => NOW },
),
).resolves.toEqual({ ok: false, detail: '回调地址 只支持 http 或 https' });
await expect(
deliverThroughChannel({ ...base, type: 'telegram', config: { chat_id: '1' } }, {}),
).resolves.toEqual({ ok: false, detail: '缺少 Telegram Bot Token' });
});
it('delegates email delivery to the injected SMTP sender', async () => {
const seen: string[] = [];
const result = await deliverThroughChannel(
{ ...base, type: 'email', config: { smtp_host: 'smtp.example.test' } },
{
now: () => NOW,
sendEmail: async (input) => {
seen.push(input.type);
return { ok: true };
},
},
);
expect(result).toEqual({ ok: true });
expect(seen).toEqual(['email']);
});
});
@@ -0,0 +1,558 @@
import { createHmac } from 'node:crypto';
import {
notificationChannelSpec,
parseKeyValueField,
type NotificationChannelConfigValue,
type NotificationChannelType,
} from '@multi-simadmin/contracts';
export type ChannelConfig = Readonly<Record<string, NotificationChannelConfigValue>>;
export interface ChannelDeliveryInput {
readonly type: NotificationChannelType;
readonly config: ChannelConfig;
readonly title: string;
readonly body: string;
readonly eventType: string;
readonly occurredAt: string;
readonly instanceName?: string;
}
export interface ChannelDeliveryResult {
readonly ok: boolean;
readonly detail?: string;
}
export interface HttpRequest {
readonly url: string;
readonly method: 'GET' | 'POST' | 'PUT' | 'PATCH';
readonly headers: Readonly<Record<string, string>>;
readonly body?: string;
}
export interface HttpResponse {
readonly status: number;
readonly body: string;
}
export type HttpRequester = (request: HttpRequest, timeoutMs: number) => Promise<HttpResponse>;
export interface ChannelDeliveryOptions {
readonly request?: HttpRequester | undefined;
readonly sendEmail?: (
input: ChannelDeliveryInput,
timeoutMs: number,
) => Promise<ChannelDeliveryResult>;
readonly timeoutMs?: number | undefined;
readonly now?: (() => Date) | undefined;
}
const DEFAULT_TIMEOUT_MS = 10_000;
const MAX_RESPONSE_BYTES = 64 * 1024;
export class ChannelDeliveryError extends Error {
constructor(message: string) {
super(message);
this.name = 'ChannelDeliveryError';
}
}
function stringConfig(config: ChannelConfig, key: string): string {
const value = config[key];
if (typeof value === 'string') return value.trim();
if (typeof value === 'number' && Number.isFinite(value)) return String(value);
if (typeof value === 'boolean') return value ? 'true' : 'false';
return '';
}
function booleanConfig(config: ChannelConfig, key: string): boolean {
const value = config[key];
return value === true || value === 'true' || value === 1;
}
function numberConfig(config: ChannelConfig, key: string, fallback: number): number {
const value = config[key];
if (typeof value === 'number' && Number.isFinite(value)) return value;
const parsed = Number(typeof value === 'string' ? value.trim() : '');
return Number.isFinite(parsed) && parsed > 0 ? parsed : fallback;
}
function trimSlash(value: string): string {
return value.replace(/\/+$/u, '');
}
function httpsUrl(raw: string, label: string): string {
let parsed: URL;
try {
parsed = new URL(raw);
} catch {
throw new ChannelDeliveryError(`${label} 不是合法地址`);
}
if (parsed.protocol !== 'https:' && parsed.protocol !== 'http:')
throw new ChannelDeliveryError(`${label} 只支持 http 或 https`);
return parsed.toString();
}
function withQuery(raw: string, parameters: Readonly<Record<string, string>>): string {
const url = new URL(raw);
for (const [key, value] of Object.entries(parameters)) {
if (value !== '' && !url.searchParams.has(key)) url.searchParams.set(key, value);
}
return url.toString();
}
function sign(algorithm: string, key: string, data: string): string {
return createHmac(algorithm, key).update(data).digest('base64');
}
function json(value: unknown): string {
return JSON.stringify(value);
}
function parseBody(raw: string): Record<string, unknown> | undefined {
if (raw.length === 0) return undefined;
try {
const value: unknown = JSON.parse(raw);
return typeof value === 'object' && value !== null && !Array.isArray(value)
? (value as Record<string, unknown>)
: undefined;
} catch {
return undefined;
}
}
/** Turns a service reply into a verdict; most vendors answer 200 even on failure. */
function verdict(
response: HttpResponse,
failurePaths: readonly string[] = ['errcode', 'code'],
successValues: Readonly<Record<string, number>> = {},
): ChannelDeliveryResult {
if (response.status < 200 || response.status >= 300) {
return { ok: false, detail: `HTTP ${response.status} ${response.body.slice(0, 200)}`.trim() };
}
const payload = parseBody(response.body);
if (!payload) return { ok: true };
for (const path of failurePaths) {
const value = payload[path];
// Most vendors use errcode 0; Bark reports success as code 200.
if (typeof value === 'number' && value !== (successValues[path] ?? 0)) {
const message = payload['msg'] ?? payload['errmsg'] ?? payload['message'];
return {
ok: false,
detail: `${path}=${value}${typeof message === 'string' ? ` ${message.slice(0, 180)}` : ''}`,
};
}
if (typeof value === 'boolean' && value === false) {
const message = payload['error_description'] ?? payload['description'];
return {
ok: false,
detail: `${path}=false${typeof message === 'string' ? ` ${message.slice(0, 180)}` : ''}`,
};
}
}
return { ok: true };
}
async function defaultRequester(request: HttpRequest, timeoutMs: number): Promise<HttpResponse> {
const response = await fetch(request.url, {
method: request.method,
headers: request.headers,
...(request.body === undefined ? {} : { body: request.body }),
signal: AbortSignal.timeout(timeoutMs),
// A signed payload must never be replayed to a redirect target.
redirect: 'manual',
});
const text = (await response.text()).slice(0, MAX_RESPONSE_BYTES);
return { status: response.status, body: text };
}
function markdownBody(input: ChannelDeliveryInput): string {
const source = input.instanceName ? `\n\n来源:${input.instanceName}` : '';
return `**${input.title}**\n\n${input.body}${source}\n\n${input.occurredAt}`;
}
function plainBody(input: ChannelDeliveryInput): string {
const source = input.instanceName ? `\n来源:${input.instanceName}` : '';
return `${input.title}\n${input.body}${source}\n${input.occurredAt}`;
}
function deliverWebhook(input: ChannelDeliveryInput, now: () => Date): HttpRequest | undefined {
const raw = stringConfig(input.config, 'url');
if (!raw) throw new ChannelDeliveryError('缺少回调地址');
const url = httpsUrl(raw, '回调地址');
const method = (stringConfig(input.config, 'http_method') || 'POST').toUpperCase();
const payload = json({
event: input.eventType,
title: input.title,
body: input.body,
instance: input.instanceName ?? null,
occurred_at: input.occurredAt,
});
const headers: Record<string, string> = { 'content-type': 'application/json' };
const secret = stringConfig(input.config, 'secret');
if (secret) {
const timestamp = String(now().getTime());
headers['x-hub-timestamp'] = timestamp;
headers['x-hub-signature'] = `sha256=${sign('sha256', secret, `${timestamp}.${payload}`)}`;
}
for (const [key, value] of Object.entries(parseKeyValueField(input.config['headers']))) {
if (/^[A-Za-z0-9!#$%&'*+.^_`|~-]+$/u.test(key)) headers[key.toLowerCase()] = value;
}
return {
url,
method: method === 'PUT' || method === 'PATCH' ? (method as 'PUT' | 'PATCH') : 'POST',
headers,
body: payload,
};
}
function deliverBark(input: ChannelDeliveryInput): HttpRequest | undefined {
const server = trimSlash(stringConfig(input.config, 'server_url') || 'https://api.day.app');
const key = stringConfig(input.config, 'device_key');
if (!key) throw new ChannelDeliveryError('缺少 Bark 设备 Key');
const body: Record<string, string | number | boolean> = {
title: input.title,
body: input.body,
group: stringConfig(input.config, 'group') || 'SimAdminHub',
autocopy: booleanConfig(input.config, 'auto_copy') ? 1 : 0,
save: booleanConfig(input.config, 'save_history') ? 1 : 0,
};
const sound = stringConfig(input.config, 'sound');
const level = stringConfig(input.config, 'level');
const icon = stringConfig(input.config, 'icon');
if (sound) body['sound'] = sound;
if (level) body['level'] = level;
if (icon) body['icon'] = icon;
return {
url: httpsUrl(`${server}/${encodeURIComponent(key)}`, 'Bark 服务器地址'),
method: 'POST',
headers: { 'content-type': 'application/json; charset=utf-8' },
body: json(body),
};
}
function deliverPushPlus(input: ChannelDeliveryInput): HttpRequest | undefined {
const token = stringConfig(input.config, 'token');
if (!token) throw new ChannelDeliveryError('缺少 PushPlus Token');
const body: Record<string, string> = {
token,
title: input.title,
content: input.body,
template: stringConfig(input.config, 'template') || 'txt',
};
const topic = stringConfig(input.config, 'topic');
const channel = stringConfig(input.config, 'channel');
const callback = stringConfig(input.config, 'callback_url');
if (topic) body['topic'] = topic;
if (channel) body['channel'] = channel;
if (callback) body['callbackUrl'] = callback;
return {
url: 'https://www.pushplus.plus/send',
method: 'POST',
headers: { 'content-type': 'application/json' },
body: json(body),
};
}
function deliverWeComRobot(input: ChannelDeliveryInput): HttpRequest | undefined {
const raw = stringConfig(input.config, 'webhook_url');
if (!raw) throw new ChannelDeliveryError('缺少企业微信机器人 Webhook 地址');
const key = stringConfig(input.config, 'key');
const url = withQuery(httpsUrl(raw, '企业微信 Webhook 地址'), key ? { key } : {});
return {
url,
method: 'POST',
headers: { 'content-type': 'application/json' },
body: json({ msgtype: 'markdown', markdown: { content: markdownBody(input) } }),
};
}
function deliverDingTalkRobot(
input: ChannelDeliveryInput,
now: () => Date,
): HttpRequest | undefined {
const raw = stringConfig(input.config, 'webhook_url');
if (!raw) throw new ChannelDeliveryError('缺少钉钉机器人 Webhook 地址');
const parameters: Record<string, string> = {};
const accessToken = stringConfig(input.config, 'access_token');
if (accessToken) parameters['access_token'] = accessToken;
const secret = stringConfig(input.config, 'secret');
if (secret) {
const timestamp = String(now().getTime());
parameters['timestamp'] = timestamp;
// withQuery encodes values already; pre-encoding here would double-escape the signature.
parameters['sign'] = sign('sha256', secret, `${timestamp}\n${secret}`);
}
const mobiles = stringConfig(input.config, 'at_mobiles')
.split(/[,\s]+/u)
.map((item) => item.trim())
.filter((item) => item !== '')
.slice(0, 50);
const body: Record<string, unknown> = {
msgtype: 'text',
text: { content: plainBody(input) },
at: { atMobiles: mobiles, isAtAll: booleanConfig(input.config, 'at_all') },
};
return {
url: withQuery(httpsUrl(raw, '钉钉 Webhook 地址'), parameters),
method: 'POST',
headers: { 'content-type': 'application/json' },
body: json(body),
};
}
function deliverFeishuRobot(input: ChannelDeliveryInput, now: () => Date): HttpRequest | undefined {
const raw = stringConfig(input.config, 'webhook_url');
if (!raw) throw new ChannelDeliveryError('缺少飞书机器人 Webhook 地址');
const body: Record<string, unknown> = {
msg_type: 'text',
content: { text: plainBody(input) },
};
const secret = stringConfig(input.config, 'secret');
if (secret) {
const timestamp = String(Math.floor(now().getTime() / 1000));
body['timestamp'] = timestamp;
body['sign'] = sign('sha256', `${timestamp}\n${secret}`, '');
}
const token = stringConfig(input.config, 'token');
const url = withQuery(httpsUrl(raw, '飞书 Webhook 地址'), token ? { token } : {});
return { url, method: 'POST', headers: { 'content-type': 'application/json' }, body: json(body) };
}
function deliverTelegram(input: ChannelDeliveryInput): HttpRequest | undefined {
const base = trimSlash(stringConfig(input.config, 'api_base_url') || 'https://api.telegram.org');
const token = stringConfig(input.config, 'bot_token');
const chatId = stringConfig(input.config, 'chat_id');
if (!token) throw new ChannelDeliveryError('缺少 Telegram Bot Token');
if (!chatId) throw new ChannelDeliveryError('缺少 Telegram Chat ID');
const body: Record<string, unknown> = {
chat_id: chatId,
text: `${input.title}\n\n${input.body}`,
disable_web_page_preview: booleanConfig(input.config, 'disable_web_page_preview'),
};
const parseMode = stringConfig(input.config, 'parse_mode');
if (parseMode) body['parse_mode'] = parseMode;
return {
url: httpsUrl(`${base}/bot${encodeURIComponent(token)}/sendMessage`, 'Telegram 接口地址'),
method: 'POST',
headers: { 'content-type': 'application/json' },
body: json(body),
};
}
function deliverServerChan(input: ChannelDeliveryInput): HttpRequest | undefined {
const sendKey = stringConfig(input.config, 'send_key');
if (!sendKey) throw new ChannelDeliveryError('缺少 Server 酱 SendKey');
const endpoint = sendKey.includes('@')
? `https://push.ft07.com/send/${encodeURIComponent(sendKey)}.send`
: `https://sctapi.ftqq.com/${encodeURIComponent(sendKey)}.send`;
const fields: Record<string, string> = { title: input.title, desp: input.body };
for (const key of ['uid', 'channel', 'openid'] as const) {
const value = stringConfig(input.config, key);
if (value) fields[key] = value;
}
const form = new URLSearchParams(fields).toString();
return {
url: httpsUrl(endpoint, 'Server 酱 SendKey'),
method: 'POST',
headers: { 'content-type': 'application/x-www-form-urlencoded' },
body: form,
};
}
/**
* Builds the outbound request for single-shot channels. Channels that need a
* token exchange are handled separately because the second call depends on the
* first response.
*/
function buildRequests(
input: ChannelDeliveryInput,
now: () => Date,
): readonly {
readonly request: HttpRequest;
readonly failurePaths: readonly string[];
readonly successValues?: Readonly<Record<string, number>>;
}[] {
switch (input.type) {
case 'webhook':
return [{ request: single(deliverWebhook(input, now)), failurePaths: [] }];
case 'bark':
return [
{
request: single(deliverBark(input)),
failurePaths: ['code'],
successValues: { code: 200 },
},
];
case 'pushplus':
return [{ request: single(deliverPushPlus(input)), failurePaths: ['code'] }];
case 'wecom_robot':
return [{ request: single(deliverWeComRobot(input)), failurePaths: ['errcode'] }];
case 'dingtalk_robot':
return [{ request: single(deliverDingTalkRobot(input, now)), failurePaths: ['errcode'] }];
case 'feishu_robot':
return [
{
request: single(deliverFeishuRobot(input, now)),
failurePaths: ['code', 'StatusCode'],
},
];
case 'telegram':
return [{ request: single(deliverTelegram(input)), failurePaths: ['ok'] }];
case 'serverchan':
return [{ request: single(deliverServerChan(input)), failurePaths: ['code'] }];
default:
return [];
}
}
/** A channel adapter either produced a request or explained why it could not. */
function single(request: HttpRequest | undefined): HttpRequest {
if (!request) throw new ChannelDeliveryError('通道配置不完整');
return request;
}
async function deliverWeComApp(
input: ChannelDeliveryInput,
request: HttpRequester,
timeoutMs: number,
): Promise<ChannelDeliveryResult> {
const base = trimSlash(
stringConfig(input.config, 'api_base_url') || 'https://qyapi.weixin.qq.com',
);
const corpId = stringConfig(input.config, 'corp_id');
const secret = stringConfig(input.config, 'secret');
const agentId = stringConfig(input.config, 'agent_id');
if (!corpId || !secret || !agentId)
throw new ChannelDeliveryError('企业微信应用需要企业 ID、应用 Secret 与 AgentId');
const tokenUrl = httpsUrl(
withQuery(`${base}/cgi-bin/gettoken`, { corpid: corpId, corpsecret: secret }),
'企业微信接口地址',
);
const tokenResponse = await request(
{ url: tokenUrl, method: 'GET', headers: { accept: 'application/json' } },
timeoutMs,
);
const tokenPayload = parseBody(tokenResponse.body);
const accessToken =
tokenResponse.status >= 200 &&
tokenResponse.status < 300 &&
typeof tokenPayload?.['access_token'] === 'string'
? tokenPayload['access_token']
: undefined;
if (!accessToken) return verdict(tokenResponse, ['errcode']);
const sendUrl = httpsUrl(
withQuery(`${base}/cgi-bin/message/send`, { access_token: accessToken }),
'企业微信接口地址',
);
const body: Record<string, unknown> = {
touser: stringConfig(input.config, 'to_user') || '@all',
msgtype: 'text',
agentid: numberConfig(input.config, 'agent_id', 0),
text: { content: plainBody(input) },
safe: booleanConfig(input.config, 'safe') ? 1 : 0,
};
const toParty = stringConfig(input.config, 'to_party');
const toTag = stringConfig(input.config, 'to_tag');
if (toParty) body['toparty'] = toParty;
if (toTag) body['totag'] = toTag;
const response = await request(
{
url: sendUrl,
method: 'POST',
headers: { 'content-type': 'application/json' },
body: json(body),
},
timeoutMs,
);
return verdict(response, ['errcode']);
}
async function deliverDingTalkApp(
input: ChannelDeliveryInput,
request: HttpRequester,
timeoutMs: number,
): Promise<ChannelDeliveryResult> {
const appKey = stringConfig(input.config, 'app_key');
const appSecret = stringConfig(input.config, 'app_secret');
const robotCode = stringConfig(input.config, 'robot_code');
const conversationId = stringConfig(input.config, 'open_conversation_id');
if (!appKey || !appSecret || !robotCode || !conversationId)
throw new ChannelDeliveryError('钉钉应用缺少 App Key、App Secret、Robot Code 或会话 ID');
const tokenResponse = await request(
{
url: 'https://api.dingtalk.com/v1.0/oauth2/accessToken',
method: 'POST',
headers: { 'content-type': 'application/json' },
body: json({ appKey, appSecret }),
},
timeoutMs,
);
const tokenPayload = parseBody(tokenResponse.body);
const accessToken =
tokenResponse.status >= 200 &&
tokenResponse.status < 300 &&
typeof tokenPayload?.['accessToken'] === 'string'
? tokenPayload['accessToken']
: undefined;
if (!accessToken) return verdict(tokenResponse, ['code', 'errcode']);
const msgKey = stringConfig(input.config, 'msg_key') || 'sampleText';
const msgParam =
msgKey === 'sampleMarkdown'
? { title: input.title, text: markdownBody(input) }
: { content: plainBody(input) };
const response = await request(
{
url: 'https://api.dingtalk.com/v1.0/robot/groupMessages/send',
method: 'POST',
headers: {
'content-type': 'application/json',
'x-acs-dingtalk-access-token': accessToken,
},
body: json({
robotCode,
openConversationId: conversationId,
msgKey,
msgParam: json(msgParam),
}),
},
timeoutMs,
);
return verdict(response, ['code', 'errcode']);
}
export function channelSupportsDelivery(type: NotificationChannelType | string): boolean {
return notificationChannelSpec(type) !== undefined;
}
/** Delivers one notification through the channel adapter that matches its type. */
export async function deliverThroughChannel(
input: ChannelDeliveryInput,
options: ChannelDeliveryOptions = {},
): Promise<ChannelDeliveryResult> {
const request = options.request ?? defaultRequester;
const now = options.now ?? (() => new Date());
const timeoutMs = options.timeoutMs ?? DEFAULT_TIMEOUT_MS;
try {
if (input.type === 'email') {
if (!options.sendEmail) return { ok: false, detail: '邮件通道未启用 SMTP 发送器' };
return await options.sendEmail(input, timeoutMs);
}
if (input.type === 'wecom_app') return await deliverWeComApp(input, request, timeoutMs);
if (input.type === 'dingtalk_app') return await deliverDingTalkApp(input, request, timeoutMs);
const steps = buildRequests(input, now);
if (steps.length === 0) return { ok: false, detail: `不支持的通道类型:${input.type}` };
let result: ChannelDeliveryResult = { ok: true };
for (const step of steps) {
const response = await request(step.request, timeoutMs);
result = verdict(response, step.failurePaths, step.successValues ?? {});
if (!result.ok) return result;
}
return result;
} catch (error) {
if (error instanceof ChannelDeliveryError) return { ok: false, detail: error.message };
const message = error instanceof Error ? error.message : '投递失败';
return { ok: false, detail: message.slice(0, 300) };
}
}
@@ -0,0 +1,713 @@
import type { Instance } from '@multi-simadmin/contracts';
import type { InstanceService } from '../instances/instance-service.js';
import type {
InstanceSessionStore,
UpstreamResponse,
UpstreamSessionClientOptions,
} from '../connections/upstream-session-client.js';
export type FleetNotificationState = 'ready' | 'unavailable' | 'failed';
export interface FleetNotificationDevice {
readonly id: string;
readonly name: string;
readonly state: FleetNotificationState;
}
export interface FleetNotificationChannelTypeSummary {
readonly type: string;
readonly total: number;
readonly enabled: number;
}
export interface FleetNotificationConfigSummary {
readonly channelCount: number;
readonly channelEnabled: number;
readonly ruleCount: number;
readonly ruleEnabled: number;
readonly channelTypes: readonly FleetNotificationChannelTypeSummary[];
}
export interface FleetNotificationLogEntry {
readonly id: string;
readonly eventType: string;
readonly status: string;
readonly ruleName?: string;
readonly channelName?: string;
readonly createdAt: string;
}
export interface FleetNotificationLogSummary {
readonly total: number;
readonly success: number;
readonly failed: number;
readonly quietHours: number;
readonly unmatched: number;
readonly noAvailableChannel: number;
readonly other: number;
readonly recent: readonly FleetNotificationLogEntry[];
}
export interface FleetNotificationQueueEntry {
readonly id: string;
readonly instanceId: string;
readonly instanceName: string;
readonly status: string;
readonly eventType: string;
readonly ruleName?: string;
readonly channelName?: string;
readonly createdAt: string;
}
export interface FleetNotificationQueueSummary {
readonly total: number;
readonly pending: number;
readonly scheduled: number;
readonly retrying: number;
readonly sending: number;
readonly failed: number;
readonly recent: readonly FleetNotificationQueueEntry[];
}
export interface InstanceNotificationSummary {
readonly observedAt: string;
readonly deviceCount: number;
readonly readyCount: number;
readonly unavailableCount: number;
readonly devices: readonly FleetNotificationDevice[];
readonly config: FleetNotificationConfigSummary;
readonly logs: FleetNotificationLogSummary;
readonly queue: FleetNotificationQueueSummary;
}
export interface FleetNotificationActionFailure {
readonly instanceId: string;
readonly instanceName: string;
readonly code: NotificationServiceErrorCode;
}
export interface FleetNotificationQueueRetryResult {
readonly requested: number;
readonly succeeded: number;
readonly failed: number;
readonly skipped: number;
readonly failures: readonly FleetNotificationActionFailure[];
}
export type NotificationServiceErrorCode =
| 'NOT_FOUND'
| 'VALIDATION_FAILED'
| 'UPSTREAM_FAILED'
| 'SESSION_INVALID';
export class NotificationServiceError extends Error {
constructor(readonly code: NotificationServiceErrorCode) {
super(code);
this.name = 'NotificationServiceError';
}
}
const MAX_DEVICES = 200;
const MAX_RESPONSE_BYTES = 262_144;
const MAX_LOGS = 200;
const MAX_QUEUE = 500;
const MAX_CHANNELS = 80;
const MAX_RULES = 200;
const MAX_RECENT_LOGS = 20;
const MAX_RECENT_QUEUE = 20;
const DEFAULT_LOGS_LIMIT = 50;
const DEFAULT_QUEUE_LIMIT = 100;
const LEGACY_CHANNEL_TYPES = [
'webhook',
'bark',
'pushplus',
'wecom_app',
'wecom_robot',
'dingtalk_robot',
'dingtalk_app',
'feishu_robot',
'telegram',
'email',
'serverchan',
] as const;
const LOG_STATUSES = new Set([
'success',
'failed',
'quiet_hours',
'unmatched',
'no_available_channel',
]);
const QUEUE_STATUSES = new Set(['pending', 'scheduled', 'retrying', 'sending', 'failed']);
const record = (value: unknown): Record<string, unknown> | undefined =>
value !== null && typeof value === 'object' && !Array.isArray(value)
? (value as Record<string, unknown>)
: undefined;
const bounded = (value: unknown, maximum: number): string | undefined =>
(typeof value === 'string' || typeof value === 'number') &&
String(value).length <= maximum &&
!/[\u0000-\u0008\u000b\u000c\u000e-\u001f\u007f]/u.test(String(value))
? String(value)
: undefined;
const safeCount = (value: unknown, maximum: number): number | undefined =>
(typeof value === 'number' || typeof value === 'string') &&
/^\d+$/u.test(String(value)) &&
Number.isSafeInteger(Number(value)) &&
Number(value) >= 0 &&
Number(value) <= maximum
? Number(value)
: undefined;
function hasSuccessStatus(body: string): Record<string, unknown> | undefined {
let root: Record<string, unknown> | undefined;
try {
root = record(JSON.parse(body));
} catch {
return undefined;
}
return root && (root.status === 'success' || root.status === 'ok' || root.status === 'OK')
? root
: undefined;
}
function parseApiResponse(response: UpstreamResponse): Record<string, unknown> {
if (
response.status < 200 ||
response.status >= 300 ||
Buffer.byteLength(response.body, 'utf8') > MAX_RESPONSE_BYTES
)
throw new NotificationServiceError('UPSTREAM_FAILED');
const root = hasSuccessStatus(response.body);
const data = record(root?.data);
if (!data) throw new NotificationServiceError('UPSTREAM_FAILED');
return data;
}
function channelTypeSummary(
current: FleetNotificationChannelTypeSummary | undefined,
type: string,
enabled: boolean,
): FleetNotificationChannelTypeSummary {
return {
type,
total: (current?.total ?? 0) + 1,
enabled: (current?.enabled ?? 0) + (enabled ? 1 : 0),
};
}
function parseConfig(data: Record<string, unknown>): FleetNotificationConfigSummary {
let channelCount = 0;
let channelEnabled = 0;
const channelTypes = new Map<string, FleetNotificationChannelTypeSummary>();
const collect = (type: string, value: unknown): void => {
const item = record(value);
if (!item || typeof item.enabled !== 'boolean') return;
channelCount += 1;
if (item.enabled) channelEnabled += 1;
channelTypes.set(type, channelTypeSummary(channelTypes.get(type), type, item.enabled));
};
if (Array.isArray(data.channels)) {
for (const value of data.channels.slice(0, MAX_CHANNELS)) {
const item = record(value);
const type = bounded(item?.type, 80);
if (!item || !type || typeof item.enabled !== 'boolean') continue;
channelCount += 1;
if (item.enabled) channelEnabled += 1;
channelTypes.set(type, channelTypeSummary(channelTypes.get(type), type, item.enabled));
}
} else {
for (const type of LEGACY_CHANNEL_TYPES) collect(type, data[type]);
}
let ruleCount = 0;
let ruleEnabled = 0;
if (Array.isArray(data.rules)) {
for (const value of data.rules.slice(0, MAX_RULES)) {
const item = record(value);
if (!item || typeof item.enabled !== 'boolean') continue;
ruleCount += 1;
if (item.enabled) ruleEnabled += 1;
}
}
return {
channelCount,
channelEnabled,
ruleCount,
ruleEnabled,
channelTypes: Object.freeze([...channelTypes.values()]),
};
}
function parseLogEntry(value: unknown): FleetNotificationLogEntry | undefined {
const item = record(value);
if (!item) return undefined;
const id = bounded(item.id, 128);
const eventType = bounded(item.event_type, 80);
const status = bounded(item.status, 40);
const ruleName = bounded(item.rule_name, 160);
const channelName = bounded(item.channel_name, 160);
const createdAt = bounded(item.created_at, 64);
if (!id || !eventType || !status || !createdAt) return undefined;
return Object.freeze({
id,
eventType,
status,
...(ruleName === undefined ? {} : { ruleName }),
...(channelName === undefined ? {} : { channelName }),
createdAt,
});
}
function parseLogs(data: Record<string, unknown>): FleetNotificationLogSummary {
const total = safeCount(data.total, 10_000_000);
if (!Array.isArray(data.logs) || data.logs.length > MAX_LOGS || total === undefined)
throw new NotificationServiceError('UPSTREAM_FAILED');
const recent: FleetNotificationLogEntry[] = [];
const counts = {
success: 0,
failed: 0,
quietHours: 0,
unmatched: 0,
noAvailableChannel: 0,
other: 0,
};
for (const value of data.logs) {
const entry = parseLogEntry(value);
if (!entry) throw new NotificationServiceError('UPSTREAM_FAILED');
if (LOG_STATUSES.has(entry.status)) {
if (entry.status === 'success') counts.success += 1;
else if (entry.status === 'failed') counts.failed += 1;
else if (entry.status === 'quiet_hours') counts.quietHours += 1;
else if (entry.status === 'unmatched') counts.unmatched += 1;
else counts.noAvailableChannel += 1;
} else {
counts.other += 1;
}
if (recent.length < MAX_RECENT_LOGS) recent.push(entry);
}
return {
total,
...counts,
recent: Object.freeze(recent),
};
}
function parseQueueEntry(
value: unknown,
instanceId: string,
instanceName: string,
): FleetNotificationQueueEntry | undefined {
const item = record(value);
if (!item) return undefined;
const id = bounded(item.id, 128);
const status = bounded(item.status, 32);
const eventType = bounded(item.event_type, 80);
const ruleName = bounded(item.rule_name, 160);
const channelName = bounded(item.channel_name, 160);
const createdAt = bounded(item.created_at, 64);
if (!id || !status || !eventType || !createdAt) return undefined;
return Object.freeze({
id,
instanceId,
instanceName,
status,
eventType,
...(ruleName === undefined ? {} : { ruleName }),
...(channelName === undefined ? {} : { channelName }),
createdAt,
});
}
function parseQueue(
data: Record<string, unknown>,
instanceId: string,
instanceName: string,
): FleetNotificationQueueSummary {
const total = safeCount(data.total, 10_000_000);
if (!Array.isArray(data.items) || data.items.length > MAX_QUEUE || total === undefined)
throw new NotificationServiceError('UPSTREAM_FAILED');
const recent: FleetNotificationQueueEntry[] = [];
const counts = { pending: 0, scheduled: 0, retrying: 0, sending: 0, failed: 0 };
for (const value of data.items) {
const entry = parseQueueEntry(value, instanceId, instanceName);
if (!entry) throw new NotificationServiceError('UPSTREAM_FAILED');
if (QUEUE_STATUSES.has(entry.status)) {
counts[entry.status as keyof typeof counts] += 1;
}
if (recent.length < MAX_RECENT_QUEUE) recent.push(entry);
}
return {
total,
...counts,
recent: Object.freeze(recent),
};
}
function aggregateLogs(
value: FleetNotificationLogSummary,
accumulator: FleetNotificationLogSummary,
): FleetNotificationLogSummary {
return {
total: value.total + accumulator.total,
success: value.success + accumulator.success,
failed: value.failed + accumulator.failed,
quietHours: value.quietHours + accumulator.quietHours,
unmatched: value.unmatched + accumulator.unmatched,
noAvailableChannel: value.noAvailableChannel + accumulator.noAvailableChannel,
other: value.other + accumulator.other,
recent: Object.freeze([...value.recent, ...accumulator.recent].slice(0, MAX_RECENT_LOGS)),
};
}
function aggregateQueue(
value: FleetNotificationQueueSummary,
accumulator: FleetNotificationQueueSummary,
): FleetNotificationQueueSummary {
return {
total: value.total + accumulator.total,
pending: value.pending + accumulator.pending,
scheduled: value.scheduled + accumulator.scheduled,
retrying: value.retrying + accumulator.retrying,
sending: value.sending + accumulator.sending,
failed: value.failed + accumulator.failed,
recent: Object.freeze([...value.recent, ...accumulator.recent].slice(0, MAX_RECENT_QUEUE)),
};
}
function emptyLogSummary(): FleetNotificationLogSummary {
return {
total: 0,
success: 0,
failed: 0,
quietHours: 0,
unmatched: 0,
noAvailableChannel: 0,
other: 0,
recent: Object.freeze([]),
};
}
function emptyQueueSummary(): FleetNotificationQueueSummary {
return {
total: 0,
pending: 0,
scheduled: 0,
retrying: 0,
sending: 0,
failed: 0,
recent: Object.freeze([]),
};
}
interface InstanceReadResult {
readonly config: FleetNotificationConfigSummary;
readonly logs: FleetNotificationLogSummary;
readonly queue: FleetNotificationQueueSummary;
}
type InstanceNotificationResult =
| {
readonly id: string;
readonly name: string;
readonly state: 'ready';
readonly value: InstanceReadResult;
}
| {
readonly id: string;
readonly name: string;
readonly state: 'unavailable' | 'failed';
};
export class InstanceNotificationService {
constructor(
private readonly options: {
readonly instances: InstanceService;
readonly sessions: InstanceSessionStore;
readonly request: UpstreamSessionClientOptions['request'];
readonly ensureSession?: (
instanceId: string,
origin: string,
force?: boolean,
) => Promise<void>;
},
) {}
async summarize(): Promise<InstanceNotificationSummary> {
const instances = await this.listInstances();
const results: readonly InstanceNotificationResult[] = await Promise.all(
instances.map(async (instance) => {
try {
return {
id: instance.id,
name: instance.name,
state: 'ready' as const,
value: await this.readInstance(instance),
};
} catch (error) {
if (error instanceof NotificationServiceError && error.code === 'NOT_FOUND') {
return {
id: instance.id,
name: instance.name,
state: 'unavailable' as const,
};
}
return { id: instance.id, name: instance.name, state: 'failed' as const };
}
}),
);
const ready = results.filter(
(result): result is Extract<InstanceNotificationResult, { state: 'ready' }> =>
result.state === 'ready',
);
const config = ready.reduce(
(accumulator, result) => {
accumulator.channelCount += result.value.config.channelCount;
accumulator.channelEnabled += result.value.config.channelEnabled;
accumulator.ruleCount += result.value.config.ruleCount;
accumulator.ruleEnabled += result.value.config.ruleEnabled;
return accumulator;
},
{ channelCount: 0, channelEnabled: 0, ruleCount: 0, ruleEnabled: 0 },
);
const channelTypes = new Map<string, FleetNotificationChannelTypeSummary>();
for (const result of ready) {
for (const typeSummary of result.value.config.channelTypes) {
const current = channelTypes.get(typeSummary.type);
channelTypes.set(
typeSummary.type,
channelTypeSummary(current, typeSummary.type, typeSummary.enabled > 0),
);
}
}
const logs = ready.reduce(
(accumulator, result) => aggregateLogs(result.value.logs, accumulator),
emptyLogSummary(),
);
const queue = ready.reduce(
(accumulator, result) => aggregateQueue(result.value.queue, accumulator),
emptyQueueSummary(),
);
return {
observedAt: new Date().toISOString(),
deviceCount: instances.length,
readyCount: ready.length,
unavailableCount: results.length - ready.length,
devices: Object.freeze(
results.map((result) =>
Object.freeze({
id: result.id,
name: result.name,
state: result.state,
}),
),
),
config: { ...config, channelTypes: Object.freeze([...channelTypes.values()]) },
logs,
queue,
};
}
private async listInstances(): Promise<readonly Instance[]> {
const instances: Instance[] = [];
let page = 1;
while (instances.length < MAX_DEVICES) {
const current = await this.options.instances.list({ page, pageSize: 100 });
instances.push(...current.items.slice(0, MAX_DEVICES - instances.length));
if (current.items.length < 100) break;
page += 1;
}
return instances;
}
private async readInstance(instance: Instance): Promise<InstanceReadResult> {
const cookie = await this.ensureOwnerCookie(instance);
const [config, logs, queue] = await Promise.all([
this.readEndpoint(instance, cookie, '/api/notifications/config').then(parseConfig),
this.readEndpoint(
instance,
cookie,
`/api/notifications/logs?limit=${DEFAULT_LOGS_LIMIT}&offset=0`,
).then(parseLogs),
this.readEndpoint(
instance,
cookie,
`/api/notifications/queue?limit=${DEFAULT_QUEUE_LIMIT}`,
).then((data) => parseQueue(data, instance.id, instance.name)),
]);
return { config, logs, queue };
}
private async ensureOwnerCookie(instance: Instance): Promise<string | undefined> {
const current = this.options.sessions.sessionFor(instance.id);
if (current && current.origin !== instance.origin)
throw new NotificationServiceError('SESSION_INVALID');
if (!current && this.options.ensureSession) {
try {
await this.options.ensureSession(instance.id, instance.origin);
} catch {
// Passwordless or anonymous reads may still succeed without a session.
}
}
const session = this.options.sessions.sessionFor(instance.id);
return session && session.origin === instance.origin ? session.cookie : undefined;
}
private async readEndpoint(
instance: Instance,
cookie: string | undefined,
path: string,
): Promise<Record<string, unknown>> {
const request = (currentCookie?: string) =>
this.options.request({
url: `${instance.origin}${path}`,
method: 'GET',
headers: {
accept: 'application/json',
...(currentCookie ? { cookie: currentCookie } : {}),
},
});
let response = await request(cookie);
if ([401, 403].includes(response.status) && this.options.ensureSession) {
await this.options.ensureSession(instance.id, instance.origin, true);
response = await request(this.options.sessions.sessionFor(instance.id)?.cookie);
}
return parseApiResponse(response);
}
async retryAllQueue(): Promise<FleetNotificationQueueRetryResult> {
const instances = await this.listInstances();
const outcomes = await Promise.all(
instances.map(async (instance) => {
try {
await this.postEndpoint(instance, '/api/notifications/queue/retry-all');
return {
instanceId: instance.id,
instanceName: instance.name,
kind: 'succeeded' as const,
};
} catch (error) {
if (error instanceof NotificationServiceError && error.code === 'NOT_FOUND')
return {
instanceId: instance.id,
instanceName: instance.name,
kind: 'skipped' as const,
};
return {
instanceId: instance.id,
instanceName: instance.name,
kind: 'failed' as const,
code:
error instanceof NotificationServiceError ? error.code : ('UPSTREAM_FAILED' as const),
};
}
}),
);
const failures = outcomes.filter(
(outcome): outcome is Extract<typeof outcome, { readonly kind: 'failed' }> =>
outcome.kind === 'failed',
);
return {
requested: instances.length,
succeeded: outcomes.filter((outcome) => outcome.kind === 'succeeded').length,
failed: failures.length,
skipped: outcomes.filter((outcome) => outcome.kind === 'skipped').length,
failures: Object.freeze(
failures.map((failure) =>
Object.freeze({
instanceId: failure.instanceId,
instanceName: failure.instanceName,
code: failure.code,
}),
),
),
};
}
async retryQueueItem(instanceId: string, queueId: string): Promise<{ readonly retried: true }> {
await this.queueItemAction(instanceId, queueId, 'retry');
return { retried: true };
}
async deleteQueueItem(instanceId: string, queueId: string): Promise<{ readonly deleted: true }> {
await this.queueItemAction(instanceId, queueId, 'delete');
return { deleted: true };
}
private async queueItemAction(
instanceId: string,
queueId: string,
action: 'retry' | 'delete',
): Promise<void> {
if (!/^[A-Za-z0-9_.-]{1,128}$/u.test(queueId))
throw new NotificationServiceError('VALIDATION_FAILED');
const instance = await this.options.instances.get(instanceId);
if (!instance) throw new NotificationServiceError('NOT_FOUND');
try {
const path =
action === 'retry'
? `/api/notifications/queue/${queueId}/retry`
: `/api/notifications/queue/${queueId}`;
if (action === 'delete') {
await this.deleteEndpoint(instance, path);
return;
}
await this.postEndpoint(instance, path);
} catch (error) {
if (error instanceof NotificationServiceError) throw error;
throw new NotificationServiceError('UPSTREAM_FAILED');
}
}
private async postEndpoint(instance: Instance, path: string): Promise<void> {
const cookie = await this.ensureOwnerCookie(instance);
const request = (currentCookie?: string) =>
this.options.request({
url: `${instance.origin}${path}`,
method: 'POST',
headers: {
accept: 'application/json',
...(currentCookie ? { cookie: currentCookie } : {}),
},
});
let response = await request(cookie);
if ([401, 403].includes(response.status) && this.options.ensureSession) {
await this.options.ensureSession(instance.id, instance.origin, true);
response = await request(this.options.sessions.sessionFor(instance.id)?.cookie);
}
assertActionSuccess(response);
}
private async deleteEndpoint(instance: Instance, path: string): Promise<void> {
const cookie = await this.ensureOwnerCookie(instance);
const request = (currentCookie?: string) =>
this.options.request({
url: `${instance.origin}${path}`,
method: 'DELETE',
headers: {
accept: 'application/json',
...(currentCookie ? { cookie: currentCookie } : {}),
},
});
let response = await request(cookie);
if ([401, 403].includes(response.status) && this.options.ensureSession) {
await this.options.ensureSession(instance.id, instance.origin, true);
response = await request(this.options.sessions.sessionFor(instance.id)?.cookie);
}
assertActionSuccess(response);
}
}
function assertActionSuccess(response: UpstreamResponse): void {
if (
response.status < 200 ||
response.status >= 300 ||
Buffer.byteLength(response.body, 'utf8') > MAX_RESPONSE_BYTES
)
throw new NotificationServiceError('UPSTREAM_FAILED');
if (!hasSuccessStatus(response.body)) throw new NotificationServiceError('UPSTREAM_FAILED');
}
@@ -0,0 +1,215 @@
import { createServer, type Server, type Socket } from 'node:net';
import { afterEach, describe, expect, it } from 'vitest';
import type { ChannelDeliveryInput } from './channel-delivery.js';
import { sendEmailNotification } from './smtp-sender.js';
interface FakePeer {
readonly port: number;
readonly lines: string[];
stop(): Promise<void>;
}
const peers: FakePeer[] = [];
afterEach(async () => {
for (const peer of peers.splice(0)) await peer.stop();
});
/**
* Scripted SMTP peer. It answers one reply per command and stays quiet while a
* message body streams in, which is how a real server behaves after `354`.
*/
function fakeServer(script: readonly string[]): Promise<FakePeer> {
return new Promise((resolve) => {
const lines: string[] = [];
const sockets = new Set<Socket>();
const server: Server = createServer((socket) => {
sockets.add(socket);
socket.once('close', () => sockets.delete(socket));
let index = 0;
let buffered = '';
let inData = false;
socket.write(`${script[index++] ?? '221 bye'}\r\n`);
socket.on('data', (chunk) => {
buffered += chunk.toString('utf8');
let boundary = buffered.indexOf('\r\n');
while (boundary >= 0) {
const line = buffered.slice(0, boundary);
buffered = buffered.slice(boundary + 2);
boundary = buffered.indexOf('\r\n');
lines.push(line);
if (inData) {
if (line === '.') inData = false;
else continue;
}
if (line === 'DATA') inData = true;
// A silent peer is deliberate: it lets a test assert the client timeout.
const reply = script[index++];
if (reply !== undefined) socket.write(`${reply}\r\n`);
}
});
});
server.listen(0, '127.0.0.1', () => {
const address = server.address();
const peer: FakePeer = {
port: typeof address === 'object' && address ? address.port : 0,
lines,
stop: () =>
new Promise<void>((done) => {
for (const socket of sockets) socket.destroy();
server.close(() => done());
}),
};
peers.push(peer);
resolve(peer);
});
});
}
function input(config: ChannelDeliveryInput['config']): ChannelDeliveryInput {
return {
type: 'email',
config,
title: '设备离线',
body: 'SIM 卡所在设备已断开',
eventType: 'device',
occurredAt: '2026-09-03T08:00:00.000Z',
};
}
// One reply per command: greeting, EHLO, AUTH LOGIN, user, password, MAIL FROM,
// two RCPT TO, DATA, the "." terminator, then QUIT.
const AUTH_SCRIPT = [
'220 smtp ready',
'250 ehlo',
'334 ' + Buffer.from('Username:').toString('base64'),
'334 ' + Buffer.from('Password:').toString('base64'),
'235 authenticated',
'250 sender ok',
'250 recipient ok',
'250 recipient ok',
'354 send data',
'250 message accepted',
'221 bye',
];
const OPEN_SCRIPT = [
'220 smtp ready',
'250 ehlo',
'250 sender ok',
'250 recipient ok',
'354 send data',
'250 message accepted',
'221 bye',
];
function messageLines(lines: readonly string[]): string[] {
const start = lines.indexOf('DATA');
return start < 0 ? [] : lines.slice(start + 1);
}
describe('sendEmailNotification', () => {
it('runs AUTH LOGIN and submits one UTF-8 message per recipient', async () => {
const peer = await fakeServer(AUTH_SCRIPT);
const result = await sendEmailNotification(
input({
smtp_host: '127.0.0.1',
smtp_port: peer.port,
smtp_security: 'none',
username: 'notify@example.test',
password: 'p@ss',
sender_address: 'notify@example.test',
sender_name: 'SimAdmin 控制台',
receiver_addresses: 'ops@example.test, admin@example.test',
message_format: 'plain',
}),
2_000,
);
expect(result).toEqual({ ok: true });
expect(peer.lines[0]).toMatch(/^EHLO /u);
expect(peer.lines[1]).toBe('AUTH LOGIN');
expect(peer.lines[2]).toBe(Buffer.from('notify@example.test').toString('base64'));
expect(peer.lines[3]).toBe(Buffer.from('p@ss').toString('base64'));
expect(peer.lines[4]).toBe('MAIL FROM:<notify@example.test>');
expect(peer.lines.filter((line) => line.startsWith('RCPT TO:<'))).toEqual([
'RCPT TO:<ops@example.test>',
'RCPT TO:<admin@example.test>',
]);
const message = messageLines(peer.lines);
expect(message.at(-1)).toBe('.');
const headers = message.slice(0, message.indexOf(''));
expect(headers.join('\n')).toContain('Subject: =?UTF-8?B?');
expect(headers.join('\n')).toContain('Content-Type: text/plain; charset=UTF-8');
const payload = Buffer.from(
message.slice(message.indexOf('') + 1, message.length - 1).join(''),
'base64',
).toString('utf8');
expect(payload).toBe('设备离线\n\nSIM 卡所在设备已断开');
});
it('switches the MIME subtype when the channel asks for HTML', async () => {
const peer = await fakeServer(OPEN_SCRIPT);
const result = await sendEmailNotification(
input({
smtp_host: '127.0.0.1',
smtp_port: peer.port,
smtp_security: 'none',
sender_address: 'a@example.test',
receiver_addresses: 'b@example.test',
message_format: 'html',
}),
2_000,
);
expect(result).toEqual({ ok: true });
expect(messageLines(peer.lines).join('\n')).toContain('Content-Type: text/html');
});
it('refuses to dial out when no receiver address is usable', async () => {
await expect(
sendEmailNotification(
input({
smtp_host: '127.0.0.1',
smtp_port: 1,
sender_address: 'notify@example.test',
receiver_addresses: 'not-an-address',
}),
500,
),
).resolves.toEqual({ ok: false, detail: '缺少有效的收件地址' });
});
it('reports a rejected handshake instead of throwing', async () => {
const peer = await fakeServer(['220 smtp ready', '421 service denied']);
await expect(
sendEmailNotification(
input({
smtp_host: '127.0.0.1',
smtp_port: peer.port,
smtp_security: 'none',
sender_address: 'a@example.test',
receiver_addresses: 'b@example.test',
}),
2_000,
),
).resolves.toMatchObject({ ok: false, detail: 'SMTP 返回 421421 service denied' });
});
it('gives up with a readable detail when the server stops answering', async () => {
const peer = await fakeServer(['220 smtp ready']);
await expect(
sendEmailNotification(
input({
smtp_host: '127.0.0.1',
smtp_port: peer.port,
smtp_security: 'none',
sender_address: 'a@example.test',
receiver_addresses: 'b@example.test',
}),
300,
),
).resolves.toMatchObject({ ok: false, detail: 'SMTP 响应超时' });
});
});
@@ -0,0 +1,295 @@
import { createConnection } from 'node:net';
import { connect as connectTls, TLSSocket } from 'node:tls';
import { hostname } from 'node:os';
import type { ChannelDeliveryInput, ChannelDeliveryResult } from './channel-delivery.js';
type Socket = ReturnType<typeof createConnection> | TLSSocket;
const CRLF = '\r\n';
const MAX_REPLY_BYTES = 64 * 1024;
function configText(config: ChannelDeliveryInput['config'], key: string): string {
const value = config[key];
if (typeof value === 'string') return value.trim();
if (typeof value === 'number') return String(value);
return '';
}
function configNumber(
config: ChannelDeliveryInput['config'],
key: string,
fallback: number,
): number {
const parsed = Number(configText(config, key));
return Number.isSafeInteger(parsed) && parsed > 0 && parsed <= 65_535 ? parsed : fallback;
}
function configFlag(config: ChannelDeliveryInput['config'], key: string): boolean {
return config[key] === true || config[key] === 'true' || config[key] === 1;
}
function splitAddresses(raw: string): string[] {
return raw
.split(/[,;\s]+/u)
.map((value) => value.trim())
.filter((value) => /^[^\s@]+@[^\s@]+$/u.test(value))
.slice(0, 50);
}
function encodeAddress(name: string, address: string): string {
if (!name) return `<${address}>`;
return `${encodeWords(name)} <${address}>`;
}
function encodeWords(value: string): string {
return `=?UTF-8?B?${Buffer.from(value, 'utf8').toString('base64')}?=`;
}
/** Header values must not smuggle extra headers through raw newlines. */
function foldHeader(value: string): string {
return value.replace(/[\r\n\u0000-\u0008\u000B\u000C\u000E-\u001F]/gu, ' ').slice(0, 500);
}
function buildMessage(input: ChannelDeliveryInput, html: boolean, sender: string): string {
const headers = [
`From: ${encodeAddress(foldHeader(configText(input.config, 'sender_name')), sender)}`,
`Subject: ${encodeWords(foldHeader(input.title))}`,
'MIME-Version: 1.0',
`Date: ${new Date(input.occurredAt).toUTCString()}`,
`Content-Type: text/${html ? 'html' : 'plain'}; charset=UTF-8`,
'Content-Transfer-Encoding: base64',
];
const body = html
? `<p>${input.title}</p><p>${input.body}</p>`
: `${input.title}\n\n${input.body}`;
const encoded =
Buffer.from(body, 'utf8')
.toString('base64')
.match(/.{1,76}/gu)
?.join(CRLF) ?? '';
return `${headers.join(CRLF)}${CRLF}${CRLF}${encoded}${CRLF}`;
}
class SmtpSession {
#socket: Socket;
#buffer = '';
#waiters: (() => void)[] = [];
#closed = false;
#failure: Error | undefined;
constructor(socket: Socket) {
this.#socket = socket;
this.#attach(socket);
}
get socket(): Socket {
return this.#socket;
}
#attach(socket: Socket): void {
socket.on('data', (chunk: Buffer) => {
this.#buffer += chunk.toString('utf8');
if (this.#buffer.length > MAX_REPLY_BYTES) this.#fail(new Error('SMTP 响应过大'));
this.#wake();
});
socket.on('error', (error: Error) => this.#fail(error));
socket.on('close', () => {
this.#closed = true;
this.#fail(new Error('SMTP 连接已关闭'));
});
}
/** Swaps the transport after STARTTLS without losing the reply reader. */
replace(socket: Socket): void {
this.#socket.removeAllListeners('data');
this.#socket.removeAllListeners('error');
this.#socket.removeAllListeners('close');
this.#buffer = '';
this.#closed = false;
this.#socket = socket;
this.#attach(socket);
}
#fail(error: Error): void {
this.#failure ??= error;
this.#wake();
}
#wake(): void {
const waiting = this.#waiters.splice(0);
for (const resolve of waiting) resolve();
}
write(line: string): void {
if (this.#failure) throw this.#failure;
this.#socket.write(line);
}
#readReply(): { readonly code: number; readonly text: string } | undefined {
const lines = this.#buffer.split(CRLF);
const collected: string[] = [];
let consumed = 0;
for (const line of lines) {
if (!/^\d{3}[- ]/u.test(line)) break;
collected.push(line);
consumed += line.length + CRLF.length;
if (line[3] === ' ') {
this.#buffer = this.#buffer.slice(consumed);
const last = collected[collected.length - 1] ?? '0';
return { code: Number(last.slice(0, 3)), text: collected.join('\n') };
}
}
return undefined;
}
async reply(timeoutMs: number): Promise<{ readonly code: number; readonly text: string }> {
const deadline = Date.now() + timeoutMs;
for (;;) {
const parsed = this.#readReply();
if (parsed) return parsed;
if (this.#failure) throw this.#failure;
if (this.#closed) throw new Error('SMTP 连接已关闭');
const remaining = deadline - Date.now();
if (remaining <= 0) throw new Error('SMTP 响应超时');
// The wake-up callback must resolve as well as clear the timer, or a reply
// that lands in this window would leave the session waiting forever.
await new Promise<void>((resolve) => {
const timer = setTimeout(resolve, Math.min(remaining, 100));
this.#waiters.push(() => {
clearTimeout(timer);
resolve();
});
});
}
}
close(): void {
this.#socket.removeAllListeners();
this.#socket.destroy();
}
}
function expect(reply: { code: number; text: string }, codes: readonly number[]): void {
if (!codes.includes(reply.code))
throw new Error(`SMTP 返回 ${reply.code}${reply.text.slice(0, 160)}`);
}
function plainSocket(host: string, port: number, timeoutMs: number): Promise<Socket> {
return new Promise((resolve, reject) => {
const socket = createConnection({ host, port });
const onTimeout = (): void => {
socket.destroy();
reject(new Error('SMTP 连接超时'));
};
socket.setTimeout(timeoutMs);
socket.once('timeout', onTimeout);
socket.once('connect', () => {
// The timer only guards the handshake. Every command afterwards carries its own reply
// deadline, and leaving an idle timer armed would destroy the session at the same
// instant that deadline expires, making the reported detail a coin flip between
// "connection closed" and "timed out".
socket.removeListener('timeout', onTimeout);
socket.setTimeout(0);
resolve(socket);
});
socket.once('error', reject);
});
}
function upgradeSocket(socket: Socket, host: string, rejectUnauthorized: boolean): Promise<Socket> {
return new Promise((resolve, reject) => {
socket.removeAllListeners('data');
const upgraded = connectTls({ socket, servername: host, rejectUnauthorized }, () =>
resolve(upgraded),
);
upgraded.once('error', reject);
});
}
/**
* Minimal SMTP client behind the email channel: implicit TLS, STARTTLS or plain,
* optional AUTH LOGIN, and one UTF-8 message per delivery.
*/
export async function sendEmailNotification(
input: ChannelDeliveryInput,
timeoutMs: number,
): Promise<ChannelDeliveryResult> {
const host = configText(input.config, 'smtp_host');
const sender = configText(input.config, 'sender_address');
const receivers = splitAddresses(configText(input.config, 'receiver_addresses'));
if (!host) return { ok: false, detail: '缺少 SMTP 地址' };
if (!sender) return { ok: false, detail: '缺少发件地址' };
// Envelope commands are line-framed; reject anything that could break the frame.
if (!/^[^\s@<>,;:"']+@[^\s@<>,;:"']+$/u.test(sender))
return { ok: false, detail: '发件地址无效' };
if (receivers.length === 0) return { ok: false, detail: '缺少有效的收件地址' };
const port = configNumber(input.config, 'smtp_port', 465);
const security = configText(input.config, 'smtp_security') || 'implicit_tls';
const rejectUnauthorized = !configFlag(input.config, 'allow_insecure_tls');
const username = configText(input.config, 'username');
const password = configText(input.config, 'password');
let session: SmtpSession | undefined;
try {
const initial: Socket =
security === 'implicit_tls'
? await new Promise<Socket>((resolve, reject) => {
const socket = connectTls({
host,
port,
servername: host,
rejectUnauthorized,
timeout: timeoutMs,
});
socket.once('secure', () => resolve(socket));
socket.once('error', reject);
})
: await plainSocket(host, port, timeoutMs);
session = new SmtpSession(initial);
expect(await session.reply(timeoutMs), [220]);
const clientName = hostname() || 'localhost';
session.write(`EHLO ${clientName}${CRLF}`);
expect(await session.reply(timeoutMs), [250]);
if (security === 'starttls') {
session.write(`STARTTLS${CRLF}`);
expect(await session.reply(timeoutMs), [220]);
session.replace(await upgradeSocket(session.socket, host, rejectUnauthorized));
session.write(`EHLO ${clientName}${CRLF}`);
expect(await session.reply(timeoutMs), [250]);
}
if (username && password) {
session.write(`AUTH LOGIN${CRLF}`);
expect(await session.reply(timeoutMs), [334]);
session.write(`${Buffer.from(username, 'utf8').toString('base64')}${CRLF}`);
expect(await session.reply(timeoutMs), [334]);
session.write(`${Buffer.from(password, 'utf8').toString('base64')}${CRLF}`);
expect(await session.reply(timeoutMs), [235]);
}
session.write(`MAIL FROM:<${sender}>${CRLF}`);
expect(await session.reply(timeoutMs), [250]);
for (const receiver of receivers) {
session.write(`RCPT TO:<${receiver}>${CRLF}`);
expect(await session.reply(timeoutMs), [250, 251]);
}
session.write(`DATA${CRLF}`);
expect(await session.reply(timeoutMs), [354]);
const message = buildMessage(
input,
configText(input.config, 'message_format') === 'html',
sender,
).replace(/^\./gmu, '..');
session.write(`${message}.${CRLF}`);
expect(await session.reply(timeoutMs), [250]);
session.write(`QUIT${CRLF}`);
return { ok: true };
} catch (error) {
const detail = error instanceof Error ? error.message : '邮件投递失败';
return { ok: false, detail: detail.slice(0, 300) };
} finally {
session?.close();
}
}
@@ -0,0 +1,141 @@
import Database from 'better-sqlite3';
import { afterEach, describe, expect, it } from 'vitest';
import { DeviceIdentityService } from '../identity/device-identity-service.js';
import { migrateDatabase } from '../../infrastructure/database/migrations.js';
import { MetricsService } from './metrics-service.js';
let db: Database.Database | undefined;
const NOW = '2026-09-05T02:00:00.000Z';
function fixture() {
db = new Database(':memory:');
db.pragma('foreign_keys=ON');
migrateDatabase(db);
const identities = new DeviceIdentityService({ db });
const metrics = new MetricsService({
db,
version: '1.9.6',
sources: {
connections: {
summarize: () => [
{
instanceId: 'node-a',
total: 40,
success: 38,
failed: 2,
averageDurationMs: 12,
lastObservedAt: NOW,
lastErrorCode: null,
availabilityPercent: 95,
},
],
},
identities,
},
now: () => new Date(NOW),
});
return { db, identities, metrics };
}
function seedNode(database: Database.Database, id: string, name: string, enabled = 1): void {
database
.prepare(
`INSERT INTO instances (id,name,base_url,auth_mode,enabled,config_revision,created_at,updated_at)
VALUES (?,?,'http://' || ? || ':8080','password',?,1,?,?)`,
)
.run(id, name, id, enabled, NOW, NOW);
}
afterEach(() => {
db?.close();
db = undefined;
});
describe('MetricsService', () => {
it('renders the fleet, queue and job gauges with every state present', () => {
const { db: database, metrics } = fixture();
seedNode(database, 'node-a', '机房 A');
seedNode(database, 'node-b', '机房 B', 0);
database
.prepare(
`INSERT INTO sms_outbox (id,instance_id,phone_number,content,status,available_at,created_at,updated_at)
VALUES ('s1','node-a','+8613800000000','hi','failed',?,?,?)`,
)
.run(NOW, NOW, NOW);
database
.prepare(
`INSERT INTO notification_queue (id,event_type,status,payload_json,available_at,created_at,updated_at)
VALUES ('q1','sms','pending','{}',?,?,?)`,
)
.run(NOW, NOW, NOW);
database
.prepare(
`INSERT INTO jobs (id,operation_id,risk_level,status,requested_by,request_id,parameters_digest,created_at,updated_at)
VALUES ('j1','node.reboot','R2','running','op','r1','digest',?,?)`,
)
.run(NOW, NOW);
const text = metrics.render();
expect(text).toContain('multi_simadmin_nodes_total 2');
expect(text).toContain('multi_simadmin_node_enabled{node="node-b"} 0');
expect(text).toContain('multi_simadmin_node_info{node="node-a"} 1');
// Operator-assigned names stay out of the session-free exposition entirely.
expect(text).not.toContain('机房 A');
expect(text).toContain('multi_simadmin_node_probe_total{node="node-a",outcome="success"} 38');
expect(text).toContain('multi_simadmin_node_availability_ratio{node="node-a"} 0.950');
expect(text).toContain('multi_simadmin_sms_outbox{status="failed"} 1');
expect(text).toContain('multi_simadmin_sms_outbox{status="queued"} 0');
expect(text).toContain('multi_simadmin_notification_queue{status="pending"} 1');
expect(text).toContain('multi_simadmin_jobs{status="running"} 1');
expect(text).toContain('multi_simadmin_jobs{status="succeeded"} 0');
expect(text).toContain('multi_simadmin_scheduled_tasks{status="armed"} 0');
expect(text).toContain('multi_simadmin_build_info{version="1.9.6"} 1');
// A node with no probe history is counted rather than silently missing.
expect(text).toContain('multi_simadmin_node_unprobed 1');
});
it('reports the identity guard for each node it has seen', () => {
const { db: database, identities, metrics } = fixture();
seedNode(database, 'node-a', 'A');
seedNode(database, 'node-b', 'B');
const report = { imei: '860000000000009', model: 'RM500Q' };
identities.observe('node-a', report);
identities.observe('node-b', report);
const text = metrics.render();
expect(text).toContain('multi_simadmin_node_identity_pending{node="node-a"} 1');
expect(text).toContain('multi_simadmin_node_identity_pending{node="node-b"} 1');
expect(text).toContain('multi_simadmin_identity_pending_total 2');
});
it('never emits a node name, so a hostile name cannot forge a series', () => {
const { db: database, metrics } = fixture();
seedNode(database, 'node-a', 'evil" name\nwith break');
const text = metrics.render();
// Names are excluded from the session-free exposition outright; escaping a
// label is not the right tool for untrusted operator input.
expect(text).not.toContain('evil"');
expect(text.split('\n').filter((line) => line.includes('with break'))).toHaveLength(0);
});
it('emits a valid exposition: help and type precede every series', () => {
const { db: database, metrics } = fixture();
seedNode(database, 'node-a', 'A');
const lines = metrics.render().trimEnd().split('\n');
const names = new Set<string>();
for (const line of lines) {
if (line.startsWith('# ')) {
const [, kind, name] = line.split(' ');
if (kind === 'TYPE') expect(names.has(name ?? ''), name).toBe(false);
continue;
}
const name = line.slice(0, line.search(/[{ ]/u));
expect(line, name).toMatch(/^[a-z_][a-z0-9_]*(\{[^{}]*\})? [-0-9.e+]+$/u);
names.add(name);
}
expect(names.size).toBeGreaterThan(10);
});
});
@@ -0,0 +1,244 @@
import type Database from 'better-sqlite3';
import type { ConnectionSummary } from '../system/connection-log-service.js';
import type { DeviceIdentity } from '../identity/device-identity-service.js';
/**
* Prometheus exposition for the fused control plane.
*
* The scrape must never reach a device: every value here comes from rows the console already
* writes (instances, probes, queues, jobs, identity records). A monitoring system polling every
* 15 seconds should not be able to wake a modem.
*/
const PREFIX = 'multi_simadmin';
export interface MetricsSource {
readonly connections: { summarize(): readonly ConnectionSummary[] };
readonly identities: { list(): readonly DeviceIdentity[] };
}
export interface MetricsOptions {
readonly db: Database.Database;
readonly version: string;
readonly sources: MetricsSource;
readonly now?: () => Date;
}
interface CountRow {
readonly status: string;
readonly count: number;
}
/** Label values are operator- or device-controlled, so they are escaped the Prometheus way. */
function label(value: string): string {
return value.replace(/\\/gu, '\\\\').replace(/"/gu, '\\"').replace(/\n/gu, '\\n');
}
const number = (value: unknown): number => {
const parsed = typeof value === 'number' ? value : Number(value);
return Number.isFinite(parsed) ? parsed : 0;
};
function sample(name: string, help: string, type: 'gauge' | 'counter', lines: readonly string[]) {
return [`# HELP ${name} ${help}`, `# TYPE ${name} ${type}`, ...lines];
}
export class MetricsService {
readonly #db: Database.Database;
readonly #version: string;
readonly #sources: MetricsSource;
readonly #clock: () => Date;
constructor(options: MetricsOptions) {
this.#db = options.db;
this.#version = options.version;
this.#sources = options.sources;
this.#clock = options.now ?? (() => new Date());
}
render(): string {
const nodes = this.#db
.prepare('SELECT id,name,enabled FROM instances ORDER BY name, id')
.all() as Array<{
id: string;
name: string;
enabled: number;
}>;
const ids = new Set(nodes.map((node) => node.id));
const lines: string[] = [];
lines.push(
...sample(`${PREFIX}_build_info`, 'Build identity of the control plane.', 'gauge', [
`${PREFIX}_build_info{version="${label(this.#version)}"} 1`,
]),
...sample(`${PREFIX}_scrape_timestamp_seconds`, 'Wall-clock time of this scrape.', 'gauge', [
`${PREFIX}_scrape_timestamp_seconds ${Math.floor(this.#clock().getTime() / 1000)}`,
]),
);
// Operator-assigned names can leak hostnames/locations to unauthenticated
// scrapers; the series intentionally stays opaque to node ids.
const info = nodes.map((node) => `${PREFIX}_node_info{node="${label(node.id)}"} 1`);
lines.push(
...sample(`${PREFIX}_node_info`, 'Registered nodes and their addresses.', 'gauge', info),
...sample(`${PREFIX}_nodes_total`, 'Number of registered nodes.', 'gauge', [
`${PREFIX}_nodes_total ${nodes.length}`,
]),
...sample(
`${PREFIX}_node_enabled`,
'Whether the node is enabled (1) or paused (0).',
'gauge',
nodes.map(
(node) => `${PREFIX}_node_enabled{node="${label(node.id)}"} ${node.enabled ? 1 : 0}`,
),
),
);
const probes = this.#sources.connections.summarize();
lines.push(
...sample(
`${PREFIX}_node_probe_total`,
'Health probes recorded in the retained window, by outcome.',
'counter',
probes.flatMap((probe) => [
`${PREFIX}_node_probe_total{node="${label(probe.instanceId)}",outcome="success"} ${probe.success}`,
`${PREFIX}_node_probe_total{node="${label(probe.instanceId)}",outcome="failed"} ${probe.failed}`,
]),
),
...sample(
`${PREFIX}_node_availability_ratio`,
'Share of successful probes in the retained window.',
'gauge',
probes.map(
(probe) =>
`${PREFIX}_node_availability_ratio{node="${label(probe.instanceId)}"} ${(probe.availabilityPercent / 100).toFixed(3)}`,
),
),
...sample(
`${PREFIX}_node_probe_duration_ms`,
'Average probe latency in the retained window.',
'gauge',
probes.map(
(probe) =>
`${PREFIX}_node_probe_duration_ms{node="${label(probe.instanceId)}"} ${number(probe.averageDurationMs)}`,
),
),
);
const identities = this.#sources.identities.list();
const pending = identities.filter((identity) => identity.status === 'pending');
lines.push(
...sample(
`${PREFIX}_node_identity_tracked`,
'Whether the node has ever reported its hardware identity.',
'gauge',
identities.map(
(identity) => `${PREFIX}_node_identity_tracked{node="${label(identity.instanceId)}"} 1`,
),
),
...sample(
`${PREFIX}_node_identity_pending`,
'1 while the identity guard holds the node and control actions are refused.',
'gauge',
identities.map(
(identity) =>
`${PREFIX}_node_identity_pending{node="${label(identity.instanceId)}"} ${identity.status === 'pending' ? 1 : 0}`,
),
),
...sample(
`${PREFIX}_identity_pending_total`,
'Nodes waiting for an operator to confirm the device behind them.',
'gauge',
[`${PREFIX}_identity_pending_total ${pending.length}`],
),
);
lines.push(
...this.#statusMetric(
`${PREFIX}_sms_outbox`,
'Queued cross-node SMS sends, by state.',
'SELECT status, COUNT(*) AS count FROM sms_outbox GROUP BY status',
['queued', 'sending', 'sent', 'failed', 'cancelled'],
),
...this.#statusMetric(
`${PREFIX}_notification_queue`,
'Central notification queue, by state.',
'SELECT status, COUNT(*) AS count FROM notification_queue GROUP BY status',
['pending', 'sending', 'succeeded', 'failed', 'cancelled'],
),
...this.#statusMetric(
`${PREFIX}_jobs`,
'Fleet operations, by state.',
'SELECT status, COUNT(*) AS count FROM jobs GROUP BY status',
[
'queued',
'running',
'cancelling',
'succeeded',
'partially-succeeded',
'failed',
'cancelled',
'unknown-result',
],
),
...this.#statusMetric(
`${PREFIX}_scheduled_tasks`,
'Automation tasks, by state.',
`SELECT CASE WHEN enabled=1 THEN 'armed' ELSE 'paused' END AS status,
COUNT(*) AS count FROM scheduled_tasks GROUP BY enabled`,
['armed', 'paused'],
),
);
const groups = this.#count('SELECT COUNT(*) AS count FROM device_groups');
const tags = this.#count('SELECT COUNT(*) AS count FROM tag_registry');
const messages = this.#count('SELECT COUNT(*) AS count FROM sms_messages');
const probed = new Set(probes.map((probe) => probe.instanceId));
const unprobed = [...ids].filter((id) => !probed.has(id)).length;
lines.push(
...sample(`${PREFIX}_device_groups`, 'Node groups.', 'gauge', [
`${PREFIX}_device_groups ${groups}`,
]),
...sample(`${PREFIX}_device_tags`, 'Labels in the tag registry.', 'gauge', [
`${PREFIX}_device_tags ${tags}`,
]),
...sample(
`${PREFIX}_sms_messages`,
'SMS rows synchronised into the central message centre.',
'gauge',
[`${PREFIX}_sms_messages ${messages}`],
),
...sample(
`${PREFIX}_node_unprobed`,
'Registered nodes without a health probe in the retained window.',
'gauge',
[`${PREFIX}_node_unprobed ${unprobed}`],
),
);
return `${lines.join('\n')}\n`;
}
#count(sql: string): number {
const row = this.#db.prepare(sql).get() as { count?: number } | undefined;
return number(row?.count ?? 0);
}
/** Every state is emitted, including the empty ones, so a query never sees a series vanish. */
#statusMetric(
name: string,
help: string,
sql: string,
states: readonly string[],
): readonly string[] {
const rows = this.#db.prepare(sql).all() as readonly CountRow[];
const counts = new Map(rows.map((row) => [String(row.status), number(row.count)]));
return sample(
name,
help,
'gauge',
states.map((state) => `${name}{status="${label(state)}"} ${counts.get(state) ?? 0}`),
);
}
}
@@ -6,11 +6,13 @@ import { migrateDatabase } from '../../infrastructure/database/migrations.js';
import type { SecretStore } from '../../infrastructure/secrets/secret-store.js';
import {
DELETE_INSTANCE_PARAMETER_SCHEMA_ID,
type BindingRelease,
DeleteInstanceOperation,
DeleteInstanceOperationError,
} from './delete-instance-operation.js';
class Store implements SecretStore {
readonly provider = 'macos-keychain';
async set() {
return '';
}
@@ -26,7 +28,9 @@ afterEach(() => {
for (const database of databases.splice(0)) database.close();
});
function fixture() {
function fixture(
releaseBinding?: (instanceId: string, requestId: string) => Promise<BindingRelease>,
) {
const db = new Database(':memory:');
db.pragma('foreign_keys=ON');
migrateDatabase(db);
@@ -43,6 +47,7 @@ function fixture() {
db,
instances,
now,
...(releaseBinding === undefined ? {} : { releaseBinding }),
idFactory: () => `operation-${++sequence}`,
tokenFactory: () => Buffer.alloc(32, 7).toString('base64url'),
});
@@ -396,4 +401,90 @@ describe('DeleteInstanceOperation', () => {
).rejects.toMatchObject({ code: 'CONFIRMATION_INVALID' });
expect(db.prepare('SELECT COUNT(*) count FROM jobs').get()).toEqual({ count: 0 });
});
it('keeps the node when the device refuses to release its binding', async () => {
const calls: string[] = [];
const { instances, operation } = fixture(async (instanceId, requestId) => {
calls.push(`${instanceId}:${requestId}`);
return { status: 'failed' };
});
const instance = await instances.create({ name: 'A', origin: 'http://192.168.1.10' });
const prepared = await operation.prepare({
operationId: 'deleteInstance',
targets: [{ instanceId: instance.id, revision: 1 }],
parameters: { parameterSchemaId: DELETE_INSTANCE_PARAMETER_SCHEMA_ID, fields: [] },
});
const job = await operation.executeDelete({
instanceId: instance.id,
revision: 1,
preparationId: prepared.id,
confirmationToken: prepared.confirmationToken,
actor: 'loopback-control-plane',
requestId: 'request-refused',
});
expect(calls).toEqual([`${instance.id}:request-refused`]);
expect(job.status).toBe('failed');
expect(job.items).toEqual([
{
id: expect.any(String),
targetId: instance.id,
state: 'failed',
error: {
type: 'about:blank',
title: 'Job item failed',
status: 502,
detail: 'The job item did not complete successfully.',
code: 'UNBIND_FAILED',
requestId: 'request-refused',
},
},
]);
expect(await instances.get(instance.id)).toBeDefined();
});
it('forgets a node that never answered, exactly as an offline device is dropped', async () => {
const { instances, operation } = fixture(async () => ({ status: 'skipped' }));
const instance = await instances.create({ name: 'A', origin: 'http://192.168.1.10' });
const prepared = await operation.prepare({
operationId: 'deleteInstance',
targets: [{ instanceId: instance.id, revision: 1 }],
parameters: { parameterSchemaId: DELETE_INSTANCE_PARAMETER_SCHEMA_ID, fields: [] },
});
const job = await operation.executeDelete({
instanceId: instance.id,
revision: 1,
preparationId: prepared.id,
confirmationToken: prepared.confirmationToken,
actor: 'loopback-control-plane',
requestId: 'request-offline',
});
expect(job.status).toBe('succeeded');
expect(job.items[0]).not.toHaveProperty('error');
expect(await instances.get(instance.id)).toBeUndefined();
});
it('treats a transport that throws while the node was online as a refusal', async () => {
const { db, instances, operation } = fixture(async () => {
throw new Error('ECONNREFUSED');
});
const instance = await instances.create({ name: 'A', origin: 'http://192.168.1.10' });
const prepared = await operation.prepare({
operationId: 'deleteInstance',
targets: [{ instanceId: instance.id, revision: 1 }],
parameters: { parameterSchemaId: DELETE_INSTANCE_PARAMETER_SCHEMA_ID, fields: [] },
});
const job = await operation.executeDelete({
instanceId: instance.id,
revision: 1,
preparationId: prepared.id,
confirmationToken: prepared.confirmationToken,
actor: 'loopback-control-plane',
requestId: 'request-thrown',
});
expect(job.items[0]).toMatchObject({ state: 'failed', error: { code: 'UNBIND_FAILED' } });
expect(db.prepare('SELECT result_code FROM job_items').get()).toEqual({
result_code: 'UNBIND_FAILED',
});
expect(await instances.get(instance.id)).toBeDefined();
});
});
@@ -1,4 +1,5 @@
import type Database from 'better-sqlite3';
import type { EventEnvelope } from '../events/event-journal.js';
import { createHash, randomBytes, randomUUID, timingSafeEqual } from 'node:crypto';
import type { Job, Preparation, PrepareOperationRequest } from '@multi-simadmin/contracts';
import { InstanceService, InstanceServiceError } from '../instances/instance-service.js';
@@ -6,6 +7,8 @@ import { InstanceService, InstanceServiceError } from '../instances/instance-ser
export const DELETE_INSTANCE_PARAMETER_SCHEMA_ID = 'deleteInstance.parameters.v1' as const;
const OPERATION_ID = 'deleteInstance';
const ACTOR = 'loopback-control-plane';
/** Job item code for "the device still holds its central binding", surfaced to the console. */
export const BINDING_REFUSAL_CODE = 'UNBIND_FAILED' as const;
const TTL_MS = 5 * 60 * 1000;
const PARAMETERS_DIGEST = createHash('sha256')
.update(JSON.stringify({ parameterSchemaId: DELETE_INSTANCE_PARAMETER_SCHEMA_ID, fields: [] }))
@@ -53,12 +56,28 @@ interface ExecuteInput {
readonly actor: string;
readonly requestId: string;
}
/**
* Answer to "release the central binding on this device before we forget it".
* `skipped` means the node never answered a heartbeat, so there is nobody to notify and the
* console deletes its own record; the device re-binds if it ever comes back. `failed` keeps the
* record, because a device that refused or timed out must not be silently orphaned.
*/
export type BindingRelease =
| { readonly status: 'released' }
| { readonly status: 'skipped' }
| { readonly status: 'failed' };
interface Options {
readonly db: Database.Database;
readonly instances: InstanceService;
/** Optional so a control plane without a device transport can still delete. */
readonly releaseBinding?: (instanceId: string, requestId: string) => Promise<BindingRelease>;
readonly now?: () => Date;
readonly idFactory?: () => string;
readonly tokenFactory?: () => string;
/** Receives job/instance envelopes so the SSE journal can invalidate fleet views. */
readonly emit?: (envelope: EventEnvelope) => void;
}
const digest = (value: string): string => createHash('sha256').update(value, 'utf8').digest('hex');
@@ -73,16 +92,20 @@ const validRevision = (value: unknown): value is number =>
export class DeleteInstanceOperation {
private readonly db: Database.Database;
private readonly instances: InstanceService;
private readonly releaseBinding: Options['releaseBinding'];
private readonly clock: () => Date;
private readonly id: () => string;
private readonly token: () => string;
private readonly emit: Options['emit'];
constructor(options: Options) {
this.db = options.db;
this.instances = options.instances;
this.releaseBinding = options.releaseBinding;
this.clock = options.now ?? (() => new Date());
this.id = options.idFactory ?? randomUUID;
this.token = options.tokenFactory ?? (() => randomBytes(32).toString('base64url'));
this.emit = options.emit;
}
async prepare(input: PrepareOperationRequest, requestId = this.id()): Promise<Preparation> {
@@ -294,6 +317,15 @@ export class DeleteInstanceOperation {
'Confirmation could not be accepted',
);
// An online device is told to release its central binding first, and a refusal keeps the
// record so the operator can retry instead of losing sight of a device that is still
// reporting to a hub. Only a node that cannot be reached at all is forgotten locally.
const release = await this.releaseBindingPort(input.instanceId, input.requestId);
if (release?.status === 'failed') {
this.closeAsFailed(ids, input.requestId, BINDING_REFUSAL_CODE);
this.#emitJob(ids.job, input.requestId);
return this.job(ids.job);
}
try {
await this.instances.delete(input.instanceId, input.revision, { allowJobHistory: true });
const finished = this.clock().toISOString();
@@ -316,6 +348,16 @@ export class DeleteInstanceOperation {
)
.run(finished, finished, ids.job);
})();
// Events only after the terminal rows are committed, so a subscriber that
// refetches on the notification sees the finished job, not 'running'.
this.#emit({
kind: 'instance',
id: this.id(),
occurredAt: this.clock().toISOString(),
requestId: input.requestId,
instanceId: input.instanceId,
});
this.#emitJob(ids.job, input.requestId);
} catch (error) {
const finished = this.clock().toISOString();
const state = error instanceof InstanceServiceError ? 'failed' : 'unknown-result';
@@ -335,17 +377,85 @@ export class DeleteInstanceOperation {
.prepare('UPDATE jobs SET status=?,finished_at=?,updated_at=? WHERE id=?')
.run(state, finished, finished, ids.job);
})();
this.#emitJob(ids.job, input.requestId);
}
return this.job(ids.job);
}
/** A transport that throws while the node was reachable counts as a refusal, never a pass. */
private async releaseBindingPort(
instanceId: string,
requestId: string,
): Promise<BindingRelease | undefined> {
if (!this.releaseBinding) return undefined;
try {
return await this.releaseBinding(instanceId, requestId);
} catch {
return { status: 'failed' };
}
}
/** Closes the job without touching the instance row, so the device stays listed and retryable. */
#emit(envelope: EventEnvelope): void {
if (!this.emit) return;
try {
this.emit(envelope);
} catch {
// A journal failure must never fail the operation itself.
}
}
#emitJob(jobId: string, requestId: string): void {
this.#emit({
kind: 'job',
id: this.id(),
occurredAt: this.clock().toISOString(),
requestId,
jobId,
});
}
private closeAsFailed(
ids: { job: string; item: string; attempt: string },
requestId: string,
resultCode: string,
): void {
const finished = this.clock().toISOString();
this.db.transaction(() => {
this.db
.prepare(
`UPDATE job_items
SET status='failed',result_code=?,error_json=?,finished_at=?,updated_at=?
WHERE id=? AND status='running'`,
)
.run(
resultCode,
JSON.stringify({ status: 502, code: resultCode, requestId }),
finished,
finished,
ids.item,
);
this.db
.prepare(
"UPDATE job_attempts SET status='failed',finished_at=? WHERE id=? AND status='running'",
)
.run(finished, ids.attempt);
this.db
.prepare(
"UPDATE jobs SET status='failed',finished_at=?,updated_at=? WHERE id=? AND status='running'",
)
.run(finished, finished, ids.job);
})();
}
reconcileInterruptedJobs(): number {
const finished = this.clock().toISOString();
return this.db.transaction(() => {
const jobs = this.db
.prepare("SELECT id FROM jobs WHERE operation_id=? AND status='running'")
.all(OPERATION_ID) as Array<{ id: string }>;
for (const { id } of jobs) {
.prepare("SELECT id, request_id FROM jobs WHERE operation_id=? AND status='running'")
.all(OPERATION_ID) as Array<{ id: string; request_id: string }>;
for (const { id, request_id: requestId } of jobs) {
this.#emitJob(id, requestId);
this.db
.prepare(
"UPDATE job_items SET status='unknown-result',result_code='INTERRUPTED',finished_at=?,updated_at=? WHERE job_id=? AND status='running'",
@@ -369,7 +479,7 @@ export class DeleteInstanceOperation {
private job(id: string): Job {
const row = this.db
.prepare(
'SELECT operation_id,status,root_job_id,retry_of_job_id,created_at FROM jobs WHERE id=?',
'SELECT operation_id,status,root_job_id,retry_of_job_id,created_at,request_id FROM jobs WHERE id=?',
)
.get(id) as {
operation_id: string;
@@ -377,13 +487,17 @@ export class DeleteInstanceOperation {
root_job_id: string;
retry_of_job_id: string | null;
created_at: string;
request_id: string;
};
const items = this.db
.prepare('SELECT id,instance_id,status FROM job_items WHERE job_id=? ORDER BY created_at,id')
.prepare(
'SELECT id,instance_id,status,result_code FROM job_items WHERE job_id=? ORDER BY created_at,id',
)
.all(id) as Array<{
id: string;
instance_id: string;
status: 'succeeded' | 'failed' | 'unknown-result';
result_code: string | null;
}>;
const attempts = this.db
.prepare(
@@ -401,7 +515,25 @@ export class DeleteInstanceOperation {
status: row.status,
...(row.retry_of_job_id ? { retryOfJobId: row.retry_of_job_id } : {}),
rootJobId: row.root_job_id,
items: items.map((item) => ({ id: item.id, targetId: item.instance_id, state: item.status })),
items: items.map((item) => ({
id: item.id,
targetId: item.instance_id,
state: item.status,
// The console reads this code straight out of the execute response to explain why the
// node it just confirmed is still in the list.
...(item.result_code === BINDING_REFUSAL_CODE
? {
error: {
type: 'about:blank',
title: 'Job item failed',
status: 502,
detail: 'The job item did not complete successfully.',
code: BINDING_REFUSAL_CODE,
requestId: row.request_id,
},
}
: {}),
})),
attempts: attempts.map((attempt) => ({
id: attempt.id,
state: attempt.status,
@@ -130,6 +130,7 @@ describe('SecureOperationExecution generic R2 slice', () => {
query: '',
contentType: '',
body: undefined,
instanceId: 'i-1',
});
await expectCode(
execution.execute(
@@ -244,10 +245,11 @@ describe('SecureOperationExecution R3 restart slice', () => {
query: '',
contentType: '',
body: undefined,
instanceId: 'i-1',
});
expect(db.prepare('SELECT risk_level FROM jobs WHERE id=?').get(job.id)).toEqual({
risk_level: 'R3',
});
expect(
db.prepare('SELECT risk_level FROM jobs WHERE id=?').get(job.id),
).toEqual({ risk_level: 'R3' });
});
it('prepares and executes system reboot only with fixed delay_seconds=3', async () => {
@@ -283,6 +285,7 @@ describe('SecureOperationExecution R3 restart slice', () => {
query: '',
contentType: 'application/json',
body: '{"delay_seconds":3}',
instanceId: 'i-1',
});
});
});
@@ -1,5 +1,6 @@
import type Database from 'better-sqlite3';
import { createHash, randomBytes, randomUUID, timingSafeEqual } from 'node:crypto';
import type { EventEnvelope } from '../events/event-journal.js';
import type {
ExecuteOperationRequest,
Job,
@@ -37,6 +38,15 @@ const EXECUTABLE_OPERATIONS: Readonly<Record<string, SecureOperationDescriptor>>
requestContentType: 'none',
parameterSchemaId: 'simadmin.58e2204.postServiceRestart.parameters.v1',
}),
postBasebandRestart: Object.freeze({
operationId: 'postBasebandRestart',
title: 'Restart Baseband',
riskLevel: 'R3',
method: 'POST',
pathTemplate: '/api/baseband/restart',
requestContentType: 'none',
parameterSchemaId: 'simadmin.58e2204.postBasebandRestart.parameters.v1',
}),
postSystemReboot: Object.freeze({
operationId: 'postSystemReboot',
title: 'Reboot System',
@@ -56,7 +66,11 @@ export const secureOperationRegistry: SecureOperationRegistry = {
},
};
const ZERO_BODY_OPERATIONS = new Set(['postNetworkRegisterAuto', 'postServiceRestart']);
const ZERO_BODY_OPERATIONS = new Set([
'postNetworkRegisterAuto',
'postServiceRestart',
'postBasebandRestart',
]);
const SYSTEM_REBOOT_OPERATION = 'postSystemReboot';
const SYSTEM_REBOOT_DELAY_SECONDS = 3;
const ACTOR = 'loopback-control-plane';
@@ -71,6 +85,8 @@ export interface SafeOperationTransportRequest {
readonly contentType: string;
/** Serialized JSON body for audited JSON operations; undefined for zero-body POSTs. */
readonly body: string | undefined;
/** Bound target used only for origin-scoped session attachment; never logged. */
readonly instanceId: string;
}
export interface SafeOperationTransport {
request(request: SafeOperationTransportRequest): Promise<{ readonly status: number }>;
@@ -106,6 +122,8 @@ interface Options {
readonly idFactory?: () => string;
readonly tokenFactory?: () => string;
readonly nonceFactory?: () => string;
/** Receives job terminal transitions so the SSE journal has a live producer. */
readonly emit?: (envelope: EventEnvelope) => void;
}
interface PreparationRow {
operation_id: string;
@@ -199,7 +217,8 @@ export class SecureOperationExecution {
let contentType = '';
let serializedBody: string | undefined;
if (ZERO_BODY_OPERATIONS.has(descriptor.operationId)) {
if (descriptor.requestContentType !== 'none' || parameters.fields.length !== 0) this.validation();
if (descriptor.requestContentType !== 'none' || parameters.fields.length !== 0)
this.validation();
} else if (descriptor.operationId === SYSTEM_REBOOT_OPERATION) {
if (descriptor.requestContentType !== 'application/json' || parameters.fields.length !== 1)
this.validation();
@@ -412,6 +431,7 @@ export class SecureOperationExecution {
bound.operation_id === SYSTEM_REBOOT_OPERATION
? JSON.stringify({ delay_seconds: SYSTEM_REBOOT_DELAY_SECONDS })
: undefined,
instanceId: bound.target_instance_id,
});
state = response.status >= 200 && response.status < 300 ? 'succeeded' : 'failed';
code = state === 'succeeded' ? 'UPSTREAM_SUCCEEDED' : 'UPSTREAM_REJECTED';
@@ -425,18 +445,40 @@ export class SecureOperationExecution {
}
}
this.finish(ids, state, code);
this.#emitTerminal(ids.job, state, requestId);
return this.job(ids.job);
}
/**
* Retention for consumed/expired/invalidated preparations. Every prepare
* (manual or a scheduled retry attempt) inserts a row, so without this the
* table grows one row per attempt forever.
*/
pruneTerminalPreparations(olderThanMs: number): number {
if (!Number.isSafeInteger(olderThanMs) || olderThanMs < 0)
throw new RangeError('olderThanMs must be a non-negative integer');
const cutoff = new Date(this.clock().getTime() - olderThanMs).toISOString();
return Number(
this.options.db
.prepare(
"DELETE FROM operation_preparations WHERE status IN ('consumed','expired','invalidated') AND updated_at <= ?",
)
.run(cutoff).changes,
);
}
reconcileInterruptedJobs(): number {
const now = this.clock().toISOString();
return this.options.db.transaction(() => {
const jobs = this.options.db
.prepare(
"SELECT id FROM jobs WHERE operation_id IN ('postNetworkRegisterAuto','postServiceRestart','postSystemReboot') AND risk_level IN ('R2','R3') AND status='running'",
"SELECT id FROM jobs WHERE operation_id IN ('postNetworkRegisterAuto','postServiceRestart','postBasebandRestart','postSystemReboot') AND risk_level IN ('R2','R3') AND status='running'",
)
.all() as Array<{ id: string }>;
for (const row of jobs) this.finishByJob(row.id, now, 'unknown-result', 'INTERRUPTED');
for (const row of jobs) {
this.finishByJob(row.id, now, 'unknown-result', 'INTERRUPTED');
this.#emitTerminal(row.id, 'unknown-result', now);
}
return jobs.length;
})();
}
@@ -462,6 +504,21 @@ export class SecureOperationExecution {
.run(state, now, now, ids.job);
})();
}
#emitTerminal(jobId: string, state: string, requestId: string): void {
if (!this.options.emit) return;
try {
this.options.emit({
kind: 'job',
id: this.id(),
occurredAt: this.clock().toISOString(),
requestId,
jobId,
});
} catch {
// A journal failure must never fail the operation itself.
}
}
private finishByJob(jobId: string, now: string, state: string, code: string): void {
this.options.db
.prepare(
@@ -0,0 +1,118 @@
import Database from 'better-sqlite3';
import { describe, expect, it } from 'vitest';
import { migrateDatabase } from '../../infrastructure/database/migrations.js';
import {
DeviceOrganizationError,
DeviceOrganizationService,
} from './device-organization-service.js';
function fixture(): { db: Database.Database; service: DeviceOrganizationService } {
const db = new Database(':memory:');
db.pragma('foreign_keys=ON');
migrateDatabase(db);
let counter = 0;
const service = new DeviceOrganizationService({
db,
idFactory: () => `group-${++counter}`,
now: () => new Date('2026-09-03T10:00:00.000Z'),
});
return { db, service };
}
function codeOf(operation: () => unknown): string {
try {
operation();
} catch (error) {
return error instanceof DeviceOrganizationError ? error.code : 'UNEXPECTED';
}
return 'NO_ERROR';
}
function insertInstance(db: Database.Database, id: string, name: string, groupId: string | null) {
db.prepare(
`INSERT INTO instances
(id,name,base_url,auth_mode,enabled,config_revision,created_at,updated_at,group_id)
VALUES (?,?,?,'password',1,1,'2026-09-03T09:00:00.000Z','2026-09-03T09:00:00.000Z',?)`,
).run(id, name, `http://${id}:8080`, groupId);
}
describe('DeviceOrganizationService', () => {
it('creates, renames, and counts group members', () => {
const { db, service } = fixture();
const created = service.createGroup({ name: '总部', description: '办公区' });
expect(created).toEqual({
id: 'group-1',
name: '总部',
description: '办公区',
deviceCount: 0,
createdAt: '2026-09-03T10:00:00.000Z',
updatedAt: '2026-09-03T10:00:00.000Z',
});
insertInstance(db, 'i-1', 'Alpha', 'group-1');
expect(service.listGroups()[0]?.deviceCount).toBe(1);
expect(service.updateGroup('group-1', { name: '上海总部' }).name).toBe('上海总部');
expect(codeOf(() => service.createGroup({ name: '上海总部' }))).toBe('DUPLICATE_GROUP');
service.deleteGroup('group-1');
expect(service.listGroups()).toEqual([]);
expect(
(
db.prepare('SELECT group_id FROM instances WHERE id=?').get('i-1') as {
group_id: string | null;
}
).group_id,
).toBeNull();
});
it('reports missing groups instead of silently succeeding', () => {
const { service } = fixture();
expect(codeOf(() => service.updateGroup('nope', { name: 'x' }))).toBe('GROUP_NOT_FOUND');
expect(codeOf(() => service.deleteGroup('nope'))).toBe('GROUP_NOT_FOUND');
});
it('synchronizes the tag registry with device tags and keeps unused entries', () => {
const { db, service } = fixture();
insertInstance(db, 'i-1', 'Alpha', null);
db.prepare('INSERT INTO instance_tags (instance_id,tag,created_at) VALUES (?,?,?)').run(
'i-1',
'office',
'2026-09-03T09:30:00.000Z',
);
service.synchronizeTags();
expect(service.listTags()).toEqual([
{
tag: 'office',
color: '',
deviceCount: 1,
createdAt: '2026-09-03T09:30:00.000Z',
updatedAt: '2026-09-03T10:00:00.000Z',
},
]);
const tagged = service.createTag({ tag: 'spare', color: 'coral' });
expect(tagged.color).toBe('coral');
service.synchronizeTags();
expect(service.listTags().map((tag) => tag.tag)).toEqual(['office', 'spare']);
});
it('detaches a deleted tag from every device', () => {
const { db, service } = fixture();
insertInstance(db, 'i-1', 'Alpha', null);
db.prepare('INSERT INTO instance_tags (instance_id,tag,created_at) VALUES (?,?,?)').run(
'i-1',
'lab',
'2026-09-03T09:30:00.000Z',
);
service.synchronizeTags();
service.deleteTag('lab');
expect(service.listTags()).toEqual([]);
expect(db.prepare('SELECT COUNT(*) count FROM instance_tags').get()).toEqual({ count: 0 });
expect(codeOf(() => service.deleteTag('lab'))).toBe('TAG_NOT_FOUND');
});
it('rejects blank and oversized names', () => {
const { service } = fixture();
expect(codeOf(() => service.createGroup({ name: ' ' }))).toBe('VALIDATION_FAILED');
expect(codeOf(() => service.createGroup({ name: 'x'.repeat(101) }))).toBe('VALIDATION_FAILED');
expect(codeOf(() => service.createTag({ tag: '' }))).toBe('VALIDATION_FAILED');
});
});
@@ -0,0 +1,239 @@
import { randomUUID } from 'node:crypto';
import type Database from 'better-sqlite3';
import type {
DeviceGroup,
DeviceGroupInput,
DeviceGroupPatch,
DeviceTag,
DeviceTagInput,
DeviceTagPatch,
} from '@multi-simadmin/contracts';
export type OrganizationErrorCode =
| 'VALIDATION_FAILED'
| 'GROUP_NOT_FOUND'
| 'TAG_NOT_FOUND'
| 'DUPLICATE_GROUP'
| 'DATABASE_FAILED';
export class DeviceOrganizationError extends Error {
constructor(
readonly code: OrganizationErrorCode,
message: string,
) {
super(message);
this.name = 'DeviceOrganizationError';
}
}
interface GroupRow {
id: string;
name: string;
description: string;
device_count: number;
created_at: string;
updated_at: string;
}
interface TagRow {
tag: string;
color: string;
device_count: number;
created_at: string;
updated_at: string;
}
export interface DeviceOrganizationOptions {
readonly db: Database.Database;
readonly idFactory?: () => string;
readonly now?: () => Date;
}
const MAX_GROUPS = 200;
const MAX_TAGS = 500;
function invalid(message: string): never {
throw new DeviceOrganizationError('VALIDATION_FAILED', message);
}
export class DeviceOrganizationService {
readonly #db: Database.Database;
readonly #id: () => string;
readonly #clock: () => Date;
constructor(options: DeviceOrganizationOptions) {
this.#db = options.db;
this.#id = options.idFactory ?? randomUUID;
this.#clock = options.now ?? (() => new Date());
}
listGroups(): readonly DeviceGroup[] {
const rows = this.#db
.prepare(
`SELECT g.id,g.name,g.description,g.created_at,g.updated_at,
(SELECT COUNT(*) FROM instances i WHERE i.group_id=g.id) device_count
FROM device_groups g ORDER BY g.name COLLATE NOCASE ASC`,
)
.all() as GroupRow[];
return Object.freeze(
rows.map((row) =>
Object.freeze({
id: row.id,
name: row.name,
description: row.description,
deviceCount: row.device_count,
createdAt: row.created_at,
updatedAt: row.updated_at,
}),
),
);
}
getGroup(groupId: string): DeviceGroup | undefined {
return this.listGroups().find((group) => group.id === groupId);
}
createGroup(input: DeviceGroupInput): DeviceGroup {
const name = requireText(input.name, '分组名称');
if (
(this.#db.prepare('SELECT COUNT(*) count FROM device_groups').get() as { count: number })
.count >= MAX_GROUPS
)
invalid('分组数量已达上限');
const description = optionalText(input.description ?? '', '分组说明', 500);
const now = this.#clock().toISOString();
const id = this.#id();
try {
this.#db
.prepare(
'INSERT INTO device_groups (id,name,description,created_at,updated_at) VALUES (?,?,?,?,?)',
)
.run(id, name, description, now, now);
} catch (error) {
if (isUnique(error)) throw new DeviceOrganizationError('DUPLICATE_GROUP', '分组名称已存在');
throw new DeviceOrganizationError('DATABASE_FAILED', '无法创建分组');
}
return this.getGroup(id)!;
}
updateGroup(groupId: string, patch: DeviceGroupPatch): DeviceGroup {
const current = this.#db
.prepare('SELECT id,name,description FROM device_groups WHERE id=?')
.get(groupId) as { id: string; name: string; description: string } | undefined;
if (!current) throw new DeviceOrganizationError('GROUP_NOT_FOUND', '分组不存在');
const name = patch.name === undefined ? current.name : requireText(patch.name, '分组名称');
const description =
patch.description === undefined
? current.description
: optionalText(patch.description, '分组说明', 500);
try {
this.#db
.prepare('UPDATE device_groups SET name=?,description=?,updated_at=? WHERE id=?')
.run(name, description, this.#clock().toISOString(), groupId);
} catch (error) {
if (isUnique(error)) throw new DeviceOrganizationError('DUPLICATE_GROUP', '分组名称已存在');
throw new DeviceOrganizationError('DATABASE_FAILED', '无法更新分组');
}
return this.getGroup(groupId)!;
}
deleteGroup(groupId: string): void {
const changed = this.#db.prepare('DELETE FROM device_groups WHERE id=?').run(groupId).changes;
if (changed === 0) throw new DeviceOrganizationError('GROUP_NOT_FOUND', '分组不存在');
}
listTags(): readonly DeviceTag[] {
const rows = this.#db
.prepare(
`SELECT r.tag,r.color,r.created_at,r.updated_at,
(SELECT COUNT(*) FROM instance_tags t WHERE t.tag=r.tag) device_count
FROM tag_registry r ORDER BY r.tag COLLATE NOCASE ASC`,
)
.all() as TagRow[];
return Object.freeze(
rows.map((row) =>
Object.freeze({
tag: row.tag,
color: row.color,
deviceCount: row.device_count,
createdAt: row.created_at,
updatedAt: row.updated_at,
}),
),
);
}
createTag(input: DeviceTagInput): DeviceTag {
const tag = requireText(input.tag, '标签');
if (
(this.#db.prepare('SELECT COUNT(*) count FROM tag_registry').get() as { count: number })
.count >= MAX_TAGS
)
invalid('标签数量已达上限');
const color = optionalText(input.color ?? '', '标签颜色', 32);
const now = this.#clock().toISOString();
try {
this.#db
.prepare('INSERT INTO tag_registry (tag,color,created_at,updated_at) VALUES (?,?,?,?)')
.run(tag, color, now, now);
} catch (error) {
if (isUnique(error)) invalid('标签已存在');
throw new DeviceOrganizationError('DATABASE_FAILED', '无法创建标签');
}
return this.listTags().find((entry) => entry.tag === tag)!;
}
updateTag(tag: string, patch: DeviceTagPatch): DeviceTag {
const existing = this.#db.prepare('SELECT tag FROM tag_registry WHERE tag=?').get(tag);
if (!existing) throw new DeviceOrganizationError('TAG_NOT_FOUND', '标签不存在');
const color = optionalText(patch.color ?? '', '标签颜色', 32);
this.#db
.prepare('UPDATE tag_registry SET color=?,updated_at=? WHERE tag=?')
.run(color, this.#clock().toISOString(), tag);
return this.listTags().find((entry) => entry.tag === tag)!;
}
/** Removing a tag also detaches it from every device that carried it. */
deleteTag(tag: string): void {
const changed = this.#db.prepare('DELETE FROM tag_registry WHERE tag=?').run(tag).changes;
if (changed === 0) throw new DeviceOrganizationError('TAG_NOT_FOUND', '标签不存在');
this.#db.prepare('DELETE FROM instance_tags WHERE tag=?').run(tag);
}
/**
* Instances may carry free-form tags that were never registered. Adopt them so the
* palette stays complete; unused registry entries are deliberate, so nothing is pruned.
*/
synchronizeTags(): void {
const now = this.#clock().toISOString();
this.#db
.prepare(
`INSERT INTO tag_registry (tag,color,created_at,updated_at)
SELECT tag,'',MIN(created_at),? FROM instance_tags
WHERE tag NOT IN (SELECT tag FROM tag_registry)
GROUP BY tag`,
)
.run(now);
}
}
function requireText(value: unknown, field: string): string {
if (typeof value !== 'string') invalid(`${field}必须是字符串`);
const trimmed = value.trim();
if (!trimmed) invalid(`${field}不能为空`);
if (trimmed.length > 100) invalid(`${field}最多 100 个字符`);
return trimmed;
}
function optionalText(value: unknown, field: string, maximum: number): string {
if (typeof value !== 'string') invalid(`${field}必须是字符串`);
const trimmed = value.trim();
if (trimmed.length > maximum) invalid(`${field}最多 ${maximum} 个字符`);
return trimmed;
}
function isUnique(error: unknown): boolean {
return error instanceof Error && /UNIQUE constraint failed/i.test(error.message);
}
@@ -1,6 +1,16 @@
import Database from 'better-sqlite3';
import { describe, expect, it } from 'vitest';
import { migrateDatabase } from '../../infrastructure/database/migrations.js';
import { InstanceResourceService, parseSim, parseStats } from './instance-resource-service.js';
import {
InstanceResourceService,
parseDevice,
parseHealth,
parseNetwork,
parseSim,
parseStats,
parseSignalStrength,
} from './instance-resource-service.js';
const response = (body: unknown) => ({
status: 200,
@@ -27,6 +37,42 @@ describe('instance resource allowlist parsing', () => {
).toEqual({ cpuPercent: 23.4, memoryPercent: 67.8, maxTemperatureCelsius: 52.6 });
});
it('falls back to version metadata exposed by stats on older SimAdmin builds', () => {
expect(
parseStats(
response({
data: {
cpu_load: { load_percent: 10 },
system: { app_version: '1.8.7', architecture: 'aarch64' },
},
}),
),
).toMatchObject({ cpuPercent: 10, version: '1.8.7', platform: 'aarch64' });
});
it('reads device uptime from the system block and rejects absurd values', () => {
expect(
parseStats(
response({
data: {
system: { uptime_seconds: 93_784, boot_id: 'secret-boot' },
},
}),
),
).toEqual({ uptimeSeconds: 93_784 });
expect(parseStats(response({ data: { uptime_seconds: 4_000_000_000 } }))).toEqual({});
expect(parseStats(response({ data: { uptime_seconds: -5 } }))).toEqual({});
expect(parseStats(response({ data: { uptime_seconds: '93784' } }))).toEqual({});
});
it('extracts the upstream SimAdmin version from the health endpoint', () => {
expect(
parseHealth(
response({ status: 'ok', version: '1.9.4', platform: 'linux-aarch64', secret: 'drop' }),
),
).toEqual({ version: '1.9.4', platform: 'linux-aarch64' });
});
it('extracts, validates, deduplicates and bounds phone numbers only', () => {
expect(
parseSim(
@@ -38,10 +84,242 @@ describe('instance resource allowlist parsing', () => {
},
}),
),
).toEqual({ phoneNumbers: ['+86 138-0000-0000'] });
).toEqual({ simPresent: true, phoneNumbers: ['+86 138-0000-0000'] });
});
it('extracts device link state without exposing hardware identity', () => {
expect(
parseDevice(response({ data: { imei: 'secret', online: true, powered: true } })),
).toEqual({ hardwareOnline: true, controlOnline: true });
expect(parseDevice(response({ data: { online: false, powered: true } }))).toEqual({
hardwareOnline: true,
controlOnline: false,
});
expect(parseDevice(response({ data: { online: true, powered: false } }))).toEqual({
hardwareOnline: false,
controlOnline: true,
});
});
it('extracts carrier, registration, access technology and signal percentage', () => {
expect(
parseNetwork(
response({
data: {
operator_name: 'China Mobile',
registration_status: 'registered_home',
technology_preference: 'LTE',
mcc: 460,
mnc: 0,
secret: 'drop',
},
}),
),
).toEqual({
carrier: '中国移动',
cellularRegistration: 'registered_home',
accessTechnology: 'LTE',
cellularOnline: true,
});
// An unassigned or foreign PLMN keeps whatever the device reported.
expect(
parseNetwork(
response({
data: {
operator_name: 'Telekom.de',
registration_status: 'registered_roaming',
mcc: 262,
mnc: 1,
},
}),
),
).toEqual({
carrier: 'Telekom.de',
cellularRegistration: 'registered_roaming',
cellularOnline: true,
});
expect(parseSignalStrength(response({ data: { strength: 80 } }))).toEqual({
signalPercent: 80,
});
});
it('marks SIM presence while retaining phone-number privacy rules', () => {
expect(
parseSim(
response({
data: {
phone_numbers: ['+8613800000000'],
imsi: 'drop',
iccid: 'drop',
},
}),
),
).toEqual({ simPresent: true, phoneNumbers: ['+8613800000000'] });
});
describe('persisted resource snapshots', () => {
it('serves the first overview after a restart from status_snapshots with zero upstream calls', async () => {
const db = new Database(':memory:');
migrateDatabase(db);
// status_snapshots carries an FK to instances; seed the node it belongs to.
db.prepare(
"INSERT INTO instances (id,name,base_url,created_at,updated_at) VALUES ('device-1','device-1','http://192.168.3.55:3000','2026-01-01T00:00:00.000Z','2026-01-01T00:00:00.000Z')",
).run();
let clock = 1_000;
let livePasses = 0;
const build = () => {
const requests: Array<{ url: string }> = [];
const service = new InstanceResourceService({
instances: { get: async () => ({ origin: 'http://192.168.3.55:3000' }) } as never,
sessions: { sessionFor: () => undefined } as never,
request: async (request: { url: string }) => {
requests.push(request);
return request.url.endsWith('/api/device')
? response({ data: { online: true } })
: { status: 503, headers: {}, body: '' };
},
db,
now: () => clock,
} as never);
return { service, requests };
};
const first = build();
await first.service.get('device-1');
livePasses = first.requests.length;
expect(livePasses).toBe(6);
// A brand-new service (process restart) reads the persisted snapshot.
const second = build();
const resources = await second.service.get('device-1');
expect(second.requests).toHaveLength(0);
expect(resources).toEqual({ controlOnline: true });
// Past the freshness window the probe goes live again and repersists.
clock += 121_000;
await second.service.get('device-1');
expect(second.requests.length).toBe(livePasses);
db.close();
});
});
describe('overview cache', () => {
const minimal = (cacheTtlMs?: number) => {
const requests: Array<{ url: string }> = [];
const service = new InstanceResourceService({
instances: { get: async () => ({ origin: 'http://192.168.3.55:3000' }) } as never,
sessions: { sessionFor: () => undefined } as never,
request: async (request: { url: string }) => {
requests.push(request);
return request.url.endsWith('/api/device')
? response({ data: { online: true } })
: { status: 503, headers: {}, body: '' };
},
...(cacheTtlMs === undefined ? {} : { cacheTtlMs }),
} as never);
return { service, requests };
};
it('serves repeat overview reads from the TTL cache and refreshes after expiry', async () => {
let clock = 1_000;
const { service, requests } = minimal();
(service as unknown as { options: { now?: () => number } }).options.now = () => clock;
await service.get('device-1');
const firstPass = requests.length;
await service.get('device-1');
expect(requests.length).toBe(firstPass);
clock += 31_000;
await service.get('device-1');
expect(requests.length).toBeGreaterThan(firstPass);
});
it('coalesces concurrent overview reads into one upstream pass', async () => {
const { service, requests } = minimal();
await Promise.all([
service.get('device-1'),
service.get('device-1'),
service.get('device-1'),
]);
expect(requests).toHaveLength(6);
});
it('probes live again when a detail view forces a refresh', async () => {
const clock = 5_000;
const { service, requests } = minimal();
(service as unknown as { options: { now?: () => number } }).options.now = () => clock;
await service.get('device-1');
const afterOverview = requests.length;
await service.get('device-1', { force: true });
expect(requests.length).toBe(afterOverview + 6);
});
it('keeps reads uncached when the TTL is disabled', async () => {
const { service, requests } = minimal(0);
await service.get('device-1');
await service.get('device-1');
expect(requests).toHaveLength(12);
});
});
it('probes passwordless instances without manufacturing a cookie', async () => {
const requests: Array<{ url: string; headers: Readonly<Record<string, string>> }> = [];
const service = new InstanceResourceService({
instances: { get: async () => ({ origin: 'http://192.168.3.55:3000' }) } as never,
sessions: { sessionFor: () => undefined } as never,
request: async (request) => {
requests.push(request);
return response(
request.url.endsWith('/api/stats')
? { data: { cpu_load: { load_percent: 12 }, memory: { used_percent: 34 } } }
: request.url.endsWith('/api/sim')
? { data: { phone_numbers: ['13800000000'] } }
: request.url.endsWith('/api/device')
? { data: { online: true, powered: true } }
: request.url.endsWith('/api/network')
? {
data: {
operator_name: 'China Mobile',
registration_status: 'registered_home',
technology_preference: 'LTE',
},
}
: request.url.endsWith('/api/network/signal-strength')
? { data: { strength: 72 } }
: { status: 'ok', version: '2.0.1', platform: 'linux' },
);
},
});
await expect(service.get('device-1')).resolves.toEqual({
cpuPercent: 12,
memoryPercent: 34,
phoneNumbers: ['13800000000'],
simPresent: true,
version: '2.0.1',
platform: 'linux',
hardwareOnline: true,
controlOnline: true,
carrier: 'China Mobile',
cellularRegistration: 'registered_home',
accessTechnology: 'LTE',
cellularOnline: true,
signalPercent: 72,
});
expect(requests.map((request) => new URL(request.url).pathname)).toEqual([
'/api/stats',
'/api/sim',
'/api/health',
'/api/device',
'/api/network',
'/api/network/signal-strength',
]);
expect(requests.map((request) => request.headers)).toEqual(
Array.from({ length: 6 }, () => ({ accept: 'application/json' })),
);
});
it('keeps partial resource behavior when device telemetry endpoints are unavailable', async () => {
const requests: Array<{ headers: Readonly<Record<string, string>> }> = [];
const service = new InstanceResourceService({
instances: { get: async () => ({ origin: 'http://192.168.3.55:3000' }) } as never,
@@ -51,7 +329,9 @@ describe('instance resource allowlist parsing', () => {
return response(
request.url.endsWith('/api/stats')
? { data: { cpu_load: { load_percent: 12 }, memory: { used_percent: 34 } } }
: { data: { phone_numbers: ['13800000000'] } },
: request.url.endsWith('/api/sim')
? { data: { phone_numbers: ['13800000000'] } }
: { status: 'ok', version: '2.0.1', platform: 'linux' },
);
},
});
@@ -59,10 +339,12 @@ describe('instance resource allowlist parsing', () => {
cpuPercent: 12,
memoryPercent: 34,
phoneNumbers: ['13800000000'],
simPresent: true,
version: '2.0.1',
platform: 'linux',
});
expect(requests.map((request) => request.headers)).toEqual([
{ accept: 'application/json' },
{ accept: 'application/json' },
...Array.from({ length: 6 }, () => ({ accept: 'application/json' })),
]);
});
@@ -1,3 +1,7 @@
import { lookupOperator } from '@multi-simadmin/contracts';
import { randomUUID } from 'node:crypto';
import type Database from 'better-sqlite3';
import type { InstanceService } from '../instances/instance-service.js';
import type {
InstanceSessionStore,
@@ -10,6 +14,18 @@ export interface InstanceResources {
readonly memoryPercent?: number;
readonly maxTemperatureCelsius?: number;
readonly phoneNumbers?: readonly string[];
readonly version?: string;
readonly platform?: string;
readonly hardwareOnline?: boolean;
readonly controlOnline?: boolean;
readonly simPresent?: boolean;
readonly carrier?: string;
readonly cellularRegistration?: string;
readonly accessTechnology?: string;
readonly cellularOnline?: boolean;
readonly signalPercent?: number;
/** Device uptime in whole seconds, reported by the node stats module. */
readonly uptimeSeconds?: number;
}
const MAX_BODY_BYTES = 32_768;
@@ -32,6 +48,39 @@ function data(response: UpstreamResponse): Record<string, unknown> | undefined {
}
}
const safeText = (value: unknown, maximum = 128): string | undefined =>
typeof value === 'string' &&
value.length > 0 &&
value.length <= maximum &&
!/[\u0000-\u001f\u007f]/u.test(value)
? value
: undefined;
const safeCode = (value: unknown, maximum = 64): string | undefined =>
typeof value === 'string' &&
value.length > 0 &&
value.length <= maximum &&
/^[A-Za-z0-9_-]+$/u.test(value)
? value
: undefined;
export function parseHealth(response: UpstreamResponse): InstanceResources {
if (response.status < 200 || response.status >= 300) return {};
if (Buffer.byteLength(response.body, 'utf8') > MAX_BODY_BYTES) return {};
try {
const root = record(JSON.parse(response.body));
if (!root) return {};
const version = safeText(root.version);
const platform = safeText(root.platform);
return {
...(version ? { version } : {}),
...(platform ? { platform } : {}),
};
} catch {
return {};
}
}
export function parseStats(response: UpstreamResponse): InstanceResources {
const value = data(response);
if (!value) return {};
@@ -43,55 +92,267 @@ export function parseStats(response: UpstreamResponse): InstanceResources {
.filter((item): item is number => item !== undefined)
: [];
const maxTemperatureCelsius = temperatures.length > 0 ? Math.max(...temperatures) : undefined;
const system = record(value.system);
const version =
safeText(value.version) ??
safeText(value.current_version) ??
safeText(system?.version) ??
safeText(system?.app_version);
const platform =
safeText(value.platform) ?? safeText(system?.platform) ?? safeText(system?.architecture);
const uptimeSeconds =
ranged(system?.uptime_seconds, 0, 680_400_000) ?? ranged(value.uptime_seconds, 0, 680_400_000);
return {
...(cpuPercent === undefined ? {} : { cpuPercent }),
...(memoryPercent === undefined ? {} : { memoryPercent }),
...(maxTemperatureCelsius === undefined ? {} : { maxTemperatureCelsius }),
...(version ? { version } : {}),
...(platform ? { platform } : {}),
...(uptimeSeconds === undefined ? {} : { uptimeSeconds: Math.floor(uptimeSeconds) }),
};
}
export function parseSim(response: UpstreamResponse): InstanceResources {
const value = data(response);
if (!value || !Array.isArray(value.phone_numbers)) return {};
const phoneNumbers = value.phone_numbers.filter(
if (!value) return {};
const numbers = Array.isArray(value.phone_numbers) ? value.phone_numbers : [];
const phoneNumbers = numbers.filter(
(item): item is string =>
typeof item === 'string' &&
item.length > 0 &&
item.length <= 32 &&
/^[+0-9 ()-]+$/.test(item),
);
return phoneNumbers.length > 0 ? { phoneNumbers: [...new Set(phoneNumbers)].slice(0, 16) } : {};
return {
simPresent: true,
...(phoneNumbers.length > 0 ? { phoneNumbers: [...new Set(phoneNumbers)].slice(0, 16) } : {}),
};
}
export function parseDevice(response: UpstreamResponse): InstanceResources {
const value = data(response);
if (!value) return {};
const hardwareOnline = typeof value.powered === 'boolean' ? value.powered : undefined;
const controlOnline = typeof value.online === 'boolean' ? value.online : undefined;
return {
...(hardwareOnline === undefined ? {} : { hardwareOnline }),
...(controlOnline === undefined ? {} : { controlOnline }),
};
}
export function parseNetwork(response: UpstreamResponse): InstanceResources {
const value = data(response);
if (!value) return {};
const reported = safeText(value.operator_name, 64);
// Devices spell the same network every which way ("China Mobile", "CHN-CMCC", "46000"); the
// shared PLMN registry settles it, exactly like the Hub does.
const known = lookupOperator(value.mcc, value.mnc);
const carrier = known?.name ?? reported;
const registration = safeCode(value.registration_status);
const accessTechnology = safeText(value.technology_preference, 32);
const cellularOnline =
registration === undefined ? undefined : registration.startsWith('registered');
return {
...(carrier ? { carrier } : {}),
...(registration ? { cellularRegistration: registration } : {}),
...(accessTechnology ? { accessTechnology } : {}),
...(cellularOnline === undefined ? {} : { cellularOnline }),
};
}
export function parseSignalStrength(response: UpstreamResponse): InstanceResources {
const signalPercent = ranged(data(response)?.strength, 0, 100);
return signalPercent === undefined ? {} : { signalPercent };
}
const DEFAULT_CACHE_TTL_MS = 30_000;
const DEFAULT_SNAPSHOT_FRESH_MS = 120_000;
const RESOURCE_SNAPSHOT_CATEGORY = 'resources';
export class InstanceResourceService {
readonly #cache = new Map<
string,
{ readonly resources: InstanceResources; readonly fetchedAt: number }
>();
readonly #inflight = new Map<string, Promise<InstanceResources>>();
constructor(
private readonly options: {
readonly instances: InstanceService;
readonly sessions: InstanceSessionStore;
readonly request: UpstreamSessionClientOptions['request'];
readonly ensureSession?: (
instanceId: string,
origin: string,
force?: boolean,
) => Promise<void>;
/** Fleet overview fan-out reads through this cache; 0 disables it. */
readonly cacheTtlMs?: number;
/**
* When set, every live fetch is persisted into status_snapshots
* (category 'resources') and a snapshot younger than snapshotFreshMs is
* served before any upstream call, so the first overview after a restart
* costs zero device requests.
*/
readonly db?: Database.Database;
readonly snapshotFreshMs?: number;
readonly now?: () => number;
},
) {}
async get(instanceId: string): Promise<InstanceResources> {
/**
* Fleet overview fans out one read per registered node; without a shared TTL
* that is six live upstream requests per device on every page render (plus a
* re-login storm when sessions expired). Detail views pass `force` to probe
* the selected device immediately.
*/
async get(
instanceId: string,
{ force = false }: { readonly force?: boolean } = {},
): Promise<InstanceResources> {
const ttl = this.options.cacheTtlMs ?? DEFAULT_CACHE_TTL_MS;
const now = this.options.now?.() ?? Date.now();
if (ttl > 0 && !force) {
const cached = this.#cache.get(instanceId);
if (cached && now - cached.fetchedAt < ttl) return cached.resources;
const persisted = this.#readSnapshot(instanceId, now);
if (persisted) {
this.#cache.set(instanceId, { resources: persisted, fetchedAt: now });
return persisted;
}
const inflight = this.#inflight.get(instanceId);
if (inflight) return inflight;
const pending = this.#fetch(instanceId)
.then((resources) => {
const fetchedAt = this.options.now?.() ?? now;
this.#cache.set(instanceId, { resources, fetchedAt });
this.#writeSnapshot(instanceId, resources, fetchedAt);
return resources;
})
.finally(() => {
this.#inflight.delete(instanceId);
});
this.#inflight.set(instanceId, pending);
return pending;
}
const resources = await this.#fetch(instanceId);
if (ttl > 0) {
this.#cache.set(instanceId, { resources, fetchedAt: now });
this.#writeSnapshot(instanceId, resources, now);
}
return resources;
}
#readSnapshot(instanceId: string, now: number): InstanceResources | undefined {
const db = this.options.db;
if (!db) return undefined;
const freshMs = this.options.snapshotFreshMs ?? DEFAULT_SNAPSHOT_FRESH_MS;
try {
const row = db
.prepare(
'SELECT payload_json, observed_at FROM status_snapshots WHERE instance_id=? AND category=?',
)
.get(instanceId, RESOURCE_SNAPSHOT_CATEGORY) as
| { payload_json: string; observed_at: string }
| undefined;
if (!row) return undefined;
const age = now - Date.parse(row.observed_at);
if (!Number.isFinite(age) || age < 0 || age >= freshMs) return undefined;
const payload: unknown = JSON.parse(row.payload_json);
if (typeof payload !== 'object' || payload === null || Array.isArray(payload))
return undefined;
return payload as InstanceResources;
} catch {
return undefined;
}
}
#writeSnapshot(instanceId: string, resources: InstanceResources, observedAt: number): void {
const db = this.options.db;
if (!db) return;
const freshMs = this.options.snapshotFreshMs ?? DEFAULT_SNAPSHOT_FRESH_MS;
const observed = new Date(observedAt).toISOString();
const expiresAt = new Date(observedAt + freshMs).toISOString();
try {
db.prepare(
`INSERT INTO status_snapshots (id, instance_id, category, state, payload_json, observed_at, expires_at, created_at)
VALUES (?, ?, ?, 'fresh', ?, ?, ?, ?)
ON CONFLICT(instance_id, category) DO UPDATE SET
id=excluded.id, state=excluded.state, payload_json=excluded.payload_json,
observed_at=excluded.observed_at, expires_at=excluded.expires_at, created_at=excluded.created_at`,
).run(
randomUUID(),
instanceId,
RESOURCE_SNAPSHOT_CATEGORY,
JSON.stringify(resources),
observed,
expiresAt,
observed,
);
} catch {
// Persistence is an optimization; a failed write must not fail the read.
}
}
async #fetch(instanceId: string): Promise<InstanceResources> {
const [instance, session] = await Promise.all([
this.options.instances.get(instanceId),
Promise.resolve(this.options.sessions.sessionFor(instanceId)),
]);
if (!instance) return {};
if (session && session.origin !== instance.origin) return {};
const get = (path: '/api/stats' | '/api/sim') =>
this.options.request({
url: `${instance.origin}${path}`,
method: 'GET',
headers: {
accept: 'application/json',
...(session ? { cookie: session.cookie } : {}),
},
});
const [stats, sim] = await Promise.allSettled([get('/api/stats'), get('/api/sim')]);
if (!session && this.options.ensureSession) {
try {
await this.options.ensureSession(instanceId, instance.origin);
} catch {
// Passwordless and temporarily unavailable credentials still permit anonymous reads.
}
}
const paths = [
'/api/stats',
'/api/sim',
'/api/health',
'/api/device',
'/api/network',
'/api/network/signal-strength',
] as const;
const readAll = () => {
const activeSession = this.options.sessions.sessionFor(instanceId);
return Promise.allSettled(
paths.map((path) =>
this.options.request({
url: `${instance.origin}${path}`,
method: 'GET',
headers: {
accept: 'application/json',
...(activeSession?.origin === instance.origin
? { cookie: activeSession.cookie }
: {}),
},
}),
),
);
};
let results = await readAll();
const unauthorized = results.some(
(result) => result.status === 'fulfilled' && [401, 403].includes(result.value.status),
);
if (unauthorized && this.options.ensureSession) {
try {
await this.options.ensureSession(instanceId, instance.origin, true);
results = await readAll();
} catch {
// Return the safe partial result when re-authentication is unavailable.
}
}
const [stats, sim, health, device, network, signal] = results;
return {
...(stats.status === 'fulfilled' ? parseStats(stats.value) : {}),
...(sim.status === 'fulfilled' ? parseSim(sim.value) : {}),
...(stats?.status === 'fulfilled' ? parseStats(stats.value) : {}),
...(sim?.status === 'fulfilled' ? parseSim(sim.value) : {}),
...(health?.status === 'fulfilled' ? parseHealth(health.value) : {}),
...(device?.status === 'fulfilled' ? parseDevice(device.value) : {}),
...(network?.status === 'fulfilled' ? parseNetwork(network.value) : {}),
...(signal?.status === 'fulfilled' ? parseSignalStrength(signal.value) : {}),
};
}
}
@@ -47,6 +47,18 @@ export interface StatusSnapshotServiceOptions {
readonly maxStaleMs: number;
readonly now?: () => Date;
readonly idFactory?: () => string;
/** Observes every probe outcome, including ones fenced from persistence by a revision change. */
readonly onProbe?: (result: HealthProbeResult) => void;
}
export interface HealthProbeResult {
readonly instanceId: string;
readonly outcome: 'success' | 'stale' | 'failed' | 'unsupported';
readonly state: HealthSnapshotState;
readonly errorCode: string | null;
readonly httpStatus: number | null;
readonly durationMs: number;
readonly observedAt: string;
}
export class StatusSnapshotError extends Error {
@@ -193,6 +205,7 @@ export class StatusSnapshotService {
readonly #maxStaleMs: number;
readonly #now: () => Date;
readonly #id: () => string;
readonly #onProbe: ((result: HealthProbeResult) => void) | undefined;
readonly #latestGeneration = new Map<string, number>();
#nextGeneration = 0;
@@ -210,6 +223,7 @@ export class StatusSnapshotService {
this.#maxStaleMs = options.maxStaleMs;
this.#now = options.now ?? (() => new Date());
this.#id = options.idFactory ?? randomUUID;
this.#onProbe = options.onProbe;
}
async refreshHealth(instanceId: string): Promise<HealthSnapshot> {
@@ -247,6 +261,7 @@ export class StatusSnapshotService {
);
const { snapshot } = classified;
this.#observe(instanceId, snapshot, classified.persisted);
if (this.#latestGeneration.get(instanceId) === generation) {
if (
isValidOwner(instanceId, currentOwner) &&
@@ -260,6 +275,35 @@ export class StatusSnapshotService {
return Object.freeze({ ...snapshot, payload: Object.freeze({ ...snapshot.payload }) });
}
/** A failed connection log write must never take a health probe down with it. */
#observe(instanceId: string, snapshot: HealthSnapshot, persisted: PersistedHealthEnvelope): void {
if (!this.#onProbe) return;
const status = persisted.httpStatus;
const outcome: HealthProbeResult['outcome'] =
persisted.errorCode === 'UNSUPPORTED'
? 'unsupported'
: snapshot.state === 'fresh'
? 'success'
: snapshot.state === 'stale'
? 'stale'
: status !== null && status >= 200 && status < 300
? 'success'
: 'failed';
try {
this.#onProbe({
instanceId,
outcome,
state: snapshot.state,
errorCode: persisted.errorCode,
httpStatus: status,
durationMs: persisted.durationMs,
observedAt: snapshot.observedAt,
});
} catch {
// Diagnostics are advisory; the snapshot result stands on its own.
}
}
#previous(instanceId: string): SnapshotRow | undefined {
return this.#db
.prepare(
@@ -0,0 +1,270 @@
import Database from 'better-sqlite3';
import { mkdtemp, readFile, readdir, rm, writeFile } from 'node:fs/promises';
import { join } from 'node:path';
import { tmpdir } from 'node:os';
import { afterEach, describe, expect, it } from 'vitest';
import { migrateDatabase } from '../../infrastructure/database/migrations.js';
import {
BACKUP_COMPONENT_KEYS,
ComponentBackupError,
ComponentBackupService,
} from './component-backup-service.js';
const roots: string[] = [];
afterEach(async () => {
await Promise.all(roots.splice(0).map((root) => rm(root, { recursive: true, force: true })));
});
function database(): Database.Database {
const db = new Database(':memory:');
db.pragma('foreign_keys = ON');
migrateDatabase(db);
return db;
}
async function directory(): Promise<string> {
const root = await mkdtemp(join(tmpdir(), 'component-backup-'));
roots.push(root);
return root;
}
function seed(db: Database.Database, tag: string): void {
db.prepare('INSERT INTO device_groups (id,name,created_at,updated_at) VALUES (?,?,?,?)').run(
`group-${tag}`,
`机房 ${tag}`,
'2026-01-01T00:00:00.000Z',
'2026-01-01T00:00:00.000Z',
);
db.prepare(
`INSERT INTO instances (id,name,base_url,auth_mode,enabled,config_revision,created_at,updated_at)
VALUES (?,?,?,?,?,?,?,?)`,
).run(
`node-${tag}`,
`节点 ${tag}`,
`http://192.168.1.${tag}:8080`,
'password',
1,
1,
'2026-01-01T00:00:00.000Z',
'2026-01-01T00:00:00.000Z',
);
db.prepare('INSERT INTO instance_tags (instance_id,tag,created_at) VALUES (?,?,?)').run(
`node-${tag}`,
`tag-${tag}`,
'2026-01-01T00:00:00.000Z',
);
db.prepare('INSERT INTO tag_registry (tag,created_at,updated_at) VALUES (?,?,?)').run(
`tag-${tag}`,
'2026-01-01T00:00:00.000Z',
'2026-01-01T00:00:00.000Z',
);
db.prepare(
`INSERT INTO audit_events
(id,actor,operation_id,risk_level,request_id,parameters_summary_json,result_code,duration_ms,created_at)
VALUES (?,?,?,?,?,?,?,?,?)`,
).run(
`audit-${tag}`,
'operator',
'probe',
'R0',
`${tag}-req`,
'[]',
'succeeded',
1,
'2026-01-01T00:00:00.000Z',
);
db.prepare(
`INSERT INTO jobs
(id,root_job_id,operation_id,risk_level,status,requested_by,request_id,parameters_digest,created_at,updated_at)
VALUES (?,?,?,?,'succeeded',?,?,?,?,?)`,
).run(
`job-${tag}`,
`job-${tag}`,
'probe',
'R0',
'operator',
`${tag}-req`,
'digest',
'2026-01-01T00:00:00.000Z',
'2026-01-01T00:00:00.000Z',
);
db.prepare(
`INSERT INTO job_items
(id,job_id,instance_id,attempt_number,status,created_at,updated_at)
VALUES (?,?,?,1,'succeeded',?,?)`,
).run(
`item-${tag}`,
`job-${tag}`,
`node-${tag}`,
'2026-01-01T00:00:00.000Z',
'2026-01-01T00:00:00.000Z',
);
db.prepare(
`INSERT INTO job_attempts (id,job_id,status,started_at,created_at)
VALUES (?,?,?,?,?)`,
).run(
`attempt-${tag}`,
`job-${tag}`,
'succeeded',
'2026-01-01T00:00:00.000Z',
'2026-01-01T00:00:00.000Z',
);
}
describe('ComponentBackupService', () => {
it('lists every catalog unit with a live row count', async () => {
const db = database();
seed(db, '1');
const service = new ComponentBackupService(db, {
version: 'test',
backupDirectory: await directory(),
});
const catalog = service.catalog();
expect(catalog.map((entry) => entry.key)).toEqual([...BACKUP_COMPONENT_KEYS]);
const devices = catalog.find((entry) => entry.key === 'devices');
expect(devices?.rows).toBe(4);
expect(devices?.label).toBe('设备与分组');
});
it('creates a component backup that the list endpoint can verify', async () => {
const db = database();
seed(db, '1');
const backupDirectory = await directory();
const service = new ComponentBackupService(db, { version: 'test', backupDirectory });
const created = await service.create(['devices', 'audit'], '升级前快照');
expect(created.integrity).toBe('ok');
expect(created.note).toBe('升级前快照');
expect(created.components.map((entry) => entry.key)).toEqual(['devices', 'audit']);
expect((await readdir(backupDirectory)).length).toBe(1);
const listed = await service.list();
expect(listed).toHaveLength(1);
expect(listed[0]?.filename).toBe(created.filename);
expect(listed[0]?.integrity).toBe('ok');
expect(listed[0]?.compatible).toBe(true);
expect(listed[0]?.components.find((entry) => entry.key === 'audit')?.rows).toBe(1);
});
it('merges a selected component into a fresh database and leaves the rest alone', async () => {
const source = database();
seed(source, '1');
seed(source, '2');
const backupDirectory = await directory();
const service = new ComponentBackupService(source, { version: 'test', backupDirectory });
const created = await service.create(['devices', 'audit']);
const target = database();
seed(target, '9');
const restoreService = new ComponentBackupService(target, {
version: 'test',
backupDirectory,
});
const result = await restoreService.restore(created.filename, ['devices']);
expect(result.written).toEqual({ devices: 8 });
// The state right before the merge is snapshotted first, so the restore stays reversible.
expect(result.safetyBackup).not.toBe(created.filename);
const safety = await restoreService.preview(result.safetyBackup);
expect(safety.note).toBe(`恢复前自动备份:${created.filename}`);
expect(safety.automatic).toBe(true);
expect(safety.components.map((component) => component.key)).toEqual(['devices']);
const instances = target.prepare('SELECT id FROM instances ORDER BY id').all() as {
id: string;
}[];
expect(instances.map((entry) => entry.id)).toEqual(['node-1', 'node-2', 'node-9']);
// The audit component was not selected, so the target keeps only its own row.
const audits = target.prepare('SELECT id FROM audit_events ORDER BY id').all() as {
id: string;
}[];
expect(audits.map((entry) => entry.id)).toEqual(['audit-9']);
});
it('refuses a tampered archive', async () => {
const db = database();
seed(db, '1');
const backupDirectory = await directory();
const service = new ComponentBackupService(db, { version: 'test', backupDirectory });
const created = await service.create(['audit']);
const path = join(backupDirectory, created.filename);
const document = JSON.parse(await readFile(path, 'utf8')) as {
components: { audit: { audit_events: { rows: unknown[][] } } };
};
const row = document.components.audit.audit_events.rows[0];
if (!row) throw new Error('expected a serialized audit row');
row[3] = 'R3';
await writeFile(path, JSON.stringify(document), 'utf8');
await expect(service.preview(created.filename)).rejects.toThrowError(ComponentBackupError);
await expect(service.restore(created.filename, ['audit'])).rejects.toMatchObject({
code: 'INTEGRITY_FAILED',
});
});
it('rejects unknown component keys and unsafe filenames', async () => {
const db = database();
const service = new ComponentBackupService(db, {
version: 'test',
backupDirectory: await directory(),
});
await expect(service.create(['nope' as never])).rejects.toMatchObject({
code: 'VALIDATION_FAILED',
});
await expect(service.preview('../secret.json')).rejects.toMatchObject({
code: 'VALIDATION_FAILED',
});
await expect(service.preview('multi-simadmin-components-missing.json')).rejects.toMatchObject({
code: 'NOT_FOUND',
});
});
it('runs one scheduled backup per period and prunes to the retained count', async () => {
const db = database();
seed(db, '1');
const backupDirectory = await directory();
let clock = new Date('2026-03-01T03:35:00.000Z');
const service = new ComponentBackupService(db, {
version: 'test',
backupDirectory,
now: () => clock,
});
service.updateAutoSettings({
enabled: true,
components: ['devices'],
timeOfDay: '02:00',
weekday: -1,
maximumCount: 2,
});
expect(await service.runDueAutoBackups()).toMatchObject({ automatic: true });
expect(await service.runDueAutoBackups()).toBeNull();
clock = new Date('2026-03-02T03:31:00.000Z');
expect(await service.runDueAutoBackups()).not.toBeNull();
clock = new Date('2026-03-03T03:31:00.000Z');
expect(await service.runDueAutoBackups()).not.toBeNull();
const files = (await readdir(backupDirectory)).filter((name) => name.includes('-auto-'));
expect(files).toHaveLength(2);
expect(service.autoSettings()).toMatchObject({ enabled: true, maximumCount: 2, weekday: -1 });
expect(service.autoSettings().lastRunAt).toBe('2026-03-03T03:31:00.000Z');
});
it('keeps the auto settings readable when the stored payload is damaged', async () => {
const db = database();
db.prepare(
'INSERT INTO app_settings (key,value_json,created_at,updated_at) VALUES (?,?,?,?)',
).run(
'system.component_backup.auto',
'{not json',
'2026-01-01T00:00:00.000Z',
'2026-01-01T00:00:00.000Z',
);
const service = new ComponentBackupService(db, {
version: 'test',
backupDirectory: await directory(),
});
expect(service.autoSettings()).toMatchObject({
enabled: false,
timeOfDay: '02:00',
components: ['devices', 'notifications', 'automation'],
});
});
});
@@ -0,0 +1,649 @@
import { createHash } from 'node:crypto';
import { mkdir, readFile, readdir, rm, stat, writeFile } from 'node:fs/promises';
import { basename, join } from 'node:path';
import type { SqliteDatabase } from '../../infrastructure/database/database.js';
/**
* Backup units the console can export and merge back. Each unit is closed under its own foreign
* keys, so restoring one never leaves a dangling reference behind.
*/
export type BackupComponentKey =
| 'devices'
| 'notifications'
| 'notificationRecords'
| 'automation'
| 'automationRecords'
| 'jobs'
| 'sms'
| 'settings'
| 'audit';
interface ComponentSpec {
readonly label: string;
readonly description: string;
/** Parent tables first so an upsert never writes a child before its owner exists. */
readonly tables: readonly string[];
}
export const BACKUP_COMPONENTS: Readonly<Record<BackupComponentKey, ComponentSpec>> = Object.freeze(
{
devices: {
label: '设备与分组',
description: '节点地址、分组、标签与能力快照;密钥材料不会被写入备份文件。',
tables: ['device_groups', 'tag_registry', 'instances', 'instance_tags', 'capabilities'],
},
notifications: {
label: '通知配置',
description: '通知渠道与转发规则。',
tables: ['notification_channels', 'notification_rules'],
},
notificationRecords: {
label: '通知记录',
description: '待办队列与历史投递结果。',
tables: ['notification_queue', 'notification_deliveries'],
},
automation: {
label: '自动化配置',
description: '定时任务定义与执行计划。',
tables: ['scheduled_tasks'],
},
automationRecords: {
label: '自动化记录',
description: '每次定时执行的状态与结果。',
tables: ['scheduled_runs'],
},
jobs: {
label: '命令历史',
description: '集中下发的设备操作命令与逐设备执行明细。',
tables: ['jobs', 'job_items', 'job_attempts'],
},
sms: {
label: '短信记录',
description: '集中保存的跨设备短信正文与会话。',
tables: ['sms_messages'],
},
settings: {
label: '系统设置',
description: '保留策略、备份计划与控制台安全配置。',
tables: ['app_settings', 'console_auth_config'],
},
audit: {
label: '操作审计',
description: '设备操作审计账本。',
tables: ['audit_events'],
},
} as const,
);
export const BACKUP_COMPONENT_KEYS = Object.freeze(
Object.keys(BACKUP_COMPONENTS) as BackupComponentKey[],
);
const FORMAT = 'multi-simadmin-component-backup';
const FORMAT_VERSION = 1;
const AUTO_SETTINGS_KEY = 'system.component_backup.auto';
const MAX_NOTE_LENGTH = 120;
const MAX_FILE_BYTES = 256 * 1024 * 1024;
const BACKUP_FILENAME = /^multi-simadmin-components-(?:auto-)?[A-Za-z0-9._-]{1,120}\.json$/u;
interface TableSnapshot {
readonly columns: readonly string[];
readonly rows: readonly (readonly unknown[])[];
}
interface BackupDocument {
readonly format: string;
readonly formatVersion: number;
readonly appVersion: string;
readonly createdAt: string;
readonly note: string;
readonly components: Readonly<Partial<Record<BackupComponentKey, Record<string, TableSnapshot>>>>;
readonly digest: string;
}
export interface BackupComponentSummary {
readonly key: BackupComponentKey;
readonly label: string;
readonly description: string;
readonly rows: number;
}
export interface BackupSummary {
readonly filename: string;
readonly createdAt: string;
readonly sizeBytes: number;
readonly appVersion: string;
readonly formatVersion: number;
readonly note: string;
readonly automatic: boolean;
readonly integrity: 'ok' | 'failed';
readonly compatible: boolean;
readonly components: readonly BackupComponentSummary[];
}
export interface AutoBackupSettings {
readonly enabled: boolean;
readonly components: readonly BackupComponentKey[];
readonly timeOfDay: string;
readonly weekday: number;
readonly maximumCount: number;
readonly lastRunAt: string | null;
}
export interface RestoreResult {
readonly written: Partial<Record<BackupComponentKey, number>>;
/** Filename of the snapshot taken just before the merge, so a restore is always reversible. */
readonly safetyBackup: string;
}
/** Download metadata without parsing an already validated archive again. */
export interface ComponentBackupFile {
readonly filename: string;
readonly path: string;
readonly sizeBytes: number;
readonly createdAt: string;
}
export class ComponentBackupError extends Error {
constructor(
readonly code:
| 'VALIDATION_FAILED'
| 'NOT_FOUND'
| 'INTEGRITY_FAILED'
| 'INCOMPATIBLE'
| 'TOO_LARGE',
) {
super(code);
this.name = 'ComponentBackupError';
}
}
const isRecord = (value: unknown): value is Record<string, unknown> =>
typeof value === 'object' && value !== null && !Array.isArray(value);
function digestOf(value: unknown): string {
return createHash('sha256').update(JSON.stringify(value), 'utf8').digest('hex');
}
function tableExists(db: SqliteDatabase, table: string): boolean {
return (
(db.prepare("SELECT name FROM sqlite_master WHERE type = 'table' AND name = ?").get(table) as
| { name: string }
| undefined) !== undefined
);
}
function primaryKeyOf(db: SqliteDatabase, table: string): readonly string[] {
const info = db.prepare(`PRAGMA table_info(${table})`).all() as { name: string; pk: number }[];
return info.filter((column) => column.pk > 0).map((column) => column.name);
}
function normalizeComponents(value: unknown): BackupComponentKey[] {
if (!Array.isArray(value) || value.length === 0)
throw new ComponentBackupError('VALIDATION_FAILED');
const seen = new Set<BackupComponentKey>();
for (const entry of value) {
if (typeof entry !== 'string' || !(entry in BACKUP_COMPONENTS))
throw new ComponentBackupError('VALIDATION_FAILED');
seen.add(entry as BackupComponentKey);
}
// Emit in catalog order so the serialized document, and therefore its digest, is stable.
return BACKUP_COMPONENT_KEYS.filter((key) => seen.has(key));
}
function normalizeNote(value: unknown): string {
if (value === undefined || value === null) return '';
if (typeof value !== 'string') throw new ComponentBackupError('VALIDATION_FAILED');
const trimmed = value.trim();
if (trimmed.length > MAX_NOTE_LENGTH) throw new ComponentBackupError('VALIDATION_FAILED');
return trimmed;
}
function normalizeTimeOfDay(value: unknown): string {
if (typeof value !== 'string' || !/^([01]\d|2[0-3]):[0-5]\d$/u.test(value))
throw new ComponentBackupError('VALIDATION_FAILED');
return value;
}
export interface ComponentBackupOptions {
readonly version: string;
readonly backupDirectory: string;
readonly now?: () => Date;
}
/** Component-scoped export and merge, mirroring the Hub backup centre. */
export class ComponentBackupService {
readonly #db: SqliteDatabase;
readonly #version: string;
readonly #directory: string;
readonly #now: () => Date;
constructor(db: SqliteDatabase, options: ComponentBackupOptions) {
this.#db = db;
this.#version = options.version;
this.#directory = options.backupDirectory;
this.#now = options.now ?? (() => new Date());
}
/** Live row counts for every unit the console knows how to back up. */
catalog(): readonly BackupComponentSummary[] {
return BACKUP_COMPONENT_KEYS.map((key) => ({
key,
label: BACKUP_COMPONENTS[key].label,
description: BACKUP_COMPONENTS[key].description,
rows: this.#countComponent(key),
}));
}
async create(
components: readonly BackupComponentKey[],
note: unknown = '',
automatic = false,
): Promise<BackupSummary> {
const keys = normalizeComponents(components);
const comment = normalizeNote(note);
const createdAt = this.#now().toISOString();
const payload: Record<string, Record<string, TableSnapshot>> = {};
for (const key of keys) {
const tables: Record<string, TableSnapshot> = {};
for (const table of BACKUP_COMPONENTS[key].tables) {
const snapshot = this.#snapshot(table);
if (snapshot) tables[table] = snapshot;
}
payload[key] = tables;
}
const body = { format: FORMAT, formatVersion: FORMAT_VERSION, createdAt, components: payload };
const document: BackupDocument = {
format: FORMAT,
formatVersion: FORMAT_VERSION,
appVersion: this.#version,
createdAt,
note: comment,
components: payload,
digest: digestOf(body),
};
const serialized = JSON.stringify(document);
if (Buffer.byteLength(serialized, 'utf8') > MAX_FILE_BYTES)
throw new ComponentBackupError('TOO_LARGE');
await mkdir(this.#directory, { recursive: true });
const filename = `multi-simadmin-components-${automatic ? 'auto-' : ''}${createdAt.replaceAll(/[:.]/gu, '-')}.json`;
await writeFile(join(this.#directory, filename), serialized, { mode: 0o600 });
return this.#summarize(filename, document, Buffer.byteLength(serialized, 'utf8'));
}
async list(): Promise<readonly BackupSummary[]> {
await mkdir(this.#directory, { recursive: true });
const names = (await readdir(this.#directory))
.filter((name) => BACKUP_FILENAME.test(name))
.sort()
.reverse();
const items: BackupSummary[] = [];
for (const filename of names) {
const parsed = await this.#read(filename).catch(() => undefined);
const details = await stat(join(this.#directory, filename)).catch(() => undefined);
if (!details?.isFile()) continue;
items.push(
parsed
? await this.#summarize(filename, parsed.document, details.size)
: {
filename,
createdAt: details.mtime.toISOString(),
sizeBytes: details.size,
appVersion: '',
formatVersion: 0,
note: '',
automatic: filename.includes('-auto-'),
integrity: 'failed',
compatible: false,
components: [],
},
);
}
return items;
}
async preview(filename: string): Promise<BackupSummary> {
const { document, sizeBytes } = await this.#read(this.#safeName(filename));
return this.#summarize(this.#safeName(filename), document, sizeBytes);
}
/** Merges the selected units back in; untouched components and rows stay exactly as they were. */
async restore(
filename: string,
components: readonly BackupComponentKey[],
): Promise<RestoreResult> {
const safe = this.#safeName(filename);
const keys = normalizeComponents(components);
const { document } = await this.#read(safe);
if (document.formatVersion > FORMAT_VERSION) throw new ComponentBackupError('INCOMPATIBLE');
// Snapshot the units that are about to be overwritten first: a restore must always be
// reversible, and the state right before the merge is the only thing worth returning to.
const safety = await this.create(keys, `恢复前自动备份:${safe}`, true);
const written: Partial<Record<BackupComponentKey, number>> = {};
const restore = this.#db.transaction(() => {
for (const key of keys) {
const tables = isRecord(document.components[key])
? (document.components[key] as Record<string, TableSnapshot>)
: {};
let count = 0;
for (const table of BACKUP_COMPONENTS[key].tables) {
const snapshot = tables[table];
if (!snapshot || !Array.isArray(snapshot.columns) || !Array.isArray(snapshot.rows))
continue;
count += this.#merge(table, snapshot);
}
written[key] = count;
}
});
restore();
return { written, safetyBackup: safety.filename };
}
/** Where the backup files live, shown so an operator can mirror the folder off-console. */
directory(): string {
return this.#directory;
}
async remove(filename: string): Promise<{ filename: string }> {
await rm(join(this.#directory, this.#safeName(filename)), { force: true });
return { filename: this.#safeName(filename) };
}
/** Resolves a listed archive to a path that stays inside the backup directory. */
async backupFile(filename: string): Promise<ComponentBackupFile | undefined> {
const safe = this.#safeName(filename);
const path = join(this.#directory, safe);
const details = await stat(path).catch((error: NodeJS.ErrnoException) => {
if (error.code === 'ENOENT') return undefined;
throw error;
});
if (!details?.isFile()) return undefined;
return {
filename: safe,
path,
sizeBytes: details.size,
createdAt: details.mtime.toISOString(),
};
}
autoSettings(): AutoBackupSettings {
return this.#readAuto();
}
updateAutoSettings(value: unknown): AutoBackupSettings {
const source = isRecord(value) ? value : {};
if (typeof source.enabled !== 'boolean') throw new ComponentBackupError('VALIDATION_FAILED');
const weekday = source.weekday;
if (
typeof weekday !== 'number' ||
!Number.isSafeInteger(weekday) ||
weekday < -1 ||
weekday > 6
)
throw new ComponentBackupError('VALIDATION_FAILED');
const maximumCount = source.maximumCount;
if (
typeof maximumCount !== 'number' ||
!Number.isSafeInteger(maximumCount) ||
maximumCount < 1 ||
maximumCount > 500
)
throw new ComponentBackupError('VALIDATION_FAILED');
const settings: AutoBackupSettings = {
enabled: source.enabled,
components: normalizeComponents(source.components),
timeOfDay: normalizeTimeOfDay(source.timeOfDay),
weekday,
maximumCount,
lastRunAt: this.#readAuto().lastRunAt,
};
const now = this.#now().toISOString();
this.#db
.prepare(
`INSERT INTO app_settings (key, value_json, created_at, updated_at)
VALUES (?, ?, ?, ?)
ON CONFLICT(key) DO UPDATE SET value_json = excluded.value_json, updated_at = excluded.updated_at`,
)
.run(AUTO_SETTINGS_KEY, JSON.stringify({ ...settings, lastRunAt: undefined }), now, now);
return settings;
}
/** Creates the scheduled backup when the current period has not run yet. */
async runDueAutoBackups(): Promise<BackupSummary | null> {
const settings = this.#readAuto();
if (!settings.enabled || settings.components.length === 0) return null;
const dueAt = this.#mostRecentOccurrence(settings.timeOfDay, settings.weekday);
if (settings.lastRunAt && Date.parse(settings.lastRunAt) >= dueAt.getTime()) return null;
const summary = await this.create(settings.components, '定时自动备份', true);
const stamped = this.#now().toISOString();
this.#db
.prepare(
`INSERT INTO app_settings (key, value_json, created_at, updated_at)
VALUES (?, ?, ?, ?)
ON CONFLICT(key) DO UPDATE SET value_json = excluded.value_json, updated_at = excluded.updated_at`,
)
.run(
AUTO_SETTINGS_KEY,
JSON.stringify({ ...settings, lastRunAt: stamped }),
stamped,
stamped,
);
await this.#prune(settings.maximumCount);
return summary;
}
#countComponent(key: BackupComponentKey): number {
let total = 0;
for (const table of BACKUP_COMPONENTS[key].tables) {
if (!tableExists(this.#db, table)) continue;
const row = this.#db.prepare(`SELECT COUNT(*) AS count FROM ${table}`).get() as
| { count: number }
| undefined;
total += Number(row?.count ?? 0);
}
return total;
}
#snapshot(table: string): TableSnapshot | undefined {
if (!tableExists(this.#db, table)) return undefined;
const info = this.#db.prepare(`PRAGMA table_info(${table})`).all() as { name: string }[];
const columns = info.map((column) => column.name);
const rows = this.#db.prepare(`SELECT * FROM ${table} ORDER BY rowid`).all() as Record<
string,
unknown
>[];
return {
columns,
rows: rows.map((row) => columns.map((column) => row[column] ?? null)),
};
}
/** Upserts one table; a conflict updates every non-key column instead of replacing the row. */
#merge(table: string, snapshot: TableSnapshot): number {
if (!tableExists(this.#db, table)) return 0;
const columns = snapshot.columns.filter((column) => typeof column === 'string');
if (columns.length === 0) return 0;
const keys = primaryKeyOf(this.#db, table);
const placeholders = columns.map(() => '?').join(', ');
const quoted = columns.map((column) => `"${column.replaceAll('"', '""')}"`).join(', ');
let sql = `INSERT INTO ${table} (${quoted}) VALUES (${placeholders})`;
const updates = columns.filter((column) => !keys.includes(column));
if (keys.length > 0 && updates.length > 0)
sql += ` ON CONFLICT(${keys.map((key) => `"${key}"`).join(', ')}) DO UPDATE SET ${updates
.map((column) => `"${column}" = excluded."${column}"`)
.join(', ')}`;
else if (keys.length > 0)
sql += ` ON CONFLICT(${keys.map((key) => `"${key}"`).join(', ')}) DO NOTHING`;
const statement = this.#db.prepare(sql);
let written = 0;
for (const row of snapshot.rows) {
if (!Array.isArray(row) || row.length !== columns.length) continue;
statement.run(...row.map((value) => (typeof value === 'boolean' ? Number(value) : value)));
written += 1;
}
return written;
}
#safeName(filename: string): string {
if (!BACKUP_FILENAME.test(filename) || basename(filename) !== filename)
throw new ComponentBackupError('VALIDATION_FAILED');
return filename;
}
async #read(filename: string): Promise<{ document: BackupDocument; sizeBytes: number }> {
const path = join(this.#directory, filename);
const content = await readFile(path).catch((error: NodeJS.ErrnoException) => {
if (error.code === 'ENOENT') throw new ComponentBackupError('NOT_FOUND');
throw error;
});
if (content.byteLength > MAX_FILE_BYTES) throw new ComponentBackupError('TOO_LARGE');
let parsed: unknown;
try {
parsed = JSON.parse(content.toString('utf8'));
} catch {
throw new ComponentBackupError('INTEGRITY_FAILED');
}
const document = this.#validate(parsed);
return { document, sizeBytes: content.byteLength };
}
#validate(value: unknown): BackupDocument {
if (!isRecord(value) || value.format !== FORMAT)
throw new ComponentBackupError('INTEGRITY_FAILED');
const formatVersion = Number(value.formatVersion);
const components = isRecord(value.components) ? value.components : {};
const document: BackupDocument = {
format: FORMAT,
formatVersion: Number.isSafeInteger(formatVersion) ? formatVersion : 0,
appVersion: typeof value.appVersion === 'string' ? value.appVersion : '',
createdAt: typeof value.createdAt === 'string' ? value.createdAt : '',
note: typeof value.note === 'string' ? value.note : '',
components: components as BackupDocument['components'],
digest: typeof value.digest === 'string' ? value.digest : '',
};
const expected = digestOf({
format: document.format,
formatVersion: document.formatVersion,
createdAt: document.createdAt,
components: document.components,
});
if (document.digest !== expected) throw new ComponentBackupError('INTEGRITY_FAILED');
return document;
}
async #summarize(
filename: string,
document: BackupDocument,
sizeBytes: number,
): Promise<BackupSummary> {
const components: BackupComponentSummary[] = [];
for (const key of BACKUP_COMPONENT_KEYS) {
const tables = isRecord(document.components[key])
? (document.components[key] as Record<string, TableSnapshot>)
: undefined;
if (!tables) continue;
let rows = 0;
for (const snapshot of Object.values(tables))
rows += Array.isArray(snapshot?.rows) ? snapshot.rows.length : 0;
components.push({
key,
label: BACKUP_COMPONENTS[key].label,
description: BACKUP_COMPONENTS[key].description,
rows,
});
}
return {
filename,
createdAt: document.createdAt,
sizeBytes,
appVersion: document.appVersion,
formatVersion: document.formatVersion,
note: document.note,
automatic: filename.includes('-auto-'),
integrity: 'ok',
compatible: document.formatVersion <= FORMAT_VERSION,
components,
};
}
#readAuto(): AutoBackupSettings {
const row = this.#db
.prepare('SELECT value_json FROM app_settings WHERE key = ?')
.get(AUTO_SETTINGS_KEY) as { value_json: string } | undefined;
let stored: Record<string, unknown> | undefined;
try {
const parsed: unknown = row ? JSON.parse(row.value_json) : undefined;
stored = isRecord(parsed) ? parsed : undefined;
} catch {
stored = undefined;
}
const fallback = {
enabled: false,
components: ['devices', 'notifications', 'automation'] as BackupComponentKey[],
timeOfDay: '02:00',
weekday: -1,
maximumCount: 10,
};
const source: Record<string, unknown> = { ...fallback, ...(stored ?? {}) };
let components: BackupComponentKey[] = fallback.components;
try {
components = normalizeComponents(source.components);
} catch {
components = fallback.components;
}
let timeOfDay = fallback.timeOfDay;
try {
timeOfDay = normalizeTimeOfDay(source.timeOfDay);
} catch {
timeOfDay = fallback.timeOfDay;
}
const weekday =
typeof source.weekday === 'number' && Number.isSafeInteger(source.weekday)
? Math.min(6, Math.max(-1, source.weekday))
: fallback.weekday;
const maximumCount =
typeof source.maximumCount === 'number' && Number.isSafeInteger(source.maximumCount)
? Math.min(500, Math.max(1, source.maximumCount))
: fallback.maximumCount;
return {
enabled: source.enabled === true,
components,
timeOfDay,
weekday,
maximumCount,
lastRunAt: typeof source.lastRunAt === 'string' ? source.lastRunAt : null,
};
}
#mostRecentOccurrence(timeOfDay: string, weekday: number): Date {
const [hour, minute] = timeOfDay.split(':').map((part) => Number(part));
const now = this.#now();
const candidate = new Date(now);
candidate.setHours(hour ?? 0, minute ?? 0, 0, 0);
if (candidate.getTime() > now.getTime()) candidate.setDate(candidate.getDate() - 1);
if (weekday >= 0) {
let guard = 0;
while (candidate.getDay() !== weekday && guard < 8) {
candidate.setDate(candidate.getDate() - 1);
guard += 1;
}
}
return candidate;
}
/** Keeps the newest `limit` scheduled backups; a limit of zero leaves manual files alone. */
async #prune(limit: number): Promise<number> {
if (limit <= 0) return 0;
const names = (await readdir(this.#directory))
.filter((name) => name.startsWith('multi-simadmin-components-auto-'))
.sort()
.reverse();
let removed = 0;
for (const filename of names.slice(limit)) {
await rm(join(this.#directory, filename), { force: true });
removed += 1;
}
return removed;
}
}
@@ -0,0 +1,179 @@
import Database from 'better-sqlite3';
import { afterEach, describe, expect, it } from 'vitest';
import { migrateDatabase } from '../../infrastructure/database/migrations.js';
import { ConnectionLogService } from './connection-log-service.js';
const databases: Database.Database[] = [];
function fixture(maximumRows?: number) {
const db = new Database(':memory:');
db.pragma('foreign_keys = ON');
migrateDatabase(db);
databases.push(db);
db.prepare(
`INSERT INTO instances (id,name,base_url,enabled,config_revision,created_at,updated_at)
VALUES ('node-a','Node A','http://node-a.local',1,1,'2026-09-01T00:00:00.000Z','2026-09-01T00:00:00.000Z'),
('node-b','Node B','http://node-b.local',1,1,'2026-09-01T00:00:00.000Z','2026-09-01T00:00:00.000Z')`,
).run();
let counter = 0;
const service = new ConnectionLogService({
db,
idFactory: () => `log-${++counter}`,
...(maximumRows === undefined ? {} : { maximumRows }),
});
return { db, service };
}
afterEach(() => {
for (const db of databases.splice(0)) db.close();
});
describe('ConnectionLogService', () => {
it('records probes and lists them newest first with paging metadata', () => {
const { service } = fixture();
service.record({
instanceId: 'node-a',
outcome: 'success',
state: 'fresh',
errorCode: null,
httpStatus: 200,
durationMs: 12,
observedAt: '2026-09-02T00:00:00.000Z',
});
service.record({
instanceId: 'node-b',
outcome: 'failed',
state: 'unknown',
errorCode: 'ECONNREFUSED',
httpStatus: null,
durationMs: 30,
observedAt: '2026-09-03T00:00:00.000Z',
});
const page = service.list({ pageSize: 1 });
expect(page.page).toEqual({ page: 1, pageSize: 1, total: 2 });
expect(page.items.map((item) => item.instanceId)).toEqual(['node-b']);
expect(page.items[0]).toMatchObject({ errorCode: 'ECONNREFUSED', outcome: 'failed' });
});
it('filters by outcome and free-text search', () => {
const { service } = fixture();
service.record({
instanceId: 'node-a',
outcome: 'stale',
state: 'stale',
errorCode: 'AUTH_REQUIRED',
httpStatus: 401,
durationMs: 8,
observedAt: '2026-09-02T00:00:00.000Z',
});
service.record({
instanceId: 'node-a',
outcome: 'success',
state: 'fresh',
errorCode: null,
httpStatus: 200,
durationMs: 9,
observedAt: '2026-09-02T01:00:00.000Z',
});
expect(service.list({ outcome: 'stale' }).page.total).toBe(1);
expect(service.list({ search: 'auth' }).page.total).toBe(1);
expect(service.list({ search: 'node-a', outcome: 'success' }).page.total).toBe(1);
});
it('summarises availability per instance', () => {
const { service } = fixture();
for (const [index, outcome] of (
['success', 'success', 'success', 'failed'] as const
).entries()) {
service.record({
instanceId: 'node-a',
outcome,
state: outcome === 'success' ? 'fresh' : 'unknown',
errorCode: outcome === 'success' ? null : 'UPSTREAM_TIMEOUT',
httpStatus: outcome === 'success' ? 200 : null,
durationMs: 10 + index,
observedAt: `2026-09-0${index + 1}T00:00:00.000Z`,
});
}
const [summary] = service.summarize(['node-a']);
expect(summary).toMatchObject({
instanceId: 'node-a',
total: 4,
success: 3,
failed: 1,
availabilityPercent: 75,
lastErrorCode: 'UPSTREAM_TIMEOUT',
lastObservedAt: '2026-09-04T00:00:00.000Z',
});
expect(service.summarize()).toHaveLength(1);
});
it('drops the oldest rows once the rolling cap is reached', () => {
const { db, service } = fixture(100);
for (let index = 0; index < 105; index += 1) {
service.record({
instanceId: 'node-a',
outcome: 'success',
state: 'fresh',
errorCode: null,
httpStatus: 200,
durationMs: index,
observedAt: new Date(Date.UTC(2026, 0, 1, 0, 0, index)).toISOString(),
});
}
const total = (
db.prepare('SELECT COUNT(*) AS count FROM connection_logs').get() as {
count: number;
}
).count;
expect(total).toBe(100);
expect(service.list({ pageSize: 100 }).items.at(-1)).toMatchObject({ durationMs: 5 });
});
it('ignores malformed drafts instead of throwing', () => {
const { service } = fixture();
expect(
service.record({
instanceId: '',
outcome: 'success',
state: 'fresh',
durationMs: 1,
observedAt: '2026-09-02T00:00:00.000Z',
}),
).toBeUndefined();
expect(
service.record({
instanceId: 'node-a',
outcome: 'nope' as never,
state: 'fresh',
durationMs: 1,
observedAt: '2026-09-02T00:00:00.000Z',
}),
).toBeUndefined();
expect(service.list().page.total).toBe(0);
});
it('prunes by cutoff and refuses an unbounded delete', () => {
const { service } = fixture();
service.record({
instanceId: 'node-a',
outcome: 'success',
state: 'fresh',
durationMs: 1,
observedAt: '2026-01-01T00:00:00.000Z',
});
service.record({
instanceId: 'node-a',
outcome: 'success',
state: 'fresh',
durationMs: 1,
observedAt: '2026-09-01T00:00:00.000Z',
});
expect(() => service.prune({})).toThrow(/before or instanceId is required/u);
expect(service.prune({ before: '2026-06-01T00:00:00.000Z' })).toBe(1);
expect(service.list().page.total).toBe(1);
expect(service.prune({ instanceId: 'node-a' })).toBe(1);
});
});
@@ -0,0 +1,305 @@
import { randomUUID } from 'node:crypto';
import type Database from 'better-sqlite3';
export type ConnectionOutcome = 'success' | 'stale' | 'failed' | 'unsupported';
/** Mirrors the status_snapshots state domain, which also covers an expired snapshot. */
export type ProbeState = 'fresh' | 'stale' | 'expired' | 'unknown';
const OUTCOMES: readonly ConnectionOutcome[] = ['success', 'stale', 'failed', 'unsupported'];
const STATES: readonly ProbeState[] = ['fresh', 'stale', 'expired', 'unknown'];
export interface ConnectionLogEntry {
readonly id: string;
readonly instanceId: string;
readonly outcome: ConnectionOutcome;
readonly state: ProbeState;
readonly errorCode: string | null;
readonly httpStatus: number | null;
readonly durationMs: number;
readonly observedAt: string;
}
/** A probe outcome as produced by the status snapshot service on every health refresh. */
export interface ConnectionLogDraft {
readonly instanceId: string;
readonly outcome: ConnectionOutcome;
readonly state: ProbeState;
readonly errorCode?: string | null;
readonly httpStatus?: number | null;
readonly durationMs: number;
readonly observedAt: string;
}
export interface ConnectionLogQuery {
readonly page?: number;
readonly pageSize?: number;
readonly instanceId?: string | undefined;
readonly outcome?: ConnectionOutcome | undefined;
readonly search?: string | undefined;
readonly from?: string | undefined;
readonly to?: string | undefined;
}
export interface ConnectionLogPage {
readonly items: readonly ConnectionLogEntry[];
readonly page: { readonly page: number; readonly pageSize: number; readonly total: number };
}
export interface ConnectionSummary {
readonly instanceId: string;
readonly total: number;
readonly success: number;
readonly failed: number;
readonly averageDurationMs: number;
readonly lastObservedAt: string | null;
readonly lastErrorCode: string | null;
readonly availabilityPercent: number;
}
export type ConnectionLogErrorCode = 'VALIDATION_FAILED';
export class ConnectionLogError extends Error {
constructor(
readonly code: ConnectionLogErrorCode,
message: string,
) {
super(message);
this.name = 'ConnectionLogError';
}
}
export interface ConnectionLogOptions {
readonly db: Database.Database;
readonly idFactory?: () => string;
/** Rolling cap so the journal can never outgrow the rest of the control-plane database. */
readonly maximumRows?: number;
}
const DEFAULT_MAXIMUM_ROWS = 20_000;
const MAX_PAGE_SIZE = 200;
function invalid(message: string): never {
throw new ConnectionLogError('VALIDATION_FAILED', message);
}
function timestamp(value: string): number {
const parsed = Date.parse(value);
return Number.isFinite(parsed) ? parsed : Number.NaN;
}
export class ConnectionLogService {
readonly #db: Database.Database;
readonly #id: () => string;
readonly #maximumRows: number;
constructor(options: ConnectionLogOptions) {
this.#db = options.db;
this.#id = options.idFactory ?? randomUUID;
this.#maximumRows = Math.max(100, options.maximumRows ?? DEFAULT_MAXIMUM_ROWS);
}
/** Best-effort write: a failed log insert must never break a health probe. */
record(draft: ConnectionLogDraft): ConnectionLogEntry | undefined {
if (typeof draft.instanceId !== 'string' || draft.instanceId.length === 0) return undefined;
if (!OUTCOMES.includes(draft.outcome) || !STATES.includes(draft.state)) return undefined;
const observedAt =
typeof draft.observedAt === 'string' && Number.isFinite(timestamp(draft.observedAt))
? draft.observedAt
: new Date().toISOString();
const entry: ConnectionLogEntry = {
id: this.#id(),
instanceId: draft.instanceId,
outcome: draft.outcome,
state: draft.state,
errorCode: typeof draft.errorCode === 'string' ? draft.errorCode.slice(0, 64) : null,
httpStatus:
Number.isSafeInteger(draft.httpStatus) && (draft.httpStatus ?? 0) >= 100
? (draft.httpStatus as number)
: null,
durationMs: Number.isSafeInteger(draft.durationMs)
? Math.max(0, Math.min(Number.MAX_SAFE_INTEGER, draft.durationMs))
: 0,
observedAt,
};
try {
this.#db
.prepare(
`INSERT INTO connection_logs
(id,instance_id,outcome,state,error_code,http_status,duration_ms,observed_at)
VALUES (@id,@instanceId,@outcome,@state,@errorCode,@httpStatus,@durationMs,@observedAt)`,
)
.run(entry);
this.#enforceCap();
} catch {
return undefined;
}
return entry;
}
list(query: ConnectionLogQuery = {}): ConnectionLogPage {
const page = query.page ?? 1;
const pageSize = query.pageSize ?? 50;
if (!Number.isSafeInteger(page) || page < 1) invalid('page must be a positive integer');
if (!Number.isSafeInteger(pageSize) || pageSize < 1 || pageSize > MAX_PAGE_SIZE)
invalid(`pageSize must be between 1 and ${MAX_PAGE_SIZE}`);
if (query.outcome !== undefined && !OUTCOMES.includes(query.outcome))
invalid('outcome is invalid');
if (query.instanceId !== undefined && (query.instanceId.trim() === '' || !query.instanceId))
invalid('instanceId is invalid');
for (const key of ['from', 'to'] as const) {
const value = query[key];
if (value !== undefined && !Number.isFinite(timestamp(value))) invalid(`${key} is invalid`);
}
if (query.search !== undefined && query.search.length > 200) invalid('search is too long');
const where: string[] = [];
const parameters: Record<string, unknown> = {};
if (query.instanceId) {
where.push('instance_id = @instanceId');
parameters.instanceId = query.instanceId;
}
if (query.outcome) {
where.push('outcome = @outcome');
parameters.outcome = query.outcome;
}
if (query.from) {
where.push('observed_at >= @from');
parameters.from = new Date(timestamp(query.from)).toISOString();
}
if (query.to) {
where.push('observed_at <= @to');
parameters.to = new Date(timestamp(query.to)).toISOString();
}
if (query.search) {
const needle = query.search.trim().toLowerCase();
if (needle !== '') {
where.push(
`(lower(instance_id) LIKE @search OR lower(COALESCE(error_code,'')) LIKE @search
OR lower(outcome) LIKE @search OR lower(state) LIKE @search)`,
);
parameters.search = `%${needle}%`;
}
}
const clause = where.length > 0 ? ` WHERE ${where.join(' AND ')}` : '';
const total = (
this.#db
.prepare(`SELECT COUNT(*) AS count FROM connection_logs${clause}`)
.get(parameters) as { count: number }
).count;
const rows = this.#db
.prepare(
`SELECT id,instance_id,outcome,state,error_code,http_status,duration_ms,observed_at
FROM connection_logs${clause}
ORDER BY observed_at DESC, rowid DESC LIMIT @limit OFFSET @offset`,
)
.all({ ...parameters, limit: pageSize, offset: (page - 1) * pageSize }) as Array<
Record<string, unknown>
>;
return Object.freeze({
items: Object.freeze(rows.map((row) => this.#entry(row))),
page: Object.freeze({ page, pageSize, total }),
});
}
/** Availability rollup over the retained window, newest probe first. */
summarize(instanceIds?: readonly string[]): readonly ConnectionSummary[] {
const ids = instanceIds ? [...instanceIds] : undefined;
// better-sqlite3 cannot expand an array binding, so the IN list is built positionally.
const placeholders = ids ? ids.map((_id, index) => `@id${index}`).join(', ') : '';
const clause = ids ? ` WHERE instance_id IN (${placeholders})` : '';
const parameters: Record<string, unknown> = {};
if (ids) ids.forEach((id, index) => (parameters[`id${index}`] = id));
const rows = this.#db
.prepare(
`SELECT instance_id,
COUNT(*) AS total,
SUM(CASE WHEN outcome = 'success' THEN 1 ELSE 0 END) AS success,
SUM(CASE WHEN outcome IN ('failed','unsupported') THEN 1 ELSE 0 END) AS failed,
AVG(duration_ms) AS average_duration,
MAX(observed_at) AS last_observed_at
FROM connection_logs${clause}
GROUP BY instance_id`,
)
.all(parameters) as Array<Record<string, unknown>>;
const summaries = rows.map((row) => {
const instanceId = String(row.instance_id ?? '');
const total = Number(row.total ?? 0);
const success = Number(row.success ?? 0);
const lastObservedAt = row.last_observed_at === null ? null : String(row.last_observed_at);
const last = this.#db
.prepare(
`SELECT error_code FROM connection_logs
WHERE instance_id = ? ORDER BY observed_at DESC, rowid DESC LIMIT 1`,
)
.get(instanceId) as { error_code: string | null } | undefined;
return Object.freeze({
instanceId,
total,
success,
failed: Number(row.failed ?? 0),
averageDurationMs: Math.round(Number(row.average_duration ?? 0)),
lastObservedAt,
lastErrorCode: last?.error_code ?? null,
availabilityPercent: total > 0 ? Math.round((success / total) * 1000) / 10 : 0,
}) satisfies ConnectionSummary;
});
return Object.freeze(summaries);
}
prune(input: {
readonly before?: string | undefined;
readonly instanceId?: string | undefined;
}): number {
const where: string[] = [];
const parameters: Record<string, unknown> = {};
if (input.before) {
const cutoff = timestamp(input.before);
if (!Number.isFinite(cutoff)) invalid('before is invalid');
where.push('observed_at < @before');
parameters.before = new Date(cutoff).toISOString();
}
if (input.instanceId) {
where.push('instance_id = @instanceId');
parameters.instanceId = input.instanceId;
}
if (where.length === 0) invalid('before or instanceId is required');
return Number(
this.#db.prepare(`DELETE FROM connection_logs WHERE ${where.join(' AND ')}`).run(parameters)
.changes,
);
}
#enforceCap(): void {
const total = (
this.#db.prepare('SELECT COUNT(*) AS count FROM connection_logs').get() as {
count: number;
}
).count;
if (total <= this.#maximumRows) return;
this.#db
.prepare(
`DELETE FROM connection_logs WHERE rowid IN (
SELECT rowid FROM connection_logs ORDER BY observed_at DESC, rowid DESC
LIMIT -1 OFFSET @keep
)`,
)
.run({ keep: this.#maximumRows });
}
#entry(row: Record<string, unknown>): ConnectionLogEntry {
return Object.freeze({
id: String(row.id),
instanceId: String(row.instance_id),
outcome: row.outcome as ConnectionOutcome,
state: row.state as ProbeState,
errorCode:
row.error_code === null || row.error_code === undefined ? null : String(row.error_code),
httpStatus:
row.http_status === null || row.http_status === undefined ? null : Number(row.http_status),
durationMs: Number(row.duration_ms ?? 0),
observedAt: String(row.observed_at),
});
}
}
@@ -0,0 +1,543 @@
import Database from 'better-sqlite3';
import { afterEach, describe, expect, it, vi } from 'vitest';
import { migrateDatabase } from '../../infrastructure/database/migrations.js';
import {
ConsoleUpdateService,
UpdateError,
createLaunchdRestartLauncher,
createPlatformRestartLauncher,
createSystemdRestartLauncher,
parseManifestVersion,
parseShortStat,
type UpdateCommandResult,
type UpdateCommandRunner,
} from './console-update-service.js';
const LOCAL = 'aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa';
const REMOTE = 'bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb';
const dbs: Database.Database[] = [];
afterEach(() => {
for (const db of dbs.splice(0)) db.close();
});
interface FakeGit {
readonly calls: string[];
runner: UpdateCommandRunner;
head: string;
remoteHead: string;
dirty: string;
fastForwardable: boolean;
/** Whether HEAD already carries the remote commit: true means the checkout is ahead. */
containsRemote: boolean;
/** Command prefixes that should fail, e.g. `git` or `git merge-base`. */
failing: Set<string>;
}
/** Matches a `command arg ...` key against the configured failing prefixes. */
function fails(patterns: ReadonlySet<string>, key: string): boolean {
const tokens = key.split(' ');
for (const pattern of patterns) {
const wanted = pattern.split(' ');
if (wanted.length <= tokens.length && wanted.every((token, index) => tokens[index] === token))
return true;
}
return false;
}
function fakeGit(overrides: Partial<Pick<FakeGit, 'head' | 'remoteHead'>> = {}): FakeGit {
const state: FakeGit = {
calls: [],
head: overrides.head ?? LOCAL,
remoteHead: overrides.remoteHead ?? REMOTE,
dirty: '',
fastForwardable: true,
containsRemote: false,
failing: new Set<string>(),
runner: { run: async () => ({ code: 0, stdout: '', stderr: '' }) },
};
const run = async (command: string, args: readonly string[]): Promise<UpdateCommandResult> => {
const key = [command, ...args].join(' ');
state.calls.push(key);
if (fails(state.failing, key)) return { code: 1, stdout: '', stderr: `${key} failed` };
if (command === 'git') {
if (args[0] === 'rev-parse' && args[1] === 'HEAD')
return { code: 0, stdout: `${state.head}\n`, stderr: '' };
if (args[0] === 'rev-parse' && args[1] === '--abbrev-ref')
return { code: 0, stdout: 'main\n', stderr: '' };
if (args[0] === 'config')
return { code: 0, stdout: 'https://example.test/repo.git\n', stderr: '' };
if (args[0] === 'ls-remote')
return { code: 0, stdout: `${state.remoteHead}\trefs/heads/main\n`, stderr: '' };
if (args[0] === 'rev-parse' && args[1] === 'FETCH_HEAD')
return { code: 0, stdout: `${state.remoteHead}\n`, stderr: '' };
if (args[0] === 'merge-base') {
// `HEAD FETCH_HEAD` asks whether we can fast-forward; `<sha> HEAD` asks whether the
// checkout already contains the remote commit.
const fastForward = args[3] === 'FETCH_HEAD';
const ok = fastForward ? state.fastForwardable : state.containsRemote;
return { code: ok ? 0 : 1, stdout: '', stderr: '' };
}
if (args[0] === 'rev-list') return { code: 0, stdout: '3\n', stderr: '' };
if (args[0] === 'diff')
return {
code: 0,
stdout: ' 2 files changed, 34 insertions(+), 6 deletions(-)\n',
stderr: '',
};
if (args[0] === 'show')
return { code: 0, stdout: '{"name":"multi-simadmin","version":"0.9.0"}', stderr: '' };
if (args[0] === 'status') return { code: 0, stdout: state.dirty, stderr: '' };
return { code: 0, stdout: '', stderr: '' };
}
return { code: 0, stdout: '', stderr: '' };
};
state.runner = { run };
return state;
}
function service(
input: {
readonly git?: FakeGit;
readonly launcher?: { supported: boolean; restart: () => Promise<void> };
readonly restartDelayMs?: number;
readonly preInstallBackup?: (targetCommit: string) => Promise<string>;
} = {},
) {
const db = new Database(':memory:');
migrateDatabase(db);
dbs.push(db);
const git = input.git ?? fakeGit();
const serviceOptions = {
db,
version: '0.1.0',
runner: git.runner,
now: () => new Date('2026-09-06T00:00:00.000Z'),
launcher: input.launcher ?? { supported: false, async restart() {} },
...(input.preInstallBackup ? { preInstallBackup: input.preInstallBackup } : {}),
...(input.restartDelayMs === undefined ? {} : { restartDelayMs: input.restartDelayMs }),
} as const;
return { db, git, updates: new ConsoleUpdateService(serviceOptions) };
}
describe('update helpers', () => {
it('adds insertions and deletions into one change count', () => {
expect(parseShortStat(' 2 files changed, 34 insertions(+), 6 deletions(-)')).toEqual({
files: 2,
lines: 40,
});
});
it('survives an empty or malformed shortstat', () => {
expect(parseShortStat('')).toEqual({ files: 0, lines: 0 });
expect(parseShortStat('1 file changed, 2 insertions(+)')).toEqual({ files: 1, lines: 2 });
});
it('reads a version out of a manifest and ignores broken JSON', () => {
expect(parseManifestVersion('{"version":"1.2.3"}')).toBe('1.2.3');
expect(parseManifestVersion('not json')).toBe('');
expect(parseManifestVersion('{"version":42}')).toBe('');
});
});
describe('ConsoleUpdateService', () => {
it('starts idle but ready to install on a git checkout', async () => {
const { updates } = service();
const status = await updates.status();
expect(status.phase).toBe('idle');
expect(status.deploymentMode).toBe('git');
expect(status.installSupported).toBe(true);
expect(status.restartSupported).toBe(false);
expect(status.release).toBeNull();
});
it('reports the manual deployment mode when git cannot resolve HEAD', async () => {
const git = fakeGit();
git.failing.add('git');
const { updates } = service({ git });
const status = await updates.status();
expect(status.deploymentMode).toBe('manual');
expect(status.installSupported).toBe(false);
expect(status.message).toContain('手动更新');
});
it('says the console is current when the remote matches HEAD', async () => {
const git = fakeGit({ remoteHead: LOCAL });
const { updates } = service({ git });
const status = await updates.check();
expect(status.phase).toBe('up_to_date');
expect(status.release?.status).toBe('latest');
expect(status.message).toContain('已是最新版本');
});
it('offers a download when the remote moved ahead', async () => {
const git = fakeGit();
const { updates } = service({ git });
const status = await updates.check();
expect(status.phase).toBe('update_available');
expect(status.release?.latestCommit).toBe(REMOTE);
expect(status.release?.latestVersion).toBe('');
});
it('says the console is ahead when the checkout already carries the remote commit', async () => {
const git = fakeGit();
git.containsRemote = true;
const { updates } = service({ git });
const status = await updates.check();
expect(status.phase).toBe('up_to_date');
expect(status.release?.status).toBe('latest');
expect(status.release?.ahead).toBe(3);
expect(status.message).toContain('领先更新源');
expect(status.release?.summary).toContain('全部提交');
// The download button stays closed because nothing was queued.
await expect(updates.install()).rejects.toMatchObject({ code: 'STATE' });
});
it('treats an unfetchable remote commit as an update rather than a local lead', async () => {
const git = fakeGit();
git.failing.add('git merge-base');
const { updates } = service({ git });
const status = await updates.check();
expect(status.phase).toBe('update_available');
expect(status.release?.ahead).toBe(0);
});
it('refuses to check a detached HEAD', async () => {
const git = fakeGit();
const original = git.runner.run.bind(git.runner);
git.runner = {
run: async (command, args, options) =>
command === 'git' && args[0] === 'rev-parse' && args[1] === '--abbrev-ref'
? { code: 0, stdout: 'HEAD\n', stderr: '' }
: original(command, args, options),
};
const { updates } = service({ git });
await expect(updates.check()).rejects.toMatchObject({ code: 'STATE' });
});
it('fetches, verifies and describes the candidate', async () => {
const git = fakeGit();
const { updates } = service({ git });
await updates.check();
const status = await updates.download();
expect(status.phase).toBe('ready');
expect(status.release?.ahead).toBe(3);
expect(status.release?.changedFiles).toBe(2);
expect(status.release?.changedLines).toBe(40);
expect(status.release?.latestVersion).toBe('0.9.0');
expect(status.message).toContain('3 个提交');
expect(git.calls.some((call) => call.startsWith('git fetch'))).toBe(true);
});
it('refuses to download before a check', async () => {
const { updates } = service();
await expect(updates.download()).rejects.toBeInstanceOf(UpdateError);
});
it('keeps the code untouched when the branch cannot fast-forward', async () => {
const git = fakeGit();
git.fastForwardable = false;
const { updates } = service({ git });
await updates.check();
const status = await updates.download();
expect(status.phase).toBe('failed');
expect(status.error).toContain('快进');
expect(git.calls.some((call) => call.startsWith('git merge --ff-only'))).toBe(false);
});
it('installs a verified candidate and queues the restart', async () => {
const git = fakeGit();
const { updates } = service({ git });
await updates.check();
await updates.download();
const status = await updates.install();
expect(status.phase).toBe('install_queued');
expect(git.calls).toContain(`git merge --ff-only ${REMOTE}`);
expect(git.calls).toContain('corepack pnpm install --frozen-lockfile');
expect(git.calls).toContain('corepack pnpm --filter @multi-simadmin/web build');
});
it('will not install over uncommitted work', async () => {
const git = fakeGit();
git.dirty = ' M apps/api/src/control-plane.ts\n';
const { updates } = service({ git });
await updates.check();
await updates.download();
await expect(updates.install()).rejects.toMatchObject({ code: 'STATE' });
expect(git.calls.some((call) => call.startsWith('git merge --ff-only'))).toBe(false);
});
it('rolls the checkout back when the build fails', async () => {
const git = fakeGit();
const original = git.runner.run.bind(git.runner);
git.runner = {
run: async (command, args, options) =>
command === 'corepack' && args.includes('build')
? { code: 2, stdout: '', stderr: 'vite exited 1' }
: original(command, args, options),
};
const { updates } = service({ git });
await updates.check();
await updates.download();
await expect(updates.install()).rejects.toMatchObject({ code: 'COMMAND' });
const status = await updates.status();
expect(status.phase).toBe('rolled_back');
expect(status.error).toContain('回滚');
expect(git.calls).toContain(`git reset --hard ${LOCAL}`);
});
it('snapshots the configuration before the checkout moves', async () => {
const snapshots: string[] = [];
const { git, updates } = service({
preInstallBackup: async (target) => {
snapshots.push(target);
return 'multi-simadmin-components-auto-preupdate.json';
},
});
await updates.check();
await updates.download();
const status = await updates.install();
expect(snapshots).toEqual([REMOTE]);
expect(status.phase).toBe('install_queued');
// The snapshot has to land before the merge, otherwise there is nothing to return to.
expect(git.calls.some((call) => call.startsWith('git merge --ff-only'))).toBe(true);
});
it('aborts the install when the safety snapshot cannot be written', async () => {
const { git, updates } = service({
preInstallBackup: async () => {
throw new Error('disk full');
},
});
await updates.check();
await updates.download();
await expect(updates.install()).rejects.toMatchObject({ code: 'COMMAND' });
const status = await updates.status();
expect(status.phase).toBe('failed');
expect(status.message).toContain('更新前备份失败');
expect(git.calls.some((call) => call.startsWith('git merge --ff-only'))).toBe(false);
});
it('requires a verified package before installing', async () => {
const { updates } = service();
await updates.check();
await expect(updates.install()).rejects.toMatchObject({ code: 'STATE' });
});
it('reports 501 when nothing supervises the process', async () => {
const { updates } = service();
await expect(updates.restart()).rejects.toMatchObject({
code: 'NOT_SUPPORTED',
statusCode: 501,
});
});
it('hands a queued install to the supervisor', async () => {
const restart = vi.fn(async () => undefined);
const { updates } = service({
launcher: { supported: true, restart },
restartDelayMs: 0,
});
await updates.check();
await updates.download();
await updates.install();
const status = await updates.restart();
expect(status.phase).toBe('restarting');
await new Promise((resolve) => setTimeout(resolve, 10));
expect(restart).toHaveBeenCalledTimes(1);
});
it('confirms its own update once the restarted process sees the new commit', async () => {
const before = fakeGit();
const first = service({
git: before,
launcher: { supported: true, restart: async () => undefined },
restartDelayMs: 0,
});
await first.updates.check();
await first.updates.download();
await first.updates.install();
await first.updates.restart();
await new Promise((resolve) => setTimeout(resolve, 10));
// Same database, new process: HEAD moved to the target commit during the restart.
const after = fakeGit({ head: REMOTE });
const reopened = new ConsoleUpdateService({
db: first.db,
version: '0.9.0',
runner: after.runner,
now: () => new Date('2026-09-06T00:05:00.000Z'),
});
const status = await reopened.status();
expect(status.phase).toBe('up_to_date');
expect(status.message).toContain('完成重启');
});
it('marks a restart that came back on the old commit as failed', async () => {
const git = fakeGit();
const { updates, db } = service({
git,
launcher: { supported: true, restart: async () => {} },
restartDelayMs: 0,
});
await updates.check();
await updates.download();
await updates.install();
await updates.restart();
const reopened = new ConsoleUpdateService({
db,
version: '0.1.0',
runner: git.runner,
now: () => new Date('2026-09-06T00:05:00.000Z'),
});
const status = await reopened.status();
expect(status.phase).toBe('failed');
expect(status.error).toContain('目标版本');
});
it('rejects a second command while one is running', async () => {
const git = fakeGit();
let release: () => void = () => {};
const gate = new Promise<void>((resolve) => {
release = resolve;
});
const original = git.runner.run.bind(git.runner);
git.runner = {
run: async (command, args, options) => {
if (command === 'git' && args[0] === 'ls-remote') await gate;
return original(command, args, options);
},
};
const { updates } = service({ git });
const first = updates.check();
await expect(updates.check()).rejects.toMatchObject({ code: 'BUSY' });
release();
await first;
});
it('keeps a corrupt stored state readable', async () => {
const { db, updates } = service();
db.prepare(
`INSERT INTO app_settings (key,value_json,created_at,updated_at)
VALUES ('console.update','{not json',?,?)`,
).run('2026-09-06T00:00:00.000Z', '2026-09-06T00:00:00.000Z');
const status = await updates.status();
expect(status.phase).toBe('idle');
});
});
describe('createLaunchdRestartLauncher', () => {
it('is unsupported without a launchd label', () => {
const launcher = createLaunchdRestartLauncher({
env: {},
runner: { run: async () => ({ code: 0, stdout: '', stderr: '' }) },
uid: () => 501,
});
expect(launcher.supported).toBe(false);
});
it('kickstarts its own launchd label', async () => {
const calls: string[][] = [];
const launcher = createLaunchdRestartLauncher({
env: { XPC_SERVICE_NAME: 'fun.chickliu.multi-simadmin-api' },
runner: {
run: async (command, args) => {
calls.push([command, ...args]);
return { code: 0, stdout: '', stderr: '' };
},
},
uid: () => 501,
});
expect(launcher.supported).toBe(true);
await launcher.restart();
expect(calls[0]).toEqual([
'launchctl',
'kickstart',
'-k',
'gui/501/fun.chickliu.multi-simadmin-api',
]);
});
it('surfaces a launchctl failure', async () => {
const launcher = createLaunchdRestartLauncher({
env: { XPC_SERVICE_NAME: 'fun.chickliu.multi-simadmin-api' },
runner: { run: async () => ({ code: 1, stdout: '', stderr: 'Could not find service' }) },
uid: () => 501,
});
await expect(launcher.restart()).rejects.toMatchObject({ code: 'COMMAND', statusCode: 500 });
});
});
describe('createSystemdRestartLauncher', () => {
it('is unsupported without a unit or with a malformed unit name', () => {
for (const unit of [undefined, '', 'evil; rm -rf /', 'multi-simadmin']) {
const launcher = createSystemdRestartLauncher({
env: unit === undefined ? {} : { MULTI_SIMADMIN_SYSTEMD_UNIT: unit },
runner: { run: async () => ({ code: 0, stdout: '', stderr: '' }) },
});
expect(launcher.supported).toBe(false);
}
});
it('restarts through the configured scope and unit', async () => {
const calls: string[][] = [];
const launcher = createSystemdRestartLauncher({
env: { MULTI_SIMADMIN_SYSTEMD_UNIT: 'multi-simadmin-api.service' },
runner: {
run: async (command, args) => {
calls.push([command, ...args]);
return { code: 0, stdout: '', stderr: '' };
},
},
});
expect(launcher.supported).toBe(true);
await launcher.restart();
expect(calls[0]).toEqual(['systemctl', 'user', 'restart', 'multi-simadmin-api.service']);
});
it('supports the system scope and surfaces systemctl failures', async () => {
const calls: string[][] = [];
const launcher = createSystemdRestartLauncher({
env: {
MULTI_SIMADMIN_SYSTEMD_UNIT: 'multi-simadmin-api.service',
MULTI_SIMADMIN_SYSTEMD_SCOPE: 'system',
},
runner: {
run: async (command, args) => {
calls.push([command, ...args]);
return { code: 1, stdout: '', stderr: 'not logged in' };
},
},
});
await launcher.restart().catch(() => {});
expect(calls[0]).toEqual(['systemctl', 'system', 'restart', 'multi-simadmin-api.service']);
await expect(launcher.restart()).rejects.toMatchObject({ code: 'COMMAND', statusCode: 500 });
});
});
describe('createPlatformRestartLauncher', () => {
const runner = { run: async () => ({ code: 0, stdout: '', stderr: '' }) };
it('prefers systemd on Linux when a unit is configured', () => {
const launcher = createPlatformRestartLauncher({
env: { MULTI_SIMADMIN_HOST_PLATFORM: 'linux', MULTI_SIMADMIN_SYSTEMD_UNIT: 'api.service' },
runner,
});
expect(launcher.supported).toBe(true);
});
it('keeps the launchd launcher everywhere else', () => {
const linux = createPlatformRestartLauncher({
env: { MULTI_SIMADMIN_HOST_PLATFORM: 'linux' },
runner,
});
const darwin = createPlatformRestartLauncher({
env: { MULTI_SIMADMIN_HOST_PLATFORM: 'darwin', XPC_SERVICE_NAME: 'x' },
runner,
uid: () => 501,
});
expect(linux.supported).toBe(false);
expect(darwin.supported).toBe(true);
});
});
@@ -0,0 +1,781 @@
import { spawn } from 'node:child_process';
import type Database from 'better-sqlite3';
/**
* Online update for the console itself, modelled on the Hub update card: check a remote
* branch, fetch and verify the candidate, install it, then hand the process back to the
* supervisor. Each step is a separate, operator-visible phase, so a half-finished update stays
* readable after a crash instead of silently half-applied.
*/
export const UPDATE_PHASES = [
'idle',
'checking',
'up_to_date',
'update_available',
'downloading',
'ready',
'installing',
'install_queued',
'restarting',
'failed',
'rolled_back',
] as const;
export type UpdatePhase = (typeof UPDATE_PHASES)[number];
/** Phases where work is in flight: the UI polls these fast and refuses a second command. */
export const BUSY_UPDATE_PHASES: ReadonlySet<UpdatePhase> = new Set<UpdatePhase>([
'checking',
'downloading',
'installing',
'install_queued',
'restarting',
]);
export interface UpdateRelease {
readonly status: 'latest' | 'update_available' | 'unavailable';
readonly remote: string;
readonly branch: string;
readonly currentCommit: string;
readonly latestCommit: string;
readonly currentVersion: string;
readonly latestVersion: string;
readonly ahead: number;
readonly changedFiles: number;
readonly changedLines: number;
readonly summary: string;
}
export interface UpdateStatus {
readonly phase: UpdatePhase;
readonly message: string;
readonly error: string;
readonly progressPercent: number;
readonly checkedAt: string;
readonly deploymentMode: 'git' | 'manual';
readonly installSupported: boolean;
readonly restartSupported: boolean;
readonly release: UpdateRelease | null;
}
export interface UpdateCommandResult {
readonly code: number;
readonly stdout: string;
readonly stderr: string;
}
export interface UpdateCommandRunner {
run(
command: string,
args: readonly string[],
options?: { readonly timeoutMs?: number },
): Promise<UpdateCommandResult>;
}
export interface RestartLauncher {
readonly supported: boolean;
restart(): Promise<void>;
}
export class UpdateError extends Error {
constructor(
readonly code: 'BUSY' | 'NOT_SUPPORTED' | 'STATE' | 'COMMAND',
message: string,
readonly statusCode = 409,
) {
super(message);
this.name = 'UpdateError';
}
}
const STATE_KEY = 'console.update';
const DEFAULT_TIMEOUT_MS = 120_000;
const INSTALL_TIMEOUT_MS = 15 * 60_000;
const MAX_OUTPUT = 4 * 1024 * 1024;
const COMMIT = /^[0-9a-f]{7,40}$/u;
type PendingAction = 'install' | 'restart' | '';
interface PersistedState {
readonly phase: UpdatePhase;
readonly message: string;
readonly error: string;
readonly progressPercent: number;
readonly checkedAt: string;
readonly release: UpdateRelease | null;
/** Where the update is aimed; lets a restarted process confirm its own success. */
readonly pendingCommit: string;
readonly pendingAction: PendingAction;
}
function text(value: unknown, maximum = 512): string {
return typeof value === 'string' ? value.slice(0, maximum) : '';
}
function count(value: unknown): number {
return typeof value === 'number' && Number.isSafeInteger(value) && value >= 0 ? value : 0;
}
function percent(value: unknown): number {
return typeof value === 'number' && Number.isFinite(value)
? Math.min(100, Math.max(0, Math.round(value)))
: 0;
}
function phaseOf(value: unknown): UpdatePhase {
return typeof value === 'string' && (UPDATE_PHASES as readonly string[]).includes(value)
? (value as UpdatePhase)
: 'idle';
}
function readRelease(value: unknown): UpdateRelease | null {
if (typeof value !== 'object' || value === null || Array.isArray(value)) return null;
const source = value as Record<string, unknown>;
const status = source.status;
return {
status:
status === 'latest' || status === 'update_available' || status === 'unavailable'
? status
: 'unavailable',
remote: text(source.remote, 256),
branch: text(source.branch, 128),
currentCommit: text(source.currentCommit, 40),
latestCommit: text(source.latestCommit, 40),
currentVersion: text(source.currentVersion, 64),
latestVersion: text(source.latestVersion, 64),
ahead: count(source.ahead),
changedFiles: count(source.changedFiles),
changedLines: count(source.changedLines),
summary: text(source.summary, 512),
};
}
/** `2 files changed, 34 insertions(+), 6 deletions(-)` from `git diff --shortstat`. */
export function parseShortStat(value: string): { files: number; lines: number } {
const files = Number(/(\d+) files? changed/u.exec(value)?.[1] ?? 0);
const insertions = Number(/(\d+) insertions?\(\+\)/u.exec(value)?.[1] ?? 0);
const deletions = Number(/(\d+) deletions?\(-\)/u.exec(value)?.[1] ?? 0);
return {
files: Number.isFinite(files) ? files : 0,
lines:
(Number.isFinite(insertions) ? insertions : 0) + (Number.isFinite(deletions) ? deletions : 0),
};
}
/** Reads the version field out of a package.json blob fetched from a commit. */
export function parseManifestVersion(value: string): string {
try {
const parsed: unknown = JSON.parse(value);
if (parsed && typeof parsed === 'object' && !Array.isArray(parsed)) {
const version = (parsed as Record<string, unknown>).version;
if (typeof version === 'string' && version.length <= 64) return version;
}
} catch {
// A malformed manifest simply means there is no version to show.
}
return '';
}
export function createCommandRunner(options: {
readonly cwd: string;
readonly timeoutMs?: number;
}): UpdateCommandRunner {
return {
run(command, args, runOptions) {
return new Promise((resolve) => {
const child = spawn(command, [...args], { cwd: options.cwd, shell: false });
let stdout = '';
let stderr = '';
const timer = setTimeout(
() => child.kill('SIGKILL'),
runOptions?.timeoutMs ?? options.timeoutMs ?? DEFAULT_TIMEOUT_MS,
);
child.stdout.on('data', (chunk: Buffer) => {
if (stdout.length < MAX_OUTPUT) stdout += chunk.toString('utf8');
});
child.stderr.on('data', (chunk: Buffer) => {
if (stderr.length < MAX_OUTPUT) stderr += chunk.toString('utf8');
});
child.once('error', (error: Error) => {
clearTimeout(timer);
resolve({ code: -1, stdout, stderr: `${stderr}${error.message}` });
});
child.once('close', (code) => {
clearTimeout(timer);
resolve({ code: code ?? -1, stdout, stderr });
});
});
},
};
}
/**
* macOS LaunchAgents learn their own label through XPC_SERVICE_NAME, which is the only
* dependable hint that this process is supervised and will be started again after it exits.
*/
export function createLaunchdRestartLauncher(input: {
readonly env: Readonly<Record<string, string | undefined>>;
readonly runner: UpdateCommandRunner;
readonly uid?: () => number | undefined;
}): RestartLauncher {
const label = text(input.env.XPC_SERVICE_NAME, 256);
const uid =
input.uid?.() ?? (typeof process.getuid === 'function' ? process.getuid() : undefined);
const supported = label.length > 0 && typeof uid === 'number';
return {
supported,
async restart() {
if (!supported) throw new UpdateError('NOT_SUPPORTED', '当前进程未受服务管理器托管。', 501);
const result = await input.runner.run('launchctl', [
'kickstart',
'-k',
`gui/${uid}/${label}`,
]);
if (result.code !== 0)
throw new UpdateError(
'COMMAND',
text(result.stderr, 200) || 'launchctl kickstart 失败。',
500,
);
},
};
}
/**
* Linux systemd counterpart to the launchd launcher. The unit and scope are
* supplied explicitly through the environment so a supervised deployment can
* hand restarts to its service manager; unset variables keep it unsupported.
*/
export function createSystemdRestartLauncher(input: {
readonly env: Readonly<Record<string, string | undefined>>;
readonly runner: UpdateCommandRunner;
}): RestartLauncher {
const unit = input.env.MULTI_SIMADMIN_SYSTEMD_UNIT ?? '';
const scope = input.env.MULTI_SIMADMIN_SYSTEMD_SCOPE === 'system' ? 'system' : 'user';
const validUnit = /^[A-Za-z0-9@_.\-]+\.service$/.test(unit);
const supported = validUnit;
return {
supported,
async restart() {
if (!supported)
throw new UpdateError(
'NOT_SUPPORTED',
'未配置 systemd 服务单元(MULTI_SIMADMIN_SYSTEMD_UNIT)。',
501,
);
const result = await input.runner.run('systemctl', [scope, 'restart', unit]);
if (result.code !== 0)
throw new UpdateError(
'COMMAND',
text(result.stderr, 200) || 'systemctl restart 失败。',
500,
);
},
};
}
/** Picks the restart launcher matching the host service manager. */
export function createPlatformRestartLauncher(input: {
readonly env: Readonly<Record<string, string | undefined>>;
readonly runner: UpdateCommandRunner;
readonly uid?: () => number | undefined;
}): RestartLauncher {
const platform = input.env.MULTI_SIMADMIN_HOST_PLATFORM ?? process.platform;
return platform === 'linux' && input.env.MULTI_SIMADMIN_SYSTEMD_UNIT !== undefined
? createSystemdRestartLauncher(input)
: createLaunchdRestartLauncher(input);
}
export interface ConsoleUpdateServiceOptions {
readonly db: Database.Database;
readonly version: string;
readonly remote?: string;
readonly runner?: UpdateCommandRunner;
readonly launcher?: RestartLauncher;
readonly now?: () => Date;
/**
* Safety net run before the checkout moves: a new version may migrate the database in a way
* that cannot be undone by rolling the code back, so the config is snapshotted first.
*/
readonly preInstallBackup?: (targetCommit: string) => Promise<string>;
/** Test hook: the real hand-off waits a beat so the HTTP response can flush first. */
readonly restartDelayMs?: number;
}
export class ConsoleUpdateService {
readonly #db: Database.Database;
readonly #version: string;
readonly #remote: string;
readonly #runner: UpdateCommandRunner;
readonly #launcher: RestartLauncher;
readonly #now: () => Date;
readonly #preInstallBackup: ((targetCommit: string) => Promise<string>) | undefined;
readonly #restartDelayMs: number;
#state: PersistedState;
#inFlight: Promise<UpdateStatus> | undefined;
#mode: 'unknown' | 'git' | 'manual' = 'unknown';
#reconciled = false;
constructor(options: ConsoleUpdateServiceOptions) {
this.#db = options.db;
this.#version = options.version;
this.#remote = options.remote ?? 'origin';
this.#runner =
options.runner ?? createCommandRunner({ cwd: process.cwd(), timeoutMs: DEFAULT_TIMEOUT_MS });
this.#launcher = options.launcher ?? { supported: false, async restart() {} };
this.#now = options.now ?? (() => new Date());
this.#preInstallBackup = options.preInstallBackup;
this.#restartDelayMs = options.restartDelayMs ?? 250;
this.#state = this.#read();
}
/** Resolves the deployment mode, because a plain status poll must not shell out to git. */
async status(): Promise<UpdateStatus> {
if (this.#mode === 'unknown') await this.#localHead();
await this.#reconcile();
return this.#view();
}
async check(): Promise<UpdateStatus> {
return this.#exclusive(async () => {
const head = await this.#localHead();
if (!head) return this.#view();
const branch = (await this.#text(['rev-parse', '--abbrev-ref', 'HEAD'])).trim();
const remoteUrl = (
await this.#text(['config', '--get', `remote.${this.#remote}.url`])
).trim();
if (!branch || branch === 'HEAD')
throw new UpdateError('STATE', '当前处于游离提交,请先切回分支再检查更新。', 409);
if (!remoteUrl)
throw new UpdateError('STATE', `未配置远端 ${this.#remote},无法检查更新。`, 409);
const listing = await this.#git(['ls-remote', remoteUrl, `refs/heads/${branch}`]);
if (listing.code !== 0)
throw new UpdateError('COMMAND', '无法连接更新源,请检查网络后重试。', 502);
const latest = text(listing.stdout).trim().split(/\s+/u)[0];
if (!latest || !COMMIT.test(latest))
throw new UpdateError('COMMAND', `远端分支 ${branch} 不存在或不可读。`, 502);
// A different SHA is not enough: a checkout that already carries the remote commit is
// ahead of the update source, not behind it. Only an ancestry test tells the two apart.
const behind = latest !== head && !(await this.#contains(latest));
const ahead = behind
? 0
: latest === head
? 0
: Number(await this.#text(['rev-list', '--count', `${latest}..HEAD`]));
const upToDate = !behind;
const release: UpdateRelease = {
status: upToDate ? 'latest' : 'update_available',
remote: remoteUrl,
branch,
currentCommit: head,
latestCommit: latest,
currentVersion: this.#version,
latestVersion: '',
ahead: behind ? 0 : Number.isFinite(ahead) ? ahead : 0,
changedFiles: 0,
changedLines: 0,
summary: behind
? '下载更新包后可以看到提交数量与改动规模。'
: upToDate && latest !== head
? `本地已包含 ${remoteUrl}/${branch} 的全部提交。`
: '',
};
this.#save({
phase: upToDate ? 'up_to_date' : 'update_available',
message: upToDate
? latest === head
? `当前已是最新版本(${head.slice(0, 7)})。`
: `本地版本已领先更新源 ${Number.isFinite(ahead) ? ahead : 0} 个提交,无需更新。`
: `发现新版本 ${latest.slice(0, 7)},可以下载更新包。`,
error: '',
progressPercent: 100,
release,
pendingCommit: '',
pendingAction: '',
});
return this.#view();
});
}
async download(): Promise<UpdateStatus> {
return this.#exclusive(async () => {
const release = this.#requireCandidate();
this.#save({
phase: 'downloading',
message: '正在下载并校验更新包…',
error: '',
progressPercent: 35,
release,
pendingCommit: release.latestCommit,
pendingAction: '',
});
const fetched = await this.#git([
'fetch',
'--quiet',
'--no-tags',
release.remote,
release.branch,
]);
if (fetched.code !== 0) {
this.#fail('下载更新包失败,请检查网络后重试。');
throw new UpdateError('COMMAND', '下载更新包失败,请检查网络后重试。', 502);
}
const candidate = (await this.#text(['rev-parse', 'FETCH_HEAD'])).trim();
if (candidate !== release.latestCommit) {
this.#fail('远端分支已变化,请重新检查更新。');
throw new UpdateError('STATE', '远端分支已变化,请重新检查更新。', 409);
}
const ancestor = await this.#git(['merge-base', '--is-ancestor', 'HEAD', 'FETCH_HEAD']);
if (ancestor.code !== 0) {
this.#fail('本地分支已偏离远端,无法快进合并,请先处理本地提交。');
return this.#view();
}
const ahead = Number(await this.#text(['rev-list', '--count', 'HEAD..FETCH_HEAD']));
const stat = parseShortStat(await this.#text(['diff', '--shortstat', 'HEAD', 'FETCH_HEAD']));
const manifest = parseManifestVersion(await this.#text(['show', 'FETCH_HEAD:package.json']));
const commits = Number.isFinite(ahead) ? ahead : 0;
const updated: UpdateRelease = {
...release,
latestVersion: manifest,
ahead: commits,
changedFiles: stat.files,
changedLines: stat.lines,
summary: `领先 ${commits} 个提交,${stat.files} 个文件、${stat.lines} 行改动。`,
};
this.#save({
phase: 'ready',
message: `更新包已校验:${updated.summary}`,
error: '',
progressPercent: 100,
release: updated,
pendingCommit: updated.latestCommit,
pendingAction: '',
});
return this.#view();
});
}
async install(): Promise<UpdateStatus> {
return this.#exclusive(async () => {
const release = this.#requireCandidate();
if (this.#state.phase !== 'ready' && this.#state.phase !== 'rolled_back')
throw new UpdateError('STATE', '请先下载并校验更新包。', 409);
// The rollback below resets the working tree, so it is only ever allowed to start from a
// clean one; otherwise an operator's uncommitted work would be the first casualty.
if ((await this.#text(['status', '--porcelain'])).trim())
throw new UpdateError('STATE', '工作区存在未提交改动,为避免丢失内容已中止安装。', 409);
const previousCommit = await this.#localHead();
if (!previousCommit) return this.#view();
if (this.#preInstallBackup) {
this.#save({
phase: 'installing',
message: '正在创建更新前备份…',
error: '',
progressPercent: 5,
release,
pendingCommit: '',
pendingAction: '',
});
try {
await this.#preInstallBackup(release.latestCommit);
} catch {
this.#fail('更新前备份失败,已中止安装,代码保持原版本。');
throw new UpdateError('COMMAND', '更新前备份失败,已中止安装。', 500);
}
}
this.#save({
phase: 'installing',
message: '正在安装更新…',
error: '',
progressPercent: 20,
release,
pendingCommit: release.latestCommit,
pendingAction: 'install',
});
const merge = await this.#git(['merge', '--ff-only', release.latestCommit]);
if (merge.code !== 0) {
this.#fail('快进合并失败,代码保持原版本。');
throw new UpdateError('COMMAND', '快进合并失败,代码保持原版本。', 500);
}
this.#progress(55, '依赖安装中…');
const dependencies = await this.#runner.run(
'corepack',
['pnpm', 'install', '--frozen-lockfile'],
{ timeoutMs: INSTALL_TIMEOUT_MS },
);
if (dependencies.code !== 0) {
await this.#rollback(previousCommit, '依赖安装失败,已回滚到原版本。');
throw new UpdateError('COMMAND', '依赖安装失败,已回滚到原版本。', 500);
}
this.#progress(80, '前端构建中…');
const build = await this.#runner.run(
'corepack',
['pnpm', '--filter', '@multi-simadmin/web', 'build'],
{ timeoutMs: INSTALL_TIMEOUT_MS },
);
if (build.code !== 0) {
await this.#rollback(previousCommit, '前端构建失败,已回滚到原版本。');
throw new UpdateError('COMMAND', '前端构建失败,已回滚到原版本。', 500);
}
this.#save({
phase: 'install_queued',
message: '安装完成,重启后生效。',
error: '',
progressPercent: 100,
release,
pendingCommit: release.latestCommit,
pendingAction: 'install',
});
return this.#view();
});
}
/**
* Hands the process back to the supervisor. The reply flushes first: killing the API in the
* middle of a request leaves the browser with a connection error instead of a result.
*/
async restart(): Promise<UpdateStatus> {
if (!this.#launcher.supported)
throw new UpdateError('NOT_SUPPORTED', '当前部署方式不支持在线重启,请手动重启服务。', 501);
if (this.#inFlight) throw new UpdateError('BUSY', '更新任务正在进行中。', 409);
const installing = this.#state.phase === 'install_queued';
this.#save({
phase: 'restarting',
message: '服务正在重启,页面连接将短暂中断。',
error: '',
progressPercent: 100,
release: this.#state.release,
pendingCommit: installing ? this.#state.pendingCommit : '',
pendingAction: installing ? 'install' : 'restart',
});
setTimeout(() => {
void this.#launcher.restart().catch(() => {
this.#save({
phase: 'failed',
message: '重启请求未能交给服务管理器。',
error: '重启请求未能交给服务管理器,请手动重启服务。',
progressPercent: 100,
release: this.#state.release,
pendingCommit: '',
pendingAction: '',
});
});
}, this.#restartDelayMs);
return this.#view();
}
async #exclusive(action: () => Promise<UpdateStatus>): Promise<UpdateStatus> {
if (this.#inFlight) throw new UpdateError('BUSY', '更新任务正在进行中,请稍候。', 409);
const pending = (async () => {
try {
return await action();
} finally {
this.#inFlight = undefined;
}
})();
this.#inFlight = pending;
return pending;
}
#requireCandidate(): UpdateRelease {
const release = this.#state.release;
if (!release || release.status !== 'update_available' || !COMMIT.test(release.latestCommit))
throw new UpdateError('STATE', '请先检查更新。', 409);
return release;
}
async #localHead(): Promise<string> {
const result = await this.#git(['rev-parse', 'HEAD']);
const head = result.stdout.trim();
if (result.code !== 0 || !COMMIT.test(head)) {
this.#mode = 'manual';
this.#save({
phase: 'idle',
message: '当前部署不是 Git 工作副本,请按安装文档手动更新后重启。',
error: '',
progressPercent: 0,
release: null,
pendingCommit: '',
pendingAction: '',
});
return '';
}
this.#mode = 'git';
return head;
}
async #git(args: readonly string[]): Promise<UpdateCommandResult> {
return this.#runner.run('git', args);
}
/** True when HEAD already carries `commit`. A missing object reads as "not contained". */
async #contains(commit: string): Promise<boolean> {
if (!COMMIT.test(commit)) return false;
const probe = await this.#git(['merge-base', '--is-ancestor', commit, 'HEAD']);
return probe.code === 0;
}
async #text(args: readonly string[]): Promise<string> {
const result = await this.#git(args);
return result.code === 0 ? text(result.stdout) : '';
}
#progress(progressPercent: number, message: string): void {
this.#save({
phase: 'installing',
message,
error: '',
progressPercent,
release: this.#state.release,
pendingCommit: this.#state.pendingCommit,
pendingAction: 'install',
});
}
async #rollback(previousCommit: string, message: string): Promise<void> {
await this.#git(['reset', '--hard', previousCommit]);
this.#save({
phase: 'rolled_back',
message,
error: message,
progressPercent: 100,
release: this.#state.release,
pendingCommit: '',
pendingAction: '',
});
}
#fail(message: string): void {
this.#save({
phase: 'failed',
message,
error: message,
progressPercent: this.#state.progressPercent,
release: this.#state.release,
pendingCommit: '',
pendingAction: '',
});
}
/**
* A restarted process has no memory of the install. When the recorded target commit is now
* HEAD, the update landed, and that is the confirmation the UI has been waiting for.
*/
async #reconcile(): Promise<void> {
if (this.#reconciled) return;
this.#reconciled = true;
const state = this.#state;
if (!state.pendingAction || !BUSY_UPDATE_PHASES.has(state.phase)) return;
const head = await this.#localHead();
if (!head) return;
if (state.pendingAction === 'install' && head !== state.pendingCommit) {
this.#save({
...state,
phase: 'failed',
message: '重启后未看到目标版本,安装可能未完成。',
error: '重启后未看到目标版本,请查看服务日志。',
pendingCommit: '',
pendingAction: '',
});
return;
}
this.#save({
...state,
phase: 'up_to_date',
message:
state.pendingAction === 'install'
? `已更新到 ${head.slice(0, 7)} 并完成重启。`
: '服务已重启完成。',
error: '',
progressPercent: 100,
release: state.release ? { ...state.release, status: 'latest', currentCommit: head } : null,
pendingCommit: '',
pendingAction: '',
});
}
#view(): UpdateStatus {
const git = this.#mode === 'git';
return {
phase: this.#state.phase,
message: this.#state.message,
error: this.#state.error,
progressPercent: this.#state.progressPercent,
checkedAt: this.#state.checkedAt,
deploymentMode: this.#mode === 'unknown' ? 'git' : this.#mode,
installSupported: git,
restartSupported: this.#launcher.supported,
release: this.#state.release,
};
}
#emptyState(message: string): PersistedState {
return {
phase: 'idle',
message,
error: '',
progressPercent: 0,
checkedAt: '',
release: null,
pendingCommit: '',
pendingAction: '',
};
}
#read(): PersistedState {
let row: { value_json?: string | null } | undefined;
try {
row = this.#db.prepare('SELECT value_json FROM app_settings WHERE key = ?').get(STATE_KEY) as
| { value_json?: string | null }
| undefined;
} catch {
row = undefined;
}
if (!row?.value_json) return this.#emptyState('尚未检查更新。');
try {
const parsed: unknown = JSON.parse(row.value_json);
const source =
parsed && typeof parsed === 'object' && !Array.isArray(parsed)
? (parsed as Record<string, unknown>)
: {};
const pendingAction = source.pendingAction;
return {
phase: phaseOf(source.phase),
message: text(source.message, 256),
error: text(source.error, 256),
progressPercent: percent(source.progressPercent),
checkedAt: text(source.checkedAt, 64),
release: readRelease(source.release),
pendingCommit: text(source.pendingCommit, 40),
pendingAction:
pendingAction === 'install' || pendingAction === 'restart' ? pendingAction : '',
};
} catch {
return this.#emptyState('尚未检查更新。');
}
}
#save(input: Omit<PersistedState, 'checkedAt'>): void {
const now = this.#now().toISOString();
this.#state = { ...input, checkedAt: now };
try {
this.#db
.prepare(
`INSERT INTO app_settings (key,value_json,created_at,updated_at)
VALUES (?,?,?,?)
ON CONFLICT(key) DO UPDATE SET value_json = excluded.value_json,
updated_at = excluded.updated_at`,
)
.run(STATE_KEY, JSON.stringify(this.#state), now, now);
} catch {
// A read-only database must not break the update flow; the in-memory state still leads.
}
}
}
@@ -0,0 +1,189 @@
import Database from 'better-sqlite3';
import { afterEach, describe, expect, it } from 'vitest';
import { migrateDatabase } from '../../infrastructure/database/migrations.js';
import { ConnectionLogService } from './connection-log-service.js';
import { LogCenterService } from './log-center-service.js';
const databases: Database.Database[] = [];
function fixture() {
const db = new Database(':memory:');
db.pragma('foreign_keys = ON');
migrateDatabase(db);
databases.push(db);
db.prepare(
`INSERT INTO instances (id,name,base_url,enabled,config_revision,created_at,updated_at)
VALUES ('node-a','Node A','http://node-a.local',1,3,'2026-09-01T00:00:00.000Z','2026-09-01T00:00:00.000Z')`,
).run();
db.prepare(
`INSERT INTO instance_tags (instance_id,tag,created_at) VALUES ('node-a','lab','2026-09-01T00:00:00.000Z')`,
).run();
const connections = new ConnectionLogService({ db });
return { db, connections, logs: new LogCenterService({ db, connections }) };
}
afterEach(() => {
for (const db of databases.splice(0)) db.close();
});
function seed(db: Database.Database): void {
db.prepare(
`INSERT INTO event_journal (public_id,envelope_json)
VALUES ('evt-1',@envelope)`,
).run({
envelope: JSON.stringify({
id: 'evt-1',
kind: 'instance.updated',
instanceId: 'node-a',
occurredAt: '2026-09-04T02:00:00.000Z',
}),
});
db.prepare(
`INSERT INTO audit_events
(id,instance_id,actor,operation_id,risk_level,request_id,parameters_summary_json,result_code,duration_ms,created_at)
VALUES ('aud-1','node-a','console','restart-service','R1','req-1','{}','failed',42,'2026-09-04T03:00:00.000Z')`,
).run();
db.prepare(
`INSERT INTO scheduled_tasks
(id,name,operation_type,enabled,version,cron_expression,timezone,target_selector_json,
misfire_policy,overlap_policy,retry_policy_json,created_by,updated_by,created_at,updated_at)
VALUES ('task-1','夜间重启','restart-service',1,1,'0 3 * * *','Asia/Shanghai','{}',
'skip','skip','{"maximumAttempts":1}','console','console','2026-09-01T00:00:00.000Z','2026-09-01T00:00:00.000Z')`,
).run();
db.prepare(
`INSERT INTO scheduled_runs
(id,scheduled_task_id,schedule_version,task_snapshot_json,due_at,claimed_at,started_at,finished_at,
target_snapshot_json,outcome,reason,job_ids_json,trigger_source,attempt)
VALUES ('run-1','task-1',1,'{"name":"夜间重启"}','2026-09-04T04:00:00.000Z','2026-09-04T04:00:00.000Z',
'2026-09-04T04:00:00.000Z','2026-09-04T04:00:05.000Z','[]','failed','UPSTREAM_TIMEOUT','[]','scheduled',1)`,
).run();
db.prepare(
`INSERT INTO notification_deliveries
(id,rule_id,channel_id,instance_id,event_type,status,detail,created_at,sent_at)
VALUES ('del-1',NULL,NULL,'node-a','sms','failed','smtp refused','2026-09-04T05:00:00.000Z',NULL)`,
).run();
}
describe('LogCenterService', () => {
it('merges every persisted activity source into one timeline', () => {
const { db, logs } = fixture();
seed(db);
const page = logs.listRuntimeLogs();
expect(page.page.total).toBe(4);
expect(page.items.map((item) => item.source)).toEqual([
'delivery',
'schedule',
'audit',
'event',
]);
expect(page.counts).toEqual({ event: 1, audit: 1, schedule: 1, delivery: 1 });
expect(page.items[1]).toMatchObject({ level: 'error', code: 'failed' });
});
it('filters the timeline by level, source and search text', () => {
const { db, logs } = fixture();
seed(db);
expect(logs.listRuntimeLogs({ level: 'error' }).items.map((item) => item.source)).toEqual([
'delivery',
'schedule',
'audit',
]);
expect(logs.listRuntimeLogs({ source: 'event' }).page.total).toBe(1);
expect(logs.listRuntimeLogs({ search: 'smtp' }).page.total).toBe(1);
expect(logs.listRuntimeLogs({ from: '2026-09-04T04:30:00.000Z' }).page.total).toBe(1);
expect(logs.listRuntimeLogs({ instanceId: 'node-a' }).page.total).toBe(3);
});
it('rejects malformed pagination and unknown enum values', () => {
const { logs } = fixture();
expect(() => logs.listRuntimeLogs({ page: 0 })).toThrow(/positive integer/u);
expect(() => logs.listRuntimeLogs({ pageSize: 500 })).toThrow(/between 1 and 200/u);
expect(() => logs.listRuntimeLogs({ source: 'nope' as never })).toThrow(/source is invalid/u);
expect(() => logs.listRuntimeLogs({ from: 'yesterday' })).toThrow(/from is invalid/u);
});
it('reports connection snapshots as device health for the live probe path', () => {
const { db, connections, logs } = fixture();
db.prepare(
`INSERT INTO status_snapshots
(id,instance_id,category,state,payload_json,observed_at,expires_at,created_at)
VALUES ('snap-1','node-a','connection','fresh','{"reachable":true,"authenticated":false}',
'2026-09-04T06:00:00.000Z','2026-09-04T06:00:30.000Z','2026-09-04T06:00:00.000Z')`,
).run();
db.prepare(
`INSERT INTO capabilities
(instance_id,operation_id,state,observed_at,created_at,updated_at)
VALUES ('node-a','restart-service','supported','2026-09-04T06:00:00.000Z',
'2026-09-04T06:00:00.000Z','2026-09-04T06:00:00.000Z'),
('node-a','reboot-system','unsupported','2026-09-04T06:00:00.000Z',
'2026-09-04T06:00:00.000Z','2026-09-04T06:00:00.000Z')`,
).run();
connections.record({
instanceId: 'node-a',
outcome: 'stale',
state: 'stale',
errorCode: 'AUTH_REQUIRED',
httpStatus: 401,
durationMs: 15,
observedAt: '2026-09-04T06:00:00.000Z',
});
const [device] = logs.diagnostics();
expect(device).toMatchObject({
instanceId: 'node-a',
name: 'Node A',
enabled: true,
revision: 3,
tags: ['lab'],
capabilities: { total: 2, supported: 1, unsupported: 1 },
});
expect(device?.health).toMatchObject({
category: 'connection',
status: 'reachable',
authenticated: false,
errorCode: 'AUTH_REQUIRED',
});
expect(device?.connections).toMatchObject({ total: 1, availabilityPercent: 0, failed: 0 });
});
it('prefers a full health envelope when both snapshot categories exist', () => {
const { db, logs } = fixture();
for (const [category, payload] of [
['connection', '{"reachable":true,"authenticated":true}'],
[
'health',
'{"schemaVersion":1,"data":{"status":"ok","version":"1.9.6","platform":"linux"},' +
'"errorCode":null,"httpStatus":200,"fetchedAt":"2026-09-04T06:00:00.000Z","durationMs":21,' +
'"freshness":"fresh","supported":true,"dataFetchedAt":null}',
],
] as const) {
db.prepare(
`INSERT INTO status_snapshots
(id,instance_id,category,state,payload_json,observed_at,expires_at,created_at)
VALUES (?, 'node-a', ?, 'fresh', ?, '2026-09-04T06:00:00.000Z', NULL, '2026-09-04T06:00:00.000Z')`,
).run(`snap-${category}`, category, payload);
}
const [device] = logs.diagnostics();
expect(device?.health).toMatchObject({
category: 'health',
version: '1.9.6',
platform: 'linux',
status: 'ok',
durationMs: 21,
httpStatus: 200,
});
expect(device?.snapshots.map((snapshot) => snapshot.category)).toEqual([
'connection',
'health',
]);
});
it('surfaces the most recent failed operations per device', () => {
const { db, logs } = fixture();
seed(db);
const [device] = logs.diagnostics();
expect(device?.recentFailures).toEqual([
{ occurredAt: '2026-09-04T03:00:00.000Z', code: 'failed', message: 'restart-service' },
]);
});
});
@@ -0,0 +1,445 @@
import type Database from 'better-sqlite3';
import type { ConnectionLogService, ConnectionSummary } from './connection-log-service.js';
export type LogSource = 'event' | 'audit' | 'schedule' | 'delivery';
export type LogLevel = 'info' | 'warning' | 'error';
const SOURCES: readonly LogSource[] = ['event', 'audit', 'schedule', 'delivery'];
const LEVELS: readonly LogLevel[] = ['info', 'warning', 'error'];
export interface RuntimeLogEntry {
readonly id: string;
readonly source: LogSource;
readonly level: LogLevel;
readonly occurredAt: string;
readonly instanceId: string | null;
readonly message: string;
readonly code: string | null;
readonly actor: string | null;
readonly durationMs: number | null;
}
export interface RuntimeLogQuery {
readonly page?: number;
readonly pageSize?: number;
readonly source?: LogSource | undefined;
readonly level?: LogLevel | undefined;
readonly instanceId?: string | undefined;
readonly search?: string | undefined;
readonly from?: string | undefined;
readonly to?: string | undefined;
}
export interface RuntimeLogPage {
readonly items: readonly RuntimeLogEntry[];
readonly page: { readonly page: number; readonly pageSize: number; readonly total: number };
readonly counts: Readonly<Record<LogSource, number>>;
}
export interface InstanceDiagnostics {
readonly instanceId: string;
readonly name: string;
readonly origin: string;
readonly enabled: boolean;
readonly revision: number;
readonly tags: readonly string[];
readonly health: {
readonly category: 'health' | 'connection';
readonly state: string;
readonly observedAt: string;
readonly errorCode: string | null;
readonly httpStatus: number | null;
readonly durationMs: number | null;
readonly version: string | null;
readonly platform: string | null;
readonly status: string | null;
readonly authenticated: boolean | null;
} | null;
readonly snapshots: readonly {
readonly category: string;
readonly state: string;
readonly observedAt: string;
}[];
readonly capabilities: {
readonly total: number;
readonly supported: number;
readonly unsupported: number;
readonly authRequired: number;
readonly degraded: number;
readonly unknown: number;
};
readonly connections: ConnectionSummary | null;
readonly recentFailures: readonly {
readonly occurredAt: string;
readonly code: string | null;
readonly message: string;
}[];
}
export type LogCenterErrorCode = 'VALIDATION_FAILED';
export class LogCenterError extends Error {
constructor(
readonly code: LogCenterErrorCode,
message: string,
) {
super(message);
this.name = 'LogCenterError';
}
}
export interface LogCenterOptions {
readonly db: Database.Database;
readonly connections: ConnectionLogService;
}
const MAX_PAGE_SIZE = 200;
function invalid(message: string): never {
throw new LogCenterError('VALIDATION_FAILED', message);
}
function instant(value: string): string | undefined {
const parsed = Date.parse(value);
return Number.isFinite(parsed) ? new Date(parsed).toISOString() : undefined;
}
/**
* One UNION over the tables that already persist control-plane activity. Nothing new is
* written here, so the timeline stays consistent with the pages that own each source.
*/
const TIMELINE_SQL = `
SELECT id, source, level, occurred_at, instance_id, message, code, actor, duration_ms FROM (
SELECT e.public_id || ':' || e.sequence AS id,
'event' AS source,
'info' AS level,
json_extract(e.envelope_json,'$.occurredAt') AS occurred_at,
json_extract(e.envelope_json,'$.instanceId') AS instance_id,
'事件 ' || json_extract(e.envelope_json,'$.kind') || ' ' || json_extract(e.envelope_json,'$.id')
AS message,
json_extract(e.envelope_json,'$.kind') AS code,
NULL AS actor,
NULL AS duration_ms
FROM event_journal e
UNION ALL
SELECT a.id,
'audit',
CASE WHEN a.result_code = 'success' THEN 'info' ELSE 'error' END,
a.created_at,
a.instance_id,
'操作 ' || a.operation_id || ' 结果 ' || a.result_code,
a.result_code,
a.actor,
a.duration_ms
FROM audit_events a
UNION ALL
SELECT r.id,
'schedule',
CASE
WHEN r.outcome IN ('failed','needs-attention') THEN 'error'
WHEN r.outcome IN ('partially-succeeded','skipped','no-targets') THEN 'warning'
ELSE 'info'
END,
COALESCE(r.finished_at, r.due_at),
NULL,
'定时任务 ' || COALESCE(json_extract(r.task_snapshot_json,'$.name'), r.scheduled_task_id)
|| ' 结果 ' || COALESCE(r.outcome, 'running')
|| COALESCE(' 原因 ' || r.reason, ''),
COALESCE(r.outcome, 'running'),
NULL,
CAST((julianday(COALESCE(r.finished_at, r.claimed_at)) - julianday(r.started_at))
* 86400000 AS INTEGER)
FROM scheduled_runs r
UNION ALL
SELECT d.id,
'delivery',
CASE
WHEN d.status = 'failed' THEN 'error'
WHEN d.status IN ('unmatched','no_available_channel','quiet_hours','rate_limited')
THEN 'warning'
ELSE 'info'
END,
d.created_at,
d.instance_id,
'通知 ' || d.event_type || ' 状态 ' || d.status
|| COALESCE(' 详情 ' || d.detail, ''),
d.status,
NULL,
NULL
FROM notification_deliveries d
)`;
export class LogCenterService {
readonly #db: Database.Database;
readonly #connections: ConnectionLogService;
constructor(options: LogCenterOptions) {
this.#db = options.db;
this.#connections = options.connections;
}
listRuntimeLogs(query: RuntimeLogQuery = {}): RuntimeLogPage {
const page = query.page ?? 1;
const pageSize = query.pageSize ?? 50;
if (!Number.isSafeInteger(page) || page < 1) invalid('page must be a positive integer');
if (!Number.isSafeInteger(pageSize) || pageSize < 1 || pageSize > MAX_PAGE_SIZE)
invalid(`pageSize must be between 1 and ${MAX_PAGE_SIZE}`);
if (query.source !== undefined && !SOURCES.includes(query.source)) invalid('source is invalid');
if (query.level !== undefined && !LEVELS.includes(query.level)) invalid('level is invalid');
if (query.search !== undefined && query.search.length > 200) invalid('search is too long');
if (query.instanceId !== undefined && query.instanceId.trim() === '')
invalid('instanceId is invalid');
const bounds: Record<string, string> = {};
for (const key of ['from', 'to'] as const) {
const value = query[key];
if (value === undefined) continue;
const normalized = instant(value);
if (!normalized) invalid(`${key} is invalid`);
bounds[key] = normalized;
}
const where: string[] = [];
const parameters: Record<string, unknown> = {};
if (query.source) {
where.push('source = @source');
parameters.source = query.source;
}
if (query.level) {
where.push('level = @level');
parameters.level = query.level;
}
if (query.instanceId) {
where.push('instance_id = @instanceId');
parameters.instanceId = query.instanceId;
}
if (bounds.from) {
where.push('occurred_at >= @from');
parameters.from = bounds.from;
}
if (bounds.to) {
where.push('occurred_at <= @to');
parameters.to = bounds.to;
}
const needle = query.search?.trim().toLowerCase() ?? '';
if (needle !== '') {
where.push(
`(lower(message) LIKE @search OR lower(COALESCE(code,'')) LIKE @search
OR lower(COALESCE(actor,'')) LIKE @search OR lower(COALESCE(instance_id,'')) LIKE @search)`,
);
parameters.search = `%${needle}%`;
}
const clause = where.length > 0 ? ` WHERE ${where.join(' AND ')}` : '';
const total = (
this.#db
.prepare(`SELECT COUNT(*) AS count FROM (${TIMELINE_SQL})${clause}`)
.get(parameters) as { count: number }
).count;
const rows = this.#db
.prepare(
`SELECT id, source, level, occurred_at, instance_id, message, code, actor, duration_ms
FROM (${TIMELINE_SQL})${clause}
ORDER BY occurred_at DESC LIMIT @limit OFFSET @offset`,
)
.all({ ...parameters, limit: pageSize, offset: (page - 1) * pageSize }) as Array<
Record<string, unknown>
>;
const counts = Object.fromEntries(SOURCES.map((source) => [source, 0])) as Record<
LogSource,
number
>;
const grouped = this.#db
.prepare(`SELECT source, COUNT(*) AS count FROM (${TIMELINE_SQL}) GROUP BY source`)
.all() as Array<{ source: string; count: number }>;
for (const row of grouped) {
if (SOURCES.includes(row.source as LogSource)) counts[row.source as LogSource] = row.count;
}
return Object.freeze({
items: Object.freeze(rows.map((row) => this.#entry(row))),
page: Object.freeze({ page, pageSize, total }),
counts: Object.freeze(counts),
});
}
/** Per-instance diagnostics for the whole fleet, newest failure surfaced first. */
diagnostics(): readonly InstanceDiagnostics[] {
const instances = this.#db
.prepare(
`SELECT id,name,base_url,enabled,config_revision FROM instances ORDER BY name COLLATE NOCASE`,
)
.all() as Array<Record<string, unknown>>;
const tagRows = this.#db.prepare('SELECT instance_id, tag FROM instance_tags').all() as Array<{
instance_id: string;
tag: string;
}>;
const tagsByInstance = new Map<string, string[]>();
for (const row of tagRows) {
const bucket = tagsByInstance.get(row.instance_id) ?? [];
bucket.push(row.tag);
tagsByInstance.set(row.instance_id, bucket);
}
const summaries = new Map(
this.#connections.summarize().map((summary) => [summary.instanceId, summary]),
);
return Object.freeze(
instances.map((instance) =>
Object.freeze(this.#diagnostics(instance, tagsByInstance, summaries)),
),
);
}
#diagnostics(
instance: Record<string, unknown>,
tagsByInstance: ReadonlyMap<string, readonly string[]>,
summaries: ReadonlyMap<string, ConnectionSummary>,
): InstanceDiagnostics {
const instanceId = String(instance.id);
// The control plane persists 'health' envelopes only when the snapshot service runs; the
// live reachability path writes 'connection' rows, so diagnostics reads whichever exists.
const snapshot = this.#db
.prepare(
`SELECT category, state, payload_json, observed_at FROM status_snapshots
WHERE instance_id = ? AND category IN ('health','connection')
ORDER BY CASE WHEN category = 'health' THEN 0 ELSE 1 END LIMIT 1`,
)
.get(instanceId) as
| { category: string; state: string; payload_json: string; observed_at: string }
| undefined;
const envelope = parseRecord(snapshot?.payload_json);
const data = parseRecord(envelope?.data) ?? envelope;
const capability = this.#db
.prepare(
`SELECT
COUNT(*) AS total,
SUM(CASE WHEN state = 'supported' THEN 1 ELSE 0 END) AS supported,
SUM(CASE WHEN state = 'unsupported' THEN 1 ELSE 0 END) AS unsupported,
SUM(CASE WHEN state = 'auth-required' THEN 1 ELSE 0 END) AS auth_required,
SUM(CASE WHEN state = 'degraded' THEN 1 ELSE 0 END) AS degraded,
SUM(CASE WHEN state = 'unknown' THEN 1 ELSE 0 END) AS unknown_state
FROM capabilities WHERE instance_id = ?`,
)
.get(instanceId) as Record<string, number | null> | undefined;
const snapshots = this.#db
.prepare(
`SELECT category, state, observed_at FROM status_snapshots
WHERE instance_id = ? ORDER BY category`,
)
.all(instanceId) as Array<{ category: string; state: string; observed_at: string }>;
const failures = this.#db
.prepare(
`SELECT id, operation_id, result_code, created_at FROM audit_events
WHERE instance_id = ? AND result_code <> 'success'
ORDER BY created_at DESC LIMIT 5`,
)
.all(instanceId) as Array<{
id: string;
operation_id: string;
result_code: string;
created_at: string;
}>;
return {
instanceId,
name: String(instance.name ?? instanceId),
origin: String(instance.base_url ?? ''),
enabled: Number(instance.enabled ?? 0) === 1,
revision: Number(instance.config_revision ?? 1),
tags: Object.freeze([...(tagsByInstance.get(instanceId) ?? [])]),
health: snapshot ? Object.freeze(this.#health(snapshot, envelope, data)) : null,
snapshots: Object.freeze(
snapshots.map((row) =>
Object.freeze({ category: row.category, state: row.state, observedAt: row.observed_at }),
),
),
capabilities: Object.freeze({
total: Number(capability?.total ?? 0),
supported: Number(capability?.supported ?? 0),
unsupported: Number(capability?.unsupported ?? 0),
authRequired: Number(capability?.auth_required ?? 0),
degraded: Number(capability?.degraded ?? 0),
unknown: Number(capability?.unknown_state ?? 0),
}),
connections: summaries.get(instanceId) ?? null,
recentFailures: Object.freeze(
failures.map((row) =>
Object.freeze({
occurredAt: row.created_at,
code: row.result_code,
message: row.operation_id,
}),
),
),
};
}
#entry(row: Record<string, unknown>): RuntimeLogEntry {
return Object.freeze({
id: String(row.id),
source: row.source as LogSource,
level: row.level as LogLevel,
occurredAt: String(row.occurred_at ?? ''),
instanceId:
row.instance_id === null || row.instance_id === undefined ? null : String(row.instance_id),
message: String(row.message ?? ''),
code: row.code === null || row.code === undefined ? null : String(row.code),
actor: row.actor === null || row.actor === undefined ? null : String(row.actor),
durationMs:
row.duration_ms === null || row.duration_ms === undefined ? null : Number(row.duration_ms),
});
}
#health(
snapshot: { category: string; state: string; observed_at: string },
envelope: Record<string, unknown> | undefined,
data: Record<string, unknown> | undefined,
): NonNullable<InstanceDiagnostics['health']> {
const category = snapshot.category === 'connection' ? 'connection' : 'health';
if (category === 'connection') {
const authenticated =
typeof envelope?.authenticated === 'boolean' ? envelope.authenticated : null;
const reachable = typeof envelope?.reachable === 'boolean' ? envelope.reachable : null;
return Object.freeze({
category,
state: snapshot.state,
observedAt: snapshot.observed_at,
errorCode: authenticated === false ? 'AUTH_REQUIRED' : null,
httpStatus: null,
durationMs: null,
version: null,
platform: null,
status: reachable === null ? null : reachable ? 'reachable' : 'unreachable',
authenticated,
});
}
return Object.freeze({
category,
state: snapshot.state,
observedAt: snapshot.observed_at,
errorCode: typeof envelope?.errorCode === 'string' ? envelope.errorCode : null,
httpStatus: typeof envelope?.httpStatus === 'number' ? envelope.httpStatus : null,
durationMs: typeof envelope?.durationMs === 'number' ? envelope.durationMs : null,
version: typeof data?.version === 'string' ? data.version : null,
platform: typeof data?.platform === 'string' ? data.platform : null,
status: typeof data?.status === 'string' ? data.status : null,
authenticated: null,
});
}
}
/** Accepts either a JSON string or an already-decoded object so both snapshot shapes work. */
function parseRecord(value: unknown): Record<string, unknown> | undefined {
if (value !== null && typeof value === 'object' && !Array.isArray(value))
return value as Record<string, unknown>;
if (typeof value !== 'string' || value === '') return undefined;
try {
const parsed: unknown = JSON.parse(value);
if (parsed === null || typeof parsed !== 'object' || Array.isArray(parsed)) return undefined;
return parsed as Record<string, unknown>;
} catch {
return undefined;
}
}
@@ -0,0 +1,243 @@
import Database from 'better-sqlite3';
import { mkdtemp, readFile, rm } from 'node:fs/promises';
import { join } from 'node:path';
import { tmpdir } from 'node:os';
import { afterEach, describe, expect, it } from 'vitest';
import { migrateDatabase } from '../../infrastructure/database/migrations.js';
import { SystemMaintenanceService } from './system-maintenance-service.js';
const roots: string[] = [];
afterEach(async () => {
await Promise.all(roots.splice(0).map((root) => rm(root, { recursive: true, force: true })));
});
function database(): Database.Database {
const db = new Database(':memory:');
db.pragma('foreign_keys = ON');
migrateDatabase(db);
return db;
}
function insertAudit(db: Database.Database, id: string, createdAt: string): void {
db.prepare(
`INSERT INTO audit_events
(id,actor,operation_id,risk_level,request_id,parameters_summary_json,result_code,duration_ms,created_at)
VALUES (?,?,?,?,?,?,?,?,?)`,
).run(id, 'operator', 'probe', 'R0', `${id}-request`, '{}', 'success', 1, createdAt);
}
function insertJob(db: Database.Database, id: string, createdAt: string): void {
db.prepare(
`INSERT INTO jobs
(id,operation_id,risk_level,status,requested_by,request_id,parameters_digest,created_at,updated_at)
VALUES (?,?,?,?,?,?,?,?,?)`,
).run(
id,
'probe',
'R0',
'succeeded',
'operator',
`${id}-request`,
'0'.repeat(64),
createdAt,
createdAt,
);
}
function insertEvent(db: Database.Database, publicId: string, envelope: unknown): void {
db.prepare('INSERT INTO event_journal (public_id,envelope_json) VALUES (?,?)').run(
publicId,
JSON.stringify(envelope),
);
}
describe('SystemMaintenanceService', () => {
it('reports runtime storage, component counts, and Hub-compatible default retention', async () => {
const db = database();
insertAudit(db, 'audit-1', '2026-01-01T00:00:00.000Z');
insertJob(db, 'job-1', '2026-01-01T00:00:00.000Z');
insertEvent(db, 'event-1', { kind: 'test' });
const service = new SystemMaintenanceService(db, { version: '1.9.9' });
const overview = await service.overview();
expect(overview.runtime).toMatchObject({
version: '1.9.9',
platform: process.platform,
arch: process.arch,
});
expect(overview.runtime.uptimeSeconds).toBeGreaterThanOrEqual(0);
expect(overview.storage.databaseBytes).toBeGreaterThan(0);
expect(
Object.fromEntries(overview.storage.components.map((item) => [item.key, item.count])),
).toEqual({
instances: 0,
statusSnapshots: 0,
jobs: 1,
auditEvents: 1,
scheduledRuns: 0,
notificationQueue: 0,
smsMessages: 0,
eventJournal: 1,
connectionLogs: 0,
});
expect(overview.retention.auditEvents).toEqual({
enabled: true,
days: 180,
maximumCount: 50_000,
});
expect(overview.retention.jobs).toEqual({
enabled: true,
days: 90,
maximumCount: 5_000,
});
});
it('persists validated retention policies and cleans selected components', async () => {
const db = database();
const task = {
name: 'Night restart',
operationType: 'restart-service' as const,
cronExpression: '0 2 * * *',
timezone: 'Asia/Shanghai' as const,
targetSelector: { mode: 'fixed' as const, instanceIds: ['instance-a'] },
misfirePolicy: 'skip' as const,
overlapPolicy: 'skip' as const,
retryPolicy: { maxRetries: 0, intervalSeconds: 60 },
enabled: false,
};
db.prepare(
`INSERT INTO scheduled_tasks
(id,name,operation_type,enabled,version,cron_expression,timezone,target_selector_json,
misfire_policy,overlap_policy,retry_policy_json,created_by,updated_by,created_at,updated_at)
VALUES ('task-1',?,?,?,?,?,?,?,?,?,?,?,?,?,?)`,
).run(
task.name,
task.operationType,
0,
1,
task.cronExpression,
task.timezone,
JSON.stringify(task.targetSelector),
task.misfirePolicy,
task.overlapPolicy,
JSON.stringify(task.retryPolicy),
'operator',
'operator',
'2026-01-01T00:00:00.000Z',
'2026-01-01T00:00:00.000Z',
);
db.prepare(
`INSERT INTO scheduled_runs
(id,scheduled_task_id,schedule_version,task_snapshot_json,due_at,claimed_at,
target_snapshot_json,job_ids_json,trigger_source,attempt)
VALUES ('run-old','task-1',1,?,'2026-01-01T02:00:00.000Z','2026-01-01T02:00:00.000Z','[]','[]','scheduled',1)`,
).run(JSON.stringify(task));
insertAudit(db, 'audit-old', '2026-01-01T00:00:00.000Z');
insertAudit(db, 'audit-new', '2026-09-01T00:00:00.000Z');
insertJob(db, 'job-old', '2026-01-01T00:00:00.000Z');
insertEvent(db, 'event-old', { kind: 'old' });
const service = new SystemMaintenanceService(db, {
version: '1.9.9',
now: () => new Date('2026-09-03T00:00:00.000Z'),
});
await service.updateRetention({
auditEvents: { enabled: true, days: 30, maximumCount: 1 },
jobs: { enabled: true, days: 1, maximumCount: 100 },
eventJournal: { enabled: true, days: 1, maximumCount: 100 },
});
const result = await service.cleanup(['auditEvents', 'jobs', 'eventJournal', 'scheduledRuns']);
expect(result).toEqual({
auditEvents: 1,
jobs: 1,
eventJournal: 1,
scheduledRuns: 1,
});
const counts = (table: string) =>
(db.prepare(`SELECT COUNT(*) AS count FROM ${table}`).get() as { count: number }).count;
expect(counts('audit_events')).toBe(1);
expect(counts('jobs')).toBe(0);
expect(counts('event_journal')).toBe(0);
expect(counts('scheduled_runs')).toBe(0);
expect((await service.overview()).retention.auditEvents).toEqual({
enabled: true,
days: 30,
maximumCount: 1,
});
});
it('rejects unknown components and invalid retention values without touching settings', async () => {
const db = database();
const service = new SystemMaintenanceService(db, { version: '1.9.9' });
await expect(service.cleanup(['unknown' as never])).rejects.toThrow('Unknown data component');
await expect(
service.updateRetention({ auditEvents: { enabled: true, days: 0, maximumCount: 1 } }),
).rejects.toThrow('Retention days must be between');
expect(await service.getRetention()).toEqual(await service.defaultRetention());
});
it('optimizes SQLite and creates validated, listed backups with retention pruning', async () => {
const db = database();
const backupRoot = await mkdtemp(join(tmpdir(), 'multi-simadmin-backup-'));
roots.push(backupRoot);
const service = new SystemMaintenanceService(db, {
version: '1.9.9',
backupDirectory: backupRoot,
maximumBackups: 2,
});
expect(await service.optimize()).toEqual({
checkpointed: true,
vacuumed: true,
analyzed: true,
});
const first = await service.createBackup();
const second = await service.createBackup();
const third = await service.createBackup();
const backups = await service.listBackups();
expect(backups).toHaveLength(2);
expect(backups.map((item) => item.filename)).toEqual([third.filename, second.filename]);
expect(first.sha256).toMatch(/^[a-f0-9]{64}$/u);
// Retention is enforced on disk, not just in the listing.
await expect(readFile(first.path)).rejects.toThrow();
expect(await readFile(second.path)).toBeTruthy();
expect(backups[0]).toEqual(
expect.objectContaining({
filename: third.filename,
sha256: third.sha256,
sizeBytes: expect.any(Number),
createdAt: expect.any(String),
}),
);
});
it('serves and deletes single backups and refuses names outside the directory', async () => {
const db = database();
const backupRoot = await mkdtemp(join(tmpdir(), 'multi-simadmin-backup-'));
roots.push(backupRoot);
const service = new SystemMaintenanceService(db, {
version: '1.9.9',
backupDirectory: backupRoot,
});
const backup = await service.createBackup();
expect(await service.backupFile(backup.filename)).toEqual({
filename: backup.filename,
path: backup.path,
sizeBytes: backup.sizeBytes,
createdAt: expect.any(String),
});
expect(await service.backupFile('multi-simadmin-1970-01-01T00-00-00-000.db')).toBeUndefined();
await expect(service.backupFile('multi-simadmin-../secret.db')).rejects.toThrow();
await expect(service.backupFile('multi-simadmin-notes.txt')).rejects.toThrow();
await expect(service.deleteBackup('multi-simadmin-%00.db')).rejects.toThrow();
expect(await service.deleteBackup(backup.filename)).toEqual({ filename: backup.filename });
expect(await service.listBackups()).toEqual([]);
});
});
@@ -0,0 +1,387 @@
import { createHash } from 'node:crypto';
import { chmod, mkdir, readdir, readFile, rm, stat } from 'node:fs/promises';
import { basename, join } from 'node:path';
import type { SqliteDatabase } from '../../infrastructure/database/database.js';
export type DataComponentKey =
| 'instances'
| 'statusSnapshots'
| 'jobs'
| 'auditEvents'
| 'scheduledRuns'
| 'notificationQueue'
| 'smsMessages'
| 'eventJournal'
| 'connectionLogs';
export interface RetentionPolicy {
readonly enabled: boolean;
readonly days: number;
readonly maximumCount: number;
}
export interface MaintenanceOverview {
readonly runtime: {
readonly version: string;
readonly platform: NodeJS.Platform;
readonly arch: string;
readonly uptimeSeconds: number;
};
readonly storage: {
readonly databaseBytes: number;
/** SQLite file path; empty for in-memory databases. */
readonly databasePath: string;
/** Write-ahead log file size; 0 when the database is not in WAL mode. */
readonly walBytes: number;
/** Total size of the retained backup files. */
readonly backupBytes: number;
/** Free pages that "整理空间" can hand back to the filesystem. */
readonly reclaimableBytes: number;
readonly components: readonly {
readonly key: DataComponentKey;
readonly count: number;
readonly bytes: number;
}[];
};
readonly retention: Readonly<Record<DataComponentKey, RetentionPolicy>>;
}
export interface MaintenanceBackup {
readonly filename: string;
readonly path: string;
readonly sizeBytes: number;
readonly sha256: string;
readonly createdAt: string;
}
/** Download metadata without the full-file hash so large backups stay cheap to serve. */
export interface MaintenanceBackupFile {
readonly filename: string;
readonly path: string;
readonly sizeBytes: number;
readonly createdAt: string;
}
interface ServiceOptions {
readonly version: string;
readonly now?: () => Date;
readonly backupDirectory?: string;
readonly maximumBackups?: number;
}
const COMPONENT_TABLES: Readonly<Record<DataComponentKey, string>> = Object.freeze({
instances: 'instances',
statusSnapshots: 'status_snapshots',
jobs: 'jobs',
auditEvents: 'audit_events',
scheduledRuns: 'scheduled_runs',
notificationQueue: 'notification_queue',
smsMessages: 'sms_messages',
eventJournal: 'event_journal',
connectionLogs: 'connection_logs',
});
const RETENTION_SETTING_KEY = 'system.retention';
const DEFAULT_RETENTION: Readonly<Record<DataComponentKey, RetentionPolicy>> = Object.freeze({
instances: Object.freeze({ enabled: false, days: 365, maximumCount: 0 }),
statusSnapshots: Object.freeze({ enabled: false, days: 30, maximumCount: 0 }),
jobs: Object.freeze({ enabled: true, days: 90, maximumCount: 5_000 }),
auditEvents: Object.freeze({ enabled: true, days: 180, maximumCount: 50_000 }),
scheduledRuns: Object.freeze({ enabled: true, days: 180, maximumCount: 5_000 }),
notificationQueue: Object.freeze({ enabled: true, days: 180, maximumCount: 10_000 }),
smsMessages: Object.freeze({ enabled: false, days: 365, maximumCount: 0 }),
eventJournal: Object.freeze({ enabled: true, days: 30, maximumCount: 10_000 }),
// The probe journal already self-caps, so retention here is opt-in by day window.
connectionLogs: Object.freeze({ enabled: true, days: 30, maximumCount: 20_000 }),
});
function cloneRetention(
value: Readonly<Record<DataComponentKey, RetentionPolicy>>,
): Record<DataComponentKey, RetentionPolicy> {
return Object.fromEntries(
Object.entries(value).map(([key, policy]) => [key, { ...policy }]),
) as Record<DataComponentKey, RetentionPolicy>;
}
function validateRetention(value: unknown): Record<DataComponentKey, RetentionPolicy> {
const source =
value !== null && typeof value === 'object' && !Array.isArray(value)
? (value as Record<string, unknown>)
: {};
const merged = cloneRetention(DEFAULT_RETENTION);
for (const key of Object.keys(DEFAULT_RETENTION) as DataComponentKey[]) {
const current = source[key];
if (current === undefined) continue;
const policy = current as Partial<RetentionPolicy>;
if (
typeof policy.enabled !== 'boolean' ||
!Number.isSafeInteger(policy.days) ||
(policy.days ?? 0) < 1 ||
(policy.days ?? 0) > 3_650 ||
!Number.isSafeInteger(policy.maximumCount) ||
(policy.maximumCount ?? 0) < 0
)
throw new RangeError('Retention days must be between 1 and 3650 and limits must be valid');
merged[key] = {
enabled: policy.enabled,
days: policy.days!,
maximumCount: policy.maximumCount!,
};
}
if (Object.keys(source).some((key) => !(key in DEFAULT_RETENTION)))
throw new TypeError('Unknown data component');
return merged;
}
function tableExists(db: SqliteDatabase, table: string): boolean {
return Boolean(
db.prepare('SELECT 1 FROM sqlite_master WHERE type = ? AND name = ?').get('table', table),
);
}
function countTable(db: SqliteDatabase, table: string): number {
if (!tableExists(db, table)) return 0;
return Number(
(db.prepare(`SELECT COUNT(*) AS count FROM "${table}"`).get() as { count?: number } | undefined)
?.count ?? 0,
);
}
/** Per-table page usage from the dbstat virtual table; 0 when SQLite was built without it. */
function tableBytes(db: SqliteDatabase, table: string): number {
if (!tableExists(db, table)) return 0;
try {
const row = db
.prepare('SELECT COALESCE(SUM(pgsize), 0) AS bytes FROM dbstat WHERE name = ?')
.get(table) as { bytes?: number } | undefined;
return Number(row?.bytes ?? 0);
} catch {
return 0;
}
}
export class SystemMaintenanceService {
readonly #db: SqliteDatabase;
readonly #version: string;
readonly #now: () => Date;
readonly #backupDirectory: string;
readonly #maximumBackups: number;
constructor(db: SqliteDatabase, options: ServiceOptions) {
this.#db = db;
this.#version = options.version;
this.#now = options.now ?? (() => new Date());
this.#backupDirectory = options.backupDirectory ?? './data/backups';
this.#maximumBackups = Math.max(1, options.maximumBackups ?? 10);
}
async overview(): Promise<MaintenanceOverview> {
const pageCount = Number(this.#db.pragma('page_count', { simple: true }));
const pageSize = Number(this.#db.pragma('page_size', { simple: true }));
const databaseFile = String(
(this.#db.prepare('PRAGMA database_list').get() as { file?: string } | undefined)?.file ?? '',
);
const walBytes = databaseFile
? await stat(`${databaseFile}-wal`)
.then((details) => (details.isFile() ? details.size : 0))
.catch(() => 0)
: 0;
const freelist = Number(this.#db.pragma('freelist_count', { simple: true }));
return {
runtime: {
version: this.#version,
platform: process.platform,
arch: process.arch,
uptimeSeconds: Math.floor(process.uptime()),
},
storage: {
databaseBytes: pageCount * pageSize,
databasePath: databaseFile,
walBytes,
backupBytes: await this.#backupBytes(),
reclaimableBytes: freelist * pageSize,
components: (Object.keys(COMPONENT_TABLES) as DataComponentKey[]).map((key) => ({
key,
count: countTable(this.#db, COMPONENT_TABLES[key]),
bytes: tableBytes(this.#db, COMPONENT_TABLES[key]),
})),
},
retention: await this.getRetention(),
};
}
async #backupBytes(): Promise<number> {
const names = await readdir(this.#backupDirectory).catch(() => [] as string[]);
let total = 0;
for (const name of names.filter((entry) => entry.startsWith('multi-simadmin-'))) {
const details = await stat(join(this.#backupDirectory, name)).catch(() => undefined);
if (details?.isFile()) total += details.size;
}
return total;
}
defaultRetention(): Readonly<Record<DataComponentKey, RetentionPolicy>> {
return cloneRetention(DEFAULT_RETENTION);
}
async getRetention(): Promise<Readonly<Record<DataComponentKey, RetentionPolicy>>> {
const row = this.#db
.prepare('SELECT value_json FROM app_settings WHERE key = ?')
.get(RETENTION_SETTING_KEY) as { value_json?: string } | undefined;
if (!row?.value_json) return this.defaultRetention();
return validateRetention(JSON.parse(row.value_json));
}
async updateRetention(
value: unknown,
): Promise<Readonly<Record<DataComponentKey, RetentionPolicy>>> {
const policy = validateRetention(value);
const now = this.#now().toISOString();
this.#db
.prepare(
`INSERT INTO app_settings (key,value_json,created_at,updated_at)
VALUES (?,?,?,?)
ON CONFLICT(key) DO UPDATE SET value_json = excluded.value_json,
updated_at = excluded.updated_at`,
)
.run(RETENTION_SETTING_KEY, JSON.stringify(policy), now, now);
return policy;
}
async cleanup(
components: readonly DataComponentKey[],
): Promise<Partial<Record<DataComponentKey, number>>> {
const known = new Set(Object.keys(COMPONENT_TABLES));
if (components.some((component) => !known.has(component)))
throw new Error('Unknown data component');
const retention = await this.getRetention();
const result: Partial<Record<DataComponentKey, number>> = {};
for (const component of components) {
const policy = retention[component];
const table = COMPONENT_TABLES[component];
if (!policy.enabled || !tableExists(this.#db, table)) continue;
const cutoff = new Date(this.#now().getTime() - policy.days * 86_400_000).toISOString();
if (component === 'auditEvents')
result[component] = this.#delete(`DELETE FROM ${table} WHERE created_at < ?`, cutoff);
else if (component === 'jobs')
result[component] = this.#delete(
`DELETE FROM ${table} WHERE created_at < ? AND status NOT IN ('running','pending')`,
cutoff,
);
else if (component === 'scheduledRuns')
result[component] = this.#delete(`DELETE FROM ${table} WHERE due_at < ?`, cutoff);
else if (component === 'eventJournal')
result[component] = this.#delete(`DELETE FROM ${table}`);
else if (component === 'connectionLogs')
result[component] = this.#delete(`DELETE FROM ${table} WHERE observed_at < ?`, cutoff);
else if (policy.maximumCount > 0)
result[component] = this.#delete(
`DELETE FROM ${table} WHERE rowid IN (
SELECT rowid FROM ${table} LIMIT ?
)`,
Math.max(0, countTable(this.#db, table) - policy.maximumCount),
);
}
return result;
}
async optimize(): Promise<{ checkpointed: boolean; vacuumed: boolean; analyzed: boolean }> {
this.#db.pragma('wal_checkpoint(TRUNCATE)');
this.#db.exec('VACUUM');
this.#db.exec('ANALYZE');
return { checkpointed: true, vacuumed: true, analyzed: true };
}
async createBackup(): Promise<MaintenanceBackup> {
await mkdir(this.#backupDirectory, { recursive: true });
const filename = `multi-simadmin-${this.#now().toISOString().replaceAll(/[:.]/gu, '-')}.db`;
const path = join(this.#backupDirectory, filename);
await this.#db.backup(path);
await chmod(path, 0o600);
const details = await stat(path);
await this.#pruneBackups();
return {
filename,
path,
sizeBytes: details.size,
sha256: await readFile(path).then((content) =>
createHash('sha256').update(content).digest('hex'),
),
createdAt: this.#now().toISOString(),
};
}
async listBackups(): Promise<readonly MaintenanceBackup[]> {
await mkdir(this.#backupDirectory, { recursive: true });
const names = (await readdir(this.#backupDirectory))
.filter((name) => name.startsWith('multi-simadmin-') && name.endsWith('.db'))
.sort()
.reverse();
const backups: MaintenanceBackup[] = [];
for (const filename of names.slice(0, this.#maximumBackups)) {
const path = join(this.#backupDirectory, filename);
const details = await stat(path);
if (!details.isFile()) continue;
backups.push({
filename,
path,
sizeBytes: details.size,
sha256: createHash('sha256')
.update(await readFile(path))
.digest('hex'),
createdAt: details.mtime.toISOString(),
});
}
return backups;
}
/** Resolves a listed backup name to a path that stays inside the backup directory. */
#backupPath(filename: string): string {
if (
!/^multi-simadmin-[A-Za-z0-9._-]{1,120}\.db$/u.test(filename) ||
basename(filename) !== filename
)
throw new TypeError('Backup filename is invalid');
return join(this.#backupDirectory, filename);
}
async backupFile(filename: string): Promise<MaintenanceBackupFile | undefined> {
const path = this.#backupPath(filename);
const details = await stat(path).catch((error: unknown) => {
if ((error as NodeJS.ErrnoException).code === 'ENOENT') return undefined;
throw error;
});
if (!details?.isFile()) return undefined;
return {
filename,
path,
sizeBytes: details.size,
createdAt: details.mtime.toISOString(),
};
}
async deleteBackup(filename: string): Promise<{ filename: string }> {
await rm(this.#backupPath(filename), { force: true });
return { filename };
}
/** Names sort newest-first because they embed an ISO timestamp. */
async #pruneBackups(): Promise<number> {
const names = (await readdir(this.#backupDirectory))
.filter((name) => name.startsWith('multi-simadmin-') && name.endsWith('.db'))
.sort()
.reverse();
let removed = 0;
for (const filename of names.slice(this.#maximumBackups)) {
await rm(join(this.#backupDirectory, filename), { force: true });
removed += 1;
}
return removed;
}
#delete(statement: string, ...parameters: readonly unknown[]): number {
return Number(this.#db.prepare(statement).run(...parameters).changes);
}
}
+30 -1
View File
@@ -75,7 +75,11 @@ async function rawSocketRequest(origin: string, request: string): Promise<string
}
async function startGateway(distDir: string, upstreamPort: number): Promise<CanaryGateway> {
const gateway = createCanaryGateway({ distDir, port: 0, upstreamPort });
const gateway = createCanaryGateway({
distDir,
port: 0,
upstreamPort,
});
gateways.push(gateway);
await gateway.start();
return gateway;
@@ -334,6 +338,31 @@ describe('same-origin canary gateway', () => {
await reader?.cancel();
});
it('flushes SSE response headers before the first upstream event', async () => {
const upstream = createServer((_request, response) => {
response.writeHead(200, {
'content-type': 'text/event-stream',
'cache-control': 'no-cache',
});
response.flushHeaders();
});
servers.push(upstream);
const gateway = await startGateway(await fixtureDist(), await listen(upstream));
const controller = new AbortController();
const timeout = setTimeout(() => controller.abort(), 500);
try {
const response = await fetch(`${gateway.origin}/api/v1/events`, {
signal: controller.signal,
});
expect(response.status).toBe(200);
expect(response.headers.get('content-type')).toBe('text/event-stream');
await response.body?.cancel();
} finally {
clearTimeout(timeout);
}
});
it('has reversible, idempotent start/stop lifecycle', async () => {
const gateway = createCanaryGateway({ distDir: await fixtureDist(), port: 0, upstreamPort: 1 });
gateways.push(gateway);
+4
View File
@@ -107,6 +107,9 @@ function trustedUpstreamHeaders(
delete sanitized[GATEWAY_CLIENT_IP_HEADER];
if (gatewayToken !== undefined) sanitized[GATEWAY_AUTH_HEADER] = gatewayToken;
if (clientIp !== undefined) sanitized[GATEWAY_CLIENT_IP_HEADER] = clientIp;
// The browser connection's protocol is the gateway's to declare; the API uses
// it to decide whether session cookies may carry the Secure flag.
sanitized['x-forwarded-proto'] = (headers['x-forwarded-proto'] as string | undefined) ?? 'http';
return sanitized;
}
@@ -177,6 +180,7 @@ async function proxyRequest(
upstreamResponse.statusMessage,
withoutHopByHop(upstreamResponse.headers),
);
response.flushHeaders();
pipeline(upstreamResponse, response)
.then(resolvePromise)
.catch(() => {
+36 -5
View File
@@ -91,6 +91,31 @@ async function expectPortFree(port: number): Promise<void> {
await closeServer(server);
}
async function isPortFree(port: number): Promise<boolean> {
try {
const server = await listenOn(port);
await closeServer(server);
return true;
} catch (error) {
if ((error as NodeJS.ErrnoException).code === 'EADDRINUSE') return false;
throw error;
}
}
/**
* Reserves a port the OS reports as free. The canary gateway accepts a configured
* port, so its test does not need to fight a live deployment for 8789.
*/
async function reserveFreePort(): Promise<number> {
const server = await listenOn(0);
const address = server.address();
if (address === null || typeof address === 'string')
throw new Error('Could not reserve a free port');
const port = address.port;
await closeServer(server);
return port;
}
async function listenerPids(port: number): Promise<readonly string[]> {
try {
const { stdout } = await execFileAsync('lsof', ['-nP', `-iTCP:${port}`, '-sTCP:LISTEN', '-t']);
@@ -106,7 +131,9 @@ afterEach(async () => {
await Promise.all(cleanup.splice(0).map((fn) => fn()));
});
describe.sequential('executable package runtimes', () => {
// The runtime lifecycle under test is macOS/POSIX-only: process-group SIGTERM,
// lsof port inspection, and corepack as a direct spawn target.
describe.sequential('executable package runtimes', { skip: process.platform === 'win32' }, () => {
it('makes the exact production gateway executable attempt 8788 only with acknowledgement', async () => {
await expectPortFree(8788);
const fixture = await listenOn(8788);
@@ -134,7 +161,10 @@ describe.sequential('executable package runtimes', () => {
expect(acknowledged.child.exitCode).not.toBe(0);
}, 20_000);
it('starts the exact production command on 8790 and shuts down on SIGTERM', async () => {
// The production executable refuses any port other than 8790 on purpose, so this
// check only runs when a live deployment is not already holding that port.
it('starts the exact production command on 8790 and shuts down on SIGTERM', async (context) => {
if (!(await isPortFree(8790))) return context.skip();
await expectPortFree(8790);
const legacyListeners = await listenerPids(8788);
const directory = await mkdtemp(join(tmpdir(), 'multi-simadmin-production-cli-'));
@@ -166,8 +196,9 @@ describe.sequential('executable package runtimes', () => {
expect(command.output()).not.toContain(gatewayToken);
}, 20_000);
it('starts the exact canary command on 8789 with synthetic dist and shuts down on SIGTERM', async () => {
const port = 8789;
it('starts the exact canary command with synthetic dist and shuts down on SIGTERM', async () => {
const port = await reserveFreePort();
const upstreamPort = await reserveFreePort();
await expectPortFree(port);
const legacyListeners = await listenerPids(8788);
const directory = await mkdtemp(join(tmpdir(), 'multi-simadmin-canary-cli-'));
@@ -177,7 +208,7 @@ describe.sequential('executable package runtimes', () => {
const command = startPackageCommand('canary', {
CANARY_DIST_DIR: directory,
CANARY_PORT: String(port),
CANARY_UPSTREAM_PORT: '8790',
CANARY_UPSTREAM_PORT: String(upstreamPort),
MULTI_SIMADMIN_GATEWAY_TOKEN: gatewayToken,
});
await waitForOutput(command, `Canary gateway listening at http://127.0.0.1:${port}`);
+10 -1
View File
@@ -9,7 +9,12 @@ it('registers the read-only operations catalog without touching upstream', async
let upstreamCalls = 0;
const app = buildControlPlaneApp({
db,
store: { set: async () => '', get: async () => undefined, delete: async () => false },
store: {
provider: 'macos-keychain',
set: async () => '',
get: async () => undefined,
delete: async () => false,
},
upstream: {
get: async () => {
upstreamCalls += 1;
@@ -27,6 +32,10 @@ it('registers the read-only operations catalog without touching upstream', async
upstreamCalls += 1;
throw new Error('unexpected');
},
postBasebandRestart: async () => {
upstreamCalls += 1;
throw new Error('unexpected');
},
postSystemReboot: async () => {
upstreamCalls += 1;
throw new Error('unexpected');
File diff suppressed because it is too large Load Diff
+524 -16
View File
@@ -12,6 +12,13 @@ import {
ConnectionProbe,
type ConnectionTransport,
} from './application/connections/connection-probe.js';
import { ConnectionSettingsService } from './application/connections/connection-settings-service.js';
import {
FLEET_STATE_CODES,
FLEET_STATE_EVENTS,
FLEET_STATE_NAMES,
FleetHeartbeatCoordinator,
} from './application/connections/fleet-heartbeat.js';
import { InstanceCredentialResolver } from './application/connections/instance-credential-resolver.js';
import { InstanceLoginService } from './application/connections/instance-login-service.js';
import {
@@ -20,11 +27,17 @@ import {
type UpstreamSessionClientOptions,
} from './application/connections/upstream-session-client.js';
import { InstanceService } from './application/instances/instance-service.js';
import { DeleteInstanceOperation } from './application/operations/delete-instance-operation.js';
import { DeviceIdentityService } from './application/identity/device-identity-service.js';
import { DeviceDiscoveryService } from './application/instances/device-discovery-service.js';
import {
type BindingRelease,
DeleteInstanceOperation,
} from './application/operations/delete-instance-operation.js';
import type { SecretStore } from './infrastructure/secrets/secret-store.js';
import { registerInstanceRoutes } from './interface/http/instance-routes.js';
import { registerEventRoutes } from './interface/http/event-routes.js';
import { JobQueryService } from './application/jobs/job-query-service.js';
import { JobReconcileService } from './application/jobs/job-reconcile-service.js';
import { registerJobRoutes } from './interface/http/job-routes.js';
import { AuditQueryService } from './application/audit/audit-query-service.js';
import { registerAuditRoutes } from './interface/http/audit-routes.js';
@@ -32,12 +45,51 @@ import { InstanceResourceService } from './application/resources/instance-resour
import { ConsoleAuthService } from './application/auth/console-auth-service.js';
import { registerConsoleAuth } from './interface/http/console-auth-routes.js';
import { InstanceMessageService } from './application/messages/instance-message-service.js';
import { HubMessageService } from './application/messages/hub-message-service.js';
import { SmsOutboxService } from './application/messages/sms-outbox-service.js';
import { ScheduledTaskRepository } from './application/automation/scheduled-task-repository.js';
import { ScheduledTaskService } from './application/automation/scheduled-task-service.js';
import { registerAutomationRoutes } from './interface/http/automation-routes.js';
import { registerFleetRoutes } from './interface/http/fleet-routes.js';
import { SystemMaintenanceService } from './application/system/system-maintenance-service.js';
import { ComponentBackupService } from './application/system/component-backup-service.js';
import { registerSystemRoutes } from './interface/http/system-routes.js';
import {
ConsoleUpdateService,
createCommandRunner,
type RestartLauncher,
type UpdateCommandRunner,
} from './application/system/console-update-service.js';
import { registerUpdateRoutes } from './interface/http/update-routes.js';
import { InstanceNotificationService } from './application/notifications/instance-notification-service.js';
import { CentralNotificationService } from './application/notifications/central-notification-service.js';
import { registerCentralNotificationRoutes } from './interface/http/central-notification-routes.js';
import { DeviceOrganizationService } from './application/organization/device-organization-service.js';
import { registerOrganizationRoutes } from './interface/http/organization-routes.js';
import { ConnectionLogService } from './application/system/connection-log-service.js';
import { LogCenterService } from './application/system/log-center-service.js';
import { registerLogCenterRoutes } from './interface/http/log-center-routes.js';
import { InstanceModuleService } from './application/instances/instance-module-service.js';
import { registerInstanceModuleRoutes } from './interface/http/instance-module-routes.js';
import { DeviceActionService } from './application/instances/device-action-service.js';
import { registerDeviceActionRoutes } from './interface/http/device-action-routes.js';
import { registerIdentityRoutes } from './interface/http/identity-routes.js';
import { MetricsService } from './application/observability/metrics-service.js';
import { registerMetricsRoutes } from './interface/http/metrics-routes.js';
import { registerDiscoveryRoutes } from './interface/http/discovery-routes.js';
import { ScheduledOperationDispatcher } from './application/automation/scheduled-operation-dispatcher.js';
import { SchedulerCoordinator } from './application/automation/scheduler-coordinator.js';
export interface SafeControlPlaneUpstream extends ConnectionTransport {
request: UpstreamSessionClientOptions['request'];
postNetworkRegisterAuto(origin: string): Promise<{ readonly status: number }>;
postServiceRestart(origin: string): Promise<{ readonly status: number }>;
postSystemReboot(origin: string, delaySeconds: number): Promise<{ readonly status: number }>;
postNetworkRegisterAuto(origin: string, cookie?: string): Promise<{ readonly status: number }>;
postServiceRestart(origin: string, cookie?: string): Promise<{ readonly status: number }>;
postBasebandRestart(origin: string, cookie?: string): Promise<{ readonly status: number }>;
postSystemReboot(
origin: string,
delaySeconds: number,
cookie?: string,
): Promise<{ readonly status: number }>;
}
export interface ControlPlaneOptions {
readonly db: Database.Database;
@@ -45,63 +97,323 @@ export interface ControlPlaneOptions {
readonly upstream: SafeControlPlaneUpstream;
readonly now?: () => Date;
readonly authenticateEventStream?: (request: FastifyRequest) => boolean;
readonly runtimeVersion?: string;
readonly backupDirectory?: string;
readonly smsSyncIntervalMs?: number;
readonly queueDrainIntervalMs?: number;
readonly logPruneIntervalMs?: number;
readonly autoBackupIntervalMs?: number;
/** Keep device online state fresh on its own; off in tests, on for the production gateway. */
readonly heartbeatEnabled?: boolean;
readonly app?: Omit<BuildAppOptions, 'registerRoutes'>;
/** Online update of the console itself; defaults to a git checkout in the current directory. */
readonly update?: {
readonly remote?: string;
readonly root?: string;
readonly runner?: UpdateCommandRunner;
readonly launcher?: RestartLauncher;
/** Overrides the safety snapshot taken before the checkout moves. */
readonly preInstallBackup?: (targetCommit: string) => Promise<string>;
};
}
const DEFAULT_SMS_SYNC_INTERVAL_MS = 300_000;
const DEFAULT_QUEUE_DRAIN_INTERVAL_MS = 15_000;
const DEFAULT_LOG_PRUNE_INTERVAL_MS = 900_000;
const DEFAULT_AUTO_BACKUP_INTERVAL_MS = 300_000;
interface HubDeviceSummary {
readonly id: string;
readonly name: string;
readonly state: 'ready' | 'unavailable' | 'unknown';
readonly tags: readonly string[];
readonly groupId: string | null;
}
// Device reachability comes from the last ConnectionProbe result, never from a live probe:
// the notification overview must stay cheap enough to poll.
function createHubDeviceReader(
db: Database.Database,
instances: InstanceService,
): () => Promise<readonly HubDeviceSummary[]> {
const readConnections = db.prepare(
`SELECT instance_id, state FROM status_snapshots
WHERE category = 'connection' AND expires_at > ?`,
);
return async () => {
const now = new Date().toISOString();
const reachable = new Set<string>(
(readConnections.all(now) as readonly { instance_id: string; state: string }[])
.filter((row) => row.state === 'fresh' || row.state === 'stale')
.map((row) => row.instance_id),
);
const devices: HubDeviceSummary[] = [];
let page = 1;
while (devices.length < 200) {
const current = await instances.list({ page, pageSize: 100 });
for (const instance of current.items.slice(0, 200 - devices.length)) {
devices.push({
id: instance.id,
name: instance.name,
state: reachable.has(instance.id) ? 'ready' : 'unknown',
tags: [...instance.tags],
groupId: instance.groupId,
});
}
if (current.items.length < 100) break;
page += 1;
}
return devices;
};
}
// Retention windows for operator-facing history. Preparations are short-lived
// protocol state and cycle faster; audit records are the longest-lived.
const RETENTION_HISTORY_MS = 30 * 24 * 60 * 60 * 1000;
const RETENTION_AUDIT_MS = 90 * 24 * 60 * 60 * 1000;
const RETENTION_PREPARATION_MS = 7 * 24 * 60 * 60 * 1000;
export interface ControlPlaneApp extends FastifyInstance {
retryPendingSecretCleanup(): Promise<unknown>;
}
export function buildControlPlaneApp(options: ControlPlaneOptions): ControlPlaneApp {
const eventJournal = new EventJournal(options.db);
const jobs = new JobQueryService(options.db);
const jobReconcile = new JobReconcileService({
db: options.db,
...(options.now ? { now: options.now } : {}),
});
// Nothing can be mid-flight before the first request is served, so a startup sweep closes every
// job row a previous process left behind.
jobReconcile.reconcile(0);
const audit = new AuditQueryService(options.db);
const scheduledTasks = new ScheduledTaskRepository(options.db);
scheduledTasks.reconcileInterruptedRuns((options.now?.() ?? new Date()).toISOString());
const automation = new ScheduledTaskService({
repository: scheduledTasks,
store: options.store,
...(options.now ? { now: options.now } : {}),
});
const instances = options.now
? new InstanceService({ db: options.db, store: options.store, now: options.now })
: new InstanceService({ db: options.db, store: options.store });
const connectionLogs = new ConnectionLogService({ db: options.db });
const connectionSettings = options.now
? new ConnectionSettingsService({ db: options.db, now: options.now })
: new ConnectionSettingsService({ db: options.db });
const connections = options.now
? new ConnectionProbe({
db: options.db,
instances,
transport: options.upstream,
now: options.now,
connectionLogs,
snapshotTtlMs: () => connectionSettings.snapshotTtlMs,
})
: new ConnectionProbe({ db: options.db, instances, transport: options.upstream });
: new ConnectionProbe({
db: options.db,
instances,
transport: options.upstream,
connectionLogs,
snapshotTtlMs: () => connectionSettings.snapshotTtlMs,
});
const sessions = new InstanceSessionStore();
const discovery = options.now
? new DeviceDiscoveryService({ transport: options.upstream, instances, now: options.now })
: new DeviceDiscoveryService({ transport: options.upstream, instances });
const client = new UpstreamSessionClient({ sessions, request: options.upstream.request });
const resolver = new InstanceCredentialResolver({ db: options.db, store: options.store });
const login = options.now
? new InstanceLoginService({ db: options.db, client, resolver, now: options.now })
: new InstanceLoginService({ db: options.db, client, resolver });
const pendingLogins = new Map<string, Promise<void>>();
const ensureSession = async (
instanceId: string,
origin: string,
force = false,
): Promise<void> => {
const existing = sessions.sessionFor(instanceId);
if (!force && existing?.origin === origin) return;
if (existing) sessions.clear(instanceId);
const current = pendingLogins.get(instanceId);
if (current) return current;
const pending = login
.login(instanceId)
.then((result) => {
if (!result.authenticated) throw new Error('INSTANCE_AUTHENTICATION_FAILED');
})
.finally(() => pendingLogins.delete(instanceId));
pendingLogins.set(instanceId, pending);
return pending;
};
const resources = new InstanceResourceService({
instances,
sessions,
request: options.upstream.request,
ensureSession,
db: options.db,
});
const messages = new InstanceMessageService({
instances,
sessions,
request: options.upstream.request,
ensureSession,
});
const resolver = new InstanceCredentialResolver({ db: options.db, store: options.store });
const login = options.now
? new InstanceLoginService({ db: options.db, client, resolver, now: options.now })
: new InstanceLoginService({ db: options.db, client, resolver });
const notifications = new InstanceNotificationService({
instances,
sessions,
request: options.upstream.request,
ensureSession,
});
const centralNotifications = new CentralNotificationService(options.db, {
store: options.store,
...(options.now ? { now: options.now } : {}),
});
// The heartbeat owns the reachability loop, so it also owns telling the rule engine when a
// node went offline, came back, or lost its device session.
const heartbeat = new FleetHeartbeatCoordinator({
instances,
probe: connections,
settings: connectionSettings,
...(options.now ? { now: options.now } : {}),
onTransition: async (transitions) => {
for (const transition of transitions) {
const instance = await instances.get(transition.instanceId);
const label = instance?.name ?? transition.instanceId;
await centralNotifications.enqueueEvent('system', {
instanceId: transition.instanceId,
instanceTags: instance?.tags ?? [],
instanceGroupIds: instance?.groupId ? [instance.groupId] : [],
fields: {
title: `${label} ${FLEET_STATE_EVENTS[transition.to]}`,
status: FLEET_STATE_CODES[transition.to],
content: `节点 ${label} 从「${FLEET_STATE_NAMES[transition.from]}」变为「${
FLEET_STATE_NAMES[transition.to]
}${FLEET_STATE_EVENTS[transition.to]}`,
},
});
}
},
});
const hubMessages = new HubMessageService(options.db, {
instances,
messages,
...(options.now ? { now: options.now } : {}),
onIncomingMessage: async (instanceId, message) => {
const instance = await instances.get(instanceId);
await centralNotifications.enqueueEvent('sms', {
instanceId,
instanceTags: instance?.tags ?? [],
instanceGroupIds: instance?.groupId ? [instance.groupId] : [],
fields: {
sender: message.phoneNumber,
content: message.content,
title: `来自 ${message.phoneNumber} 的新短信`,
status: message.status,
},
});
},
});
// Offline sends wait here instead of failing, exactly like the Hub hands a message to a device
// that has just come back on the LAN.
const smsOutbox = new SmsOutboxService(options.db, {
send: (instanceId, input) => messages.send(instanceId, input),
offlineInstances: () => {
const offline = new Set<string>();
for (const [instanceId, state] of connections.reachability())
if (!state.reachable) offline.add(instanceId);
return offline;
},
...(options.now ? { now: options.now } : {}),
});
smsOutbox.reconcileInterrupted();
centralNotifications.reconcileInterrupted();
// Removal follows the hub rule the operator already knows: an online device is told to release
// its own binding first and a refusal keeps the record, while a device that cannot be reached
// at all is forgotten locally. The closure is late-bound because the device action service is
// built further down, next to the session store it needs.
const releaseBinding = async (instanceId: string, requestId: string): Promise<BindingRelease> => {
try {
await connections.test(instanceId);
} catch {
return { status: 'skipped' };
}
const result = await deviceActions.execute(
instanceId,
'hub.unbind',
{},
{ actor: 'loopback-control-plane', requestId, confirm: true },
);
// A device too old to expose the endpoint never had a binding to release.
if (result.status === 404 || result.status === 405) return { status: 'released' };
return result.ok ? { status: 'released' } : { status: 'failed' };
};
const deletion = options.now
? new DeleteInstanceOperation({ db: options.db, instances, now: options.now })
: new DeleteInstanceOperation({ db: options.db, instances });
? new DeleteInstanceOperation({
db: options.db,
instances,
releaseBinding,
now: options.now,
emit: (envelope) => eventJournal.append(envelope),
})
: new DeleteInstanceOperation({
db: options.db,
instances,
releaseBinding,
emit: (envelope) => eventJournal.append(envelope),
});
deletion.reconcileInterruptedJobs();
const listHubDevices = createHubDeviceReader(options.db, instances);
const resolveOperationCookie = async (
instanceId: string,
origin: string,
): Promise<string | undefined> => {
// Attach only an origin-bound in-memory session. Never invent cookies, never log them.
const existing = sessions.sessionFor(instanceId);
if (
existing &&
existing.origin === origin &&
/^simadmin_session=[^;\s,]+$/u.test(existing.cookie)
)
return existing.cookie;
// Password-protected instances need a session for reboot/restart. Best-effort Keychain login
// keeps the control plane from silently dispatching unauthenticated upstream mutations.
try {
const result = await login.login(instanceId);
if (!result.authenticated) return undefined;
} catch {
return undefined;
}
const refreshed = sessions.sessionFor(instanceId);
return refreshed &&
refreshed.origin === origin &&
/^simadmin_session=[^;\s,]+$/u.test(refreshed.cookie)
? refreshed.cookie
: undefined;
};
const secureExecution = new SecureOperationExecution({
db: options.db,
registry: secureOperationRegistry,
emit: (envelope) => eventJournal.append(envelope),
transport: {
request: async ({ origin, path, body, contentType }) => {
request: async ({ origin, path, body, contentType, instanceId }) => {
const cookie = await resolveOperationCookie(instanceId, origin);
if (path === '/api/network/register-auto') {
const response = await options.upstream.postNetworkRegisterAuto(origin);
const response = await options.upstream.postNetworkRegisterAuto(origin, cookie);
return { status: response.status };
}
if (path === '/api/service/restart') {
const response = await options.upstream.postServiceRestart(origin);
const response = await options.upstream.postServiceRestart(origin, cookie);
return { status: response.status };
}
if (path === '/api/baseband/restart') {
const response = await options.upstream.postBasebandRestart(origin, cookie);
return { status: response.status };
}
if (path === '/api/system/reboot') {
if (contentType !== 'application/json' || body !== JSON.stringify({ delay_seconds: 3 }))
throw new Error('UPSTREAM_REQUEST_INVALID');
const response = await options.upstream.postSystemReboot(origin, 3);
const response = await options.upstream.postSystemReboot(origin, 3, cookie);
return { status: response.status };
}
throw new Error('UPSTREAM_REQUEST_INVALID');
@@ -110,10 +422,96 @@ export function buildControlPlaneApp(options: ControlPlaneOptions): ControlPlane
...(options.now ? { now: options.now } : {}),
});
secureExecution.reconcileInterruptedJobs();
const maintenance = new SystemMaintenanceService(options.db, {
version: options.runtimeVersion ?? '0.1.0',
...(options.now ? { now: options.now } : {}),
...(options.backupDirectory ? { backupDirectory: options.backupDirectory } : {}),
});
const componentBackups = new ComponentBackupService(options.db, {
version: options.runtimeVersion ?? '0.1.0',
backupDirectory: options.backupDirectory ?? './data/backups',
...(options.now ? { now: options.now } : {}),
});
// The console updates the same way the Hub does: a git remote is consulted, the candidate is
// fetched and verified, and only then does the supervisor get asked for a restart.
const updateRunner =
options.update?.runner ??
createCommandRunner({ cwd: options.update?.root ?? process.cwd(), timeoutMs: 120_000 });
const updates = new ConsoleUpdateService({
db: options.db,
version: options.runtimeVersion ?? '0.1.0',
runner: updateRunner,
launcher: options.update?.launcher ?? { supported: false, async restart() {} },
...(options.update?.remote ? { remote: options.update.remote } : {}),
...(options.now ? { now: options.now } : {}),
preInstallBackup:
options.update?.preInstallBackup ??
(async (targetCommit: string) => {
const created = await componentBackups.create(
['devices', 'notifications', 'automation', 'settings'],
`更新前自动备份:${targetCommit.slice(0, 7)}`,
true,
);
return created.filename;
}),
});
const scheduledDispatcher = new ScheduledOperationDispatcher({
db: options.db,
operations: secureExecution,
messages,
maintenance,
store: options.store,
repository: scheduledTasks,
...(options.now ? { now: options.now } : {}),
});
const scheduler = new SchedulerCoordinator({
db: options.db,
repository: scheduledTasks,
dispatch: (task, targets, _run, context) =>
scheduledDispatcher.dispatch(task, targets, context),
...(options.now ? { now: options.now } : {}),
});
const auth = new ConsoleAuthService({
db: options.db,
...(options.now ? { now: options.now } : {}),
});
const organization = new DeviceOrganizationService({
db: options.db,
...(options.now ? { now: options.now } : {}),
});
const logCenter = new LogCenterService({ db: options.db, connections: connectionLogs });
// The guard learns identities from the device reads the console already makes, and in return it
// gets to stop control traffic when a node stops matching its own record.
const identities = new DeviceIdentityService({
db: options.db,
...(options.now ? { now: options.now } : {}),
authorizationMode: () => connectionSettings.get().authorizationMode,
});
const instanceModules = new InstanceModuleService({
instances,
sessions,
request: options.upstream.request,
ensureSession,
onIdentity: (instanceId, evidence, origin) => {
identities.observe(instanceId, { ...evidence, origin });
},
});
const deviceActions = new DeviceActionService({
instances,
sessions,
request: options.upstream.request,
db: options.db,
ensureSession,
identities,
});
// Scraped by the operator's monitoring rather than by the console, and read-only by
// construction: it aggregates rows the control plane already writes instead of probing devices.
const metrics = new MetricsService({
db: options.db,
version: options.runtimeVersion ?? '0.1.0',
sources: { connections: connectionLogs, identities },
...(options.now ? { now: options.now } : {}),
});
const app = buildApp({
...options.app,
registerRoutes: (app) => {
@@ -126,20 +524,130 @@ export function buildControlPlaneApp(options: ControlPlaneOptions): ControlPlane
resources,
messages,
registerDeletionPreparationRoute: false,
onEvent: (envelope) => eventJournal.append(envelope),
});
registerFleetRoutes(app, {
instances,
resources,
messages: hubMessages,
notifications,
connections,
outbox: smsOutbox,
identities,
});
registerCentralNotificationRoutes(app, {
notifications: centralNotifications,
devices: listHubDevices,
});
registerOperationRoutes(app, operationCatalogRegistry, secureExecution, deletion);
registerJobRoutes(app, { jobs });
registerJobRoutes(app, { jobs, reconcile: jobReconcile });
registerAuditRoutes(app, { audit });
registerAutomationRoutes(app, {
service: automation,
repository: scheduledTasks,
runNow: (taskId, actor, requestId) => scheduler.runNow(taskId, actor, requestId),
});
registerEventRoutes(app, {
journal: eventJournal,
...(options.authenticateEventStream
? { authenticate: options.authenticateEventStream }
: {}),
});
registerSystemRoutes(app, {
maintenance,
componentBackups,
connectionSettings,
heartbeat,
});
registerUpdateRoutes(app, { updates });
registerOrganizationRoutes(app, { organization });
registerLogCenterRoutes(app, { logs: logCenter, connections: connectionLogs });
registerInstanceModuleRoutes(app, { modules: instanceModules });
registerDeviceActionRoutes(app, { actions: deviceActions, identities });
registerIdentityRoutes(app, { identities, instances, modules: instanceModules });
registerMetricsRoutes(app, { metrics });
registerDiscoveryRoutes(app, { discovery });
},
});
Object.assign(app, {
retryPendingSecretCleanup: () => instances.retryPendingSecretCleanup(),
});
// Central Hub loops: pull device SMS into the archive and drain the delivery queue.
const smsSyncIntervalMs = options.smsSyncIntervalMs ?? DEFAULT_SMS_SYNC_INTERVAL_MS;
const queueDrainIntervalMs = options.queueDrainIntervalMs ?? DEFAULT_QUEUE_DRAIN_INTERVAL_MS;
const logPruneIntervalMs = options.logPruneIntervalMs ?? DEFAULT_LOG_PRUNE_INTERVAL_MS;
const autoBackupIntervalMs = options.autoBackupIntervalMs ?? DEFAULT_AUTO_BACKUP_INTERVAL_MS;
const messageSyncInterval =
smsSyncIntervalMs > 0
? setInterval(() => {
void hubMessages.syncAll().catch(() => undefined);
}, smsSyncIntervalMs)
: undefined;
const queueDrainInterval =
queueDrainIntervalMs > 0
? setInterval(() => {
void centralNotifications.processQueue().catch(() => undefined);
// Queued SMS ride the same drain tick so a returning device is picked up quickly.
void smsOutbox.flush().catch(() => undefined);
}, queueDrainIntervalMs)
: undefined;
const logPruneInterval =
logPruneIntervalMs > 0
? setInterval(() => {
void Promise.resolve()
.then(() => centralNotifications.pruneLogs())
.catch(() => undefined);
// Delivered and abandoned outbox rows otherwise accumulate forever.
void Promise.resolve()
.then(() => smsOutbox.prune(RETENTION_HISTORY_MS))
.catch(() => undefined);
void Promise.resolve()
.then(() => eventJournal.prune())
.catch(() => undefined);
// Connection probes write a log row per beat per node; without a sweep
// that alone is thousands of rows a day.
void Promise.resolve()
.then(() =>
connectionLogs.prune({
before: new Date(Date.now() - RETENTION_HISTORY_MS).toISOString(),
}),
)
.catch(() => undefined);
void Promise.resolve()
.then(() => audit.pruneBefore(new Date(Date.now() - RETENTION_AUDIT_MS).toISOString()))
.catch(() => undefined);
void Promise.resolve()
.then(() => secureExecution.pruneTerminalPreparations(RETENTION_PREPARATION_MS))
.catch(() => undefined);
void Promise.resolve()
.then(() => centralNotifications.pruneFinishedQueue(RETENTION_HISTORY_MS))
.catch(() => undefined);
}, logPruneIntervalMs)
: undefined;
// Scheduled component backups: the tick is cheap and only writes when a period is overdue.
const autoBackupInterval =
autoBackupIntervalMs > 0
? setInterval(() => {
void Promise.resolve()
.then(() => componentBackups.runDueAutoBackups())
.catch(() => undefined);
}, autoBackupIntervalMs)
: undefined;
messageSyncInterval?.unref?.();
queueDrainInterval?.unref?.();
logPruneInterval?.unref?.();
autoBackupInterval?.unref?.();
app.addHook('onClose', async () => {
scheduler.stop();
if (messageSyncInterval) clearInterval(messageSyncInterval);
if (queueDrainInterval) clearInterval(queueDrainInterval);
if (logPruneInterval) clearInterval(logPruneInterval);
if (autoBackupInterval) clearInterval(autoBackupInterval);
});
scheduler.start();
if (options.heartbeatEnabled) heartbeat.start();
app.addHook('onClose', async () => {
heartbeat.stop();
});
return app as ControlPlaneApp;
}
+11 -7
View File
@@ -23,7 +23,7 @@ const STATE_FILE = 'cutover-state.json';
const GATEWAY_ARGV = [
process.execPath,
'--import',
resolve(process.cwd(), 'node_modules/.pnpm/tsx@4.22.4/node_modules/tsx/dist/loader.mjs'),
'tsx',
resolve(process.cwd(), 'apps/api/src/production-gateway-cli.ts'),
] as const;
const DIGEST = /^[a-f0-9]{64}$/u;
@@ -289,11 +289,14 @@ async function durableState(plan: CutoverPlan, state: CutoverState): Promise<voi
await handle.close();
}
await rename(temporary, target);
const directory = await open(plan.stateDir, constants.O_RDONLY);
try {
await directory.sync();
} finally {
await directory.close();
if (process.platform !== 'win32') {
// directory fsync only exists on POSIX
const directory = await open(plan.stateDir, constants.O_RDONLY);
try {
await directory.sync();
} finally {
await directory.close();
}
}
}
@@ -311,7 +314,8 @@ function preparedState(plan: CutoverPlan): CutoverState {
async function readState(plan: CutoverPlan): Promise<CutoverState> {
const path = join(plan.stateDir, STATE_FILE);
const info = await lstat(path);
if (!info.isFile() || (info.mode & 0o777) !== 0o600)
// Windows keeps no POSIX mode bits (chmod is a no-op); ACLs govern access there.
if (!info.isFile() || (process.platform !== 'win32' && (info.mode & 0o777) !== 0o600))
throw new Error('Cutover state must be a mode-0600 regular file');
const input = record(JSON.parse(await readFile(path, 'utf8')) as unknown, 'Cutover state');
const base = ['version', 'phase', 'legacyPid', 'legacyCommand', 'legacyIdentity', 'legacyStart'];
+2
View File
@@ -66,4 +66,6 @@ export type {
InstanceServiceOptions,
} from './application/instances/instance-service.js';
export { MIGRATIONS, migrateDatabase } from './infrastructure/database/migrations.js';
export { ScheduledTaskRepository } from './application/automation/scheduled-task-repository.js';
export { ScheduledTaskService } from './application/automation/scheduled-task-service.js';
export * as databaseSchema from './infrastructure/database/schema.js';
@@ -48,10 +48,13 @@ describe('verified backup, restore, and rollback foundation', () => {
]);
copy.close();
expect(snapshot.sha256).toMatch(/^[a-f0-9]{64}$/);
expect((await stat(backupPath)).mode & 0o777).toBe(0o600);
if (process.platform !== 'win32') expect((await stat(backupPath)).mode & 0o777).toBe(0o600);
});
it('verifies a separate candidate and digest-binds activation', async () => {
// Activation renames a WAL file that a live connection still holds open;
// POSIX permits that, Windows file locking does not.
const itPosix = process.platform === 'win32' ? it.skip : it;
itPosix('verifies a separate candidate and digest-binds activation', async () => {
const { livePath, backupPath, database } = await fixture();
const snapshot = await backupDatabase(database, backupPath);
database.exec('UPDATE app_settings SET value_json = \'"after"\'');
@@ -118,7 +121,7 @@ describe('verified backup, restore, and rollback foundation', () => {
expect(await readFile(livePath)).toEqual(before);
});
it('rejects in-place staged-byte mutation even when metadata is preserved', async () => {
itPosix('rejects in-place staged-byte mutation even when metadata is preserved', async () => {
const { livePath, backupPath, database } = await fixture();
await backupDatabase(database, backupPath);
database.close();
@@ -159,7 +159,8 @@ async function digestFileHandle(handle: FileHandle): Promise<string> {
}
async function syncFile(path: string): Promise<void> {
const handle = await open(path, 'r');
// Windows FlushFileBuffers requires a writable handle; POSIX allows fsync on 'r'.
const handle = await open(path, process.platform === 'win32' ? 'r+' : 'r');
try {
await handle.sync();
} finally {
@@ -168,6 +169,7 @@ async function syncFile(path: string): Promise<void> {
}
async function syncDirectory(path: string): Promise<void> {
if (process.platform === 'win32') return; // directory fsync only exists on POSIX
const handle = await open(path, 'r');
try {
await handle.sync();
@@ -264,7 +266,9 @@ export async function activateRestoreCandidate(
await assertValidDatabase(stagedPath);
if ((await digestFileHandle(stagedHandle)) !== stagedDigest)
throw new Error('Staged restore candidate changed during validation');
await stagedHandle.sync();
// The staged handle is intentionally read-only after chmod(0400); Windows
// FlushFileBuffers requires a writable handle, so skip the durability sync there.
if (process.platform !== 'win32') await stagedHandle.sync();
await activateStagedDatabase(
databasePath,
stagedPath,
@@ -62,19 +62,31 @@ describe('database migrations', () => {
'app_settings',
'audit_events',
'capabilities',
'connection_logs',
'console_auth_config',
'console_auth_sessions',
'device_groups',
'device_identities',
'event_journal',
'instance_tags',
'instances',
'job_attempts',
'job_items',
'jobs',
'notification_channels',
'notification_deliveries',
'notification_queue',
'notification_rules',
'operation_preparations',
'scheduled_runs',
'scheduled_tasks',
'schema_migrations',
'secret_cleanup_tasks',
'secret_references',
'sms_messages',
'sms_outbox',
'status_snapshots',
'tag_registry',
]);
expect(database.pragma('foreign_keys', { simple: true })).toBe(1);
@@ -350,7 +362,8 @@ describe('database backup and restore', () => {
migrateDatabase(database);
const backupPath = join(directory, 'backup.sqlite');
await backupDatabase(database, backupPath);
expect((await lstat(backupPath)).mode & 0o777).toBe(0o600);
// Windows keeps no POSIX mode bits; the 0600 guarantee is POSIX-only.
if (process.platform !== 'win32') expect((await lstat(backupPath)).mode & 0o777).toBe(0o600);
database.close();
});
@@ -0,0 +1,103 @@
import Database from 'better-sqlite3';
import { describe, expect, it } from 'vitest';
import { MIGRATIONS, migrateDatabase } from './migrations.js';
function seeded(): Database.Database {
const db = new Database(':memory:');
db.pragma('foreign_keys=ON');
migrateDatabase(db, MIGRATIONS.slice(0, 10));
db.prepare(
`INSERT INTO instances (id,name,base_url,auth_mode,enabled,config_revision,created_at,updated_at)
VALUES ('i-1','Node One','http://a','password',1,1,'2026-01-01T00:00:00.000Z','2026-01-01T00:00:00.000Z')`,
).run();
db.prepare(
`INSERT INTO instance_tags (instance_id,tag,created_at) VALUES ('i-1','office','2026-01-01T00:00:00.000Z')`,
).run();
db.prepare(
`INSERT INTO scheduled_tasks (id,name,operation_type,enabled,version,cron_expression,timezone,
target_selector_json,sms_secret_reference,sms_recipient_count,effective_start_at,effective_end_at,
misfire_policy,overlap_policy,retry_policy_json,next_due_at,last_evaluated_at,created_by,updated_by,
created_at,updated_at,deleted_at)
VALUES ('t-1','Nightly','reboot-system',1,3,'0 3 * * *','Asia/Shanghai','{"mode":"fixed","instanceIds":["i-1"]}',
NULL,NULL,NULL,NULL,'skip','skip','{"maxRetries":2,"intervalSeconds":60}',
'2026-09-04T03:00:00.000Z',NULL,'sys','sys','2026-01-01T00:00:00.000Z','2026-01-01T00:00:00.000Z',NULL)`,
).run();
db.prepare(
`INSERT INTO scheduled_runs (id,scheduled_task_id,schedule_version,task_snapshot_json,due_at,claimed_at,
started_at,finished_at,target_snapshot_json,outcome,reason,job_ids_json,trigger_source,attempt)
VALUES ('r-1','t-1',3,'{}','2026-09-03T03:00:00.000Z','2026-09-03T03:00:00.000Z',
'2026-09-03T03:00:01.000Z','2026-09-03T03:00:02.000Z','["i-1"]','succeeded',NULL,'[]','scheduled',1)`,
).run();
return db;
}
describe('migration upgrade path', () => {
it('preserves automation rows and registers existing tags when upgrading from migration 10', () => {
const db = seeded();
expect(() => migrateDatabase(db)).not.toThrow();
const task = db.prepare('SELECT * FROM scheduled_tasks WHERE id=?').get('t-1') as Record<
string,
unknown
>;
expect(task).toMatchObject({ name: 'Nightly', operation_type: 'reboot-system', version: 3 });
expect(task.trigger_json).toBeNull();
const run = db.prepare('SELECT * FROM scheduled_runs WHERE id=?').get('r-1') as Record<
string,
unknown
>;
expect(run).toMatchObject({ scheduled_task_id: 't-1', outcome: 'succeeded' });
expect(db.prepare('SELECT tag FROM tag_registry ORDER BY tag').all()).toEqual([
{ tag: 'office' },
]);
expect(db.prepare('PRAGMA foreign_key_check').all()).toEqual([]);
db.prepare(
`INSERT INTO scheduled_tasks (id,name,operation_type,enabled,version,cron_expression,timezone,
target_selector_json,sms_secret_reference,sms_recipient_count,effective_start_at,effective_end_at,
misfire_policy,overlap_policy,retry_policy_json,next_due_at,last_evaluated_at,created_by,updated_by,
created_at,updated_at,deleted_at,trigger_json)
VALUES ('t-2','Baseband','restart-baseband',1,1,'0 4 * * *','Asia/Shanghai','{"mode":"all"}',
NULL,NULL,NULL,NULL,'skip','skip','{"maxRetries":0,"intervalSeconds":60}',NULL,NULL,'sys','sys',
'2026-01-01T00:00:00.000Z','2026-01-01T00:00:00.000Z',NULL,'{"kind":"interval","value":6,"unit":"hours"}')`,
).run();
expect(
(
db.prepare('SELECT trigger_json FROM scheduled_tasks WHERE id=?').get('t-2') as {
trigger_json: string;
}
).trigger_json,
).toContain('interval');
const names = (
db
.prepare(
"SELECT name FROM sqlite_master WHERE type='index' AND name LIKE 'idx_scheduled%' ORDER BY name",
)
.all() as Array<{
name: string;
}>
).map((row) => row.name);
expect(names).toEqual([
'idx_scheduled_runs_outcome_finished',
'idx_scheduled_runs_task_due',
'idx_scheduled_tasks_enabled_next_due',
]);
db.close();
});
it('applies cleanly to an empty database', () => {
const db = new Database(':memory:');
db.pragma('foreign_keys=ON');
migrateDatabase(db);
expect(db.prepare('PRAGMA foreign_key_check').all()).toEqual([]);
const applied = db.prepare('SELECT id,name FROM schema_migrations ORDER BY id').all() as Array<{
id: number;
name: string;
}>;
// The ledger must match the declared migration list exactly, in order, so a new migration
// can never be added without being applied here.
expect(applied).toEqual(
MIGRATIONS.map((migration) => ({ id: migration.id, name: migration.name })),
);
db.close();
});
});
@@ -304,6 +304,382 @@ export const MIGRATIONS: readonly Migration[] = [
'CREATE INDEX idx_console_auth_sessions_expires_at ON console_auth_sessions(expires_at)',
],
},
{
id: 8,
name: 'scheduled-automation',
statements: [
`CREATE TABLE scheduled_tasks (
id TEXT PRIMARY KEY,
name TEXT NOT NULL,
operation_type TEXT NOT NULL CHECK (operation_type IN ('restart-service','reboot-system','send-sms')),
enabled INTEGER NOT NULL DEFAULT 1 CHECK (enabled IN (0, 1)),
version INTEGER NOT NULL DEFAULT 1 CHECK (version > 0),
cron_expression TEXT NOT NULL,
timezone TEXT NOT NULL CHECK (timezone = 'Asia/Shanghai'),
target_selector_json TEXT NOT NULL,
sms_secret_reference TEXT,
sms_recipient_count INTEGER CHECK (sms_recipient_count IS NULL OR sms_recipient_count > 0),
effective_start_at TEXT,
effective_end_at TEXT,
misfire_policy TEXT NOT NULL CHECK (misfire_policy IN ('skip','catch-up-once')),
overlap_policy TEXT NOT NULL CHECK (overlap_policy IN ('skip','queue-once')),
retry_policy_json TEXT NOT NULL,
next_due_at TEXT,
last_evaluated_at TEXT,
created_by TEXT NOT NULL,
updated_by TEXT NOT NULL,
created_at TEXT NOT NULL,
updated_at TEXT NOT NULL,
deleted_at TEXT,
CHECK ((operation_type = 'send-sms') = (sms_secret_reference IS NOT NULL)),
CHECK ((sms_secret_reference IS NULL) = (sms_recipient_count IS NULL))
)`,
`CREATE TABLE scheduled_runs (
id TEXT PRIMARY KEY,
scheduled_task_id TEXT NOT NULL REFERENCES scheduled_tasks(id) ON DELETE RESTRICT,
schedule_version INTEGER NOT NULL CHECK (schedule_version > 0),
task_snapshot_json TEXT NOT NULL,
due_at TEXT NOT NULL,
claimed_at TEXT NOT NULL,
started_at TEXT,
finished_at TEXT,
target_snapshot_json TEXT NOT NULL,
outcome TEXT CHECK (outcome IS NULL OR outcome IN ('succeeded','partially-succeeded','failed','skipped','no-targets','needs-attention')),
reason TEXT,
job_ids_json TEXT NOT NULL DEFAULT '[]',
trigger_source TEXT NOT NULL CHECK (trigger_source IN ('scheduled','manual')),
attempt INTEGER NOT NULL DEFAULT 1 CHECK (attempt > 0),
UNIQUE (scheduled_task_id, schedule_version, due_at, trigger_source)
)`,
'CREATE INDEX idx_scheduled_tasks_enabled_next_due ON scheduled_tasks(enabled, next_due_at) WHERE deleted_at IS NULL',
'CREATE INDEX idx_scheduled_runs_task_due ON scheduled_runs(scheduled_task_id, due_at DESC)',
'CREATE INDEX idx_scheduled_runs_outcome_finished ON scheduled_runs(outcome, finished_at DESC)',
],
},
{
id: 9,
name: 'central-notifications',
statements: [
`CREATE TABLE notification_channels (
id TEXT PRIMARY KEY,
name TEXT NOT NULL,
type TEXT NOT NULL CHECK (type IN (
'webhook','bark','pushplus','wecom_app','wecom_robot','dingtalk_robot','dingtalk_app',
'feishu_robot','telegram','email','serverchan'
)),
enabled INTEGER NOT NULL DEFAULT 1 CHECK (enabled IN (0, 1)),
config_json TEXT NOT NULL,
secret_reference TEXT,
created_at TEXT NOT NULL,
updated_at TEXT NOT NULL
)`,
`CREATE TABLE notification_rules (
id TEXT PRIMARY KEY,
name TEXT NOT NULL,
event_type TEXT NOT NULL CHECK (event_type IN (
'sms','ddns','version','system','device','automation'
)),
enabled INTEGER NOT NULL DEFAULT 1 CHECK (enabled IN (0, 1)),
condition_json TEXT NOT NULL,
scope_json TEXT NOT NULL,
channel_ids_json TEXT NOT NULL,
templates_json TEXT NOT NULL,
created_at TEXT NOT NULL,
updated_at TEXT NOT NULL
)`,
`CREATE TABLE notification_deliveries (
id TEXT PRIMARY KEY,
rule_id TEXT REFERENCES notification_rules(id) ON DELETE SET NULL,
channel_id TEXT NOT NULL REFERENCES notification_channels(id) ON DELETE CASCADE,
instance_id TEXT,
event_type TEXT NOT NULL,
status TEXT NOT NULL CHECK (status IN (
'success','failed','pending','sending','retrying','unmatched','no_available_channel','quiet_hours'
)),
detail TEXT,
created_at TEXT NOT NULL,
sent_at TEXT
)`,
'CREATE INDEX idx_notification_deliveries_created_at ON notification_deliveries(created_at DESC)',
'CREATE INDEX idx_notification_deliveries_status_created_at ON notification_deliveries(status, created_at DESC)',
],
},
{
id: 10,
name: 'hub-message-and-notification-queue-persistence',
statements: [
`CREATE TABLE sms_messages (
id TEXT PRIMARY KEY,
instance_id TEXT NOT NULL,
upstream_id TEXT NOT NULL,
direction TEXT NOT NULL CHECK (direction IN ('incoming','outgoing','received','sent','unknown')),
phone_number TEXT NOT NULL,
content TEXT NOT NULL,
timestamp TEXT NOT NULL,
status TEXT NOT NULL,
transport TEXT NOT NULL,
synced_at TEXT NOT NULL,
UNIQUE (instance_id, upstream_id)
)`,
'CREATE INDEX idx_sms_messages_instance_timestamp ON sms_messages(instance_id, timestamp DESC)',
'CREATE INDEX idx_sms_messages_timestamp ON sms_messages(timestamp DESC)',
'CREATE INDEX idx_sms_messages_phone_timestamp ON sms_messages(phone_number, timestamp DESC)',
`CREATE TABLE notification_queue (
id TEXT PRIMARY KEY,
instance_id TEXT,
event_type TEXT NOT NULL,
rule_id TEXT REFERENCES notification_rules(id) ON DELETE SET NULL,
channel_id TEXT REFERENCES notification_channels(id) ON DELETE CASCADE,
status TEXT NOT NULL DEFAULT 'pending' CHECK (status IN (
'pending','sending','succeeded','failed','cancelled'
)),
attempts INTEGER NOT NULL DEFAULT 0 CHECK (attempts >= 0),
max_attempts INTEGER NOT NULL DEFAULT 3 CHECK (max_attempts > 0),
payload_json TEXT NOT NULL,
last_error TEXT,
available_at TEXT NOT NULL,
delivered_at TEXT,
created_at TEXT NOT NULL,
updated_at TEXT NOT NULL
)`,
'CREATE INDEX idx_notification_queue_status_available_at ON notification_queue(status, available_at)',
'CREATE INDEX idx_notification_queue_created_at ON notification_queue(created_at DESC)',
],
},
{
id: 11,
name: 'hub-groups-tags-and-automation-triggers',
statements: [
`CREATE TABLE device_groups (
id TEXT PRIMARY KEY,
name TEXT NOT NULL UNIQUE,
description TEXT NOT NULL DEFAULT '',
created_at TEXT NOT NULL,
updated_at TEXT NOT NULL
)`,
'ALTER TABLE instances ADD COLUMN group_id TEXT REFERENCES device_groups(id) ON DELETE SET NULL',
'CREATE INDEX idx_instances_group_id ON instances(group_id)',
`CREATE TABLE tag_registry (
tag TEXT PRIMARY KEY,
color TEXT NOT NULL DEFAULT '',
created_at TEXT NOT NULL,
updated_at TEXT NOT NULL
)`,
`INSERT INTO tag_registry (tag, created_at, updated_at)
SELECT tag, MIN(created_at), MAX(created_at) FROM instance_tags GROUP BY tag`,
// scheduled_tasks carries an operation_type CHECK that has to widen, so both the
// task table and its only child are rebuilt; renaming the child first keeps the
// RESTRICT edge from firing while the parent is swapped out.
'ALTER TABLE scheduled_runs RENAME TO scheduled_runs_old',
'ALTER TABLE scheduled_tasks RENAME TO scheduled_tasks_old',
`CREATE TABLE scheduled_tasks (
id TEXT PRIMARY KEY,
name TEXT NOT NULL,
operation_type TEXT NOT NULL CHECK (operation_type IN (
'restart-service','reboot-system','send-sms','restart-baseband','backup-data'
)),
enabled INTEGER NOT NULL DEFAULT 1 CHECK (enabled IN (0, 1)),
version INTEGER NOT NULL DEFAULT 1 CHECK (version > 0),
cron_expression TEXT NOT NULL,
timezone TEXT NOT NULL CHECK (timezone = 'Asia/Shanghai'),
target_selector_json TEXT NOT NULL,
sms_secret_reference TEXT,
sms_recipient_count INTEGER CHECK (sms_recipient_count IS NULL OR sms_recipient_count > 0),
effective_start_at TEXT,
effective_end_at TEXT,
misfire_policy TEXT NOT NULL CHECK (misfire_policy IN ('skip','catch-up-once')),
overlap_policy TEXT NOT NULL CHECK (overlap_policy IN ('skip','queue-once')),
retry_policy_json TEXT NOT NULL,
next_due_at TEXT,
last_evaluated_at TEXT,
created_by TEXT NOT NULL,
updated_by TEXT NOT NULL,
created_at TEXT NOT NULL,
updated_at TEXT NOT NULL,
deleted_at TEXT,
trigger_json TEXT,
CHECK ((operation_type = 'send-sms') = (sms_secret_reference IS NOT NULL)),
CHECK ((sms_secret_reference IS NULL) = (sms_recipient_count IS NULL))
)`,
`CREATE TABLE scheduled_runs (
id TEXT PRIMARY KEY,
scheduled_task_id TEXT NOT NULL REFERENCES scheduled_tasks(id) ON DELETE RESTRICT,
schedule_version INTEGER NOT NULL CHECK (schedule_version > 0),
task_snapshot_json TEXT NOT NULL,
due_at TEXT NOT NULL,
claimed_at TEXT NOT NULL,
started_at TEXT,
finished_at TEXT,
target_snapshot_json TEXT NOT NULL,
outcome TEXT CHECK (outcome IS NULL OR outcome IN ('succeeded','partially-succeeded','failed','skipped','no-targets','needs-attention')),
reason TEXT,
job_ids_json TEXT NOT NULL DEFAULT '[]',
trigger_source TEXT NOT NULL CHECK (trigger_source IN ('scheduled','manual')),
attempt INTEGER NOT NULL DEFAULT 1 CHECK (attempt > 0),
UNIQUE (scheduled_task_id, schedule_version, due_at, trigger_source)
)`,
`INSERT INTO scheduled_tasks (id,name,operation_type,enabled,version,cron_expression,timezone,
target_selector_json,sms_secret_reference,sms_recipient_count,effective_start_at,
effective_end_at,misfire_policy,overlap_policy,retry_policy_json,next_due_at,
last_evaluated_at,created_by,updated_by,created_at,updated_at,deleted_at,trigger_json)
SELECT id,name,operation_type,enabled,version,cron_expression,timezone,target_selector_json,
sms_secret_reference,sms_recipient_count,effective_start_at,effective_end_at,misfire_policy,
overlap_policy,retry_policy_json,next_due_at,last_evaluated_at,created_by,updated_by,
created_at,updated_at,deleted_at,NULL FROM scheduled_tasks_old`,
`INSERT INTO scheduled_runs (id,scheduled_task_id,schedule_version,task_snapshot_json,due_at,
claimed_at,started_at,finished_at,target_snapshot_json,outcome,reason,job_ids_json,
trigger_source,attempt)
SELECT id,scheduled_task_id,schedule_version,task_snapshot_json,due_at,claimed_at,started_at,
finished_at,target_snapshot_json,outcome,reason,job_ids_json,trigger_source,attempt
FROM scheduled_runs_old`,
'DROP TABLE scheduled_runs_old',
'DROP TABLE scheduled_tasks_old',
'CREATE INDEX idx_scheduled_tasks_enabled_next_due ON scheduled_tasks(enabled, next_due_at) WHERE deleted_at IS NULL',
'CREATE INDEX idx_scheduled_runs_task_due ON scheduled_runs(scheduled_task_id, due_at DESC)',
'CREATE INDEX idx_scheduled_runs_outcome_finished ON scheduled_runs(outcome, finished_at DESC)',
],
},
{
id: 12,
name: 'schedule-action-delay',
statements: [
`ALTER TABLE scheduled_tasks ADD COLUMN delay_seconds INTEGER
CHECK (delay_seconds IS NULL OR (delay_seconds >= 0 AND delay_seconds <= 3600))`,
],
},
{
id: 13,
name: 'notification-rule-rate-limit-and-quiet-hours',
statements: [
`ALTER TABLE notification_rules ADD COLUMN rate_limit_json TEXT NOT NULL DEFAULT
'{"enabled":false,"maxMessages":20,"windowSeconds":60}'`,
`ALTER TABLE notification_rules ADD COLUMN quiet_hours_json TEXT NOT NULL DEFAULT '[]'`,
// Suppressions (quiet hours, rate limit) are attributed to a rule rather than a channel,
// so channel_id becomes optional. The status CHECK also has to widen, which means the
// table is rebuilt; nothing references notification_deliveries, so the swap is contained.
`CREATE TABLE notification_deliveries_new (
id TEXT PRIMARY KEY,
rule_id TEXT REFERENCES notification_rules(id) ON DELETE SET NULL,
channel_id TEXT REFERENCES notification_channels(id) ON DELETE CASCADE,
instance_id TEXT,
event_type TEXT NOT NULL,
status TEXT NOT NULL CHECK (status IN (
'success','failed','pending','sending','retrying','unmatched','no_available_channel',
'quiet_hours','rate_limited'
)),
detail TEXT,
created_at TEXT NOT NULL,
sent_at TEXT
)`,
`INSERT INTO notification_deliveries_new
(id,rule_id,channel_id,instance_id,event_type,status,detail,created_at,sent_at)
SELECT id,rule_id,channel_id,instance_id,event_type,status,detail,created_at,sent_at
FROM notification_deliveries`,
'DROP TABLE notification_deliveries',
'ALTER TABLE notification_deliveries_new RENAME TO notification_deliveries',
'CREATE INDEX idx_notification_deliveries_created_at ON notification_deliveries(created_at DESC)',
'CREATE INDEX idx_notification_deliveries_status_created_at ON notification_deliveries(status, created_at DESC)',
],
},
{
id: 14,
name: 'connection-log-history',
statements: [
// The Hub keeps a rolling connection history per device; the snapshot table only holds
// the latest probe, so outcomes need their own bounded journal.
`CREATE TABLE connection_logs (
id TEXT PRIMARY KEY,
instance_id TEXT NOT NULL REFERENCES instances(id) ON DELETE CASCADE,
outcome TEXT NOT NULL CHECK (outcome IN ('success','stale','failed','unsupported')),
state TEXT NOT NULL CHECK (state IN ('fresh','stale','expired','unknown')),
error_code TEXT,
http_status INTEGER,
duration_ms INTEGER NOT NULL CHECK (duration_ms >= 0),
observed_at TEXT NOT NULL
)`,
'CREATE INDEX idx_connection_logs_observed_at ON connection_logs(observed_at DESC)',
'CREATE INDEX idx_connection_logs_instance_observed_at ON connection_logs(instance_id, observed_at DESC)',
'CREATE INDEX idx_connection_logs_outcome_observed_at ON connection_logs(outcome, observed_at DESC)',
],
},
{
id: 15,
name: 'sms-offline-outbox',
statements: [
// The Hub keeps sending while a device is offline and delivers when it returns; the
// central archive alone cannot hold a write, so outbound SMS need their own queue.
`CREATE TABLE sms_outbox (
id TEXT PRIMARY KEY,
instance_id TEXT NOT NULL REFERENCES instances(id) ON DELETE CASCADE,
phone_number TEXT NOT NULL,
content TEXT NOT NULL,
status TEXT NOT NULL DEFAULT 'queued' CHECK (status IN (
'queued','sending','sent','failed','cancelled'
)),
attempts INTEGER NOT NULL DEFAULT 0 CHECK (attempts >= 0),
max_attempts INTEGER NOT NULL DEFAULT 24 CHECK (max_attempts > 0),
last_error TEXT,
available_at TEXT NOT NULL,
sent_at TEXT,
created_at TEXT NOT NULL,
updated_at TEXT NOT NULL
)`,
'CREATE INDEX idx_sms_outbox_status_available_at ON sms_outbox(status, available_at)',
'CREATE INDEX idx_sms_outbox_instance_status ON sms_outbox(instance_id, status)',
'CREATE INDEX idx_sms_outbox_created_at ON sms_outbox(created_at DESC)',
],
},
{
id: 16,
name: 'notification-channel-secret-fields',
statements: [
// Hub channels keep several credentials (DingTalk needs an access token *and* a signing
// key), so the secret store holds a JSON map and the row records which keys it covers.
"ALTER TABLE notification_channels ADD COLUMN secret_fields TEXT NOT NULL DEFAULT ''",
],
},
{
id: 17,
name: 'device-identity-guard',
statements: [
// The Hub refuses to control a device whose identity drifted: the same IMEI claimed by
// two records, or hardware swapped behind an address that is still registered. Both are
// recorded here and only an operator can clear them, because the machine cannot tell
// which of two claimants is the real device.
`CREATE TABLE device_identities (
instance_id TEXT PRIMARY KEY REFERENCES instances(id) ON DELETE CASCADE,
status TEXT NOT NULL DEFAULT 'confirmed' CHECK (status IN ('confirmed','pending')),
reasons TEXT NOT NULL DEFAULT '[]',
imei TEXT NOT NULL DEFAULT '',
manufacturer TEXT NOT NULL DEFAULT '',
model TEXT NOT NULL DEFAULT '',
revision TEXT NOT NULL DEFAULT '',
agent TEXT NOT NULL DEFAULT '',
origin TEXT NOT NULL DEFAULT '',
fingerprint TEXT NOT NULL DEFAULT '',
confirmed_fingerprint TEXT NOT NULL DEFAULT '',
confirmed_values_json TEXT NOT NULL DEFAULT '{}',
confirmed_peers_json TEXT NOT NULL DEFAULT '[]',
detail_json TEXT NOT NULL DEFAULT '{}',
observed_at TEXT NOT NULL,
confirmed_at TEXT NOT NULL,
created_at TEXT NOT NULL,
updated_at TEXT NOT NULL
)`,
'CREATE INDEX idx_device_identities_status ON device_identities(status, instance_id)',
'CREATE INDEX idx_device_identities_imei ON device_identities(imei, instance_id)',
],
},
{
id: 18,
name: 'console-auth-scrypt-params',
statements: [
// Legacy hashes were derived with N=16384; new derivations use stronger
// parameters. The column records the derivation per row; NULL means legacy
// and is transparently upgraded after the next successful login.
'ALTER TABLE console_auth_config ADD COLUMN password_kdf TEXT',
],
},
];
const createMigrationsTable = `CREATE TABLE schema_migrations (
@@ -255,3 +255,67 @@ export const secretCleanupTasks = sqliteTable(
},
(table) => [index('idx_secret_cleanup_tasks_queued_at').on(table.queuedAt)],
);
export const scheduledTasks = sqliteTable(
'scheduled_tasks',
{
id: text('id').primaryKey(),
name: text('name').notNull(),
operationType: text('operation_type').notNull(),
enabled: integer('enabled', { mode: 'boolean' }).notNull().default(true),
version: integer('version').notNull().default(1),
cronExpression: text('cron_expression').notNull(),
timezone: text('timezone').notNull(),
targetSelectorJson: text('target_selector_json').notNull(),
smsSecretReference: text('sms_secret_reference'),
smsRecipientCount: integer('sms_recipient_count'),
effectiveStartAt: text('effective_start_at'),
effectiveEndAt: text('effective_end_at'),
misfirePolicy: text('misfire_policy').notNull(),
overlapPolicy: text('overlap_policy').notNull(),
retryPolicyJson: text('retry_policy_json').notNull(),
nextDueAt: text('next_due_at'),
lastEvaluatedAt: text('last_evaluated_at'),
createdBy: text('created_by').notNull(),
updatedBy: text('updated_by').notNull(),
createdAt: text('created_at').notNull(),
updatedAt: text('updated_at').notNull(),
deletedAt: text('deleted_at'),
},
(table) => [
check('scheduled_tasks_enabled_check', sql`${table.enabled} in (0, 1)`),
check('scheduled_tasks_version_check', sql`${table.version} > 0`),
index('idx_scheduled_tasks_enabled_next_due').on(table.enabled, table.nextDueAt),
],
);
export const scheduledRuns = sqliteTable(
'scheduled_runs',
{
id: text('id').primaryKey(),
scheduledTaskId: text('scheduled_task_id')
.notNull()
.references(() => scheduledTasks.id, { onDelete: 'restrict' }),
scheduleVersion: integer('schedule_version').notNull(),
taskSnapshotJson: text('task_snapshot_json').notNull(),
dueAt: text('due_at').notNull(),
claimedAt: text('claimed_at').notNull(),
startedAt: text('started_at'),
finishedAt: text('finished_at'),
targetSnapshotJson: text('target_snapshot_json').notNull(),
outcome: text('outcome'),
reason: text('reason'),
jobIdsJson: text('job_ids_json').notNull().default('[]'),
triggerSource: text('trigger_source').notNull(),
attempt: integer('attempt').notNull().default(1),
},
(table) => [
unique('scheduled_runs_occurrence_unique').on(
table.scheduledTaskId,
table.scheduleVersion,
table.dueAt,
table.triggerSource,
),
index('idx_scheduled_runs_task_due').on(table.scheduledTaskId, desc(table.dueAt)),
],
);
@@ -0,0 +1,174 @@
import { randomUUID } from 'node:crypto';
import { chmod, mkdir, open, readFile, rename, rm, stat } from 'node:fs/promises';
import { dirname, isAbsolute } from 'node:path';
import {
accountFor,
parseSecretAccount,
validateKey,
SecretReferenceError,
} from './secret-reference.js';
import type { SecretKey, SecretStore } from './secret-store.js';
const PROVIDER = 'secret-file';
const SERVICE = 'multi-simadmin';
const MAX_SECRET_BYTES = 16_384;
const MAX_STORE_BYTES = 4 * 1024 * 1024;
export type FileSecretStoreErrorCode =
| 'INVALID_KEY'
| 'INVALID_SECRET'
| 'INVALID_REFERENCE'
| 'INVALID_STORE_PATH'
| 'SIZE_LIMIT'
| 'PAYLOAD_INVALID';
export class FileSecretStoreError extends Error {
constructor(
readonly code: FileSecretStoreErrorCode,
message: string,
) {
super(message);
this.name = 'FileSecretStoreError';
}
}
function invalid(code: FileSecretStoreErrorCode, message: string): never {
throw new FileSecretStoreError(code, message);
}
/**
* POSIX filesystem-backed secret store for hosts without a system keyring
* (e.g. Linux servers). Secrets live in a single 0600 JSON map under the data
* root, replaced atomically via temp file + rename, with writes serialized
* in-process.
*/
export class FileSecretStore implements SecretStore {
readonly provider = PROVIDER;
private queue: Promise<unknown> = Promise.resolve();
constructor(readonly filePath: string) {
if (typeof filePath !== 'string' || filePath.length === 0 || !isAbsolute(filePath))
invalid('INVALID_STORE_PATH', 'Secret file path must be absolute');
}
async set(key: SecretKey, value: string): Promise<string> {
if (
typeof value !== 'string' ||
value.length === 0 ||
/[\0\r\n]/.test(value) ||
Buffer.byteLength(value, 'utf8') > MAX_SECRET_BYTES
) {
invalid('INVALID_SECRET', 'Secret is not valid for file storage');
}
validateKey(key);
const account = accountFor(key);
await this.enqueue(() => this.withStore((store) => ({ ...store, [account]: value })));
return `secret-file://${SERVICE}/${account}`;
}
async get(reference: string): Promise<string | undefined> {
const { account } = this.parse(reference);
const store = await this.readStore();
return store[account];
}
async delete(reference: string): Promise<boolean> {
const { account } = this.parse(reference);
let removed = false;
await this.enqueue(() =>
this.withStore((store) => {
if (!(account in store)) return store;
removed = true;
const next = { ...store };
delete next[account];
return next;
}),
);
return removed;
}
private parse(reference: string): { account: string } {
if (typeof reference !== 'string' || reference.length > 512)
invalid('INVALID_REFERENCE', 'Secret file reference is invalid');
const match = /^secret-file:\/\/multi-simadmin\/([A-Za-z0-9_-]+)$/.exec(reference);
if (!match?.[1]) invalid('INVALID_REFERENCE', 'Secret file reference is invalid');
try {
parseSecretAccount(match[1], PROVIDER);
} catch (error) {
if (error instanceof SecretReferenceError)
invalid('INVALID_REFERENCE', 'Secret file reference is invalid');
throw error;
}
return { account: match[1] };
}
private enqueue<T>(task: () => Promise<T>): Promise<T> {
const run = this.queue.then(task, task);
this.queue = run.catch(() => {});
return run;
}
private async readStore(): Promise<Record<string, string>> {
let raw: string;
try {
raw = await readFile(this.filePath, 'utf8');
} catch (error) {
if ((error as NodeJS.ErrnoException).code === 'ENOENT') return {};
throw error;
}
if (Buffer.byteLength(raw, 'utf8') > MAX_STORE_BYTES)
invalid('SIZE_LIMIT', 'Secret file exceeds the size limit');
let parsed: unknown;
try {
parsed = JSON.parse(raw);
} catch {
invalid('PAYLOAD_INVALID', 'Secret file payload is invalid');
}
if (!isPlainStringMap(parsed)) invalid('PAYLOAD_INVALID', 'Secret file payload is invalid');
return parsed;
}
/** Reads the current map, applies `mutate`, and durably replaces the file. */
private async withStore(
mutate: (store: Record<string, string>) => Record<string, string>,
): Promise<void> {
const current = await this.readStore();
const next = mutate(current);
const directory = dirname(this.filePath);
await mkdir(directory, { recursive: true, mode: 0o700 });
if (process.platform !== 'win32') {
const info = await stat(directory);
if (info.isDirectory() && (info.mode & 0o077) !== 0) await chmod(directory, 0o700);
}
const temporary = `${this.filePath}.${randomUUID()}.tmp`;
const handle = await open(temporary, 'wx', 0o600);
try {
await handle.writeFile(`${JSON.stringify(next, null, 2)}\n`, 'utf8');
// Windows FlushFileBuffers is skipped along with the directory fsync below;
// durability anchoring is POSIX-only by design.
if (process.platform !== 'win32') await handle.sync();
} finally {
await handle.close();
}
try {
await rename(temporary, this.filePath);
if (process.platform !== 'win32') {
const directoryHandle = await open(directory, 'r');
try {
await directoryHandle.sync();
} finally {
await directoryHandle.close();
}
}
} catch (error) {
await rm(temporary, { force: true });
throw error;
}
}
}
function isPlainStringMap(value: unknown): value is Record<string, string> {
if (typeof value !== 'object' || value === null || Array.isArray(value)) return false;
return Object.values(value).every((entry) => typeof entry === 'string');
}
@@ -151,7 +151,23 @@ describe('MacOSKeychainSecretStore', () => {
}
});
it.each(['line\nbreak', 'line\rbreak', 'nul\0break', 'x'.repeat(4097)])(
it('stores a maximum-sized scheduled SMS payload', async () => {
const runner = new FakeRunner();
const store = new MacOSKeychainSecretStore(runner);
const payload = JSON.stringify({
recipients: Array.from(
{ length: 50 },
(_, index) => `1380013${String(index).padStart(4, '0')}`,
),
content: '字'.repeat(2_000),
});
await expect(
store.set({ instanceId: 'task-1', purpose: 'scheduled-sms', slot: 'rotation-1' }, payload),
).resolves.toMatch(/^keychain:/);
});
it.each(['line\nbreak', 'line\rbreak', 'nul\0break', 'x'.repeat(16_385)])(
'rejects non-line-safe or oversized secret input before invoking the runner',
async (value) => {
const runner = new FakeRunner();
@@ -1,11 +1,11 @@
import { spawn } from 'node:child_process';
import { accountFor, parseSecretAccount, SecretReferenceError } from './secret-reference.js';
import type { SecretKey, SecretStore } from './secret-store.js';
const SECURITY_PATH = '/usr/bin/security';
const SERVICE = 'multi-simadmin';
const TIMEOUT_MS = 5_000;
const MAX_STDOUT_BYTES = 65_536;
const safeComponent = /^[A-Za-z0-9_.-]+$/;
export type SecretStoreErrorCode =
| 'INVALID_KEY'
@@ -105,43 +105,6 @@ function invalid(code: 'INVALID_KEY' | 'INVALID_REFERENCE', message: string): ne
throw new SecretStoreError(code, message);
}
function validateKey(key: SecretKey): void {
if (
typeof key.instanceId !== 'string' ||
key.instanceId.length === 0 ||
key.instanceId.length > 128 ||
!safeComponent.test(key.instanceId)
) {
invalid('INVALID_KEY', 'Secret key instance id is invalid');
}
if (
typeof key.purpose !== 'string' ||
key.purpose.length === 0 ||
key.purpose.length > 64 ||
!safeComponent.test(key.purpose)
) {
invalid('INVALID_KEY', 'Secret key purpose is invalid');
}
if (
key.slot !== undefined &&
(typeof key.slot !== 'string' ||
key.slot.length === 0 ||
key.slot.length > 64 ||
!safeComponent.test(key.slot))
) {
invalid('INVALID_KEY', 'Secret key slot is invalid');
}
}
function accountFor(key: SecretKey): string {
validateKey(key);
const components =
key.slot === undefined
? [key.instanceId, key.purpose]
: [key.instanceId, key.purpose, key.slot];
return Buffer.from(JSON.stringify(components), 'utf8').toString('base64url');
}
function referenceFor(account: string): string {
return `keychain://${SERVICE}/${account}`;
}
@@ -152,28 +115,26 @@ export function parseKeychainReference(reference: string): ParsedKeychainReferen
const match = /^keychain:\/\/multi-simadmin\/([A-Za-z0-9_-]+)$/.exec(reference);
if (!match?.[1]) invalid('INVALID_REFERENCE', 'Keychain reference is invalid');
const account = match[1];
let decoded: unknown;
let parsed;
try {
decoded = JSON.parse(Buffer.from(account, 'base64url').toString('utf8'));
} catch {
invalid('INVALID_REFERENCE', 'Keychain reference is invalid');
parsed = parseSecretAccount(account, 'macos-keychain');
} catch (error) {
if (error instanceof SecretReferenceError)
invalid('INVALID_REFERENCE', 'Keychain reference is invalid');
throw error;
}
if (!Array.isArray(decoded) || (decoded.length !== 2 && decoded.length !== 3))
invalid('INVALID_REFERENCE', 'Keychain reference is invalid');
const [instanceId, purpose, slot] = decoded;
const key = { instanceId, purpose, ...(slot === undefined ? {} : { slot }) } as SecretKey;
validateKey(key);
if (accountFor(key) !== account) invalid('INVALID_REFERENCE', 'Keychain reference is invalid');
return {
service: SERVICE,
account,
instanceId: key.instanceId,
purpose: key.purpose,
...(key.slot === undefined ? {} : { slot: key.slot }),
instanceId: parsed.instanceId,
purpose: parsed.purpose,
...(parsed.slot === undefined ? {} : { slot: parsed.slot }),
};
}
export class MacOSKeychainSecretStore implements SecretStore {
readonly provider = 'macos-keychain';
constructor(private readonly runner: CommandRunner = new SpawnCommandRunner()) {}
async set(key: SecretKey, value: string): Promise<string> {
@@ -181,7 +142,7 @@ export class MacOSKeychainSecretStore implements SecretStore {
typeof value !== 'string' ||
value.length === 0 ||
/[\0\r\n]/.test(value) ||
Buffer.byteLength(value, 'utf8') > 4096
Buffer.byteLength(value, 'utf8') > 16_384
) {
throw new SecretStoreError('INVALID_SECRET', 'Secret is not valid for Keychain storage');
}
@@ -0,0 +1,165 @@
import { describe, expect, it } from 'vitest';
import { mkdtemp, readFile, readdir, rm, stat, writeFile, chmod } from 'node:fs/promises';
import { tmpdir } from 'node:os';
import { join } from 'node:path';
import { FileSecretStore, FileSecretStoreError } from './file-secret-store.js';
import { SecretReferenceError } from './secret-reference.js';
import {
createDefaultSecretStore,
defaultSecretFileMetadataCheck,
resolveSecretBackend,
SecretBackendError,
} from './secret-backend.js';
import { MacOSKeychainSecretStore } from './keychain-secret-store.js';
describe('FileSecretStore', () => {
it('round-trips set/get/delete with stable secret-file references', async () => {
const directory = await mkdtemp(join(tmpdir(), 'msa-file-secrets-'));
try {
const store = new FileSecretStore(join(directory, 'secrets.json'));
const reference = await store.set(
{ instanceId: 'device-1', purpose: 'instance-password', slot: 'slot-1' },
'pa$$word',
);
expect(reference).toMatch(/^secret-file:\/\/multi-simadmin\/[A-Za-z0-9_-]+$/);
expect(await store.get(reference)).toBe('pa$$word');
expect(await store.delete(reference)).toBe(true);
expect(await store.get(reference)).toBeUndefined();
expect(await store.delete(reference)).toBe(false);
} finally {
await rm(directory, { recursive: true, force: true });
}
});
it('rejects empty, framing, oversize, and malformed keys/values', async () => {
const directory = await mkdtemp(join(tmpdir(), 'msa-file-secrets-'));
try {
const store = new FileSecretStore(join(directory, 'secrets.json'));
await expect(store.set({ instanceId: 'x', purpose: 'p' }, '')).rejects.toThrow(
FileSecretStoreError,
);
await expect(store.set({ instanceId: 'x', purpose: 'p' }, 'a\nb')).rejects.toThrow(
FileSecretStoreError,
);
await expect(
store.set({ instanceId: 'x', purpose: 'p' }, 'a'.repeat(16_385)),
).rejects.toThrow(FileSecretStoreError);
await expect(store.set({ instanceId: '', purpose: 'p' }, 'v')).rejects.toThrow(
SecretReferenceError,
);
await expect(store.get('https://evil.example/one')).rejects.toThrow(FileSecretStoreError);
await expect(store.get('secret-file://other-service/abc')).rejects.toThrow(
FileSecretStoreError,
);
} finally {
await rm(directory, { recursive: true, force: true });
}
});
it('keeps the map private (0600) and leaves no temporary files behind', async () => {
const directory = await mkdtemp(join(tmpdir(), 'msa-file-secrets-'));
try {
const path = join(directory, 'nested', 'secrets.json');
const store = new FileSecretStore(path);
const reference = await store.set({ instanceId: 'x', purpose: 'p', slot: 's' }, 'secret');
if (process.platform !== 'win32') expect((await stat(path)).mode & 0o777).toBe(0o600);
expect(await readdir(join(directory, 'nested'))).toEqual(['secrets.json']);
const raw = await readFile(path, 'utf8');
expect(JSON.parse(raw)).toEqual({
[reference.slice('secret-file://multi-simadmin/'.length)]: 'secret',
});
expect(await store.get(reference)).toBe('secret');
} finally {
await rm(directory, { recursive: true, force: true });
}
});
it('serializes concurrent writes so every mutation lands', async () => {
const directory = await mkdtemp(join(tmpdir(), 'msa-file-secrets-'));
try {
const store = new FileSecretStore(join(directory, 'secrets.json'));
const references = await Promise.all(
Array.from({ length: 25 }, (_, index) =>
store.set({ instanceId: 'x', purpose: `p${index}`, slot: 's' }, `v${index}`),
),
);
for (const [index, reference] of references.entries())
expect(await store.get(reference)).toBe(`v${index}`);
} finally {
await rm(directory, { recursive: true, force: true });
}
});
it('refuses a corrupted or non-map payload instead of failing open', async () => {
const directory = await mkdtemp(join(tmpdir(), 'msa-file-secrets-'));
try {
const path = join(directory, 'secrets.json');
await writeFile(path, 'not json', 'utf8');
const store = new FileSecretStore(path);
await expect(store.get('secret-file://multi-simadmin/abc')).rejects.toThrow(
FileSecretStoreError,
);
await writeFile(path, '{"account": 42}', 'utf8');
await expect(store.get('secret-file://multi-simadmin/abc')).rejects.toThrow(
FileSecretStoreError,
);
expect(() => new FileSecretStore('relative/secrets.json')).toThrow(FileSecretStoreError);
} finally {
await rm(directory, { recursive: true, force: true });
}
});
});
describe('secret backend selection', () => {
it('defaults to the keychain on darwin and the file store elsewhere', () => {
expect(resolveSecretBackend({}, 'darwin')).toBe('macos-keychain');
expect(resolveSecretBackend({}, 'linux')).toBe('secret-file');
expect(resolveSecretBackend({ MULTI_SIMADMIN_SECRET_BACKEND: 'secret-file' }, 'darwin')).toBe(
'secret-file',
);
expect(() =>
resolveSecretBackend({ MULTI_SIMADMIN_SECRET_BACKEND: 'vault' }, 'darwin'),
).toThrow(SecretBackendError);
});
it('builds the matching store and refuses keychain off macOS', () => {
const directory = join(tmpdir(), 'msa-backend-selection');
expect(
createDefaultSecretStore({
env: {},
secretFilePath: join(directory, 's.json'),
platform: 'linux',
}),
).toBeInstanceOf(FileSecretStore);
expect(
createDefaultSecretStore({
env: { MULTI_SIMADMIN_SECRET_BACKEND: 'macos-keychain' },
secretFilePath: join(directory, 's.json'),
platform: 'darwin',
}),
).toBeInstanceOf(MacOSKeychainSecretStore);
expect(() =>
createDefaultSecretStore({
env: { MULTI_SIMADMIN_SECRET_BACKEND: 'macos-keychain' },
secretFilePath: join(directory, 's.json'),
platform: 'linux',
}),
).toThrow(SecretBackendError);
});
it('file metadata probe accepts a missing or private file and rejects a public one', async () => {
const directory = await mkdtemp(join(tmpdir(), 'msa-file-secrets-meta-'));
try {
const path = join(directory, 'secrets.json');
expect(await defaultSecretFileMetadataCheck(path)).toBe(true);
await writeFile(path, '{}\n', { mode: 0o600 });
expect(await defaultSecretFileMetadataCheck(path)).toBe(true);
if (process.platform !== 'win32') {
await chmod(path, 0o644);
expect(await defaultSecretFileMetadataCheck(path)).toBe(false);
}
} finally {
await rm(directory, { recursive: true, force: true });
}
});
});

Some files were not shown because too many files have changed in this diff Show More