Compare commits
39
Commits
bbd94b5672
..
main
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
61a5aee48d | ||
|
|
70e598208a | ||
|
|
c1be714ba4 | ||
|
|
29a0eee854 | ||
|
|
2977f75129 | ||
|
|
236e78327c | ||
|
|
3c5971e167 | ||
|
|
e4c4bce75b | ||
|
|
c57dc81e42 | ||
|
|
96dd9e2ba6 | ||
|
|
4f42a6f835 | ||
|
|
eed134a25c | ||
|
|
f6319374cd | ||
|
|
6d251dab7b | ||
|
|
864e1bd90c | ||
|
|
635527dd26 | ||
|
|
7168e9dba2 | ||
|
|
961eb928c4 | ||
|
|
8406f14469 | ||
|
|
8b7df8cb65 | ||
|
|
5016586922 | ||
|
|
a75badeb58 | ||
|
|
f9186bd851 | ||
|
|
f11877f13e | ||
|
|
03fa2e6c7b | ||
|
|
9bc0f27f12 | ||
|
|
95abd09c39 | ||
|
|
4fbc902325 | ||
|
|
25502a52bb | ||
|
|
144e17c220 | ||
|
|
3a6652be0f | ||
|
|
af5abb4b82 | ||
|
|
3bbd227c58 | ||
|
|
d60cff1dd8 | ||
|
|
6ec8e4f6a6 | ||
|
|
cf2cf6827d | ||
|
|
570edf6bb2 | ||
|
|
25c853dea8 | ||
|
|
f57ec1f48d |
@@ -0,0 +1 @@
|
||||
* text=auto eol=lf
|
||||
@@ -7,3 +7,6 @@ npm-debug.log*
|
||||
coverage/
|
||||
apps/web/dist/
|
||||
.hermes/
|
||||
.local-run/
|
||||
artifacts/
|
||||
data/
|
||||
|
||||
@@ -11,9 +11,10 @@
|
||||
- 支持无密码实例状态读取;有密码实例可通过本地服务代登录并保持会话。
|
||||
- 统一代理 SimAdmin API,内置 API 工作台,便于读取/调试设备、SIM、网络、短信、eSIM、OTA 等接口。
|
||||
- 保留原始 SimAdmin 页面 iframe 嵌入;若目标站禁止 iframe,可一键在原站打开。
|
||||
- 融合 [SimAdminHub](https://github.com/3899/SimAdminHub) 的管理思路:跨节点短信、通知、自动化与跨域健康自然融入 Fleet 节点总览体系,无需部署或接入外部 Hub。
|
||||
- 敏感配置留在本地 `config.json`,仓库只提交 `config.example.json`。
|
||||
|
||||
## 一键安装(macOS)
|
||||
## 一键安装(macOS / Linux)
|
||||
|
||||
```bash
|
||||
curl -fsSL https://gitea.chickliu.fun/Hermes/multi-simadmin/raw/branch/main/scripts/install.sh | sh
|
||||
@@ -27,7 +28,7 @@ curl -fsSL https://gitea.chickliu.fun/Hermes/multi-simadmin/raw/branch/main/scri
|
||||
|
||||
安装完成后按终端输出访问 `http://<本机局域网 IP>:8788/fleet`。首次密码可在当前管理台直接设置。Gateway 默认监听所有网络接口,因此必须确保主机仅接入可信内网或已通过防火墙限制 8788 的来源;HTTP 部署不得直接暴露到公网,公网开放必须由前置代理提供 HTTPS 和访问控制。
|
||||
|
||||
> 当前生产秘密存储使用 macOS Keychain,因此一键安装脚本暂只支持 macOS。脚本不会覆盖已有源码目录或数据库,也不会占用已被其他进程监听的 8788/8790 端口。
|
||||
> 秘密存储按平台自动选择:macOS 使用 Keychain,Linux 使用数据目录下 0600 权限的 `secrets.json` 文件;也可用 `MULTI_SIMADMIN_SECRET_BACKEND=macos-keychain|secret-file` 显式指定。脚本不会覆盖已有源码目录或数据库,也不会占用已被其他进程监听的 8788/8790 端口。
|
||||
|
||||
### 服务管理
|
||||
|
||||
@@ -42,7 +43,46 @@ sh /tmp/multi-simadmin-install.sh start
|
||||
sh /tmp/multi-simadmin-install.sh uninstall
|
||||
```
|
||||
|
||||
`uninstall` 默认只移除程序源码,保留数据库、Gateway token 和日志。默认安装位置为 `~/Library/Application Support/multi-simadmin`,可通过 `MULTI_SIMADMIN_HOME` 覆盖;额外 LAN Host 可通过 `MULTI_SIMADMIN_ALLOWED_HOSTS`(逗号分隔)配置。
|
||||
`uninstall` 默认只移除程序源码,保留数据库、Gateway token 和日志。默认安装位置 macOS 为 `~/Library/Application Support/multi-simadmin`,Linux 为 `~/.local/share/multi-simadmin`(遵循 XDG),均可通过 `MULTI_SIMADMIN_HOME` 覆盖;额外 LAN Host 可通过 `MULTI_SIMADMIN_ALLOWED_HOSTS`(逗号分隔)配置。
|
||||
|
||||
#### Linux systemd(可选,但推荐)
|
||||
|
||||
`sh install.sh unit` 会生成 `~/.config/systemd/user/` 下的 API 与 Gateway 用户单元(`Restart=on-failure`、开机自启、密钥经 0600 的 `service-env` 注入)。随后:
|
||||
|
||||
```bash
|
||||
systemctl --user enable --now multi-simadmin-api.service multi-simadmin-gateway.service
|
||||
```
|
||||
|
||||
单元内声明了 `MULTI_SIMADMIN_SYSTEMD_UNIT`,因此 Web 控制台的“在线更新”在 Linux 上会把重启交给 systemd 完成,形成自更新闭环。由 systemd 托管后请用 `systemctl` 管理进程,不要再用本脚本 `start/stop`,避免双进程争抢端口。
|
||||
|
||||
### Fleet 节点总览与跨域健康
|
||||
|
||||
`/fleet` 读取本机 Control Plane 中的节点状态、资源快照与设备号码,形成节点总览。侧栏在实例总数 / 在线 / 需处理之外,还聚合跨域健康指标:
|
||||
|
||||
- 短信异常:标记哪些节点的短信接口不可用,点击直达 `/fleet/messages`。
|
||||
- 通知异常:汇总跨节点通知队列的 pending + failed 数量,点击直达 `/fleet/notifications`。
|
||||
|
||||
节点卡片直接进入对应实例详情,查看 CPU、内存、温度、SIM 号码、平台与版本。
|
||||
|
||||
### 跨节点短信中心
|
||||
|
||||
`/fleet/messages` 融合官方 SimAdminHub 的短信中心形态:它通过本地 Control Plane 按节点读取 SimAdmin 的 `/api/sms/list`,再跨设备聚合搜索、展示与发送。
|
||||
|
||||
- 跨设备记录:汇总各节点最新短信,支持按号码、内容或节点名称搜索,并分页加载。
|
||||
- 会话视图:按号码分组为会话,节点栏 / 会话栏 / 消息栏三栏工作流,默认打开最新会话。
|
||||
- 发送短信:从可用节点下拉选择目标设备,提交到 `/api/v1/fleet/messages/send` 后由 Control Plane 转发到对应 SimAdmin。
|
||||
- 批量删除:可在短信记录中勾选选择并确认删除,Control Plane 通过严格白名单的 `/api/sms/batch-delete` 按节点批量清理,未成功项会以逐条失败原因返回。
|
||||
- 安全边界:短信正文与号码只在本地页面展示;批量删除请求最多 500 条,且不接收外部 Hub 地址、Webhook 或任何可注入字段。
|
||||
|
||||
### 跨节点通知中心
|
||||
|
||||
`/fleet/notifications` 融合官方 SimAdminHub 的通知中心能力:它按本地节点聚合阅读 SimAdmin 的通知接口,再跨设备汇总转发通道、转发规则、转发日志与失败重试队列。
|
||||
|
||||
- 设备可用性:自动登录后读取 `/api/notifications/config`、`/api/notifications/logs` 与 `/api/notifications/queue`,无法读取的实例单独标记。
|
||||
- 转发通道:统计通道总数、启用数以及常见的 webhook、Bark、PushPlus、企业微信、钉钉、飞书、Telegram、邮件等类型。
|
||||
- 转发日志:展示成功、失败、免打扰、未匹配、无可用通道与最近记录。
|
||||
- 失败重试:展示待发送、定时、重试中、发送中和失败的队列摘要,并支持整队列重试、单条重试或删除积压项。
|
||||
- 安全边界:只返回经过白名单清洗的计数与最近条目,不暴露通知原文、webhook/token/secret 或模板配置。
|
||||
|
||||
## 旧版开发入口
|
||||
|
||||
@@ -50,10 +90,12 @@ sh /tmp/multi-simadmin-install.sh uninstall
|
||||
|
||||
```bash
|
||||
cp config.example.json config.json
|
||||
npm install
|
||||
npm start
|
||||
corepack pnpm install
|
||||
corepack pnpm start
|
||||
```
|
||||
|
||||
> 仓库统一使用 pnpm(`pnpm-lock.yaml`)作为唯一 lockfile;陈旧的 `package-lock.json` 已移除。
|
||||
|
||||
## 配置
|
||||
|
||||
编辑 `config.json`:
|
||||
|
||||
@@ -16,6 +16,7 @@
|
||||
"@multi-simadmin/contracts": "workspace:*",
|
||||
"@multi-simadmin/operation-registry": "workspace:*",
|
||||
"better-sqlite3": "12.11.1",
|
||||
"cron-parser": "^5.6.2",
|
||||
"drizzle-orm": "0.45.2",
|
||||
"fastify": "5.10.0",
|
||||
"tsx": "4.22.4"
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
import type Database from 'better-sqlite3';
|
||||
import {
|
||||
AUDIT_OUTCOMES,
|
||||
type AuditOutcome,
|
||||
type AuditEvent,
|
||||
type AuditPage,
|
||||
type AuditPageQuery,
|
||||
@@ -19,6 +20,26 @@ export class AuditQueryError extends Error {
|
||||
}
|
||||
}
|
||||
|
||||
export interface ControlPlaneAuditSummary {
|
||||
readonly total: number;
|
||||
readonly succeeded: number;
|
||||
readonly failed: number;
|
||||
readonly partiallySucceeded: number;
|
||||
readonly denied: number;
|
||||
readonly recent: readonly ControlPlaneAuditSummaryEvent[];
|
||||
}
|
||||
|
||||
export interface ControlPlaneAuditSummaryEvent {
|
||||
readonly id: string;
|
||||
readonly occurredAt: string;
|
||||
readonly actorId: string;
|
||||
readonly action: string;
|
||||
readonly outcome: AuditEvent['outcome'];
|
||||
readonly requestId: string;
|
||||
readonly instanceId?: string;
|
||||
readonly jobId?: string;
|
||||
}
|
||||
|
||||
interface AuditRow {
|
||||
id: unknown;
|
||||
instance_id: unknown;
|
||||
@@ -118,6 +139,74 @@ function parameterSummary(value: unknown): readonly RedactedParameterSummaryItem
|
||||
export class AuditQueryService {
|
||||
constructor(private readonly db: Database.Database) {}
|
||||
|
||||
/** Retention hook: drops audit records older than the cutoff. */
|
||||
pruneBefore(cutoffIso: string): number {
|
||||
if (typeof cutoffIso !== 'string' || !Number.isFinite(Date.parse(cutoffIso)))
|
||||
throw new RangeError('cutoffIso must be a valid date-time');
|
||||
return Number(
|
||||
this.db
|
||||
.prepare('DELETE FROM audit_events WHERE created_at < ?')
|
||||
.run(new Date(cutoffIso).toISOString()).changes,
|
||||
);
|
||||
}
|
||||
|
||||
summary(limit = 5): ControlPlaneAuditSummary {
|
||||
if (!Number.isSafeInteger(limit) || limit < 1 || limit > 20) validation('limit is invalid');
|
||||
const totalRow = this.db.prepare('SELECT COUNT(*) AS total FROM audit_events').get() as
|
||||
| {
|
||||
total: unknown;
|
||||
}
|
||||
| undefined;
|
||||
if (
|
||||
!totalRow ||
|
||||
typeof totalRow.total !== 'number' ||
|
||||
!Number.isSafeInteger(totalRow.total) ||
|
||||
totalRow.total < 0
|
||||
)
|
||||
fail('Persisted audit count is invalid');
|
||||
const outcomeRows = this.db
|
||||
.prepare('SELECT result_code, COUNT(*) AS count FROM audit_events GROUP BY result_code')
|
||||
.all() as Array<{ result_code: unknown; count: unknown }>;
|
||||
const counts = new Map<string, number>();
|
||||
for (const row of outcomeRows) {
|
||||
const outcome = boundedString(row.result_code, 'result_code');
|
||||
if (!(AUDIT_OUTCOMES as readonly string[]).includes(outcome))
|
||||
fail('Persisted audit result_code is unrepresentable');
|
||||
if (!Number.isSafeInteger(row.count) || (row.count as number) < 0)
|
||||
fail('Persisted audit outcome count is invalid');
|
||||
counts.set(outcome, row.count as number);
|
||||
}
|
||||
const count = (outcome: AuditOutcome): number => counts.get(outcome) ?? 0;
|
||||
const recentRows = this.db
|
||||
.prepare(`SELECT ${COLUMNS} FROM audit_events ORDER BY created_at DESC, id ASC LIMIT ?`)
|
||||
.all(limit) as AuditRow[];
|
||||
const recent = recentRows.map((row) => {
|
||||
const outcome = boundedString(row.result_code, 'result_code');
|
||||
if (!(AUDIT_OUTCOMES as readonly string[]).includes(outcome))
|
||||
fail('Persisted audit result_code is unrepresentable');
|
||||
const instanceId = optionalString(row.instance_id, 'instance_id');
|
||||
const jobId = optionalString(row.job_id, 'job_id');
|
||||
return freeze({
|
||||
id: boundedString(row.id, 'id'),
|
||||
occurredAt: canonicalTimestamp(row.created_at, 'created_at'),
|
||||
actorId: boundedString(row.actor, 'actor'),
|
||||
action: boundedString(row.operation_id, 'operation_id'),
|
||||
outcome: outcome as ControlPlaneAuditSummaryEvent['outcome'],
|
||||
requestId: boundedString(row.request_id, 'request_id'),
|
||||
...(instanceId === undefined ? {} : { instanceId }),
|
||||
...(jobId === undefined ? {} : { jobId }),
|
||||
});
|
||||
});
|
||||
return freeze({
|
||||
total: totalRow.total,
|
||||
succeeded: count('succeeded'),
|
||||
failed: count('failed'),
|
||||
partiallySucceeded: count('partially-succeeded'),
|
||||
denied: count('denied'),
|
||||
recent: freeze(recent),
|
||||
});
|
||||
}
|
||||
|
||||
get(id: string): AuditEvent {
|
||||
inputString(id, 'eventId');
|
||||
const row = this.db.prepare(`SELECT ${COLUMNS} FROM audit_events WHERE id = ?`).get(id) as
|
||||
|
||||
@@ -6,18 +6,43 @@ const SESSION_TTL_MS = 7 * 24 * 60 * 60 * 1000;
|
||||
const MIN_PASSWORD_LENGTH = 8;
|
||||
const MAX_PASSWORD_BYTES = 1024;
|
||||
const SCRYPT_KEY_LENGTH = 32;
|
||||
// Legacy derivations used N=16384; OWASP guidance for interactive login sits at
|
||||
// N=2^16/r=8. Rows keep their derivation tag so verification stays correct and
|
||||
// the upgrade happens transparently on the next successful login.
|
||||
interface ScryptParams {
|
||||
readonly N: number;
|
||||
readonly r: number;
|
||||
readonly p: number;
|
||||
}
|
||||
const SCRYPT_LEGACY: ScryptParams = { N: 16_384, r: 8, p: 1 };
|
||||
const SCRYPT_CURRENT: ScryptParams = { N: 65_536, r: 8, p: 1 };
|
||||
const SCRYPT_MAXMEM = 256 * 1024 * 1024;
|
||||
const scryptTag = (params: ScryptParams): string => `scrypt:${params.N}:${params.r}:${params.p}`;
|
||||
const parseScryptTag = (value: string | null | undefined): ScryptParams => {
|
||||
if (!value) return SCRYPT_LEGACY;
|
||||
const match = /^scrypt:(\d+):(\d+):(\d+)$/.exec(value);
|
||||
if (!match) return SCRYPT_LEGACY;
|
||||
const [N, r, p] = match.slice(1).map(Number) as [number, number, number];
|
||||
return { N, r, p };
|
||||
};
|
||||
const IDLE_TIMEOUT_SETTING_KEY = 'console-auth.idle-timeout-minutes';
|
||||
const DEFAULT_IDLE_TIMEOUT_MINUTES = 30;
|
||||
const MIN_IDLE_TIMEOUT_MINUTES = 5;
|
||||
const MAX_IDLE_TIMEOUT_MINUTES = 1440;
|
||||
|
||||
interface AuthConfigRow {
|
||||
readonly protection_enabled: 0 | 1;
|
||||
readonly password_salt: string | null;
|
||||
readonly password_hash: string | null;
|
||||
readonly password_revision: number;
|
||||
readonly password_kdf: string | null;
|
||||
}
|
||||
|
||||
export interface ConsoleAuthStatus {
|
||||
readonly configured: boolean;
|
||||
readonly protectionEnabled: boolean;
|
||||
readonly authenticated: boolean;
|
||||
readonly sessionIdleTimeoutMinutes: number;
|
||||
}
|
||||
|
||||
export class ConsoleAuthError extends Error {
|
||||
@@ -40,6 +65,7 @@ export interface ConsoleAuthServiceOptions {
|
||||
export class ConsoleAuthService {
|
||||
private readonly db: Database.Database;
|
||||
private readonly now: () => Date;
|
||||
#idleTimeoutMinutes: number = DEFAULT_IDLE_TIMEOUT_MINUTES;
|
||||
|
||||
constructor(options: ConsoleAuthServiceOptions) {
|
||||
this.db = options.db;
|
||||
@@ -49,17 +75,19 @@ export class ConsoleAuthService {
|
||||
status(sessionToken?: string): ConsoleAuthStatus {
|
||||
this.pruneExpired();
|
||||
const config = this.config();
|
||||
const sessionIdleTimeoutMinutes = this.idleTimeoutMinutes();
|
||||
const configured = config?.password_hash !== null && config?.password_hash !== undefined;
|
||||
const protectionEnabled = config?.protection_enabled === 1;
|
||||
return {
|
||||
configured,
|
||||
protectionEnabled,
|
||||
authenticated: !protectionEnabled || this.isAuthenticated(sessionToken, config),
|
||||
};
|
||||
const authenticated = !protectionEnabled || this.isAuthenticated(sessionToken, config);
|
||||
return { configured, protectionEnabled, authenticated, sessionIdleTimeoutMinutes };
|
||||
}
|
||||
|
||||
isProtectedPath(pathname: string): boolean {
|
||||
if (!pathname.startsWith('/api/v1/')) return false;
|
||||
// Monitoring scrapers cannot hold a browser session. Like the Hub's /health and /metrics,
|
||||
// this one stays session-free and is meant for the trusted network the console already
|
||||
// assumes; it carries no device addresses, only opaque node ids and counters.
|
||||
if (pathname === '/api/v1/metrics') return false;
|
||||
return !pathname.startsWith('/api/v1/auth/');
|
||||
}
|
||||
|
||||
@@ -67,6 +95,7 @@ export class ConsoleAuthService {
|
||||
const status = this.status(sessionToken);
|
||||
if (status.protectionEnabled && !status.authenticated)
|
||||
throw new ConsoleAuthError('AUTH_REQUIRED');
|
||||
this.touchSession(sessionToken);
|
||||
}
|
||||
|
||||
async login(password: string): Promise<string> {
|
||||
@@ -78,19 +107,37 @@ export class ConsoleAuthService {
|
||||
!config.password_hash
|
||||
)
|
||||
throw new ConsoleAuthError('INVALID_CREDENTIALS');
|
||||
const actual = await this.derive(password, config.password_salt);
|
||||
const params = parseScryptTag(config.password_kdf);
|
||||
const actual = await this.derive(password, config.password_salt, params);
|
||||
const expected = Buffer.from(config.password_hash, 'hex');
|
||||
if (actual.length !== expected.length || !timingSafeEqual(actual, expected))
|
||||
throw new ConsoleAuthError('INVALID_CREDENTIALS');
|
||||
if (params !== SCRYPT_CURRENT) {
|
||||
// Transparent upgrade: same salt, stronger derivation, same revision so
|
||||
// live sessions survive.
|
||||
const hash = (await this.derive(password, config.password_salt, SCRYPT_CURRENT)).toString(
|
||||
'hex',
|
||||
);
|
||||
this.db
|
||||
.prepare(
|
||||
'UPDATE console_auth_config SET password_hash=?,password_kdf=?,updated_at=? WHERE singleton=1',
|
||||
)
|
||||
.run(hash, scryptTag(SCRYPT_CURRENT), this.now().toISOString());
|
||||
}
|
||||
return this.createSession(config.password_revision);
|
||||
}
|
||||
|
||||
async updateSettings(
|
||||
input: { readonly enabled: boolean; readonly newPassword?: string },
|
||||
input: {
|
||||
readonly enabled: boolean;
|
||||
readonly newPassword?: string;
|
||||
readonly idleTimeoutMinutes?: number;
|
||||
},
|
||||
sessionToken?: string,
|
||||
): Promise<{ readonly status: ConsoleAuthStatus; readonly sessionToken?: string }> {
|
||||
const current = this.config();
|
||||
if (current?.protection_enabled === 1) this.requireSession(sessionToken);
|
||||
if (input.idleTimeoutMinutes !== undefined) this.saveIdleTimeout(input.idleTimeoutMinutes);
|
||||
|
||||
if (input.enabled) {
|
||||
if (input.newPassword !== undefined) {
|
||||
@@ -107,14 +154,14 @@ export class ConsoleAuthService {
|
||||
this.db
|
||||
.prepare(
|
||||
`INSERT INTO console_auth_config
|
||||
(singleton,protection_enabled,password_salt,password_hash,password_revision,updated_at)
|
||||
VALUES (1,1,?,?,?,?)
|
||||
(singleton,protection_enabled,password_salt,password_hash,password_revision,password_kdf,updated_at)
|
||||
VALUES (1,1,?,?,?,?,?)
|
||||
ON CONFLICT(singleton) DO UPDATE SET
|
||||
protection_enabled=1,password_salt=excluded.password_salt,
|
||||
password_hash=excluded.password_hash,password_revision=excluded.password_revision,
|
||||
updated_at=excluded.updated_at`,
|
||||
password_kdf=excluded.password_kdf,updated_at=excluded.updated_at`,
|
||||
)
|
||||
.run(salt, hash, revision, this.now().toISOString());
|
||||
.run(salt, hash, revision, scryptTag(SCRYPT_CURRENT), this.now().toISOString());
|
||||
this.db.prepare('DELETE FROM console_auth_sessions').run();
|
||||
return this.createSession(revision);
|
||||
})();
|
||||
@@ -164,12 +211,37 @@ export class ConsoleAuthService {
|
||||
private config(): AuthConfigRow | undefined {
|
||||
return this.db
|
||||
.prepare(
|
||||
`SELECT protection_enabled,password_salt,password_hash,password_revision
|
||||
`SELECT protection_enabled,password_salt,password_hash,password_revision,password_kdf
|
||||
FROM console_auth_config WHERE singleton=1`,
|
||||
)
|
||||
.get() as AuthConfigRow | undefined;
|
||||
}
|
||||
|
||||
private idleTimeoutMinutes(): number {
|
||||
const row = this.db
|
||||
.prepare('SELECT value_json FROM app_settings WHERE key = ?')
|
||||
.get(IDLE_TIMEOUT_SETTING_KEY) as { readonly value_json: unknown } | undefined;
|
||||
const value = row?.value_json;
|
||||
return typeof value === 'number' ? this.normalizeIdleTimeout(value) : this.#idleTimeoutMinutes;
|
||||
}
|
||||
|
||||
private saveIdleTimeout(value: number): void {
|
||||
const minutes = this.normalizeIdleTimeout(value);
|
||||
const now = this.now().toISOString();
|
||||
this.db
|
||||
.prepare(
|
||||
`INSERT INTO app_settings (key,value_json,created_at,updated_at)
|
||||
VALUES (?,?,?,?)
|
||||
ON CONFLICT(key) DO UPDATE SET value_json=excluded.value_json,updated_at=excluded.updated_at`,
|
||||
)
|
||||
.run(IDLE_TIMEOUT_SETTING_KEY, Math.round(minutes), now, now);
|
||||
this.#idleTimeoutMinutes = minutes;
|
||||
}
|
||||
|
||||
private normalizeIdleTimeout(value: number): number {
|
||||
return Math.min(MAX_IDLE_TIMEOUT_MINUTES, Math.max(MIN_IDLE_TIMEOUT_MINUTES, value));
|
||||
}
|
||||
|
||||
private validatePassword(password: string): void {
|
||||
const bytes = Buffer.byteLength(password, 'utf8');
|
||||
if (
|
||||
@@ -182,13 +254,17 @@ export class ConsoleAuthService {
|
||||
throw new ConsoleAuthError('PASSWORD_POLICY_FAILED');
|
||||
}
|
||||
|
||||
private derive(password: string, salt: string): Promise<Buffer> {
|
||||
private derive(
|
||||
password: string,
|
||||
salt: string,
|
||||
params: ScryptParams = SCRYPT_CURRENT,
|
||||
): Promise<Buffer> {
|
||||
return new Promise((resolvePromise, reject) => {
|
||||
deriveScrypt(
|
||||
password,
|
||||
Buffer.from(salt, 'hex'),
|
||||
SCRYPT_KEY_LENGTH,
|
||||
{ N: 16_384, r: 8, p: 1, maxmem: 64 * 1024 * 1024 },
|
||||
{ ...params, maxmem: SCRYPT_MAXMEM },
|
||||
(error, derivedKey) => {
|
||||
if (error) reject(error);
|
||||
else resolvePromise(derivedKey);
|
||||
@@ -204,7 +280,12 @@ export class ConsoleAuthService {
|
||||
private createSession(passwordRevision: number): string {
|
||||
const token = randomBytes(32).toString('base64url');
|
||||
const now = this.now();
|
||||
const expires = new Date(now.getTime() + SESSION_TTL_MS);
|
||||
const expires = new Date(
|
||||
Math.min(
|
||||
now.getTime() + this.idleTimeoutMinutes() * 60 * 1000,
|
||||
now.getTime() + SESSION_TTL_MS,
|
||||
),
|
||||
);
|
||||
this.db
|
||||
.prepare(
|
||||
`INSERT INTO console_auth_sessions
|
||||
@@ -230,4 +311,29 @@ export class ConsoleAuthService {
|
||||
.prepare('DELETE FROM console_auth_sessions WHERE expires_at<=?')
|
||||
.run(this.now().toISOString());
|
||||
}
|
||||
|
||||
touchSession(token: string | undefined): void {
|
||||
if (!token) return;
|
||||
const config = this.config();
|
||||
if (!config || config.password_revision <= 0) return;
|
||||
const row = this.db
|
||||
.prepare('SELECT created_at FROM console_auth_sessions WHERE session_hash=? AND expires_at>?')
|
||||
.get(this.digest(token), this.now().toISOString()) as
|
||||
| { readonly created_at: string }
|
||||
| undefined;
|
||||
if (!row) return;
|
||||
const timeoutMs = this.idleTimeoutMinutes() * 60 * 1000;
|
||||
const now = this.now();
|
||||
this.db
|
||||
.prepare(
|
||||
'UPDATE console_auth_sessions SET expires_at=? WHERE session_hash=? AND expires_at>?',
|
||||
)
|
||||
.run(
|
||||
new Date(
|
||||
Math.min(now.getTime() + timeoutMs, new Date(row.created_at).getTime() + SESSION_TTL_MS),
|
||||
).toISOString(),
|
||||
this.digest(token),
|
||||
this.now().toISOString(),
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -0,0 +1,118 @@
|
||||
import { describe, expect, it } from 'vitest';
|
||||
|
||||
import {
|
||||
nextOccurrence,
|
||||
nextOccurrenceFor,
|
||||
previewCron,
|
||||
previewTrigger,
|
||||
reconcileOccurrence,
|
||||
} from './schedule-time.js';
|
||||
|
||||
describe('schedule time', () => {
|
||||
it('calculates future occurrences in Beijing time across the UTC day boundary', () => {
|
||||
expect(previewCron('0 9 * * *', 2, new Date('2026-07-30T00:30:00.000Z'))).toEqual([
|
||||
'2026-07-30T01:00:00.000Z',
|
||||
'2026-07-31T01:00:00.000Z',
|
||||
]);
|
||||
expect(nextOccurrence('30 0 * * *', new Date('2026-07-30T15:59:00.000Z'))).toBe(
|
||||
'2026-07-30T16:30:00.000Z',
|
||||
);
|
||||
});
|
||||
|
||||
it('rejects seconds fields and invalid cron grammar', () => {
|
||||
expect(() => previewCron('* * * * * *', 1)).toThrow(/five-field/i);
|
||||
expect(() => previewCron('99 99 * * *', 1)).toThrow(/cron/i);
|
||||
});
|
||||
|
||||
it('reconciles a missed due time according to the per-task policy', () => {
|
||||
const common = {
|
||||
cronExpression: '*/10 * * * *',
|
||||
nextDueAt: '2026-07-30T01:00:00.000Z',
|
||||
};
|
||||
expect(
|
||||
reconcileOccurrence(
|
||||
{ ...common, misfirePolicy: 'skip' },
|
||||
new Date('2026-07-30T01:25:00.000Z'),
|
||||
),
|
||||
).toEqual({
|
||||
action: 'skip',
|
||||
dueAt: '2026-07-30T01:20:00.000Z',
|
||||
nextDueAt: '2026-07-30T01:30:00.000Z',
|
||||
});
|
||||
expect(
|
||||
reconcileOccurrence(
|
||||
{ ...common, misfirePolicy: 'catch-up-once' },
|
||||
new Date('2026-07-30T01:25:00.000Z'),
|
||||
),
|
||||
).toEqual({
|
||||
action: 'run',
|
||||
dueAt: '2026-07-30T01:20:00.000Z',
|
||||
nextDueAt: '2026-07-30T01:30:00.000Z',
|
||||
});
|
||||
});
|
||||
|
||||
it('expands Hub fixed schedules over the Beijing weekday and time grid', () => {
|
||||
const weekdays = { kind: 'fixed' as const, weekdays: [4, 5], times: ['09:00', '21:30'] };
|
||||
// 2026-07-30 is a Thursday; 01:25Z is 09:25 in Beijing, so the same-day 21:30 slot is next.
|
||||
expect(previewTrigger(weekdays, 2, new Date('2026-07-30T01:25:00.000Z'))).toEqual([
|
||||
'2026-07-30T13:30:00.000Z',
|
||||
'2026-07-31T01:00:00.000Z',
|
||||
]);
|
||||
expect(
|
||||
nextOccurrenceFor(
|
||||
{ kind: 'fixed', weekdays: [6], times: ['00:30'] },
|
||||
new Date('2026-07-30T01:25:00.000Z'),
|
||||
),
|
||||
).toBe('2026-07-31T16:30:00.000Z');
|
||||
});
|
||||
|
||||
it('keeps interval schedules anchored on the previous occurrence', () => {
|
||||
const interval = { kind: 'interval' as const, value: 90, unit: 'mins' as const };
|
||||
expect(nextOccurrenceFor(interval, new Date('2026-07-30T01:00:00.000Z'))).toBe(
|
||||
'2026-07-30T02:30:00.000Z',
|
||||
);
|
||||
expect(
|
||||
nextOccurrenceFor(
|
||||
interval,
|
||||
new Date('2026-07-30T05:00:00.000Z'),
|
||||
new Date('2026-07-30T01:00:00.000Z'),
|
||||
),
|
||||
).toBe('2026-07-30T05:30:00.000Z');
|
||||
expect(previewTrigger(interval, 3, new Date('2026-07-30T01:00:00.000Z'))).toEqual([
|
||||
'2026-07-30T02:30:00.000Z',
|
||||
'2026-07-30T04:00:00.000Z',
|
||||
'2026-07-30T05:30:00.000Z',
|
||||
]);
|
||||
});
|
||||
|
||||
it('reconciles fixed and interval misfires against the stored due time', () => {
|
||||
expect(
|
||||
reconcileOccurrence(
|
||||
{
|
||||
trigger: { kind: 'interval', value: 90, unit: 'mins' },
|
||||
nextDueAt: '2026-07-30T04:00:00.000Z',
|
||||
misfirePolicy: 'catch-up-once',
|
||||
},
|
||||
new Date('2026-07-30T05:00:00.000Z'),
|
||||
),
|
||||
).toEqual({
|
||||
action: 'run',
|
||||
dueAt: '2026-07-30T04:00:00.000Z',
|
||||
nextDueAt: '2026-07-30T05:30:00.000Z',
|
||||
});
|
||||
expect(
|
||||
reconcileOccurrence(
|
||||
{
|
||||
trigger: { kind: 'fixed', weekdays: [4, 5], times: ['09:00', '21:30'] },
|
||||
nextDueAt: '2026-07-30T01:00:00.000Z',
|
||||
misfirePolicy: 'skip',
|
||||
},
|
||||
new Date('2026-07-30T01:25:00.000Z'),
|
||||
),
|
||||
).toEqual({
|
||||
action: 'skip',
|
||||
dueAt: '2026-07-30T01:00:00.000Z',
|
||||
nextDueAt: '2026-07-30T13:30:00.000Z',
|
||||
});
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,171 @@
|
||||
import type {
|
||||
ScheduleMisfirePolicy,
|
||||
ScheduleTrigger,
|
||||
ScheduledTask,
|
||||
} from '@multi-simadmin/contracts';
|
||||
import { intervalMilliseconds } from '@multi-simadmin/contracts';
|
||||
import { CronExpressionParser } from 'cron-parser';
|
||||
|
||||
const TIMEZONE = 'Asia/Shanghai';
|
||||
/** Asia/Shanghai has observed UTC+8 without DST since 1991, so fixed-offset arithmetic is exact. */
|
||||
const ZONE_OFFSET_MS = 8 * 60 * 60 * 1_000;
|
||||
|
||||
function validateExpression(expression: string): string {
|
||||
const clean = expression.trim().replace(/\s+/g, ' ');
|
||||
if (clean.split(' ').length !== 5)
|
||||
throw new TypeError('Cron must use standard five-field syntax');
|
||||
return clean;
|
||||
}
|
||||
|
||||
function parser(expression: string, currentDate: Date) {
|
||||
try {
|
||||
return CronExpressionParser.parse(validateExpression(expression), {
|
||||
currentDate,
|
||||
tz: TIMEZONE,
|
||||
});
|
||||
} catch (error) {
|
||||
if (error instanceof TypeError && /five-field/.test(error.message)) throw error;
|
||||
throw new TypeError('Cron expression is invalid');
|
||||
}
|
||||
}
|
||||
|
||||
export function previewCron(
|
||||
expression: string,
|
||||
count: number,
|
||||
from = new Date(),
|
||||
): readonly string[] {
|
||||
if (!Number.isSafeInteger(count) || count < 1 || count > 10)
|
||||
throw new TypeError('Cron preview count must be between 1 and 10');
|
||||
const interval = parser(expression, from);
|
||||
return Array.from({ length: count }, () => interval.next().toDate().toISOString());
|
||||
}
|
||||
|
||||
function shanghaiParts(instant: Date) {
|
||||
const shifted = new Date(instant.getTime() + ZONE_OFFSET_MS);
|
||||
const weekday = shifted.getUTCDay();
|
||||
return {
|
||||
year: shifted.getUTCFullYear(),
|
||||
month: shifted.getUTCMonth(),
|
||||
date: shifted.getUTCDate(),
|
||||
hour: shifted.getUTCHours(),
|
||||
minute: shifted.getUTCMinutes(),
|
||||
// ISO weekday: Monday is 1 and Sunday is 7, matching the Hub weekday vocabulary.
|
||||
weekday: weekday === 0 ? 7 : weekday,
|
||||
};
|
||||
}
|
||||
|
||||
function shanghaiInstant(year: number, month: number, date: number, hour: number, minute: number) {
|
||||
return new Date(Date.UTC(year, month, date, hour, minute) - ZONE_OFFSET_MS);
|
||||
}
|
||||
|
||||
/**
|
||||
* Minutes since local midnight in Asia/Shanghai. Shared with the notification engine so quiet
|
||||
* hours and scheduled windows read from one clock.
|
||||
*/
|
||||
export function shanghaiMinuteOfDay(instant: Date): number {
|
||||
const parts = shanghaiParts(instant);
|
||||
return parts.hour * 60 + parts.minute;
|
||||
}
|
||||
|
||||
function fixedOccurrenceAfter(trigger: Extract<ScheduleTrigger, { kind: 'fixed' }>, after: Date) {
|
||||
const start = shanghaiParts(after);
|
||||
for (let offset = 0; offset <= 8; offset += 1) {
|
||||
const day = shanghaiInstant(start.year, start.month, start.date + offset, 0, 0);
|
||||
const parts = shanghaiParts(day);
|
||||
if (!trigger.weekdays.includes(parts.weekday)) continue;
|
||||
for (const time of trigger.times) {
|
||||
const [hour, minute] = time.split(':');
|
||||
const candidate = shanghaiInstant(
|
||||
parts.year,
|
||||
parts.month,
|
||||
parts.date,
|
||||
Number(hour),
|
||||
Number(minute),
|
||||
);
|
||||
if (candidate.getTime() > after.getTime()) return candidate;
|
||||
}
|
||||
}
|
||||
return undefined;
|
||||
}
|
||||
|
||||
function intervalOccurrenceAfter(
|
||||
trigger: Extract<ScheduleTrigger, { kind: 'interval' }>,
|
||||
after: Date,
|
||||
anchor?: Date,
|
||||
) {
|
||||
const period = intervalMilliseconds(trigger);
|
||||
if (anchor && anchor.getTime() < after.getTime()) {
|
||||
// Keep the period anchored on the previous occurrence instead of drifting with the clock.
|
||||
const steps = Math.floor((after.getTime() - anchor.getTime()) / period) + 1;
|
||||
return new Date(anchor.getTime() + steps * period);
|
||||
}
|
||||
return new Date(after.getTime() + period);
|
||||
}
|
||||
|
||||
export function nextOccurrenceFor(trigger: ScheduleTrigger, after: Date, anchor?: Date): string {
|
||||
if (trigger.kind === 'cron') return nextOccurrence(trigger.expression, after);
|
||||
if (trigger.kind === 'fixed') {
|
||||
const next = fixedOccurrenceAfter(trigger, after);
|
||||
if (!next) throw new TypeError('Fixed schedule has no occurrence within eight days');
|
||||
return next.toISOString();
|
||||
}
|
||||
return intervalOccurrenceAfter(trigger, after, anchor).toISOString();
|
||||
}
|
||||
|
||||
export function previewTrigger(
|
||||
trigger: ScheduleTrigger,
|
||||
count: number,
|
||||
from = new Date(),
|
||||
): readonly string[] {
|
||||
if (!Number.isSafeInteger(count) || count < 1 || count > 10)
|
||||
throw new TypeError('Schedule preview count must be between 1 and 10');
|
||||
const occurrences: string[] = [];
|
||||
let cursor = from;
|
||||
for (let index = 0; index < count; index += 1) {
|
||||
const next = nextOccurrenceFor(trigger, cursor, from);
|
||||
occurrences.push(next);
|
||||
cursor = new Date(next);
|
||||
}
|
||||
return occurrences;
|
||||
}
|
||||
|
||||
export function nextOccurrence(expression: string, after: Date): string {
|
||||
return parser(expression, after).next().toDate().toISOString();
|
||||
}
|
||||
|
||||
export type ReconciledOccurrence =
|
||||
| { readonly action: 'wait'; readonly dueAt: string; readonly nextDueAt: string }
|
||||
| { readonly action: 'run' | 'skip'; readonly dueAt: string; readonly nextDueAt: string };
|
||||
|
||||
export function reconcileOccurrence(
|
||||
task: {
|
||||
readonly trigger?: ScheduledTask['trigger'];
|
||||
readonly cronExpression?: string;
|
||||
readonly nextDueAt: string;
|
||||
readonly misfirePolicy: ScheduleMisfirePolicy;
|
||||
},
|
||||
now: Date,
|
||||
): ReconciledOccurrence {
|
||||
const nextDue = new Date(task.nextDueAt);
|
||||
if (!Number.isFinite(nextDue.getTime())) throw new TypeError('nextDueAt is invalid');
|
||||
if (nextDue.getTime() > now.getTime())
|
||||
return { action: 'wait', dueAt: task.nextDueAt, nextDueAt: task.nextDueAt };
|
||||
|
||||
const trigger = task.trigger ?? {
|
||||
kind: 'cron' as const,
|
||||
expression: task.cronExpression ?? task.nextDueAt,
|
||||
};
|
||||
let dueAt: string;
|
||||
if (trigger.kind === 'cron') {
|
||||
dueAt = parser(trigger.expression, now).prev().toDate().toISOString();
|
||||
} else {
|
||||
// For fixed and interval triggers the stored due time is the authoritative occurrence.
|
||||
dueAt = task.nextDueAt;
|
||||
}
|
||||
const nextDueAt = nextOccurrenceFor(trigger, now, nextDue);
|
||||
return {
|
||||
action: task.misfirePolicy === 'catch-up-once' ? 'run' : 'skip',
|
||||
dueAt,
|
||||
nextDueAt,
|
||||
};
|
||||
}
|
||||
@@ -0,0 +1,252 @@
|
||||
import Database from 'better-sqlite3';
|
||||
import { describe, expect, it, vi } from 'vitest';
|
||||
|
||||
import { migrateDatabase } from '../../infrastructure/database/migrations.js';
|
||||
import { ScheduledTaskRepository } from './scheduled-task-repository.js';
|
||||
import { ScheduledOperationDispatcher } from './scheduled-operation-dispatcher.js';
|
||||
|
||||
describe('ScheduledOperationDispatcher', () => {
|
||||
it('renders SMS time and random macros separately for each target', async () => {
|
||||
const db = new Database(':memory:');
|
||||
migrateDatabase(db);
|
||||
const send = vi.fn(async (_instanceId: string, _input: { content: string }) => ({
|
||||
sent: true as const,
|
||||
}));
|
||||
const dispatcher = new ScheduledOperationDispatcher({
|
||||
db,
|
||||
operations: { prepare: vi.fn(), execute: vi.fn() },
|
||||
messages: { send },
|
||||
store: {
|
||||
provider: 'macos-keychain',
|
||||
set: vi.fn(),
|
||||
get: vi.fn(async () =>
|
||||
JSON.stringify({
|
||||
recipients: ['13800138000'],
|
||||
content: 'Status ${time} id ${random}',
|
||||
}),
|
||||
),
|
||||
delete: vi.fn(),
|
||||
},
|
||||
repository: new ScheduledTaskRepository(db),
|
||||
now: () => new Date('2026-09-06T04:05:06.000Z'),
|
||||
});
|
||||
for (const [id, name, address] of [
|
||||
['a', 'Alpha', 'http://10.0.0.1'],
|
||||
['b', 'Beta', 'http://10.0.0.2'],
|
||||
] as const) {
|
||||
db.prepare(
|
||||
'INSERT INTO instances (id,name,base_url,enabled,config_revision,created_at,updated_at) VALUES (?,?,?,?,?,?,?)',
|
||||
).run(id, name, address, 1, 1, '2026-07-30T00:00:00.000Z', '2026-07-30T00:00:00.000Z');
|
||||
}
|
||||
const task = new ScheduledTaskRepository(db).create({
|
||||
id: 'task-template',
|
||||
task: {
|
||||
name: 'Status',
|
||||
operationType: 'send-sms',
|
||||
cronExpression: '0 9 * * *',
|
||||
timezone: 'Asia/Shanghai',
|
||||
targetSelector: { mode: 'fixed', instanceIds: ['a', 'b'] },
|
||||
sms: { recipients: ['13800138000'], content: 'Status ${time} id ${random}' },
|
||||
misfirePolicy: 'skip',
|
||||
overlapPolicy: 'skip',
|
||||
retryPolicy: { maxRetries: 0, intervalSeconds: 60 },
|
||||
enabled: true,
|
||||
},
|
||||
smsSecretReference: 'memory://sms',
|
||||
createdBy: 'operator',
|
||||
now: '2026-09-06T04:05:06.000Z',
|
||||
});
|
||||
|
||||
const result = await dispatcher.dispatch(
|
||||
task,
|
||||
[
|
||||
{ id: 'a', revision: 1 },
|
||||
{ id: 'b', revision: 1 },
|
||||
],
|
||||
{ actor: 'operator', requestId: 'request-1' },
|
||||
);
|
||||
|
||||
expect(result.outcome).toBe('succeeded');
|
||||
expect(send).toHaveBeenCalledTimes(2);
|
||||
for (const call of send.mock.calls) {
|
||||
expect(call[1]?.content).toMatch(/^Status 2026-09-06 12:05:06 id [A-Za-z0-9]{12}$/u);
|
||||
}
|
||||
db.close();
|
||||
});
|
||||
|
||||
it('executes restart targets independently and aggregates partial success', async () => {
|
||||
const db = new Database(':memory:');
|
||||
migrateDatabase(db);
|
||||
const execute = vi
|
||||
.fn()
|
||||
.mockResolvedValueOnce({ id: 'job-a', status: 'succeeded' })
|
||||
.mockResolvedValueOnce({ id: 'job-b', status: 'failed' });
|
||||
const dispatcher = new ScheduledOperationDispatcher({
|
||||
db,
|
||||
operations: {
|
||||
prepare: vi.fn(async (input) => ({
|
||||
id: `prep-${input.targets[0]?.instanceId}`,
|
||||
confirmationToken: 'token',
|
||||
})),
|
||||
execute,
|
||||
},
|
||||
messages: { send: vi.fn() },
|
||||
store: { provider: 'macos-keychain', set: vi.fn(), get: vi.fn(), delete: vi.fn() },
|
||||
repository: new ScheduledTaskRepository(db),
|
||||
});
|
||||
const result = await dispatcher.dispatch(
|
||||
{
|
||||
id: 'task-1',
|
||||
name: 'Restart',
|
||||
operationType: 'restart-service',
|
||||
cronExpression: '0 9 * * *',
|
||||
timezone: 'Asia/Shanghai',
|
||||
targetSelector: { mode: 'fixed', instanceIds: ['a', 'b'] },
|
||||
misfirePolicy: 'skip',
|
||||
overlapPolicy: 'skip',
|
||||
retryPolicy: { maxRetries: 0, intervalSeconds: 60 },
|
||||
enabled: true,
|
||||
version: 1,
|
||||
createdBy: 'operator',
|
||||
updatedBy: 'operator',
|
||||
createdAt: '2026-07-30T00:00:00.000Z',
|
||||
updatedAt: '2026-07-30T00:00:00.000Z',
|
||||
},
|
||||
[
|
||||
{ id: 'a', revision: 1 },
|
||||
{ id: 'b', revision: 2 },
|
||||
],
|
||||
{ actor: 'operator', requestId: 'request-1' },
|
||||
);
|
||||
expect(result).toEqual({ outcome: 'partially-succeeded', jobIds: ['job-a', 'job-b'] });
|
||||
expect(execute).toHaveBeenCalledTimes(2);
|
||||
db.close();
|
||||
});
|
||||
|
||||
it('retries only failed SMS recipients using the task retry interval', async () => {
|
||||
const db = new Database(':memory:');
|
||||
migrateDatabase(db);
|
||||
db.prepare(
|
||||
'INSERT INTO instances (id,name,base_url,enabled,config_revision,created_at,updated_at) VALUES (?,?,?,?,?,?,?)',
|
||||
).run(
|
||||
'a',
|
||||
'Alpha',
|
||||
'http://10.0.0.1',
|
||||
1,
|
||||
1,
|
||||
'2026-07-30T00:00:00.000Z',
|
||||
'2026-07-30T00:00:00.000Z',
|
||||
);
|
||||
db.prepare(
|
||||
'INSERT INTO instances (id,name,base_url,enabled,config_revision,created_at,updated_at) VALUES (?,?,?,?,?,?,?)',
|
||||
).run(
|
||||
'b',
|
||||
'Beta',
|
||||
'http://10.0.0.2',
|
||||
1,
|
||||
1,
|
||||
'2026-07-30T00:00:00.000Z',
|
||||
'2026-07-30T00:00:00.000Z',
|
||||
);
|
||||
const repository = new ScheduledTaskRepository(db);
|
||||
const task = repository.create({
|
||||
id: 'task-sms',
|
||||
task: {
|
||||
name: 'Notify lab',
|
||||
operationType: 'send-sms',
|
||||
cronExpression: '0 9 * * *',
|
||||
timezone: 'Asia/Shanghai',
|
||||
targetSelector: { mode: 'fixed', instanceIds: ['a'] },
|
||||
sms: { recipients: ['13800138000', '13900139000'], content: 'Maintenance' },
|
||||
misfirePolicy: 'skip',
|
||||
overlapPolicy: 'skip',
|
||||
retryPolicy: { maxRetries: 1, intervalSeconds: 30 },
|
||||
enabled: true,
|
||||
},
|
||||
smsSecretReference: 'memory://sms',
|
||||
createdBy: 'operator',
|
||||
now: '2026-07-30T00:00:00.000Z',
|
||||
});
|
||||
const attempts = new Map<string, number>();
|
||||
const send = vi.fn(async (_instanceId: string, input: { phoneNumber: string }) => {
|
||||
const count = (attempts.get(input.phoneNumber) ?? 0) + 1;
|
||||
attempts.set(input.phoneNumber, count);
|
||||
if (input.phoneNumber === '13800138000' && count === 1) throw new Error('temporary');
|
||||
return { sent: true as const };
|
||||
});
|
||||
const sleep = vi.fn(async () => undefined);
|
||||
const dispatcher = new ScheduledOperationDispatcher({
|
||||
db,
|
||||
operations: { prepare: vi.fn(), execute: vi.fn() },
|
||||
messages: { send },
|
||||
store: {
|
||||
provider: 'macos-keychain',
|
||||
set: vi.fn(),
|
||||
get: vi.fn(async () =>
|
||||
JSON.stringify({ recipients: ['13800138000', '13900139000'], content: 'Maintenance' }),
|
||||
),
|
||||
delete: vi.fn(),
|
||||
},
|
||||
repository,
|
||||
sleep,
|
||||
});
|
||||
|
||||
const result = await dispatcher.dispatch(task, [{ id: 'a', revision: 1 }], {
|
||||
actor: 'operator',
|
||||
requestId: 'request-1',
|
||||
});
|
||||
|
||||
expect(result.outcome).toBe('succeeded');
|
||||
expect(send).toHaveBeenCalledTimes(3);
|
||||
expect(sleep).toHaveBeenCalledTimes(1);
|
||||
expect(sleep).toHaveBeenCalledWith(30_000);
|
||||
db.close();
|
||||
});
|
||||
|
||||
it('dispatches a baseband task against the baseband restart operation', async () => {
|
||||
const db = new Database(':memory:');
|
||||
migrateDatabase(db);
|
||||
const prepare = vi.fn(async (input: { operationId: string; parameters: unknown }) => ({
|
||||
id: 'prep-a',
|
||||
confirmationToken: 'token',
|
||||
...input,
|
||||
}));
|
||||
const dispatcher = new ScheduledOperationDispatcher({
|
||||
db,
|
||||
operations: {
|
||||
prepare,
|
||||
execute: vi.fn().mockResolvedValue({ id: 'job-a', status: 'succeeded' }),
|
||||
},
|
||||
messages: { send: vi.fn() },
|
||||
store: { provider: 'macos-keychain', set: vi.fn(), get: vi.fn(), delete: vi.fn() },
|
||||
repository: new ScheduledTaskRepository(db),
|
||||
});
|
||||
const result = await dispatcher.dispatch(
|
||||
{
|
||||
id: 'task-baseband',
|
||||
name: 'Baseband',
|
||||
operationType: 'restart-baseband',
|
||||
cronExpression: '0 4 * * *',
|
||||
timezone: 'Asia/Shanghai',
|
||||
targetSelector: { mode: 'fixed', instanceIds: ['a'] },
|
||||
misfirePolicy: 'skip',
|
||||
overlapPolicy: 'skip',
|
||||
retryPolicy: { maxRetries: 0, intervalSeconds: 60 },
|
||||
enabled: true,
|
||||
version: 1,
|
||||
createdBy: 'operator',
|
||||
updatedBy: 'operator',
|
||||
createdAt: '2026-07-30T00:00:00.000Z',
|
||||
updatedAt: '2026-07-30T00:00:00.000Z',
|
||||
},
|
||||
[{ id: 'a', revision: 1 }],
|
||||
{ actor: 'operator', requestId: 'request-1' },
|
||||
);
|
||||
expect(result.outcome).toBe('succeeded');
|
||||
expect(prepare.mock.calls[0]?.[0]).toMatchObject({
|
||||
operationId: 'postBasebandRestart',
|
||||
parameters: { parameterSchemaId: 'simadmin.58e2204.postBasebandRestart.parameters.v1' },
|
||||
});
|
||||
db.close();
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,331 @@
|
||||
import { createHash, randomInt, randomUUID } from 'node:crypto';
|
||||
import type { Job, PrepareOperationRequest, ScheduledTask } from '@multi-simadmin/contracts';
|
||||
|
||||
import type { InstanceMessageService } from '../messages/instance-message-service.js';
|
||||
import type { SecureOperationExecution } from '../operations/secure-operation-execution.js';
|
||||
import type { SqliteDatabase } from '../../infrastructure/database/database.js';
|
||||
import type { SecretStore } from '../../infrastructure/secrets/secret-store.js';
|
||||
import type { DispatchResult } from './scheduler-coordinator.js';
|
||||
import { ScheduledTaskRepository } from './scheduled-task-repository.js';
|
||||
import type { ResolvedTarget } from './target-resolver.js';
|
||||
|
||||
interface OperationExecutor {
|
||||
prepare(
|
||||
input: PrepareOperationRequest,
|
||||
requestId?: string,
|
||||
): Promise<{ id: string; confirmationToken: string }>;
|
||||
execute(
|
||||
input: { preparationId: string; confirmationToken: string },
|
||||
actor: string,
|
||||
requestId: string,
|
||||
): Promise<Pick<Job, 'id' | 'status'>>;
|
||||
}
|
||||
|
||||
interface MessageExecutor {
|
||||
send(
|
||||
instanceId: string,
|
||||
input: { phoneNumber: string; content: string },
|
||||
): Promise<{ readonly sent: true }>;
|
||||
}
|
||||
|
||||
interface BackupExecutor {
|
||||
createBackup(): Promise<{ readonly filename: string; readonly sizeBytes: number }>;
|
||||
}
|
||||
|
||||
export interface ScheduledOperationDispatcherOptions {
|
||||
readonly db: SqliteDatabase;
|
||||
readonly operations: Pick<SecureOperationExecution, 'prepare' | 'execute'> | OperationExecutor;
|
||||
readonly messages: Pick<InstanceMessageService, 'send'> | MessageExecutor;
|
||||
readonly maintenance?: BackupExecutor;
|
||||
readonly store: SecretStore;
|
||||
readonly repository: ScheduledTaskRepository;
|
||||
readonly now?: () => Date;
|
||||
readonly id?: () => string;
|
||||
readonly sleep?: (milliseconds: number) => Promise<void>;
|
||||
}
|
||||
|
||||
const operationSchemas: Record<string, string> = {
|
||||
'restart-service': 'simadmin.58e2204.postServiceRestart.parameters.v1',
|
||||
'reboot-system': 'simadmin.58e2204.postSystemReboot.parameters.v1',
|
||||
'restart-baseband': 'simadmin.58e2204.postBasebandRestart.parameters.v1',
|
||||
};
|
||||
|
||||
const operationNames: Record<string, string> = {
|
||||
'restart-service': 'postServiceRestart',
|
||||
'reboot-system': 'postSystemReboot',
|
||||
'restart-baseband': 'postBasebandRestart',
|
||||
};
|
||||
|
||||
const SMS_RANDOM_LENGTH = 12;
|
||||
const SMS_RANDOM_ALPHABET = 'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789';
|
||||
|
||||
/**
|
||||
* Hub-compatible SMS macros are filled in just before delivery. Rendering per target keeps
|
||||
* `${time}` meaningful when a large batch spans seconds and gives every message its own nonce.
|
||||
*/
|
||||
export function renderSmsTemplate(
|
||||
content: string,
|
||||
now: Date,
|
||||
random: () => string = () =>
|
||||
Array.from({ length: SMS_RANDOM_LENGTH }, () =>
|
||||
SMS_RANDOM_ALPHABET.charAt(randomInt(SMS_RANDOM_ALPHABET.length)),
|
||||
).join(''),
|
||||
): string {
|
||||
const formatter = new Intl.DateTimeFormat('en-CA', {
|
||||
timeZone: 'Asia/Shanghai',
|
||||
year: 'numeric',
|
||||
month: '2-digit',
|
||||
day: '2-digit',
|
||||
hour: '2-digit',
|
||||
minute: '2-digit',
|
||||
second: '2-digit',
|
||||
hour12: false,
|
||||
});
|
||||
const timestamp = formatter.format(now).replace(', ', ' ');
|
||||
return content.replaceAll('${time}', timestamp).replaceAll('${random}', random());
|
||||
}
|
||||
|
||||
function aggregate(successes: number, failures: number): DispatchResult['outcome'] {
|
||||
if (successes > 0 && failures > 0) return 'partially-succeeded';
|
||||
return failures > 0 ? 'failed' : 'succeeded';
|
||||
}
|
||||
|
||||
export class ScheduledOperationDispatcher {
|
||||
private readonly now: () => Date;
|
||||
private readonly id: () => string;
|
||||
private readonly sleep: (milliseconds: number) => Promise<void>;
|
||||
|
||||
constructor(private readonly options: ScheduledOperationDispatcherOptions) {
|
||||
this.now = options.now ?? (() => new Date());
|
||||
this.id = options.id ?? randomUUID;
|
||||
this.sleep =
|
||||
options.sleep ??
|
||||
((milliseconds) => new Promise((resolve) => setTimeout(resolve, milliseconds)));
|
||||
}
|
||||
|
||||
async dispatch(
|
||||
task: ScheduledTask,
|
||||
targets: readonly ResolvedTarget[],
|
||||
context: { readonly actor: string; readonly requestId: string },
|
||||
): Promise<DispatchResult> {
|
||||
if (task.operationType === 'send-sms') return this.dispatchSms(task, targets, context);
|
||||
if (task.operationType === 'backup-data') return this.dispatchBackup(task);
|
||||
const operationId = operationNames[task.operationType];
|
||||
const schema = operationSchemas[task.operationType];
|
||||
if (!operationId || !schema)
|
||||
return { outcome: 'needs-attention', reason: 'Unsupported scheduled operation', jobIds: [] };
|
||||
// The execution baseline pins delay_seconds to 3; anything else could never
|
||||
// succeed, so refuse up front instead of exhausting the retry policy.
|
||||
if (task.operationType === 'reboot-system' && (task.delaySeconds ?? 3) !== 3)
|
||||
return {
|
||||
outcome: 'needs-attention',
|
||||
reason: 'Scheduled reboots only support the pinned 3-second delay',
|
||||
jobIds: [],
|
||||
};
|
||||
const fields =
|
||||
task.operationType === 'reboot-system'
|
||||
? [
|
||||
{
|
||||
fieldId: 'delay_seconds',
|
||||
kind: 'number' as const,
|
||||
value: task.delaySeconds ?? 3,
|
||||
},
|
||||
]
|
||||
: [];
|
||||
const jobIds: string[] = [];
|
||||
let successes = 0;
|
||||
let failures = 0;
|
||||
for (const target of targets) {
|
||||
let succeeded = false;
|
||||
for (let attempt = 0; attempt <= task.retryPolicy.maxRetries; attempt += 1) {
|
||||
try {
|
||||
const preparation = await this.options.operations.prepare(
|
||||
{
|
||||
operationId,
|
||||
targets: [{ instanceId: target.id, revision: target.revision }],
|
||||
parameters: { parameterSchemaId: schema, fields },
|
||||
},
|
||||
context.requestId,
|
||||
);
|
||||
const job = await this.options.operations.execute(
|
||||
{ preparationId: preparation.id, confirmationToken: preparation.confirmationToken },
|
||||
context.actor,
|
||||
context.requestId,
|
||||
);
|
||||
jobIds.push(job.id);
|
||||
if (job.status === 'succeeded') {
|
||||
succeeded = true;
|
||||
break;
|
||||
}
|
||||
} catch {
|
||||
// Continue only within the task's explicit bounded retry policy.
|
||||
}
|
||||
if (!succeeded && attempt < task.retryPolicy.maxRetries)
|
||||
await this.sleep(task.retryPolicy.intervalSeconds * 1_000);
|
||||
}
|
||||
if (succeeded) successes += 1;
|
||||
else failures += 1;
|
||||
}
|
||||
return { outcome: aggregate(successes, failures), jobIds };
|
||||
}
|
||||
|
||||
private async dispatchSms(
|
||||
task: ScheduledTask,
|
||||
targets: readonly ResolvedTarget[],
|
||||
context: { readonly actor: string; readonly requestId: string },
|
||||
): Promise<DispatchResult> {
|
||||
const reference = this.options.repository.getSmsSecretReference(task.id);
|
||||
if (!reference)
|
||||
return { outcome: 'needs-attention', reason: 'SMS secret is missing', jobIds: [] };
|
||||
const secret = await this.options.store.get(reference);
|
||||
if (!secret) return { outcome: 'needs-attention', reason: 'SMS secret is missing', jobIds: [] };
|
||||
let payload: { recipients: string[]; content: string; randomDelaySeconds?: number };
|
||||
try {
|
||||
payload = JSON.parse(secret) as typeof payload;
|
||||
if (
|
||||
!Array.isArray(payload.recipients) ||
|
||||
!payload.recipients.length ||
|
||||
typeof payload.content !== 'string' ||
|
||||
!payload.content
|
||||
)
|
||||
throw new Error('invalid');
|
||||
} catch {
|
||||
return { outcome: 'needs-attention', reason: 'SMS secret is invalid', jobIds: [] };
|
||||
}
|
||||
const jobIds: string[] = [];
|
||||
let successes = 0;
|
||||
let failures = 0;
|
||||
for (const target of targets) {
|
||||
const result = await this.recordSmsJob(
|
||||
target.id,
|
||||
{ ...payload, content: renderSmsTemplate(payload.content, this.now()) },
|
||||
task.retryPolicy,
|
||||
context,
|
||||
);
|
||||
jobIds.push(result.id);
|
||||
if (result.succeeded) successes += 1;
|
||||
else failures += 1;
|
||||
}
|
||||
return { outcome: aggregate(successes, failures), jobIds };
|
||||
}
|
||||
|
||||
private async recordSmsJob(
|
||||
instanceId: string,
|
||||
payload: {
|
||||
readonly recipients: readonly string[];
|
||||
readonly content: string;
|
||||
readonly randomDelaySeconds?: number;
|
||||
},
|
||||
retryPolicy: ScheduledTask['retryPolicy'],
|
||||
context: { readonly actor: string; readonly requestId: string },
|
||||
): Promise<{ readonly id: string; readonly succeeded: boolean }> {
|
||||
const ids = { job: this.id(), item: this.id(), attempt: this.id(), audit: this.id() };
|
||||
const startedAt = this.now().toISOString();
|
||||
const digest = createHash('sha256').update(JSON.stringify(payload)).digest('hex');
|
||||
this.options.db.transaction(() => {
|
||||
this.options.db
|
||||
.prepare(
|
||||
`INSERT INTO jobs (id,root_job_id,operation_id,risk_level,status,requested_by,request_id,parameters_digest,created_at,started_at,updated_at)
|
||||
VALUES (?,?,'sendSms','R2','running',?,?,?,?,?,?)`,
|
||||
)
|
||||
.run(
|
||||
ids.job,
|
||||
ids.job,
|
||||
context.actor,
|
||||
context.requestId,
|
||||
digest,
|
||||
startedAt,
|
||||
startedAt,
|
||||
startedAt,
|
||||
);
|
||||
this.options.db
|
||||
.prepare(
|
||||
"INSERT INTO job_attempts (id,job_id,status,started_at,created_at) VALUES (?,?,'running',?,?)",
|
||||
)
|
||||
.run(ids.attempt, ids.job, startedAt, startedAt);
|
||||
this.options.db
|
||||
.prepare(
|
||||
"INSERT INTO job_items (id,job_id,instance_id,attempt_number,status,created_at,started_at,updated_at) VALUES (?,?,?,1,'running',?,?,?)",
|
||||
)
|
||||
.run(ids.item, ids.job, instanceId, startedAt, startedAt, startedAt);
|
||||
})();
|
||||
let succeeded = true;
|
||||
for (const recipient of payload.recipients) {
|
||||
let delivered = false;
|
||||
// Hub parity: per-recipient jitter so bulk schedules do not fire in one burst.
|
||||
const jitter = Math.max(0, Math.floor(payload.randomDelaySeconds ?? 0));
|
||||
if (jitter > 0) await this.sleep(Math.floor(Math.random() * (jitter + 1)) * 1_000);
|
||||
for (let attempt = 0; attempt <= retryPolicy.maxRetries; attempt += 1) {
|
||||
try {
|
||||
await this.options.messages.send(instanceId, {
|
||||
phoneNumber: recipient,
|
||||
content: payload.content,
|
||||
});
|
||||
delivered = true;
|
||||
break;
|
||||
} catch {
|
||||
delivered = false;
|
||||
}
|
||||
if (!delivered && attempt < retryPolicy.maxRetries)
|
||||
await this.sleep(retryPolicy.intervalSeconds * 1_000);
|
||||
}
|
||||
if (!delivered) succeeded = false;
|
||||
}
|
||||
const finishedAt = this.now().toISOString();
|
||||
const status = succeeded ? 'succeeded' : 'failed';
|
||||
this.options.db.transaction(() => {
|
||||
this.options.db
|
||||
.prepare('UPDATE jobs SET status=?,finished_at=?,updated_at=? WHERE id=?')
|
||||
.run(status, finishedAt, finishedAt, ids.job);
|
||||
this.options.db
|
||||
.prepare(
|
||||
'UPDATE job_items SET status=?,result_code=?,finished_at=?,updated_at=? WHERE id=?',
|
||||
)
|
||||
.run(status, succeeded ? 'SMS_SENT' : 'SMS_FAILED', finishedAt, finishedAt, ids.item);
|
||||
this.options.db
|
||||
.prepare('UPDATE job_attempts SET status=?,finished_at=? WHERE id=?')
|
||||
.run(status, finishedAt, ids.attempt);
|
||||
this.options.db
|
||||
.prepare(
|
||||
`INSERT INTO audit_events
|
||||
(id,instance_id,job_id,actor,operation_id,risk_level,request_id,parameters_summary_json,body_digest,result_code,duration_ms,created_at)
|
||||
VALUES (?,?,?,?,'sendSms','R2',?,?,?,?,0,?)`,
|
||||
)
|
||||
.run(
|
||||
ids.audit,
|
||||
instanceId,
|
||||
ids.job,
|
||||
context.actor,
|
||||
context.requestId,
|
||||
JSON.stringify([
|
||||
{ fieldId: 'recipients', displayValue: '[REDACTED]', redacted: true },
|
||||
{ fieldId: 'content', displayValue: '[REDACTED]', redacted: true },
|
||||
]),
|
||||
digest,
|
||||
succeeded ? 'SMS_SENT' : 'SMS_FAILED',
|
||||
finishedAt,
|
||||
);
|
||||
})();
|
||||
return { id: ids.job, succeeded };
|
||||
}
|
||||
|
||||
private async dispatchBackup(task: ScheduledTask): Promise<DispatchResult> {
|
||||
const maintenance = this.options.maintenance;
|
||||
if (!maintenance)
|
||||
return {
|
||||
outcome: 'needs-attention',
|
||||
reason: 'Backup maintenance service is unavailable',
|
||||
jobIds: [],
|
||||
};
|
||||
try {
|
||||
const backup = await maintenance.createBackup();
|
||||
return {
|
||||
outcome: 'succeeded',
|
||||
reason: `Backup ${backup.filename} (${backup.sizeBytes} bytes) created for schedule ${task.name}`,
|
||||
jobIds: [],
|
||||
};
|
||||
} catch {
|
||||
return { outcome: 'failed', reason: 'Control-plane backup failed', jobIds: [] };
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,173 @@
|
||||
import Database from 'better-sqlite3';
|
||||
import { describe, expect, it } from 'vitest';
|
||||
|
||||
import { migrateDatabase } from '../../infrastructure/database/migrations.js';
|
||||
import { ScheduledTaskRepository } from './scheduled-task-repository.js';
|
||||
|
||||
function fixture() {
|
||||
const db = new Database(':memory:');
|
||||
db.pragma('foreign_keys = ON');
|
||||
migrateDatabase(db);
|
||||
return { db, repository: new ScheduledTaskRepository(db) };
|
||||
}
|
||||
|
||||
const task = {
|
||||
name: 'Morning restart',
|
||||
operationType: 'restart-service' as const,
|
||||
cronExpression: '0 9 * * *',
|
||||
timezone: 'Asia/Shanghai' as const,
|
||||
targetSelector: { mode: 'fixed' as const, instanceIds: ['instance-a', 'instance-b'] },
|
||||
misfirePolicy: 'skip' as const,
|
||||
overlapPolicy: 'skip' as const,
|
||||
retryPolicy: { maxRetries: 0, intervalSeconds: 60 },
|
||||
enabled: true,
|
||||
};
|
||||
|
||||
describe('ScheduledTaskRepository', () => {
|
||||
it('round-trips a versioned task without exposing secret payload data', () => {
|
||||
const { db, repository } = fixture();
|
||||
const created = repository.create({
|
||||
id: 'task-1',
|
||||
task,
|
||||
createdBy: 'operator',
|
||||
now: '2026-07-30T00:00:00.000Z',
|
||||
nextDueAt: '2026-07-30T01:00:00.000Z',
|
||||
});
|
||||
|
||||
expect(created).toMatchObject({
|
||||
id: 'task-1',
|
||||
version: 1,
|
||||
nextDueAt: '2026-07-30T01:00:00.000Z',
|
||||
});
|
||||
expect(repository.get('task-1')).toEqual(created);
|
||||
expect(JSON.stringify(created)).not.toMatch(/secretReference|content|recipient/i);
|
||||
db.close();
|
||||
});
|
||||
|
||||
it('claims the same scheduled occurrence once and preserves its immutable snapshot', () => {
|
||||
const { db, repository } = fixture();
|
||||
repository.create({
|
||||
id: 'task-1',
|
||||
task,
|
||||
createdBy: 'operator',
|
||||
now: '2026-07-30T00:00:00.000Z',
|
||||
});
|
||||
const dueAt = '2026-07-30T01:00:00.000Z';
|
||||
const first = repository.claimOccurrence({
|
||||
id: 'run-1',
|
||||
scheduledTaskId: 'task-1',
|
||||
scheduleVersion: 1,
|
||||
dueAt,
|
||||
claimedAt: '2026-07-30T01:00:01.000Z',
|
||||
triggerSource: 'scheduled',
|
||||
targetSnapshot: ['instance-a', 'instance-b'],
|
||||
taskSnapshot: task,
|
||||
});
|
||||
const duplicate = repository.claimOccurrence({
|
||||
id: 'run-2',
|
||||
scheduledTaskId: 'task-1',
|
||||
scheduleVersion: 1,
|
||||
dueAt,
|
||||
claimedAt: '2026-07-30T01:00:02.000Z',
|
||||
triggerSource: 'scheduled',
|
||||
targetSnapshot: ['changed'],
|
||||
taskSnapshot: task,
|
||||
});
|
||||
|
||||
expect(first).toMatchObject({ id: 'run-1', targetSnapshot: ['instance-a', 'instance-b'] });
|
||||
expect(duplicate).toBeNull();
|
||||
expect(repository.getRun('run-1')?.targetSnapshot).toEqual(['instance-a', 'instance-b']);
|
||||
db.close();
|
||||
});
|
||||
|
||||
it('rolls back due-time advancement when scheduled claim preparation fails', () => {
|
||||
const { db, repository } = fixture();
|
||||
repository.create({
|
||||
id: 'task-1',
|
||||
task,
|
||||
createdBy: 'operator',
|
||||
now: '2026-07-30T00:00:00.000Z',
|
||||
nextDueAt: '2026-07-30T01:00:00.000Z',
|
||||
});
|
||||
|
||||
expect(() =>
|
||||
repository.claimScheduledOccurrence({
|
||||
id: 'run-1',
|
||||
scheduledTaskId: 'task-1',
|
||||
scheduleVersion: 1,
|
||||
dueAt: '2026-07-30T01:00:00.000Z',
|
||||
nextDueAt: '2026-07-31T01:00:00.000Z',
|
||||
claimedAt: '2026-07-30T01:00:00.000Z',
|
||||
taskSnapshot: task,
|
||||
overlapPolicy: 'skip',
|
||||
resolveTargets: () => {
|
||||
throw new Error('target resolution failed');
|
||||
},
|
||||
}),
|
||||
).toThrow('target resolution failed');
|
||||
expect(repository.get('task-1')?.nextDueAt).toBe('2026-07-30T01:00:00.000Z');
|
||||
expect(repository.listRuns()).toHaveLength(0);
|
||||
db.close();
|
||||
});
|
||||
|
||||
it('soft deletes task configuration while retaining completed run history', () => {
|
||||
const { db, repository } = fixture();
|
||||
repository.create({
|
||||
id: 'task-1',
|
||||
task,
|
||||
createdBy: 'operator',
|
||||
now: '2026-07-30T00:00:00.000Z',
|
||||
});
|
||||
repository.claimOccurrence({
|
||||
id: 'run-1',
|
||||
scheduledTaskId: 'task-1',
|
||||
scheduleVersion: 1,
|
||||
dueAt: '2026-07-30T01:00:00.000Z',
|
||||
claimedAt: '2026-07-30T01:00:01.000Z',
|
||||
triggerSource: 'scheduled',
|
||||
targetSnapshot: [],
|
||||
taskSnapshot: task,
|
||||
});
|
||||
repository.finishRun('run-1', {
|
||||
outcome: 'no-targets',
|
||||
reason: 'No enabled instances matched',
|
||||
jobIds: [],
|
||||
finishedAt: '2026-07-30T01:00:02.000Z',
|
||||
});
|
||||
repository.softDelete('task-1', 1, 'operator', '2026-07-30T02:00:00.000Z');
|
||||
|
||||
expect(repository.get('task-1')).toBeNull();
|
||||
expect(repository.getRun('run-1')).toMatchObject({ outcome: 'no-targets' });
|
||||
db.close();
|
||||
});
|
||||
|
||||
it('reconciles interrupted started runs to needs-attention', () => {
|
||||
const { db, repository } = fixture();
|
||||
repository.create({
|
||||
id: 'task-1',
|
||||
task,
|
||||
createdBy: 'operator',
|
||||
now: '2026-07-30T00:00:00.000Z',
|
||||
});
|
||||
repository.claimOccurrence({
|
||||
id: 'run-1',
|
||||
scheduledTaskId: 'task-1',
|
||||
scheduleVersion: 1,
|
||||
dueAt: '2026-07-30T01:00:00.000Z',
|
||||
claimedAt: '2026-07-30T01:00:00.000Z',
|
||||
triggerSource: 'scheduled',
|
||||
targetSnapshot: ['instance-a'],
|
||||
taskSnapshot: task,
|
||||
});
|
||||
repository.startRun('run-1', '2026-07-30T01:00:01.000Z');
|
||||
|
||||
expect(repository.reconcileInterruptedRuns('2026-07-30T02:00:00.000Z')).toBe(1);
|
||||
expect(repository.getRun('run-1')).toMatchObject({
|
||||
outcome: 'needs-attention',
|
||||
reason: 'Scheduler stopped before the run outcome was known',
|
||||
finishedAt: '2026-07-30T02:00:00.000Z',
|
||||
});
|
||||
expect(repository.hasActiveRun('task-1')).toBe(false);
|
||||
db.close();
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,502 @@
|
||||
import type {
|
||||
CreateScheduledTaskRequest,
|
||||
ScheduleTrigger,
|
||||
ScheduledRun,
|
||||
ScheduledRunOutcome,
|
||||
ScheduledRunTriggerSource,
|
||||
ScheduledTask,
|
||||
} from '@multi-simadmin/contracts';
|
||||
import { scheduleTriggerOf } from '@multi-simadmin/contracts';
|
||||
|
||||
import type { SqliteDatabase } from '../../infrastructure/database/database.js';
|
||||
|
||||
interface ScheduledTaskRow {
|
||||
id: string;
|
||||
name: string;
|
||||
operation_type: ScheduledTask['operationType'];
|
||||
enabled: number;
|
||||
version: number;
|
||||
cron_expression: string;
|
||||
timezone: ScheduledTask['timezone'];
|
||||
target_selector_json: string;
|
||||
sms_secret_reference: string | null;
|
||||
sms_recipient_count: number | null;
|
||||
effective_start_at: string | null;
|
||||
effective_end_at: string | null;
|
||||
misfire_policy: ScheduledTask['misfirePolicy'];
|
||||
overlap_policy: ScheduledTask['overlapPolicy'];
|
||||
retry_policy_json: string;
|
||||
trigger_json: string | null;
|
||||
delay_seconds: number | null;
|
||||
next_due_at: string | null;
|
||||
last_evaluated_at: string | null;
|
||||
created_by: string;
|
||||
updated_by: string;
|
||||
created_at: string;
|
||||
updated_at: string;
|
||||
}
|
||||
|
||||
interface ScheduledRunRow {
|
||||
id: string;
|
||||
scheduled_task_id: string;
|
||||
schedule_version: number;
|
||||
task_snapshot_json: string;
|
||||
due_at: string;
|
||||
claimed_at: string;
|
||||
started_at: string | null;
|
||||
finished_at: string | null;
|
||||
target_snapshot_json: string;
|
||||
outcome: ScheduledRunOutcome | null;
|
||||
reason: string | null;
|
||||
job_ids_json: string;
|
||||
trigger_source: ScheduledRunTriggerSource;
|
||||
attempt: number;
|
||||
}
|
||||
|
||||
export interface CreateTaskRecord {
|
||||
readonly id: string;
|
||||
readonly task: CreateScheduledTaskRequest;
|
||||
readonly smsSecretReference?: string;
|
||||
readonly createdBy: string;
|
||||
readonly now: string;
|
||||
readonly nextDueAt?: string;
|
||||
}
|
||||
|
||||
export interface UpdateTaskRecord {
|
||||
readonly id: string;
|
||||
readonly version: number;
|
||||
readonly task: CreateScheduledTaskRequest;
|
||||
readonly smsSecretReference?: string;
|
||||
readonly updatedBy: string;
|
||||
readonly now: string;
|
||||
readonly nextDueAt?: string;
|
||||
}
|
||||
|
||||
export interface ClaimOccurrenceRecord {
|
||||
readonly id: string;
|
||||
readonly scheduledTaskId: string;
|
||||
readonly scheduleVersion: number;
|
||||
readonly dueAt: string;
|
||||
readonly claimedAt: string;
|
||||
readonly triggerSource: ScheduledRunTriggerSource;
|
||||
readonly targetSnapshot: readonly string[];
|
||||
readonly taskSnapshot: CreateScheduledTaskRequest;
|
||||
readonly attempt?: number;
|
||||
}
|
||||
|
||||
export interface ScheduledClaimTarget {
|
||||
readonly id: string;
|
||||
readonly revision: number;
|
||||
}
|
||||
|
||||
export interface ClaimScheduledOccurrenceRecord {
|
||||
readonly id: string;
|
||||
readonly scheduledTaskId: string;
|
||||
readonly scheduleVersion: number;
|
||||
readonly dueAt: string;
|
||||
readonly nextDueAt: string;
|
||||
readonly claimedAt: string;
|
||||
readonly taskSnapshot: CreateScheduledTaskRequest;
|
||||
readonly overlapPolicy: ScheduledTask['overlapPolicy'];
|
||||
readonly resolveTargets: () => {
|
||||
readonly targets: readonly ScheduledClaimTarget[];
|
||||
readonly targetSnapshot?: readonly string[];
|
||||
readonly attentionReason?: string;
|
||||
};
|
||||
readonly terminal?: {
|
||||
readonly outcome: ScheduledRunOutcome;
|
||||
readonly reason: string;
|
||||
};
|
||||
}
|
||||
|
||||
export interface ScheduledClaimResult {
|
||||
readonly run: ScheduledRun;
|
||||
readonly targets: readonly ScheduledClaimTarget[];
|
||||
readonly disposition: 'execute' | 'queued' | 'finished';
|
||||
}
|
||||
|
||||
export interface FinishRunRecord {
|
||||
readonly outcome: ScheduledRunOutcome;
|
||||
readonly reason?: string;
|
||||
readonly jobIds: readonly string[];
|
||||
readonly finishedAt: string;
|
||||
}
|
||||
|
||||
function parseJson<T>(value: string, label: string): T {
|
||||
try {
|
||||
return JSON.parse(value) as T;
|
||||
} catch {
|
||||
throw new Error(`Stored ${label} is invalid`);
|
||||
}
|
||||
}
|
||||
|
||||
function projectTask(row: ScheduledTaskRow): ScheduledTask {
|
||||
const trigger: ScheduleTrigger = row.trigger_json
|
||||
? parseJson<ScheduleTrigger>(row.trigger_json, 'schedule trigger')
|
||||
: { kind: 'cron', expression: row.cron_expression };
|
||||
return {
|
||||
id: row.id,
|
||||
name: row.name,
|
||||
operationType: row.operation_type,
|
||||
trigger,
|
||||
cronExpression: row.cron_expression,
|
||||
timezone: row.timezone,
|
||||
targetSelector: parseJson(row.target_selector_json, 'target selector'),
|
||||
...(row.sms_secret_reference
|
||||
? { sms: { configured: true, recipientCount: row.sms_recipient_count ?? 0 } }
|
||||
: {}),
|
||||
...(row.delay_seconds === null || row.delay_seconds === undefined
|
||||
? {}
|
||||
: { delaySeconds: row.delay_seconds }),
|
||||
...(row.effective_start_at ? { effectiveStartAt: row.effective_start_at } : {}),
|
||||
...(row.effective_end_at ? { effectiveEndAt: row.effective_end_at } : {}),
|
||||
misfirePolicy: row.misfire_policy,
|
||||
overlapPolicy: row.overlap_policy,
|
||||
retryPolicy: parseJson(row.retry_policy_json, 'retry policy'),
|
||||
enabled: row.enabled === 1,
|
||||
version: row.version,
|
||||
...(row.next_due_at ? { nextDueAt: row.next_due_at } : {}),
|
||||
...(row.last_evaluated_at ? { lastEvaluatedAt: row.last_evaluated_at } : {}),
|
||||
createdBy: row.created_by,
|
||||
updatedBy: row.updated_by,
|
||||
createdAt: row.created_at,
|
||||
updatedAt: row.updated_at,
|
||||
};
|
||||
}
|
||||
|
||||
function projectRun(row: ScheduledRunRow): ScheduledRun {
|
||||
const task = parseJson<CreateScheduledTaskRequest>(row.task_snapshot_json, 'task snapshot');
|
||||
return {
|
||||
id: row.id,
|
||||
scheduledTaskId: row.scheduled_task_id,
|
||||
scheduleVersion: row.schedule_version,
|
||||
taskName: task.name,
|
||||
operationType: task.operationType,
|
||||
dueAt: row.due_at,
|
||||
claimedAt: row.claimed_at,
|
||||
...(row.started_at ? { startedAt: row.started_at } : {}),
|
||||
...(row.finished_at ? { finishedAt: row.finished_at } : {}),
|
||||
targetSnapshot: parseJson(row.target_snapshot_json, 'target snapshot'),
|
||||
...(row.outcome ? { outcome: row.outcome } : {}),
|
||||
...(row.reason ? { reason: row.reason } : {}),
|
||||
jobIds: parseJson(row.job_ids_json, 'job ids'),
|
||||
triggerSource: row.trigger_source,
|
||||
attempt: row.attempt,
|
||||
};
|
||||
}
|
||||
|
||||
export class ScheduledTaskRepository {
|
||||
constructor(private readonly db: SqliteDatabase) {}
|
||||
|
||||
create(input: CreateTaskRecord): ScheduledTask {
|
||||
const task = input.task;
|
||||
this.db
|
||||
.prepare(
|
||||
`INSERT INTO scheduled_tasks
|
||||
(id,name,operation_type,enabled,version,cron_expression,timezone,target_selector_json,
|
||||
sms_secret_reference,sms_recipient_count,effective_start_at,effective_end_at,misfire_policy,
|
||||
overlap_policy,retry_policy_json,trigger_json,delay_seconds,next_due_at,created_by,
|
||||
updated_by,created_at,updated_at)
|
||||
VALUES (?,?,?,?,1,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)`,
|
||||
)
|
||||
.run(
|
||||
input.id,
|
||||
task.name,
|
||||
task.operationType,
|
||||
task.enabled ? 1 : 0,
|
||||
task.cronExpression,
|
||||
task.timezone,
|
||||
JSON.stringify(task.targetSelector),
|
||||
input.smsSecretReference ?? null,
|
||||
task.sms?.recipients.length ?? null,
|
||||
task.effectiveStartAt ?? null,
|
||||
task.effectiveEndAt ?? null,
|
||||
task.misfirePolicy,
|
||||
task.overlapPolicy,
|
||||
JSON.stringify(task.retryPolicy),
|
||||
JSON.stringify(scheduleTriggerOf(task)),
|
||||
task.delaySeconds ?? null,
|
||||
input.nextDueAt ?? null,
|
||||
input.createdBy,
|
||||
input.createdBy,
|
||||
input.now,
|
||||
input.now,
|
||||
);
|
||||
const created = this.get(input.id);
|
||||
if (!created) throw new Error('Scheduled task was not persisted');
|
||||
return created;
|
||||
}
|
||||
|
||||
update(input: UpdateTaskRecord): ScheduledTask {
|
||||
const task = input.task;
|
||||
const result = this.db
|
||||
.prepare(
|
||||
`UPDATE scheduled_tasks
|
||||
SET name = ?, operation_type = ?, enabled = ?, version = version + 1,
|
||||
cron_expression = ?, timezone = ?, target_selector_json = ?,
|
||||
sms_secret_reference = ?, sms_recipient_count = ?, effective_start_at = ?,
|
||||
effective_end_at = ?, misfire_policy = ?, overlap_policy = ?, retry_policy_json = ?,
|
||||
trigger_json = ?, delay_seconds = ?,
|
||||
next_due_at = ?, last_evaluated_at = NULL, updated_by = ?, updated_at = ?
|
||||
WHERE id = ? AND version = ? AND deleted_at IS NULL`,
|
||||
)
|
||||
.run(
|
||||
task.name,
|
||||
task.operationType,
|
||||
task.enabled ? 1 : 0,
|
||||
task.cronExpression,
|
||||
task.timezone,
|
||||
JSON.stringify(task.targetSelector),
|
||||
input.smsSecretReference ?? null,
|
||||
task.sms?.recipients.length ?? null,
|
||||
task.effectiveStartAt ?? null,
|
||||
task.effectiveEndAt ?? null,
|
||||
task.misfirePolicy,
|
||||
task.overlapPolicy,
|
||||
JSON.stringify(task.retryPolicy),
|
||||
JSON.stringify(scheduleTriggerOf(task)),
|
||||
task.delaySeconds ?? null,
|
||||
input.nextDueAt ?? null,
|
||||
input.updatedBy,
|
||||
input.now,
|
||||
input.id,
|
||||
input.version,
|
||||
);
|
||||
if (result.changes !== 1) throw new Error('Scheduled task was not found or version changed');
|
||||
const updated = this.get(input.id);
|
||||
if (!updated) throw new Error('Scheduled task was not persisted');
|
||||
return updated;
|
||||
}
|
||||
|
||||
get(id: string): ScheduledTask | null {
|
||||
const row = this.db
|
||||
.prepare('SELECT * FROM scheduled_tasks WHERE id = ? AND deleted_at IS NULL')
|
||||
.get(id) as ScheduledTaskRow | undefined;
|
||||
return row ? projectTask(row) : null;
|
||||
}
|
||||
|
||||
list(): readonly ScheduledTask[] {
|
||||
return (
|
||||
this.db
|
||||
.prepare(
|
||||
'SELECT * FROM scheduled_tasks WHERE deleted_at IS NULL ORDER BY updated_at DESC, id ASC',
|
||||
)
|
||||
.all() as ScheduledTaskRow[]
|
||||
).map(projectTask);
|
||||
}
|
||||
|
||||
getSmsSecretReference(id: string): string | undefined {
|
||||
const row = this.db
|
||||
.prepare(
|
||||
'SELECT sms_secret_reference FROM scheduled_tasks WHERE id = ? AND deleted_at IS NULL',
|
||||
)
|
||||
.get(id) as { sms_secret_reference: string | null } | undefined;
|
||||
return row?.sms_secret_reference ?? undefined;
|
||||
}
|
||||
|
||||
setEnabled(
|
||||
id: string,
|
||||
version: number,
|
||||
enabled: boolean,
|
||||
actor: string,
|
||||
now: string,
|
||||
): ScheduledTask {
|
||||
const result = this.db
|
||||
.prepare(
|
||||
`UPDATE scheduled_tasks
|
||||
SET enabled = ?, version = version + 1, updated_by = ?, updated_at = ?
|
||||
WHERE id = ? AND version = ? AND deleted_at IS NULL`,
|
||||
)
|
||||
.run(enabled ? 1 : 0, actor, now, id, version);
|
||||
if (result.changes !== 1) throw new Error('Scheduled task was not found or version changed');
|
||||
const task = this.get(id);
|
||||
if (!task) throw new Error('Scheduled task was not persisted');
|
||||
return task;
|
||||
}
|
||||
|
||||
advanceNextDue(id: string, version: number, nextDueAt: string, evaluatedAt: string): boolean {
|
||||
return (
|
||||
this.db
|
||||
.prepare(
|
||||
`UPDATE scheduled_tasks SET next_due_at = ?, last_evaluated_at = ?
|
||||
WHERE id = ? AND version = ? AND enabled = 1 AND deleted_at IS NULL`,
|
||||
)
|
||||
.run(nextDueAt, evaluatedAt, id, version).changes === 1
|
||||
);
|
||||
}
|
||||
|
||||
startRun(id: string, startedAt: string): void {
|
||||
const result = this.db
|
||||
.prepare(
|
||||
'UPDATE scheduled_runs SET started_at = ? WHERE id = ? AND started_at IS NULL AND finished_at IS NULL',
|
||||
)
|
||||
.run(startedAt, id);
|
||||
if (result.changes !== 1) throw new Error('Scheduled run is missing or already started');
|
||||
}
|
||||
|
||||
hasActiveRun(scheduledTaskId: string): boolean {
|
||||
return !!this.db
|
||||
.prepare(
|
||||
'SELECT 1 FROM scheduled_runs WHERE scheduled_task_id = ? AND started_at IS NOT NULL AND finished_at IS NULL LIMIT 1',
|
||||
)
|
||||
.get(scheduledTaskId);
|
||||
}
|
||||
|
||||
reconcileInterruptedRuns(finishedAt: string): number {
|
||||
return this.db
|
||||
.prepare(
|
||||
`UPDATE scheduled_runs
|
||||
SET outcome = 'needs-attention',
|
||||
reason = 'Scheduler stopped before the run outcome was known',
|
||||
finished_at = ?
|
||||
WHERE started_at IS NOT NULL AND finished_at IS NULL`,
|
||||
)
|
||||
.run(finishedAt).changes;
|
||||
}
|
||||
|
||||
getQueuedRun(scheduledTaskId: string): ScheduledRun | null {
|
||||
const row = this.db
|
||||
.prepare(
|
||||
`SELECT * FROM scheduled_runs
|
||||
WHERE scheduled_task_id = ? AND started_at IS NULL AND finished_at IS NULL
|
||||
ORDER BY claimed_at ASC, id ASC LIMIT 1`,
|
||||
)
|
||||
.get(scheduledTaskId) as ScheduledRunRow | undefined;
|
||||
return row ? projectRun(row) : null;
|
||||
}
|
||||
|
||||
claimOccurrence(input: ClaimOccurrenceRecord): ScheduledRun | null {
|
||||
const result = this.db
|
||||
.prepare(
|
||||
`INSERT OR IGNORE INTO scheduled_runs
|
||||
(id,scheduled_task_id,schedule_version,task_snapshot_json,due_at,claimed_at,target_snapshot_json,
|
||||
job_ids_json,trigger_source,attempt)
|
||||
VALUES (?,?,?,?,?,?,?,?,?,?)`,
|
||||
)
|
||||
.run(
|
||||
input.id,
|
||||
input.scheduledTaskId,
|
||||
input.scheduleVersion,
|
||||
JSON.stringify(input.taskSnapshot),
|
||||
input.dueAt,
|
||||
input.claimedAt,
|
||||
JSON.stringify(input.targetSnapshot),
|
||||
'[]',
|
||||
input.triggerSource,
|
||||
input.attempt ?? 1,
|
||||
);
|
||||
return result.changes === 0 ? null : this.getRun(input.id);
|
||||
}
|
||||
|
||||
claimScheduledOccurrence(input: ClaimScheduledOccurrenceRecord): ScheduledClaimResult | null {
|
||||
return this.db.transaction((): ScheduledClaimResult | null => {
|
||||
const advanced = this.db
|
||||
.prepare(
|
||||
`UPDATE scheduled_tasks SET next_due_at = ?, last_evaluated_at = ?
|
||||
WHERE id = ? AND version = ? AND enabled = 1 AND deleted_at IS NULL AND next_due_at = ?`,
|
||||
)
|
||||
.run(
|
||||
input.nextDueAt,
|
||||
input.claimedAt,
|
||||
input.scheduledTaskId,
|
||||
input.scheduleVersion,
|
||||
input.dueAt,
|
||||
);
|
||||
if (advanced.changes !== 1) return null;
|
||||
|
||||
const resolution = input.terminal
|
||||
? { targets: [] as readonly ScheduledClaimTarget[] }
|
||||
: input.resolveTargets();
|
||||
const targets = [...resolution.targets];
|
||||
let disposition: ScheduledClaimResult['disposition'] = 'execute';
|
||||
let terminal =
|
||||
input.terminal ??
|
||||
(resolution.attentionReason
|
||||
? { outcome: 'needs-attention' as const, reason: resolution.attentionReason }
|
||||
: undefined);
|
||||
if (!terminal && this.hasActiveRun(input.scheduledTaskId)) {
|
||||
const queued = this.getQueuedRun(input.scheduledTaskId);
|
||||
if (input.overlapPolicy === 'queue-once' && !queued) {
|
||||
disposition = 'queued';
|
||||
} else {
|
||||
disposition = 'finished';
|
||||
terminal = {
|
||||
outcome: 'skipped',
|
||||
reason:
|
||||
input.overlapPolicy === 'queue-once'
|
||||
? 'Overlap queue is full'
|
||||
: 'Overlapping occurrence skipped by task policy',
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
const claimed = this.claimOccurrence({
|
||||
id: input.id,
|
||||
scheduledTaskId: input.scheduledTaskId,
|
||||
scheduleVersion: input.scheduleVersion,
|
||||
dueAt: input.dueAt,
|
||||
claimedAt: input.claimedAt,
|
||||
triggerSource: 'scheduled',
|
||||
targetSnapshot: resolution.targetSnapshot ?? targets.map((target) => target.id),
|
||||
taskSnapshot: input.taskSnapshot,
|
||||
});
|
||||
if (!claimed) return null;
|
||||
if (terminal) {
|
||||
return {
|
||||
run: this.finishRun(claimed.id, {
|
||||
outcome: terminal.outcome,
|
||||
reason: terminal.reason,
|
||||
jobIds: [],
|
||||
finishedAt: input.claimedAt,
|
||||
}),
|
||||
targets,
|
||||
disposition: 'finished',
|
||||
};
|
||||
}
|
||||
return { run: claimed, targets, disposition };
|
||||
})();
|
||||
}
|
||||
|
||||
getRun(id: string): ScheduledRun | null {
|
||||
const row = this.db.prepare('SELECT * FROM scheduled_runs WHERE id = ?').get(id) as
|
||||
| ScheduledRunRow
|
||||
| undefined;
|
||||
return row ? projectRun(row) : null;
|
||||
}
|
||||
|
||||
listRuns(scheduledTaskId?: string): readonly ScheduledRun[] {
|
||||
const rows = scheduledTaskId
|
||||
? (this.db
|
||||
.prepare(
|
||||
'SELECT * FROM scheduled_runs WHERE scheduled_task_id = ? ORDER BY due_at DESC, id ASC',
|
||||
)
|
||||
.all(scheduledTaskId) as ScheduledRunRow[])
|
||||
: (this.db
|
||||
.prepare('SELECT * FROM scheduled_runs ORDER BY due_at DESC, id ASC')
|
||||
.all() as ScheduledRunRow[]);
|
||||
return rows.map(projectRun);
|
||||
}
|
||||
|
||||
finishRun(id: string, input: FinishRunRecord): ScheduledRun {
|
||||
const result = this.db
|
||||
.prepare(
|
||||
`UPDATE scheduled_runs SET outcome = ?, reason = ?, job_ids_json = ?, finished_at = ?
|
||||
WHERE id = ? AND finished_at IS NULL`,
|
||||
)
|
||||
.run(input.outcome, input.reason ?? null, JSON.stringify(input.jobIds), input.finishedAt, id);
|
||||
if (result.changes !== 1) throw new Error('Scheduled run is missing or already finished');
|
||||
const run = this.getRun(id);
|
||||
if (!run) throw new Error('Scheduled run was not persisted');
|
||||
return run;
|
||||
}
|
||||
|
||||
softDelete(id: string, version: number, actor: string, now: string): void {
|
||||
const result = this.db
|
||||
.prepare(
|
||||
`UPDATE scheduled_tasks SET enabled = 0, version = version + 1, updated_by = ?, updated_at = ?, deleted_at = ?
|
||||
WHERE id = ? AND version = ? AND deleted_at IS NULL`,
|
||||
)
|
||||
.run(actor, now, now, id, version);
|
||||
if (result.changes !== 1) throw new Error('Scheduled task was not found or version changed');
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,236 @@
|
||||
import Database from 'better-sqlite3';
|
||||
import { describe, expect, it } from 'vitest';
|
||||
|
||||
import { migrateDatabase } from '../../infrastructure/database/migrations.js';
|
||||
import type { SecretKey, SecretStore } from '../../infrastructure/secrets/secret-store.js';
|
||||
import { ScheduledTaskRepository } from './scheduled-task-repository.js';
|
||||
import { ScheduledTaskService } from './scheduled-task-service.js';
|
||||
|
||||
class MemorySecrets implements SecretStore {
|
||||
readonly provider = 'macos-keychain';
|
||||
readonly values = new Map<string, string>();
|
||||
readonly setCalls: Array<{ key: SecretKey; value: string }> = [];
|
||||
blockRotations = false;
|
||||
private rotationWaiters: Array<() => void> = [];
|
||||
|
||||
async set(key: SecretKey, value: string): Promise<string> {
|
||||
const reference = `memory://${key.instanceId}/${key.purpose}/${key.slot ?? 'default'}`;
|
||||
this.setCalls.push({ key, value });
|
||||
this.values.set(reference, value);
|
||||
if (this.blockRotations && key.slot) {
|
||||
await new Promise<void>((resolve) => {
|
||||
this.rotationWaiters.push(resolve);
|
||||
if (this.rotationWaiters.length === 2) {
|
||||
for (const waiter of this.rotationWaiters.splice(0)) waiter();
|
||||
}
|
||||
});
|
||||
}
|
||||
return reference;
|
||||
}
|
||||
async get(reference: string): Promise<string | undefined> {
|
||||
return this.values.get(reference);
|
||||
}
|
||||
async delete(reference: string): Promise<boolean> {
|
||||
return this.values.delete(reference);
|
||||
}
|
||||
}
|
||||
|
||||
function fixture() {
|
||||
const db = new Database(':memory:');
|
||||
db.pragma('foreign_keys = ON');
|
||||
migrateDatabase(db);
|
||||
const store = new MemorySecrets();
|
||||
const repository = new ScheduledTaskRepository(db);
|
||||
let id = 0;
|
||||
const service = new ScheduledTaskService({
|
||||
repository,
|
||||
store,
|
||||
now: () => new Date('2026-07-30T00:00:00.000Z'),
|
||||
id: () => `task-${++id}`,
|
||||
});
|
||||
return { db, repository, service, store };
|
||||
}
|
||||
|
||||
const base = {
|
||||
name: 'Morning SMS',
|
||||
operationType: 'send-sms',
|
||||
cronExpression: '0 9 * * *',
|
||||
targetSelector: { mode: 'tags', match: 'all', tags: ['lab'] },
|
||||
sms: { recipients: ['13800138000', '13900139000'], content: 'Maintenance complete' },
|
||||
};
|
||||
|
||||
describe('ScheduledTaskService', () => {
|
||||
it('stores SMS payload only in the secret store and returns a redacted summary', async () => {
|
||||
const { db, service, store } = fixture();
|
||||
const created = await service.create('operator', base);
|
||||
|
||||
expect(created.sms).toEqual({ configured: true, recipientCount: 2 });
|
||||
expect(JSON.stringify(created)).not.toContain('13800138000');
|
||||
expect(JSON.stringify(created)).not.toContain('Maintenance complete');
|
||||
expect(store.setCalls[0]?.value).toBe(
|
||||
JSON.stringify({
|
||||
recipients: ['13800138000', '13900139000'],
|
||||
content: 'Maintenance complete',
|
||||
}),
|
||||
);
|
||||
const persisted = db.prepare('SELECT * FROM scheduled_tasks WHERE id = ?').get('task-1');
|
||||
expect(JSON.stringify(persisted)).not.toContain('13800138000');
|
||||
expect(JSON.stringify(persisted)).not.toContain('Maintenance complete');
|
||||
db.close();
|
||||
});
|
||||
|
||||
it('increments the optimistic version when pausing and rejects a stale update', async () => {
|
||||
const { db, service } = fixture();
|
||||
const created = await service.create('operator', {
|
||||
...base,
|
||||
operationType: 'restart-service',
|
||||
sms: undefined,
|
||||
});
|
||||
expect(service.setEnabled('operator', created.id, created.version, false).version).toBe(2);
|
||||
expect(() => service.setEnabled('operator', created.id, created.version, true)).toThrow(
|
||||
/version/i,
|
||||
);
|
||||
db.close();
|
||||
});
|
||||
|
||||
it('does not schedule the first occurrence before the effective start', async () => {
|
||||
const { db, service } = fixture();
|
||||
const created = await service.create('operator', {
|
||||
...base,
|
||||
operationType: 'restart-service',
|
||||
sms: undefined,
|
||||
effectiveStartAt: '2026-08-05T00:00:00.000Z',
|
||||
});
|
||||
|
||||
expect(created.nextDueAt).toBe('2026-08-05T01:00:00.000Z');
|
||||
db.close();
|
||||
});
|
||||
|
||||
it('keeps immutable run history after deleting its schedule', async () => {
|
||||
const { db, repository, service } = fixture();
|
||||
const created = await service.create('operator', {
|
||||
...base,
|
||||
operationType: 'restart-service',
|
||||
sms: undefined,
|
||||
});
|
||||
repository.claimOccurrence({
|
||||
id: 'run-1',
|
||||
scheduledTaskId: created.id,
|
||||
scheduleVersion: created.version,
|
||||
dueAt: '2026-07-30T01:00:00.000Z',
|
||||
claimedAt: '2026-07-30T01:00:00.000Z',
|
||||
triggerSource: 'manual',
|
||||
targetSnapshot: [],
|
||||
taskSnapshot: {
|
||||
name: base.name,
|
||||
cronExpression: base.cronExpression,
|
||||
targetSelector: { mode: 'tags' as const, match: 'all' as const, tags: ['lab'] },
|
||||
operationType: 'restart-service',
|
||||
timezone: 'Asia/Shanghai',
|
||||
misfirePolicy: 'skip',
|
||||
overlapPolicy: 'skip',
|
||||
retryPolicy: { maxRetries: 0, intervalSeconds: 60 },
|
||||
enabled: true,
|
||||
},
|
||||
});
|
||||
await service.remove('operator', created.id, created.version);
|
||||
|
||||
expect(service.get(created.id)).toBeNull();
|
||||
expect(repository.getRun('run-1')).not.toBeNull();
|
||||
db.close();
|
||||
});
|
||||
|
||||
it('updates the complete task, preserves an unchanged SMS secret, and advances the version', async () => {
|
||||
const { db, service, store } = fixture();
|
||||
const created = await service.create('operator', base);
|
||||
const reference = store.setCalls[0]?.key;
|
||||
|
||||
const updated = await service.update('editor', created.id, created.version, {
|
||||
name: 'Evening SMS',
|
||||
cronExpression: '30 18 * * *',
|
||||
effectiveStartAt: '2026-08-01T00:00:00.000Z',
|
||||
effectiveEndAt: '2026-09-01T00:00:00.000Z',
|
||||
overlapPolicy: 'queue-once',
|
||||
});
|
||||
|
||||
expect(updated).toMatchObject({
|
||||
name: 'Evening SMS',
|
||||
cronExpression: '30 18 * * *',
|
||||
overlapPolicy: 'queue-once',
|
||||
version: 2,
|
||||
updatedBy: 'editor',
|
||||
sms: { configured: true, recipientCount: 2 },
|
||||
});
|
||||
expect(store.setCalls).toHaveLength(1);
|
||||
expect(store.setCalls[0]?.key).toEqual(reference);
|
||||
db.close();
|
||||
});
|
||||
|
||||
it('rotates edited SMS secrets and validates retry policy against the effective operation', async () => {
|
||||
const { db, service, store } = fixture();
|
||||
const created = await service.create('operator', base);
|
||||
const oldReference = [...store.values.keys()][0];
|
||||
|
||||
const updated = await service.update('editor', created.id, created.version, {
|
||||
sms: { recipients: ['13700137000'], content: 'Updated content' },
|
||||
});
|
||||
expect(updated.sms).toEqual({ configured: true, recipientCount: 1 });
|
||||
expect(store.setCalls).toHaveLength(2);
|
||||
expect(oldReference ? store.values.has(oldReference) : true).toBe(false);
|
||||
|
||||
const reboot = await service.create('operator', {
|
||||
...base,
|
||||
name: 'Reboot',
|
||||
operationType: 'reboot-system',
|
||||
sms: undefined,
|
||||
});
|
||||
await expect(
|
||||
service.update('editor', reboot.id, reboot.version, {
|
||||
retryPolicy: { maxRetries: 1, intervalSeconds: 60 },
|
||||
}),
|
||||
).rejects.toThrow(/reboot/i);
|
||||
db.close();
|
||||
});
|
||||
|
||||
it('does not delete the winning SMS secret when concurrent updates race', async () => {
|
||||
const { db, repository, service, store } = fixture();
|
||||
const created = await service.create('operator', base);
|
||||
store.blockRotations = true;
|
||||
|
||||
const results = await Promise.allSettled([
|
||||
service.update('editor-a', created.id, created.version, {
|
||||
sms: { recipients: ['13700137000'], content: 'Update A' },
|
||||
}),
|
||||
service.update('editor-b', created.id, created.version, {
|
||||
sms: { recipients: ['13600136000'], content: 'Update B' },
|
||||
}),
|
||||
]);
|
||||
|
||||
expect(results.filter((result) => result.status === 'fulfilled')).toHaveLength(1);
|
||||
expect(results.filter((result) => result.status === 'rejected')).toHaveLength(1);
|
||||
const currentReference = repository.getSmsSecretReference(created.id);
|
||||
expect(currentReference).toBeDefined();
|
||||
expect(currentReference ? store.values.has(currentReference) : false).toBe(true);
|
||||
expect(store.values).toHaveLength(1);
|
||||
db.close();
|
||||
});
|
||||
|
||||
it('duplicates configuration and SMS secrets into a disabled independent task', async () => {
|
||||
const { db, service, store } = fixture();
|
||||
const created = await service.create('operator', base);
|
||||
|
||||
const duplicate = await service.duplicate('operator', created.id, created.version);
|
||||
|
||||
expect(duplicate).toMatchObject({
|
||||
id: 'task-2',
|
||||
name: 'Morning SMS copy',
|
||||
operationType: 'send-sms',
|
||||
enabled: false,
|
||||
version: 1,
|
||||
sms: { configured: true, recipientCount: 2 },
|
||||
});
|
||||
expect(store.setCalls).toHaveLength(2);
|
||||
expect(store.setCalls[1]?.value).toBe(store.setCalls[0]?.value);
|
||||
db.close();
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,241 @@
|
||||
import { randomUUID } from 'node:crypto';
|
||||
import {
|
||||
parseCreateScheduledTaskRequest,
|
||||
parseScheduleTrigger,
|
||||
parseUpdateScheduledTaskRequest,
|
||||
scheduleTriggerOf,
|
||||
type CreateScheduledTaskRequest,
|
||||
type ScheduleTrigger,
|
||||
type ScheduledTask,
|
||||
type ScheduledSmsInput,
|
||||
} from '@multi-simadmin/contracts';
|
||||
|
||||
import type { SecretStore } from '../../infrastructure/secrets/secret-store.js';
|
||||
import { nextOccurrenceFor, previewTrigger } from './schedule-time.js';
|
||||
import { ScheduledTaskRepository } from './scheduled-task-repository.js';
|
||||
|
||||
export interface ScheduledTaskServiceOptions {
|
||||
readonly repository: ScheduledTaskRepository;
|
||||
readonly store: SecretStore;
|
||||
readonly now?: () => Date;
|
||||
readonly id?: () => string;
|
||||
readonly secretSlot?: () => string;
|
||||
}
|
||||
|
||||
export class ScheduledTaskService {
|
||||
private readonly repository: ScheduledTaskRepository;
|
||||
private readonly store: SecretStore;
|
||||
private readonly now: () => Date;
|
||||
private readonly id: () => string;
|
||||
private readonly secretSlot: () => string;
|
||||
|
||||
constructor(options: ScheduledTaskServiceOptions) {
|
||||
this.repository = options.repository;
|
||||
this.store = options.store;
|
||||
this.now = options.now ?? (() => new Date());
|
||||
this.id = options.id ?? randomUUID;
|
||||
this.secretSlot = options.secretSlot ?? randomUUID;
|
||||
}
|
||||
|
||||
async create(actor: string, value: unknown): Promise<ScheduledTask> {
|
||||
const task = parseCreateScheduledTaskRequest(value);
|
||||
const now = this.now();
|
||||
previewTrigger(scheduleTriggerOf(task), 1, now);
|
||||
const id = this.id();
|
||||
let smsSecretReference: string | undefined;
|
||||
if (task.sms) {
|
||||
smsSecretReference = await this.store.set(
|
||||
{ instanceId: id, purpose: 'scheduled-sms' },
|
||||
JSON.stringify(task.sms),
|
||||
);
|
||||
}
|
||||
try {
|
||||
return this.repository.create({
|
||||
id,
|
||||
task,
|
||||
...(smsSecretReference ? { smsSecretReference } : {}),
|
||||
createdBy: actor,
|
||||
now: now.toISOString(),
|
||||
nextDueAt: this.nextDueAt(task, now),
|
||||
});
|
||||
} catch (error) {
|
||||
if (smsSecretReference) await this.store.delete(smsSecretReference).catch(() => false);
|
||||
throw error;
|
||||
}
|
||||
}
|
||||
|
||||
async update(actor: string, id: string, version: number, value: unknown): Promise<ScheduledTask> {
|
||||
const source = this.updateObject(value);
|
||||
const change = parseUpdateScheduledTaskRequest({ ...source, version });
|
||||
const current = this.repository.get(id);
|
||||
if (!current || current.version !== version)
|
||||
throw new Error('Scheduled task was not found or version changed');
|
||||
|
||||
const oldReference = this.repository.getSmsSecretReference(id);
|
||||
const operationType = change.operationType ?? current.operationType;
|
||||
let smsPayload: ScheduledSmsInput | undefined;
|
||||
if (operationType === 'send-sms') {
|
||||
if (change.sms === null) throw new TypeError('SMS configuration is required');
|
||||
if (change.sms) smsPayload = change.sms;
|
||||
else smsPayload = await this.readSms(oldReference);
|
||||
} else if (change.sms) {
|
||||
throw new TypeError('SMS is only valid for send-sms');
|
||||
}
|
||||
|
||||
const merged = parseCreateScheduledTaskRequest({
|
||||
name: change.name ?? current.name,
|
||||
operationType,
|
||||
trigger:
|
||||
(change.trigger as ScheduleTrigger | undefined) ??
|
||||
(change.cronExpression
|
||||
? ({ kind: 'cron', expression: change.cronExpression } as ScheduleTrigger)
|
||||
: current.trigger),
|
||||
cronExpression: change.cronExpression ?? current.cronExpression,
|
||||
timezone: change.timezone ?? current.timezone,
|
||||
targetSelector: change.targetSelector ?? current.targetSelector,
|
||||
...(smsPayload ? { sms: smsPayload } : {}),
|
||||
...(change.delaySeconds === null
|
||||
? {}
|
||||
: change.delaySeconds !== undefined
|
||||
? { delaySeconds: change.delaySeconds }
|
||||
: current.delaySeconds !== undefined
|
||||
? { delaySeconds: current.delaySeconds }
|
||||
: {}),
|
||||
...(change.effectiveStartAt === null
|
||||
? {}
|
||||
: change.effectiveStartAt
|
||||
? { effectiveStartAt: change.effectiveStartAt }
|
||||
: current.effectiveStartAt
|
||||
? { effectiveStartAt: current.effectiveStartAt }
|
||||
: {}),
|
||||
...(change.effectiveEndAt === null
|
||||
? {}
|
||||
: change.effectiveEndAt
|
||||
? { effectiveEndAt: change.effectiveEndAt }
|
||||
: current.effectiveEndAt
|
||||
? { effectiveEndAt: current.effectiveEndAt }
|
||||
: {}),
|
||||
misfirePolicy: change.misfirePolicy ?? current.misfirePolicy,
|
||||
overlapPolicy: change.overlapPolicy ?? current.overlapPolicy,
|
||||
retryPolicy: change.retryPolicy ?? current.retryPolicy,
|
||||
enabled: change.enabled ?? current.enabled,
|
||||
});
|
||||
const now = this.now();
|
||||
previewTrigger(scheduleTriggerOf(merged), 1, now);
|
||||
|
||||
let nextReference = operationType === 'send-sms' ? oldReference : undefined;
|
||||
let wroteReference: string | undefined;
|
||||
if (operationType === 'send-sms' && change.sms) {
|
||||
wroteReference = await this.store.set(
|
||||
{ instanceId: id, purpose: 'scheduled-sms', slot: this.secretSlot() },
|
||||
JSON.stringify(change.sms),
|
||||
);
|
||||
nextReference = wroteReference;
|
||||
}
|
||||
if (operationType === 'send-sms' && !nextReference)
|
||||
throw new Error('Scheduled SMS secret is missing');
|
||||
|
||||
try {
|
||||
const updated = this.repository.update({
|
||||
id,
|
||||
version,
|
||||
task: merged,
|
||||
...(nextReference ? { smsSecretReference: nextReference } : {}),
|
||||
updatedBy: actor,
|
||||
now: now.toISOString(),
|
||||
nextDueAt: this.nextDueAt(merged, now),
|
||||
});
|
||||
if (oldReference && oldReference !== nextReference)
|
||||
await this.store.delete(oldReference).catch(() => false);
|
||||
return updated;
|
||||
} catch (error) {
|
||||
if (wroteReference) await this.store.delete(wroteReference).catch(() => false);
|
||||
throw error;
|
||||
}
|
||||
}
|
||||
|
||||
async duplicate(actor: string, id: string, version: number): Promise<ScheduledTask> {
|
||||
const current = this.repository.get(id);
|
||||
if (!current || current.version !== version)
|
||||
throw new Error('Scheduled task was not found or version changed');
|
||||
const smsPayload =
|
||||
current.operationType === 'send-sms'
|
||||
? await this.readSms(this.repository.getSmsSecretReference(id))
|
||||
: undefined;
|
||||
const suffix = ' copy';
|
||||
return this.create(actor, {
|
||||
name: `${current.name.slice(0, 120 - suffix.length)}${suffix}`,
|
||||
operationType: current.operationType,
|
||||
trigger: current.trigger,
|
||||
cronExpression: current.cronExpression,
|
||||
timezone: current.timezone,
|
||||
targetSelector: current.targetSelector,
|
||||
...(smsPayload ? { sms: smsPayload } : {}),
|
||||
...(current.delaySeconds === undefined ? {} : { delaySeconds: current.delaySeconds }),
|
||||
...(current.effectiveStartAt ? { effectiveStartAt: current.effectiveStartAt } : {}),
|
||||
...(current.effectiveEndAt ? { effectiveEndAt: current.effectiveEndAt } : {}),
|
||||
misfirePolicy: current.misfirePolicy,
|
||||
overlapPolicy: current.overlapPolicy,
|
||||
retryPolicy: current.retryPolicy,
|
||||
enabled: false,
|
||||
});
|
||||
}
|
||||
|
||||
get(id: string): ScheduledTask | null {
|
||||
return this.repository.get(id);
|
||||
}
|
||||
|
||||
list(): readonly ScheduledTask[] {
|
||||
return this.repository.list();
|
||||
}
|
||||
|
||||
preview(value: unknown, count = 5, from = this.now()): readonly string[] {
|
||||
const trigger =
|
||||
typeof value === 'string'
|
||||
? ({ kind: 'cron', expression: value } as ScheduleTrigger)
|
||||
: parseScheduleTrigger(value);
|
||||
return previewTrigger(trigger, count, from);
|
||||
}
|
||||
|
||||
setEnabled(actor: string, id: string, version: number, enabled: boolean): ScheduledTask {
|
||||
return this.repository.setEnabled(id, version, enabled, actor, this.now().toISOString());
|
||||
}
|
||||
|
||||
async remove(actor: string, id: string, version: number): Promise<void> {
|
||||
const reference = this.repository.getSmsSecretReference(id);
|
||||
this.repository.softDelete(id, version, actor, this.now().toISOString());
|
||||
if (reference) await this.store.delete(reference).catch(() => false);
|
||||
}
|
||||
|
||||
private updateObject(value: unknown): Record<string, unknown> {
|
||||
if (typeof value !== 'object' || value === null || Array.isArray(value))
|
||||
throw new TypeError('Schedule update must be an object');
|
||||
return value as Record<string, unknown>;
|
||||
}
|
||||
|
||||
private async readSms(reference: string | undefined): Promise<ScheduledSmsInput> {
|
||||
if (!reference) throw new Error('Scheduled SMS secret is missing');
|
||||
const value = await this.store.get(reference);
|
||||
if (!value) throw new Error('Scheduled SMS secret is missing');
|
||||
try {
|
||||
const parsed = JSON.parse(value) as unknown;
|
||||
return parseCreateScheduledTaskRequest({
|
||||
name: 'SMS validation',
|
||||
operationType: 'send-sms',
|
||||
cronExpression: '0 0 * * *',
|
||||
timezone: 'Asia/Shanghai',
|
||||
targetSelector: { mode: 'fixed', instanceIds: ['secret-validation'] },
|
||||
sms: parsed,
|
||||
}).sms as ScheduledSmsInput;
|
||||
} catch {
|
||||
throw new Error('Scheduled SMS secret is invalid');
|
||||
}
|
||||
}
|
||||
|
||||
private nextDueAt(task: CreateScheduledTaskRequest, now: Date): string {
|
||||
const anchor = task.effectiveStartAt
|
||||
? new Date(Math.max(now.getTime(), Date.parse(task.effectiveStartAt)))
|
||||
: now;
|
||||
return nextOccurrenceFor(scheduleTriggerOf(task), anchor);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,321 @@
|
||||
import Database from 'better-sqlite3';
|
||||
import { describe, expect, it, vi } from 'vitest';
|
||||
|
||||
import { migrateDatabase } from '../../infrastructure/database/migrations.js';
|
||||
import { ScheduledTaskRepository } from './scheduled-task-repository.js';
|
||||
import { SchedulerCoordinator } from './scheduler-coordinator.js';
|
||||
import type { ScheduledTask } from '@multi-simadmin/contracts';
|
||||
import type { ResolvedTarget } from './target-resolver.js';
|
||||
|
||||
function fixture() {
|
||||
const db = new Database(':memory:');
|
||||
db.pragma('foreign_keys = ON');
|
||||
migrateDatabase(db);
|
||||
const now = '2026-07-30T01:00:00.000Z';
|
||||
db.prepare(
|
||||
'INSERT INTO instances (id,name,base_url,enabled,config_revision,created_at,updated_at) VALUES (?,?,?,?,?,?,?)',
|
||||
).run('a', 'Alpha', 'http://10.0.0.1', 1, 2, now, now);
|
||||
db.prepare(
|
||||
'INSERT INTO instances (id,name,base_url,enabled,config_revision,created_at,updated_at) VALUES (?,?,?,?,?,?,?)',
|
||||
).run('b', 'Beta', 'http://10.0.0.2', 1, 3, now, now);
|
||||
const capability = db.prepare(
|
||||
"INSERT INTO capabilities (instance_id,operation_id,state,observed_at,created_at,updated_at) VALUES (?,'postServiceRestart','supported',?,?,?)",
|
||||
);
|
||||
capability.run('a', now, now, now);
|
||||
capability.run('b', now, now, now);
|
||||
const repository = new ScheduledTaskRepository(db);
|
||||
repository.create({
|
||||
id: 'task-1',
|
||||
task: {
|
||||
name: 'Restart lab',
|
||||
operationType: 'restart-service',
|
||||
cronExpression: '0 9 * * *',
|
||||
timezone: 'Asia/Shanghai',
|
||||
targetSelector: { mode: 'fixed', instanceIds: ['a', 'b'] },
|
||||
misfirePolicy: 'skip',
|
||||
overlapPolicy: 'skip',
|
||||
retryPolicy: { maxRetries: 0, intervalSeconds: 60 },
|
||||
enabled: true,
|
||||
},
|
||||
createdBy: 'operator',
|
||||
now,
|
||||
nextDueAt: now,
|
||||
});
|
||||
return { db, repository };
|
||||
}
|
||||
|
||||
describe('SchedulerCoordinator', () => {
|
||||
it('resolves multiple targets, persists the snapshot, and finishes a manual run', async () => {
|
||||
const { db, repository } = fixture();
|
||||
const dispatch = vi.fn<
|
||||
(
|
||||
task: ScheduledTask,
|
||||
targets: readonly ResolvedTarget[],
|
||||
) => Promise<{ outcome: 'succeeded'; jobIds: string[] }>
|
||||
>(async () => ({
|
||||
outcome: 'succeeded',
|
||||
jobIds: ['job-a', 'job-b'],
|
||||
}));
|
||||
const coordinator = new SchedulerCoordinator({
|
||||
db,
|
||||
repository,
|
||||
dispatch,
|
||||
now: () => new Date('2026-07-30T01:00:00.000Z'),
|
||||
id: () => 'run-1',
|
||||
});
|
||||
|
||||
const run = await coordinator.runNow('task-1', 'operator', 'request-1');
|
||||
expect(run).toMatchObject({
|
||||
id: 'run-1',
|
||||
triggerSource: 'manual',
|
||||
targetSnapshot: ['a', 'b'],
|
||||
outcome: 'succeeded',
|
||||
jobIds: ['job-a', 'job-b'],
|
||||
});
|
||||
expect(dispatch.mock.calls[0]?.[1]).toEqual([
|
||||
{ id: 'a', revision: 2 },
|
||||
{ id: 'b', revision: 3 },
|
||||
]);
|
||||
db.close();
|
||||
});
|
||||
|
||||
it('does not manually dispatch when a selected target lacks the required capability', async () => {
|
||||
const { db, repository } = fixture();
|
||||
db.prepare(
|
||||
"UPDATE capabilities SET state = 'unsupported' WHERE instance_id = 'b' AND operation_id = 'postServiceRestart'",
|
||||
).run();
|
||||
const dispatch = vi.fn();
|
||||
const coordinator = new SchedulerCoordinator({
|
||||
db,
|
||||
repository,
|
||||
dispatch,
|
||||
now: () => new Date('2026-07-30T01:00:00.000Z'),
|
||||
id: () => 'manual-capability-run',
|
||||
});
|
||||
|
||||
const run = await coordinator.runNow('task-1', 'operator', 'request-1');
|
||||
expect(run).toMatchObject({ outcome: 'needs-attention', targetSnapshot: ['a', 'b'] });
|
||||
expect(dispatch).not.toHaveBeenCalled();
|
||||
db.close();
|
||||
});
|
||||
|
||||
it('records no-targets without dispatching and prevents duplicate scheduled claims', async () => {
|
||||
const { db, repository } = fixture();
|
||||
db.prepare('UPDATE instances SET enabled = 0').run();
|
||||
const dispatch = vi.fn();
|
||||
const coordinator = new SchedulerCoordinator({
|
||||
db,
|
||||
repository,
|
||||
dispatch,
|
||||
now: () => new Date('2026-07-30T01:00:00.000Z'),
|
||||
id: () => 'run-1',
|
||||
});
|
||||
const first = await coordinator.tick();
|
||||
const second = await coordinator.tick();
|
||||
expect(first).toHaveLength(1);
|
||||
expect(first[0]).toMatchObject({ outcome: 'no-targets' });
|
||||
expect(second).toHaveLength(0);
|
||||
expect(dispatch).not.toHaveBeenCalled();
|
||||
db.close();
|
||||
});
|
||||
|
||||
it('records needs-attention when a matched target loses its required capability', async () => {
|
||||
const { db, repository } = fixture();
|
||||
db.prepare(
|
||||
"UPDATE capabilities SET state = 'unsupported' WHERE instance_id = 'b' AND operation_id = 'postServiceRestart'",
|
||||
).run();
|
||||
const dispatch = vi.fn();
|
||||
const coordinator = new SchedulerCoordinator({
|
||||
db,
|
||||
repository,
|
||||
dispatch,
|
||||
now: () => new Date('2026-07-30T01:00:00.000Z'),
|
||||
id: () => 'capability-run',
|
||||
});
|
||||
|
||||
const runs = await coordinator.tick();
|
||||
expect(runs).toHaveLength(1);
|
||||
expect(runs[0]).toMatchObject({
|
||||
outcome: 'needs-attention',
|
||||
targetSnapshot: ['a', 'b'],
|
||||
});
|
||||
expect(runs[0]?.reason).toContain('b');
|
||||
expect(dispatch).not.toHaveBeenCalled();
|
||||
db.close();
|
||||
});
|
||||
|
||||
it('queues one overlapping occurrence, skips a second, and dispatches the queued run later', async () => {
|
||||
const { db, repository } = fixture();
|
||||
db.prepare("UPDATE scheduled_tasks SET overlap_policy = 'queue-once'").run();
|
||||
const task = repository.get('task-1')!;
|
||||
const taskSnapshot = {
|
||||
name: task.name,
|
||||
operationType: task.operationType,
|
||||
cronExpression: task.cronExpression,
|
||||
timezone: task.timezone,
|
||||
targetSelector: task.targetSelector,
|
||||
misfirePolicy: task.misfirePolicy,
|
||||
overlapPolicy: task.overlapPolicy,
|
||||
retryPolicy: task.retryPolicy,
|
||||
enabled: task.enabled,
|
||||
};
|
||||
repository.claimOccurrence({
|
||||
id: 'active-run',
|
||||
scheduledTaskId: task.id,
|
||||
scheduleVersion: task.version,
|
||||
dueAt: '2026-07-30T00:00:00.000Z',
|
||||
claimedAt: '2026-07-30T00:00:00.000Z',
|
||||
triggerSource: 'manual',
|
||||
targetSnapshot: ['a'],
|
||||
taskSnapshot,
|
||||
});
|
||||
repository.startRun('active-run', '2026-07-30T00:00:00.000Z');
|
||||
let now = new Date('2026-07-30T01:00:00.000Z');
|
||||
let id = 0;
|
||||
const dispatch = vi.fn(async () => ({ outcome: 'succeeded' as const, jobIds: ['job-1'] }));
|
||||
const coordinator = new SchedulerCoordinator({
|
||||
db,
|
||||
repository,
|
||||
dispatch,
|
||||
now: () => now,
|
||||
id: () => `queued-${++id}`,
|
||||
});
|
||||
|
||||
expect(await coordinator.tick()).toHaveLength(0);
|
||||
expect(repository.getQueuedRun('task-1')).toMatchObject({
|
||||
id: 'queued-1',
|
||||
targetSnapshot: ['a', 'b'],
|
||||
});
|
||||
|
||||
now = new Date('2026-07-31T01:00:00.000Z');
|
||||
const queueFull = await coordinator.tick();
|
||||
expect(queueFull).toHaveLength(1);
|
||||
expect(queueFull[0]).toMatchObject({ outcome: 'skipped', reason: 'Overlap queue is full' });
|
||||
expect(dispatch).not.toHaveBeenCalled();
|
||||
|
||||
repository.finishRun('active-run', {
|
||||
outcome: 'succeeded',
|
||||
jobIds: [],
|
||||
finishedAt: now.toISOString(),
|
||||
});
|
||||
const completed = await coordinator.tick();
|
||||
expect(completed[0]).toMatchObject({ id: 'queued-1', outcome: 'succeeded' });
|
||||
expect(dispatch).toHaveBeenCalledTimes(1);
|
||||
db.close();
|
||||
});
|
||||
|
||||
it('records and advances an occurrence outside the effective window', async () => {
|
||||
const { db, repository } = fixture();
|
||||
db.prepare("UPDATE scheduled_tasks SET effective_end_at = '2026-07-29T23:59:00.000Z'").run();
|
||||
const dispatch = vi.fn();
|
||||
const coordinator = new SchedulerCoordinator({
|
||||
db,
|
||||
repository,
|
||||
dispatch,
|
||||
now: () => new Date('2026-07-30T01:00:00.000Z'),
|
||||
id: () => 'window-skip',
|
||||
});
|
||||
|
||||
const runs = await coordinator.tick();
|
||||
expect(runs).toHaveLength(1);
|
||||
expect(runs[0]).toMatchObject({
|
||||
outcome: 'skipped',
|
||||
reason: 'Occurrence is outside the effective window',
|
||||
});
|
||||
expect(repository.get('task-1')?.nextDueAt).toBe('2026-07-31T01:00:00.000Z');
|
||||
expect(dispatch).not.toHaveBeenCalled();
|
||||
db.close();
|
||||
});
|
||||
|
||||
it('does not advance a due occurrence when its claim cannot be persisted', async () => {
|
||||
const { db, repository } = fixture();
|
||||
vi.spyOn(repository, 'claimOccurrence').mockImplementationOnce(() => {
|
||||
throw new Error('claim failed');
|
||||
});
|
||||
const coordinator = new SchedulerCoordinator({
|
||||
db,
|
||||
repository,
|
||||
dispatch: vi.fn(),
|
||||
now: () => new Date('2026-07-30T01:00:00.000Z'),
|
||||
id: () => 'failed-claim',
|
||||
});
|
||||
|
||||
await expect(coordinator.tick()).rejects.toThrow('claim failed');
|
||||
expect(repository.get('task-1')?.nextDueAt).toBe('2026-07-30T01:00:00.000Z');
|
||||
db.close();
|
||||
});
|
||||
|
||||
it('executes at most the queued occurrence for a task during one tick', async () => {
|
||||
const { db, repository } = fixture();
|
||||
const task = repository.get('task-1')!;
|
||||
repository.claimOccurrence({
|
||||
id: 'queued-run',
|
||||
scheduledTaskId: task.id,
|
||||
scheduleVersion: task.version,
|
||||
dueAt: '2026-07-29T01:00:00.000Z',
|
||||
claimedAt: '2026-07-29T01:00:00.000Z',
|
||||
triggerSource: 'scheduled',
|
||||
targetSnapshot: ['a'],
|
||||
taskSnapshot: {
|
||||
name: task.name,
|
||||
operationType: task.operationType,
|
||||
cronExpression: task.cronExpression,
|
||||
timezone: task.timezone,
|
||||
targetSelector: task.targetSelector,
|
||||
misfirePolicy: task.misfirePolicy,
|
||||
overlapPolicy: task.overlapPolicy,
|
||||
retryPolicy: task.retryPolicy,
|
||||
enabled: task.enabled,
|
||||
},
|
||||
});
|
||||
let id = 0;
|
||||
const dispatch = vi.fn(async () => ({ outcome: 'succeeded' as const, jobIds: ['job-1'] }));
|
||||
const coordinator = new SchedulerCoordinator({
|
||||
db,
|
||||
repository,
|
||||
dispatch,
|
||||
now: () => new Date('2026-07-30T01:00:00.000Z'),
|
||||
id: () => `run-${++id}`,
|
||||
});
|
||||
|
||||
const completed = await coordinator.tick();
|
||||
expect(completed).toHaveLength(1);
|
||||
expect(completed[0]?.id).toBe('queued-run');
|
||||
expect(dispatch).toHaveBeenCalledTimes(1);
|
||||
expect(repository.get('task-1')?.nextDueAt).toBe('2026-07-30T01:00:00.000Z');
|
||||
db.close();
|
||||
});
|
||||
|
||||
it('does not reschedule after stop while a tick is still running', async () => {
|
||||
vi.useFakeTimers();
|
||||
const { db, repository } = fixture();
|
||||
let finishDispatch!: (value: { outcome: 'succeeded'; jobIds: string[] }) => void;
|
||||
const dispatch = vi.fn(
|
||||
() =>
|
||||
new Promise<{ outcome: 'succeeded'; jobIds: string[] }>((resolve) => {
|
||||
finishDispatch = resolve;
|
||||
}),
|
||||
);
|
||||
const coordinator = new SchedulerCoordinator({
|
||||
db,
|
||||
repository,
|
||||
dispatch,
|
||||
now: () => new Date('2026-07-30T01:00:00.000Z'),
|
||||
id: () => 'in-flight-run',
|
||||
intervalMs: 1_000,
|
||||
});
|
||||
|
||||
try {
|
||||
coordinator.start();
|
||||
expect(dispatch).toHaveBeenCalledTimes(1);
|
||||
coordinator.stop();
|
||||
finishDispatch({ outcome: 'succeeded', jobIds: ['job-1'] });
|
||||
await vi.advanceTimersByTimeAsync(0);
|
||||
expect(vi.getTimerCount()).toBe(0);
|
||||
} finally {
|
||||
coordinator.stop();
|
||||
vi.useRealTimers();
|
||||
db.close();
|
||||
}
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,329 @@
|
||||
import { randomUUID } from 'node:crypto';
|
||||
import type {
|
||||
CreateScheduledTaskRequest,
|
||||
ScheduledRun,
|
||||
ScheduledRunOutcome,
|
||||
ScheduledTask,
|
||||
} from '@multi-simadmin/contracts';
|
||||
import { scheduleTriggerOf } from '@multi-simadmin/contracts';
|
||||
|
||||
import type { SqliteDatabase } from '../../infrastructure/database/database.js';
|
||||
import { nextOccurrenceFor, reconcileOccurrence } from './schedule-time.js';
|
||||
import { ScheduledTaskRepository } from './scheduled-task-repository.js';
|
||||
import { resolveOperationTargets, type ResolvedTarget } from './target-resolver.js';
|
||||
|
||||
export interface DispatchResult {
|
||||
readonly outcome: ScheduledRunOutcome;
|
||||
readonly reason?: string;
|
||||
readonly jobIds: readonly string[];
|
||||
}
|
||||
|
||||
export type ScheduledDispatch = (
|
||||
task: ScheduledTask,
|
||||
targets: readonly ResolvedTarget[],
|
||||
run: ScheduledRun,
|
||||
context: { readonly actor: string; readonly requestId: string },
|
||||
) => Promise<DispatchResult>;
|
||||
|
||||
export interface SchedulerCoordinatorOptions {
|
||||
readonly db: SqliteDatabase;
|
||||
readonly repository: ScheduledTaskRepository;
|
||||
readonly dispatch: ScheduledDispatch;
|
||||
readonly now?: () => Date;
|
||||
readonly id?: () => string;
|
||||
readonly intervalMs?: number;
|
||||
}
|
||||
|
||||
function snapshot(task: ScheduledTask): CreateScheduledTaskRequest {
|
||||
return {
|
||||
name: task.name,
|
||||
operationType: task.operationType,
|
||||
trigger: scheduleTriggerOf(task),
|
||||
cronExpression: task.cronExpression,
|
||||
timezone: task.timezone,
|
||||
targetSelector: task.targetSelector,
|
||||
...(task.delaySeconds === undefined ? {} : { delaySeconds: task.delaySeconds }),
|
||||
...(task.effectiveStartAt ? { effectiveStartAt: task.effectiveStartAt } : {}),
|
||||
...(task.effectiveEndAt ? { effectiveEndAt: task.effectiveEndAt } : {}),
|
||||
misfirePolicy: task.misfirePolicy,
|
||||
overlapPolicy: task.overlapPolicy,
|
||||
retryPolicy: task.retryPolicy,
|
||||
enabled: task.enabled,
|
||||
};
|
||||
}
|
||||
|
||||
export class SchedulerCoordinator {
|
||||
private readonly now: () => Date;
|
||||
private readonly id: () => string;
|
||||
private readonly intervalMs: number;
|
||||
private timer: ReturnType<typeof setTimeout> | undefined;
|
||||
private started = false;
|
||||
private lifecycle = 0;
|
||||
|
||||
constructor(private readonly options: SchedulerCoordinatorOptions) {
|
||||
this.now = options.now ?? (() => new Date());
|
||||
this.id = options.id ?? randomUUID;
|
||||
this.intervalMs = options.intervalMs ?? 30_000;
|
||||
}
|
||||
|
||||
start(): void {
|
||||
if (this.started) return;
|
||||
this.started = true;
|
||||
const lifecycle = ++this.lifecycle;
|
||||
const schedule = () => {
|
||||
if (!this.started || lifecycle !== this.lifecycle) return;
|
||||
this.timer = setTimeout(() => {
|
||||
this.timer = undefined;
|
||||
void this.tick().finally(schedule);
|
||||
}, this.intervalMs);
|
||||
this.timer.unref?.();
|
||||
};
|
||||
void this.tick().finally(schedule);
|
||||
}
|
||||
|
||||
stop(): void {
|
||||
this.started = false;
|
||||
this.lifecycle += 1;
|
||||
if (this.timer) clearTimeout(this.timer);
|
||||
this.timer = undefined;
|
||||
}
|
||||
|
||||
async runNow(taskId: string, actor: string, requestId: string): Promise<ScheduledRun> {
|
||||
const task = this.options.repository.get(taskId);
|
||||
if (!task) throw new Error('Scheduled task was not found');
|
||||
return this.runOccurrence(task, this.now().toISOString(), 'manual', actor, requestId);
|
||||
}
|
||||
|
||||
async tick(): Promise<readonly ScheduledRun[]> {
|
||||
const now = this.now();
|
||||
const nowIso = now.toISOString();
|
||||
const completed: ScheduledRun[] = [];
|
||||
for (const task of this.options.repository.list()) {
|
||||
const queued = this.options.repository.getQueuedRun(task.id);
|
||||
if (queued && !this.options.repository.hasActiveRun(task.id)) {
|
||||
if (queued.scheduleVersion !== task.version) {
|
||||
completed.push(
|
||||
this.options.repository.finishRun(queued.id, {
|
||||
outcome: 'needs-attention',
|
||||
reason: 'Queued occurrence belongs to an outdated schedule version',
|
||||
jobIds: [],
|
||||
finishedAt: nowIso,
|
||||
}),
|
||||
);
|
||||
} else if (!task.enabled) {
|
||||
completed.push(
|
||||
this.options.repository.finishRun(queued.id, {
|
||||
outcome: 'skipped',
|
||||
reason: 'Schedule was disabled while the occurrence was queued',
|
||||
jobIds: [],
|
||||
finishedAt: nowIso,
|
||||
}),
|
||||
);
|
||||
} else {
|
||||
const resolution = resolveOperationTargets(
|
||||
this.options.db,
|
||||
{ mode: 'fixed', instanceIds: queued.targetSnapshot },
|
||||
task.operationType,
|
||||
);
|
||||
if (resolution.unavailableInstanceIds.length > 0) {
|
||||
completed.push(
|
||||
this.options.repository.finishRun(queued.id, {
|
||||
outcome: 'needs-attention',
|
||||
reason: this.capabilityReason(resolution.unavailableInstanceIds),
|
||||
jobIds: [],
|
||||
finishedAt: nowIso,
|
||||
}),
|
||||
);
|
||||
continue;
|
||||
}
|
||||
completed.push(
|
||||
await this.executeClaimed(
|
||||
task,
|
||||
queued,
|
||||
resolution.targets,
|
||||
'scheduled-automation',
|
||||
`queue:${queued.id}`,
|
||||
),
|
||||
);
|
||||
}
|
||||
continue;
|
||||
}
|
||||
if (!task.enabled || !task.nextDueAt || task.nextDueAt > nowIso) continue;
|
||||
const outsideWindow =
|
||||
(task.effectiveStartAt !== undefined && nowIso < task.effectiveStartAt) ||
|
||||
(task.effectiveEndAt !== undefined && nowIso > task.effectiveEndAt);
|
||||
if (outsideWindow) {
|
||||
const anchor =
|
||||
task.effectiveStartAt && nowIso < task.effectiveStartAt
|
||||
? new Date(task.effectiveStartAt)
|
||||
: now;
|
||||
const nextDueAt = nextOccurrenceFor(scheduleTriggerOf(task), anchor);
|
||||
const claimed = this.options.repository.claimScheduledOccurrence({
|
||||
id: this.id(),
|
||||
scheduledTaskId: task.id,
|
||||
scheduleVersion: task.version,
|
||||
dueAt: task.nextDueAt,
|
||||
nextDueAt,
|
||||
claimedAt: nowIso,
|
||||
taskSnapshot: snapshot(task),
|
||||
overlapPolicy: task.overlapPolicy,
|
||||
resolveTargets: () => ({ targets: [] }),
|
||||
terminal: {
|
||||
outcome: 'skipped',
|
||||
reason: 'Occurrence is outside the effective window',
|
||||
},
|
||||
});
|
||||
if (claimed) completed.push(claimed.run);
|
||||
continue;
|
||||
}
|
||||
|
||||
const exact = task.nextDueAt === nowIso;
|
||||
const reconciled = exact
|
||||
? {
|
||||
action: 'run' as const,
|
||||
dueAt: task.nextDueAt,
|
||||
nextDueAt: nextOccurrenceFor(scheduleTriggerOf(task), now, new Date(task.nextDueAt)),
|
||||
}
|
||||
: reconcileOccurrence(
|
||||
{
|
||||
trigger: task.trigger,
|
||||
nextDueAt: task.nextDueAt,
|
||||
misfirePolicy: task.misfirePolicy,
|
||||
},
|
||||
now,
|
||||
);
|
||||
if (reconciled.action === 'wait') continue;
|
||||
if (reconciled.action === 'skip') {
|
||||
const claimed = this.options.repository.claimScheduledOccurrence({
|
||||
id: this.id(),
|
||||
scheduledTaskId: task.id,
|
||||
scheduleVersion: task.version,
|
||||
dueAt: reconciled.dueAt,
|
||||
nextDueAt: reconciled.nextDueAt,
|
||||
claimedAt: nowIso,
|
||||
taskSnapshot: snapshot(task),
|
||||
overlapPolicy: task.overlapPolicy,
|
||||
resolveTargets: () => ({ targets: [] }),
|
||||
terminal: {
|
||||
outcome: 'skipped',
|
||||
reason: 'Missed occurrence skipped by task policy',
|
||||
},
|
||||
});
|
||||
if (claimed) completed.push(claimed.run);
|
||||
continue;
|
||||
}
|
||||
const claimed = this.options.repository.claimScheduledOccurrence({
|
||||
id: this.id(),
|
||||
scheduledTaskId: task.id,
|
||||
scheduleVersion: task.version,
|
||||
dueAt: reconciled.dueAt,
|
||||
nextDueAt: reconciled.nextDueAt,
|
||||
claimedAt: nowIso,
|
||||
taskSnapshot: snapshot(task),
|
||||
overlapPolicy: task.overlapPolicy,
|
||||
resolveTargets: () => {
|
||||
const resolution = resolveOperationTargets(
|
||||
this.options.db,
|
||||
task.targetSelector,
|
||||
task.operationType,
|
||||
);
|
||||
return {
|
||||
targets: resolution.targets,
|
||||
targetSnapshot: resolution.targetSnapshot,
|
||||
...(resolution.unavailableInstanceIds.length > 0
|
||||
? {
|
||||
attentionReason: this.capabilityReason(resolution.unavailableInstanceIds),
|
||||
}
|
||||
: {}),
|
||||
};
|
||||
},
|
||||
});
|
||||
if (!claimed || claimed.disposition === 'queued') continue;
|
||||
if (claimed.disposition === 'finished') {
|
||||
completed.push(claimed.run);
|
||||
continue;
|
||||
}
|
||||
completed.push(
|
||||
await this.executeClaimed(
|
||||
task,
|
||||
claimed.run,
|
||||
claimed.targets,
|
||||
'scheduled-automation',
|
||||
`schedule:${task.id}:${reconciled.dueAt}`,
|
||||
),
|
||||
);
|
||||
}
|
||||
return completed;
|
||||
}
|
||||
|
||||
private async runOccurrence(
|
||||
task: ScheduledTask,
|
||||
dueAt: string,
|
||||
triggerSource: 'scheduled' | 'manual',
|
||||
actor: string,
|
||||
requestId: string,
|
||||
): Promise<ScheduledRun> {
|
||||
const resolution = resolveOperationTargets(
|
||||
this.options.db,
|
||||
task.targetSelector,
|
||||
task.operationType,
|
||||
);
|
||||
const claimed = this.options.repository.claimOccurrence({
|
||||
id: this.id(),
|
||||
scheduledTaskId: task.id,
|
||||
scheduleVersion: task.version,
|
||||
dueAt,
|
||||
claimedAt: this.now().toISOString(),
|
||||
triggerSource,
|
||||
targetSnapshot: resolution.targetSnapshot,
|
||||
taskSnapshot: snapshot(task),
|
||||
});
|
||||
if (!claimed) throw new Error('Scheduled occurrence was already claimed');
|
||||
if (resolution.unavailableInstanceIds.length > 0)
|
||||
return this.options.repository.finishRun(claimed.id, {
|
||||
outcome: 'needs-attention',
|
||||
reason: this.capabilityReason(resolution.unavailableInstanceIds),
|
||||
jobIds: [],
|
||||
finishedAt: this.now().toISOString(),
|
||||
});
|
||||
return this.executeClaimed(task, claimed, resolution.targets, actor, requestId);
|
||||
}
|
||||
|
||||
private capabilityReason(instanceIds: readonly string[]): string {
|
||||
return `Required capability is unavailable for: ${instanceIds.join(', ')}`;
|
||||
}
|
||||
|
||||
private async executeClaimed(
|
||||
task: ScheduledTask,
|
||||
claimed: ScheduledRun,
|
||||
targets: readonly ResolvedTarget[],
|
||||
actor: string,
|
||||
requestId: string,
|
||||
): Promise<ScheduledRun> {
|
||||
const now = this.now().toISOString();
|
||||
if (targets.length === 0)
|
||||
return this.options.repository.finishRun(claimed.id, {
|
||||
outcome: 'no-targets',
|
||||
reason: 'No enabled instances matched the current selector',
|
||||
jobIds: [],
|
||||
finishedAt: now,
|
||||
});
|
||||
this.options.repository.startRun(claimed.id, now);
|
||||
try {
|
||||
const result = await this.options.dispatch(task, targets, claimed, { actor, requestId });
|
||||
return this.options.repository.finishRun(claimed.id, {
|
||||
outcome: result.outcome,
|
||||
...(result.reason ? { reason: result.reason } : {}),
|
||||
jobIds: result.jobIds,
|
||||
finishedAt: this.now().toISOString(),
|
||||
});
|
||||
} catch {
|
||||
return this.options.repository.finishRun(claimed.id, {
|
||||
outcome: 'failed',
|
||||
reason: 'Scheduled dispatch failed',
|
||||
jobIds: [],
|
||||
finishedAt: this.now().toISOString(),
|
||||
});
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,128 @@
|
||||
import Database from 'better-sqlite3';
|
||||
import { describe, expect, it } from 'vitest';
|
||||
|
||||
import { migrateDatabase } from '../../infrastructure/database/migrations.js';
|
||||
import { resolveOperationTargets, resolveTargets } from './target-resolver.js';
|
||||
|
||||
function fixture() {
|
||||
const db = new Database(':memory:');
|
||||
db.pragma('foreign_keys = ON');
|
||||
migrateDatabase(db);
|
||||
const now = '2026-07-30T00:00:00.000Z';
|
||||
const insert = db.prepare(
|
||||
'INSERT INTO instances (id,name,base_url,enabled,config_revision,created_at,updated_at) VALUES (?,?,?,?,?,?,?)',
|
||||
);
|
||||
insert.run('a', 'Alpha', 'http://10.0.0.1', 1, 3, now, now);
|
||||
insert.run('b', 'Beta', 'http://10.0.0.2', 1, 2, now, now);
|
||||
insert.run('c', 'Disabled', 'http://10.0.0.3', 0, 1, now, now);
|
||||
const tag = db.prepare('INSERT INTO instance_tags (instance_id,tag,created_at) VALUES (?,?,?)');
|
||||
tag.run('a', 'lab', now);
|
||||
tag.run('a', 'east', now);
|
||||
tag.run('b', 'lab', now);
|
||||
tag.run('c', 'east', now);
|
||||
return db;
|
||||
}
|
||||
|
||||
describe('resolveTargets', () => {
|
||||
it('resolves enabled fixed targets in selector order with current revisions', () => {
|
||||
const db = fixture();
|
||||
expect(resolveTargets(db, { mode: 'fixed', instanceIds: ['b', 'c', 'a'] })).toEqual([
|
||||
{ id: 'b', revision: 2 },
|
||||
{ id: 'a', revision: 3 },
|
||||
]);
|
||||
db.close();
|
||||
});
|
||||
|
||||
it('resolves dynamic all and any tag matches on every call', () => {
|
||||
const db = fixture();
|
||||
expect(resolveTargets(db, { mode: 'tags', match: 'all', tags: ['lab', 'east'] })).toEqual([
|
||||
{ id: 'a', revision: 3 },
|
||||
]);
|
||||
expect(resolveTargets(db, { mode: 'tags', match: 'any', tags: ['lab', 'east'] })).toEqual([
|
||||
{ id: 'a', revision: 3 },
|
||||
{ id: 'b', revision: 2 },
|
||||
]);
|
||||
db.prepare(
|
||||
'INSERT INTO instances (id,name,base_url,enabled,config_revision,created_at,updated_at) VALUES (?,?,?,?,?,?,?)',
|
||||
).run(
|
||||
'd',
|
||||
'Dynamic',
|
||||
'http://10.0.0.4',
|
||||
1,
|
||||
1,
|
||||
'2026-07-30T00:00:00.000Z',
|
||||
'2026-07-30T00:00:00.000Z',
|
||||
);
|
||||
db.prepare('INSERT INTO instance_tags (instance_id,tag,created_at) VALUES (?,?,?)').run(
|
||||
'd',
|
||||
'east',
|
||||
'2026-07-30T00:00:00.000Z',
|
||||
);
|
||||
expect(
|
||||
resolveTargets(db, { mode: 'tags', match: 'any', tags: ['east'] }).map((x) => x.id),
|
||||
).toEqual(['a', 'd']);
|
||||
db.close();
|
||||
});
|
||||
|
||||
it('resolves the Hub all-devices and group selectors', () => {
|
||||
const db = fixture();
|
||||
const now = '2026-07-30T00:00:00.000Z';
|
||||
db.prepare(
|
||||
'INSERT INTO device_groups (id,name,description,created_at,updated_at) VALUES (?,?,?,?,?)',
|
||||
).run('g-lab', '实验室', '', now, now);
|
||||
db.prepare('UPDATE instances SET group_id = ? WHERE id IN (?,?)').run('g-lab', 'a', 'c');
|
||||
expect(resolveTargets(db, { mode: 'all' })).toEqual([
|
||||
{ id: 'a', revision: 3 },
|
||||
{ id: 'b', revision: 2 },
|
||||
]);
|
||||
expect(resolveTargets(db, { mode: 'group', groupId: 'g-lab' })).toEqual([
|
||||
{ id: 'a', revision: 3 },
|
||||
]);
|
||||
expect(resolveTargets(db, { mode: 'groups', groupIds: ['g-lab', 'g-office'] })).toEqual([
|
||||
{ id: 'a', revision: 3 },
|
||||
]);
|
||||
db.close();
|
||||
});
|
||||
|
||||
it('dispatches control-plane backups without resolving device targets', () => {
|
||||
const db = fixture();
|
||||
expect(resolveOperationTargets(db, { mode: 'all' }, 'backup-data')).toEqual({
|
||||
targets: [{ id: 'control-plane', revision: 1 }],
|
||||
targetSnapshot: [],
|
||||
unavailableInstanceIds: [],
|
||||
});
|
||||
db.close();
|
||||
});
|
||||
|
||||
it('dispatches devices whose capability was never probed', () => {
|
||||
const db = fixture();
|
||||
expect(resolveOperationTargets(db, { mode: 'all' }, 'restart-baseband')).toEqual({
|
||||
targets: [
|
||||
{ id: 'a', revision: 3 },
|
||||
{ id: 'b', revision: 2 },
|
||||
],
|
||||
targetSnapshot: ['a', 'b'],
|
||||
unavailableInstanceIds: [],
|
||||
});
|
||||
db.close();
|
||||
});
|
||||
|
||||
it('separates targets whose required operation capability is unavailable', () => {
|
||||
const db = fixture();
|
||||
const now = '2026-07-30T00:00:00.000Z';
|
||||
db.prepare(
|
||||
'INSERT INTO capabilities (instance_id,operation_id,state,observed_at,created_at,updated_at) VALUES (?,?,?,?,?,?)',
|
||||
).run('a', 'postServiceRestart', 'supported', now, now, now);
|
||||
db.prepare(
|
||||
'INSERT INTO capabilities (instance_id,operation_id,state,observed_at,created_at,updated_at) VALUES (?,?,?,?,?,?)',
|
||||
).run('b', 'postServiceRestart', 'unsupported', now, now, now);
|
||||
expect(
|
||||
resolveOperationTargets(db, { mode: 'fixed', instanceIds: ['a', 'b'] }, 'restart-service'),
|
||||
).toEqual({
|
||||
targets: [{ id: 'a', revision: 3 }],
|
||||
targetSnapshot: ['a', 'b'],
|
||||
unavailableInstanceIds: ['b'],
|
||||
});
|
||||
db.close();
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,129 @@
|
||||
import type { ScheduledOperationType, ScheduleTargetSelector } from '@multi-simadmin/contracts';
|
||||
|
||||
import type { SqliteDatabase } from '../../infrastructure/database/database.js';
|
||||
|
||||
export interface ResolvedTarget {
|
||||
readonly id: string;
|
||||
readonly revision: number;
|
||||
}
|
||||
|
||||
interface TargetRow {
|
||||
id: string;
|
||||
config_revision: number;
|
||||
}
|
||||
|
||||
export interface OperationTargetResolution {
|
||||
readonly targets: readonly ResolvedTarget[];
|
||||
readonly targetSnapshot: readonly string[];
|
||||
readonly unavailableInstanceIds: readonly string[];
|
||||
}
|
||||
|
||||
/** Sentinel target for control-plane-local actions; it is never an instance id. */
|
||||
export const LOCAL_TARGET: ResolvedTarget = { id: 'control-plane', revision: 1 };
|
||||
|
||||
const operationIds: Record<ScheduledOperationType, string> = {
|
||||
'restart-service': 'postServiceRestart',
|
||||
'reboot-system': 'postSystemReboot',
|
||||
'restart-baseband': 'postBasebandRestart',
|
||||
'send-sms': 'postSmsSend',
|
||||
'backup-data': 'localBackup',
|
||||
};
|
||||
|
||||
export function resolveTargets(
|
||||
db: SqliteDatabase,
|
||||
selector: ScheduleTargetSelector,
|
||||
): readonly ResolvedTarget[] {
|
||||
if (selector.mode === 'fixed') {
|
||||
const placeholders = selector.instanceIds.map(() => '?').join(',');
|
||||
const rows = db
|
||||
.prepare(
|
||||
`SELECT id, config_revision FROM instances
|
||||
WHERE enabled = 1 AND id IN (${placeholders})`,
|
||||
)
|
||||
.all(...selector.instanceIds) as TargetRow[];
|
||||
const byId = new Map(rows.map((row) => [row.id, row]));
|
||||
return selector.instanceIds.flatMap((id) => {
|
||||
const row = byId.get(id);
|
||||
return row ? [{ id: row.id, revision: row.config_revision }] : [];
|
||||
});
|
||||
}
|
||||
|
||||
if (selector.mode === 'all') {
|
||||
const rows = db
|
||||
.prepare('SELECT id, config_revision FROM instances WHERE enabled = 1 ORDER BY id ASC')
|
||||
.all() as TargetRow[];
|
||||
return rows.map((row) => ({ id: row.id, revision: row.config_revision }));
|
||||
}
|
||||
|
||||
if (selector.mode === 'group') {
|
||||
const rows = db
|
||||
.prepare(
|
||||
'SELECT id, config_revision FROM instances WHERE enabled = 1 AND group_id = ? ORDER BY id ASC',
|
||||
)
|
||||
.all(selector.groupId) as TargetRow[];
|
||||
return rows.map((row) => ({ id: row.id, revision: row.config_revision }));
|
||||
}
|
||||
|
||||
if (selector.mode === 'groups') {
|
||||
const placeholders = selector.groupIds.map(() => '?').join(',');
|
||||
const rows = db
|
||||
.prepare(
|
||||
`SELECT id, config_revision FROM instances
|
||||
WHERE enabled = 1 AND group_id IN (${placeholders})
|
||||
ORDER BY id ASC`,
|
||||
)
|
||||
.all(...selector.groupIds) as TargetRow[];
|
||||
return rows.map((row) => ({ id: row.id, revision: row.config_revision }));
|
||||
}
|
||||
|
||||
const placeholders = selector.tags.map(() => '?').join(',');
|
||||
const comparison = selector.match === 'all' ? '= ?' : '> 0';
|
||||
const parameters: Array<string | number> = [...selector.tags];
|
||||
if (selector.match === 'all') parameters.push(selector.tags.length);
|
||||
const rows = db
|
||||
.prepare(
|
||||
`SELECT i.id, i.config_revision
|
||||
FROM instances i
|
||||
JOIN instance_tags t ON t.instance_id = i.id
|
||||
WHERE i.enabled = 1 AND t.tag IN (${placeholders})
|
||||
GROUP BY i.id, i.config_revision
|
||||
HAVING COUNT(DISTINCT t.tag) ${comparison}
|
||||
ORDER BY i.id ASC`,
|
||||
)
|
||||
.all(...parameters) as TargetRow[];
|
||||
return rows.map((row) => ({ id: row.id, revision: row.config_revision }));
|
||||
}
|
||||
|
||||
export function resolveOperationTargets(
|
||||
db: SqliteDatabase,
|
||||
selector: ScheduleTargetSelector,
|
||||
operationType: ScheduledOperationType,
|
||||
): OperationTargetResolution {
|
||||
if (operationType === 'backup-data')
|
||||
return { targets: [LOCAL_TARGET], targetSnapshot: [], unavailableInstanceIds: [] };
|
||||
const candidates = resolveTargets(db, selector);
|
||||
if (candidates.length === 0)
|
||||
return { targets: [], targetSnapshot: [], unavailableInstanceIds: [] };
|
||||
const placeholders = candidates.map(() => '?').join(',');
|
||||
const rows = db
|
||||
.prepare(
|
||||
`SELECT instance_id, state FROM capabilities
|
||||
WHERE operation_id = ? AND instance_id IN (${placeholders})`,
|
||||
)
|
||||
.all(operationIds[operationType], ...candidates.map((target) => target.id)) as Array<{
|
||||
instance_id: string;
|
||||
state: string;
|
||||
}>;
|
||||
const states = new Map(rows.map((row) => [row.instance_id, row.state]));
|
||||
// Only a probed "unsupported" verdict blocks a device. Capability rows are written by active
|
||||
// probes, so an absent row means "never observed", which must dispatch optimistically and let
|
||||
// the job result report the truth; otherwise every scheduled task stalls on missing evidence.
|
||||
const isBlocked = (target: ResolvedTarget): boolean => states.get(target.id) === 'unsupported';
|
||||
return {
|
||||
targets: candidates.filter((target) => !isBlocked(target)),
|
||||
targetSnapshot: candidates.map((target) => target.id),
|
||||
unavailableInstanceIds: candidates
|
||||
.filter((target) => isBlocked(target))
|
||||
.map((target) => target.id),
|
||||
};
|
||||
}
|
||||
@@ -5,8 +5,10 @@ import { migrateDatabase } from '../../infrastructure/database/migrations.js';
|
||||
import type { SecretStore } from '../../infrastructure/secrets/secret-store.js';
|
||||
import type { TransportResponse } from '../../infrastructure/transport/safe-instance-transport.js';
|
||||
import { ConnectionProbe } from './connection-probe.js';
|
||||
import { ConnectionLogService } from '../system/connection-log-service.js';
|
||||
|
||||
class Store implements SecretStore {
|
||||
readonly provider = 'macos-keychain';
|
||||
async set(key: { instanceId: string; purpose: string; slot?: string }) {
|
||||
return `keychain://multi-simadmin/${Buffer.from(JSON.stringify([key.instanceId, key.purpose, key.slot])).toString('base64url')}`;
|
||||
}
|
||||
@@ -21,7 +23,7 @@ const dbs: Database.Database[] = [];
|
||||
afterEach(() => {
|
||||
for (const db of dbs.splice(0)) db.close();
|
||||
});
|
||||
const fixture = (response: TransportResponse) => {
|
||||
const fixture = (response: TransportResponse, options: { logs?: boolean } = {}) => {
|
||||
const db = new Database(':memory:');
|
||||
db.pragma('foreign_keys=ON');
|
||||
migrateDatabase(db);
|
||||
@@ -32,6 +34,7 @@ const fixture = (response: TransportResponse) => {
|
||||
idFactory: () => 'instance-1',
|
||||
now: () => new Date('2026-07-16T12:00:00.000Z'),
|
||||
});
|
||||
const connectionLogs = options.logs ? new ConnectionLogService({ db }) : undefined;
|
||||
const transport = {
|
||||
get: async (url: string) => {
|
||||
expect(url).toBe('http://192.168.1.20:3000/api/health');
|
||||
@@ -40,12 +43,14 @@ const fixture = (response: TransportResponse) => {
|
||||
};
|
||||
return {
|
||||
db,
|
||||
connectionLogs,
|
||||
instances,
|
||||
probe: new ConnectionProbe({
|
||||
db,
|
||||
instances,
|
||||
transport,
|
||||
now: () => new Date('2026-07-16T12:00:00.000Z'),
|
||||
...(connectionLogs ? { connectionLogs } : {}),
|
||||
}),
|
||||
};
|
||||
};
|
||||
@@ -92,4 +97,116 @@ describe('ConnectionProbe', () => {
|
||||
const { probe } = fixture({ status: 200, headers: {}, body: '{}' });
|
||||
await expect(probe.test('missing')).rejects.toMatchObject({ code: 'NOT_FOUND' });
|
||||
});
|
||||
|
||||
it('journals a successful probe and marks an unauthenticated one as stale', async () => {
|
||||
const { db, instances, probe } = fixture(
|
||||
{ status: 401, headers: {}, body: '{}' },
|
||||
{ logs: true },
|
||||
);
|
||||
await instances.create({ name: 'LAN', origin: 'http://192.168.1.20:3000' });
|
||||
await probe.test('instance-1');
|
||||
const rows = db
|
||||
.prepare('SELECT outcome, state, error_code, http_status, duration_ms FROM connection_logs')
|
||||
.all() as {
|
||||
outcome: string;
|
||||
state: string;
|
||||
error_code: string | null;
|
||||
http_status: number;
|
||||
duration_ms: number;
|
||||
}[];
|
||||
expect(rows).toEqual([
|
||||
{
|
||||
outcome: 'stale',
|
||||
state: 'stale',
|
||||
error_code: 'HTTP_401',
|
||||
http_status: 401,
|
||||
duration_ms: 0,
|
||||
},
|
||||
]);
|
||||
});
|
||||
|
||||
it('journals a transport failure before rethrowing it', async () => {
|
||||
const db = new Database(':memory:');
|
||||
db.pragma('foreign_keys=ON');
|
||||
migrateDatabase(db);
|
||||
dbs.push(db);
|
||||
const instances = new InstanceService({
|
||||
db,
|
||||
store: new Store(),
|
||||
idFactory: () => 'instance-1',
|
||||
now: () => new Date('2026-07-16T12:00:00.000Z'),
|
||||
});
|
||||
await instances.create({ name: 'LAN', origin: 'http://192.168.1.20:3000' });
|
||||
const connectionLogs = new ConnectionLogService({ db });
|
||||
const probe = new ConnectionProbe({
|
||||
db,
|
||||
instances,
|
||||
connectionLogs,
|
||||
transport: {
|
||||
get: async () => {
|
||||
throw Object.assign(new Error('boom'), { code: 'ECONNREFUSED' });
|
||||
},
|
||||
},
|
||||
});
|
||||
await expect(probe.test('instance-1')).rejects.toThrow('boom');
|
||||
expect(connectionLogs.list().items[0]).toMatchObject({
|
||||
outcome: 'failed',
|
||||
state: 'unknown',
|
||||
errorCode: 'ECONNREFUSED',
|
||||
httpStatus: null,
|
||||
});
|
||||
});
|
||||
|
||||
it('honours a configured snapshot lifetime so the offline window drives expiry', async () => {
|
||||
const db = new Database(':memory:');
|
||||
db.pragma('foreign_keys=ON');
|
||||
migrateDatabase(db);
|
||||
dbs.push(db);
|
||||
const instances = new InstanceService({
|
||||
db,
|
||||
store: new Store(),
|
||||
idFactory: () => 'instance-1',
|
||||
now: () => new Date('2026-07-16T12:00:00.000Z'),
|
||||
});
|
||||
await instances.create({ name: 'LAN', origin: 'http://192.168.1.20:3000' });
|
||||
let clock = new Date('2026-07-16T12:00:00.000Z');
|
||||
const probe = new ConnectionProbe({
|
||||
db,
|
||||
instances,
|
||||
transport: { get: async () => ({ status: 200, headers: {}, body: '{}' }) },
|
||||
now: () => clock,
|
||||
snapshotTtlMs: () => 120_000,
|
||||
});
|
||||
await probe.test('instance-1');
|
||||
expect(probe.reachability().get('instance-1')).toEqual({
|
||||
reachable: true,
|
||||
authenticated: true,
|
||||
checkedAt: '2026-07-16T12:00:00.000Z',
|
||||
});
|
||||
|
||||
clock = new Date('2026-07-16T12:01:30.000Z');
|
||||
expect(probe.reachability().get('instance-1')).toMatchObject({ reachable: true });
|
||||
|
||||
clock = new Date('2026-07-16T12:02:01.000Z');
|
||||
expect(probe.reachability().get('instance-1')).toEqual({
|
||||
reachable: false,
|
||||
authenticated: false,
|
||||
checkedAt: '2026-07-16T12:00:00.000Z',
|
||||
});
|
||||
});
|
||||
|
||||
it('reports an expired authenticated snapshot as offline and never as authenticated', async () => {
|
||||
const { db, instances, probe } = fixture({ status: 200, headers: {}, body: '{}' });
|
||||
await instances.create({ name: 'LAN', origin: 'http://192.168.1.20:3000' });
|
||||
expect(probe.reachability().size).toBe(0);
|
||||
await probe.test('instance-1');
|
||||
db.prepare(
|
||||
`UPDATE status_snapshots SET expires_at = '2020-01-01T00:00:00.000Z' WHERE instance_id = 'instance-1'`,
|
||||
).run();
|
||||
expect(probe.reachability().get('instance-1')).toEqual({
|
||||
reachable: false,
|
||||
authenticated: false,
|
||||
checkedAt: '2026-07-16T12:00:00.000Z',
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
@@ -3,6 +3,7 @@ import type Database from 'better-sqlite3';
|
||||
import type { SessionMetadata } from '@multi-simadmin/contracts';
|
||||
import { InstanceService, InstanceServiceError } from '../instances/instance-service.js';
|
||||
import type { TransportResponse } from '../../infrastructure/transport/safe-instance-transport.js';
|
||||
import type { ConnectionLogService } from '../system/connection-log-service.js';
|
||||
|
||||
export interface ConnectionTransport {
|
||||
readonly get: (url: string) => Promise<TransportResponse>;
|
||||
@@ -12,21 +13,59 @@ export interface ConnectionProbeOptions {
|
||||
readonly instances: InstanceService;
|
||||
readonly transport: ConnectionTransport;
|
||||
readonly now?: () => Date;
|
||||
/** Optional journal of every reachability attempt, used by the log centre. */
|
||||
readonly connectionLogs?: ConnectionLogService;
|
||||
/**
|
||||
* How long a probe stays authoritative. Defaults to one heartbeat period; the fleet heartbeat
|
||||
* passes the configured offline window so a lost probe is tolerated exactly as long as the
|
||||
* operator asked for.
|
||||
*/
|
||||
readonly snapshotTtlMs?: number | (() => number);
|
||||
}
|
||||
const CONNECTION_SNAPSHOT_TTL_MS = 30_000;
|
||||
|
||||
/** Last known reachability for one instance, read from the snapshot journal without probing. */
|
||||
export interface ConnectionState {
|
||||
readonly reachable: boolean;
|
||||
readonly authenticated: boolean;
|
||||
readonly checkedAt: string | null;
|
||||
}
|
||||
|
||||
interface SnapshotRow {
|
||||
readonly instance_id: string;
|
||||
readonly payload_json: string | null;
|
||||
readonly observed_at: string;
|
||||
readonly expires_at: string | null;
|
||||
}
|
||||
|
||||
export class ConnectionProbe {
|
||||
private readonly now: () => Date;
|
||||
constructor(private readonly options: ConnectionProbeOptions) {
|
||||
this.now = options.now ?? (() => new Date());
|
||||
}
|
||||
|
||||
#ttlMs(): number {
|
||||
const value = this.options.snapshotTtlMs;
|
||||
const resolved = typeof value === 'function' ? value() : value;
|
||||
return Number.isFinite(resolved) && (resolved ?? 0) > 0
|
||||
? (resolved as number)
|
||||
: CONNECTION_SNAPSHOT_TTL_MS;
|
||||
}
|
||||
|
||||
async test(instanceId: string): Promise<SessionMetadata> {
|
||||
const instance = await this.options.instances.get(instanceId);
|
||||
if (!instance) throw new InstanceServiceError('NOT_FOUND', 'Instance was not found');
|
||||
const response = await this.options.transport.get(`${instance.origin}/api/health`);
|
||||
const startedAt = this.now().getTime();
|
||||
let response: TransportResponse;
|
||||
try {
|
||||
response = await this.options.transport.get(`${instance.origin}/api/health`);
|
||||
} catch (error) {
|
||||
this.#record(instanceId, null, 'failed', startedAt, errorCode(error));
|
||||
throw error;
|
||||
}
|
||||
const observed = this.now();
|
||||
const observedAt = observed.toISOString();
|
||||
const expiresAt = new Date(observed.getTime() + CONNECTION_SNAPSHOT_TTL_MS).toISOString();
|
||||
const expiresAt = new Date(observed.getTime() + this.#ttlMs()).toISOString();
|
||||
const authenticated = response.status >= 200 && response.status < 300;
|
||||
this.options.db
|
||||
.prepare(
|
||||
@@ -48,6 +87,68 @@ export class ConnectionProbe {
|
||||
expiresAt,
|
||||
observedAt,
|
||||
);
|
||||
this.#record(
|
||||
instanceId,
|
||||
response.status,
|
||||
authenticated ? 'success' : 'stale',
|
||||
startedAt,
|
||||
authenticated ? null : `HTTP_${response.status}`,
|
||||
);
|
||||
return { instanceId, authenticated, checkedAt: observedAt };
|
||||
}
|
||||
|
||||
/**
|
||||
* Reachability for every instance that has ever been probed. Anything missing, or whose
|
||||
* snapshot outlived the offline window, is reported as unreachable.
|
||||
*/
|
||||
reachability(): ReadonlyMap<string, ConnectionState> {
|
||||
const rows = this.options.db
|
||||
.prepare(
|
||||
`SELECT instance_id, payload_json, observed_at, expires_at
|
||||
FROM status_snapshots WHERE category = 'connection'`,
|
||||
)
|
||||
.all() as unknown as readonly SnapshotRow[];
|
||||
const now = this.now().toISOString();
|
||||
const states = new Map<string, ConnectionState>();
|
||||
for (const row of rows) {
|
||||
let authenticated = false;
|
||||
try {
|
||||
const payload = JSON.parse(row.payload_json ?? '{}') as { authenticated?: unknown };
|
||||
authenticated = payload.authenticated === true;
|
||||
} catch {
|
||||
authenticated = false;
|
||||
}
|
||||
const reachable = Boolean(row.expires_at) && (row.expires_at as string) > now;
|
||||
states.set(row.instance_id, {
|
||||
reachable,
|
||||
authenticated: reachable ? authenticated : false,
|
||||
checkedAt: row.observed_at,
|
||||
});
|
||||
}
|
||||
return states;
|
||||
}
|
||||
|
||||
/** Best-effort: the journal is diagnostics, never a reason to fail a probe. */
|
||||
#record(
|
||||
instanceId: string,
|
||||
httpStatus: number | null,
|
||||
outcome: 'success' | 'stale' | 'failed',
|
||||
startedAt: number,
|
||||
errorCode: string | null,
|
||||
): void {
|
||||
this.options.connectionLogs?.record({
|
||||
instanceId,
|
||||
outcome,
|
||||
state: outcome === 'success' ? 'fresh' : outcome === 'stale' ? 'stale' : 'unknown',
|
||||
errorCode,
|
||||
httpStatus,
|
||||
durationMs: Math.max(0, this.now().getTime() - startedAt),
|
||||
observedAt: this.now().toISOString(),
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
function errorCode(error: unknown): string {
|
||||
const code = (error as { code?: unknown })?.code;
|
||||
return typeof code === 'string' ? code.slice(0, 64) : 'UPSTREAM_UNAVAILABLE';
|
||||
}
|
||||
|
||||
@@ -0,0 +1,160 @@
|
||||
import Database from 'better-sqlite3';
|
||||
import { afterEach, describe, expect, it } from 'vitest';
|
||||
|
||||
import { migrateDatabase } from '../../infrastructure/database/migrations.js';
|
||||
import {
|
||||
ConnectionSettingsService,
|
||||
DEFAULT_CONNECTION_SETTINGS,
|
||||
validateConnectionSettings,
|
||||
} from './connection-settings-service.js';
|
||||
|
||||
const dbs: Database.Database[] = [];
|
||||
afterEach(() => {
|
||||
for (const db of dbs.splice(0)) db.close();
|
||||
});
|
||||
|
||||
function service() {
|
||||
const db = new Database(':memory:');
|
||||
db.pragma('foreign_keys = ON');
|
||||
migrateDatabase(db);
|
||||
dbs.push(db);
|
||||
return {
|
||||
db,
|
||||
settings: new ConnectionSettingsService({
|
||||
db,
|
||||
now: () => new Date('2026-09-05T00:00:00.000Z'),
|
||||
}),
|
||||
};
|
||||
}
|
||||
|
||||
function stored(db: Database.Database): string | undefined {
|
||||
const row = db
|
||||
.prepare('SELECT value_json FROM app_settings WHERE key = ?')
|
||||
.get('connection.settings') as { value_json: string } | undefined;
|
||||
return row?.value_json;
|
||||
}
|
||||
|
||||
describe('validateConnectionSettings', () => {
|
||||
it('fills the missing field from the defaults', () => {
|
||||
expect(validateConnectionSettings({ heartbeatSeconds: 60 })).toEqual({
|
||||
heartbeatSeconds: 60,
|
||||
offlineSeconds: 120,
|
||||
authorizationMode: 'auto',
|
||||
});
|
||||
expect(validateConnectionSettings({ offlineSeconds: 300 })).toEqual({
|
||||
heartbeatSeconds: 30,
|
||||
offlineSeconds: 300,
|
||||
authorizationMode: 'auto',
|
||||
});
|
||||
});
|
||||
|
||||
it('rejects payloads that are not objects or carry unknown keys', () => {
|
||||
expect(() => validateConnectionSettings(null)).toThrow(TypeError);
|
||||
expect(() => validateConnectionSettings([])).toThrow(TypeError);
|
||||
expect(() => validateConnectionSettings('30')).toThrow(TypeError);
|
||||
expect(() => validateConnectionSettings({ heartbeat_interval: 30 })).toThrow(TypeError);
|
||||
});
|
||||
|
||||
it('rejects non-integer or out-of-range cadence', () => {
|
||||
expect(() => validateConnectionSettings({ heartbeatSeconds: 30.5 })).toThrow(RangeError);
|
||||
expect(() => validateConnectionSettings({ heartbeatSeconds: 4 })).toThrow(RangeError);
|
||||
expect(() => validateConnectionSettings({ heartbeatSeconds: 301 })).toThrow(RangeError);
|
||||
expect(() => validateConnectionSettings({ offlineSeconds: 1_801 })).toThrow(RangeError);
|
||||
});
|
||||
|
||||
it('requires the offline window to tolerate at least two missed beats', () => {
|
||||
expect(() => validateConnectionSettings({ heartbeatSeconds: 30, offlineSeconds: 59 })).toThrow(
|
||||
RangeError,
|
||||
);
|
||||
expect(validateConnectionSettings({ heartbeatSeconds: 30, offlineSeconds: 60 })).toEqual({
|
||||
heartbeatSeconds: 30,
|
||||
offlineSeconds: 60,
|
||||
authorizationMode: 'auto',
|
||||
});
|
||||
});
|
||||
|
||||
it('requires a known authorization mode and keeps the mode omitted on update', () => {
|
||||
expect(() => validateConnectionSettings({ authorizationMode: 'operator' })).toThrow(TypeError);
|
||||
expect(validateConnectionSettings({ authorizationMode: 'manual' })).toEqual({
|
||||
heartbeatSeconds: 30,
|
||||
offlineSeconds: 90,
|
||||
authorizationMode: 'manual',
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
describe('ConnectionSettingsService', () => {
|
||||
it('starts from the shipped defaults', () => {
|
||||
const { settings } = service();
|
||||
expect(settings.get()).toEqual({
|
||||
heartbeatSeconds: 30,
|
||||
offlineSeconds: 90,
|
||||
authorizationMode: 'auto',
|
||||
});
|
||||
expect(settings.heartbeatMs).toBe(30_000);
|
||||
expect(settings.snapshotTtlMs).toBe(90_000);
|
||||
});
|
||||
|
||||
it('persists an update and reads it back', () => {
|
||||
const { db, settings } = service();
|
||||
expect(settings.update({ heartbeatSeconds: 45, offlineSeconds: 180 })).toEqual({
|
||||
heartbeatSeconds: 45,
|
||||
offlineSeconds: 180,
|
||||
authorizationMode: 'auto',
|
||||
});
|
||||
expect(stored(db)).toBe(
|
||||
'{"heartbeatSeconds":45,"offlineSeconds":180,"authorizationMode":"auto"}',
|
||||
);
|
||||
expect(settings.get()).toEqual({
|
||||
heartbeatSeconds: 45,
|
||||
offlineSeconds: 180,
|
||||
authorizationMode: 'auto',
|
||||
});
|
||||
expect(settings.heartbeatMs).toBe(45_000);
|
||||
expect(settings.snapshotTtlMs).toBe(180_000);
|
||||
});
|
||||
|
||||
it('rewrites the same row instead of inserting a second one', () => {
|
||||
const { db, settings } = service();
|
||||
settings.update({ heartbeatSeconds: 10, offlineSeconds: 20 });
|
||||
settings.update({ heartbeatSeconds: 20, offlineSeconds: 40, authorizationMode: 'manual' });
|
||||
const rows = db.prepare('SELECT COUNT(*) AS count FROM app_settings').get() as {
|
||||
count: number;
|
||||
};
|
||||
expect(rows.count).toBe(1);
|
||||
expect(settings.get()).toEqual({
|
||||
heartbeatSeconds: 20,
|
||||
offlineSeconds: 40,
|
||||
authorizationMode: 'manual',
|
||||
});
|
||||
});
|
||||
|
||||
it('falls back to the defaults when the stored row is corrupt', () => {
|
||||
const { db, settings } = service();
|
||||
const now = new Date('2026-09-05T00:00:00.000Z').toISOString();
|
||||
db.prepare(
|
||||
'INSERT INTO app_settings (key,value_json,created_at,updated_at) VALUES (?,?,?,?)',
|
||||
).run('connection.settings', 'not json', now, now);
|
||||
expect(settings.get()).toEqual(DEFAULT_CONNECTION_SETTINGS);
|
||||
|
||||
db.prepare('UPDATE app_settings SET value_json = ? WHERE key = ?').run(
|
||||
'{"heartbeatSeconds":9999,"offlineSeconds":10}',
|
||||
'connection.settings',
|
||||
);
|
||||
expect(settings.get()).toEqual(DEFAULT_CONNECTION_SETTINGS);
|
||||
});
|
||||
|
||||
it('keeps a rejected update from touching the stored row', () => {
|
||||
const { db, settings } = service();
|
||||
settings.update({ heartbeatSeconds: 60, offlineSeconds: 120 });
|
||||
expect(() => settings.update({ heartbeatSeconds: 60, offlineSeconds: 61 })).toThrow(RangeError);
|
||||
expect(settings.get()).toEqual({
|
||||
heartbeatSeconds: 60,
|
||||
offlineSeconds: 120,
|
||||
authorizationMode: 'auto',
|
||||
});
|
||||
expect(stored(db)).toBe(
|
||||
'{"heartbeatSeconds":60,"offlineSeconds":120,"authorizationMode":"auto"}',
|
||||
);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,126 @@
|
||||
import type Database from 'better-sqlite3';
|
||||
|
||||
/**
|
||||
* Reachability cadence, the Hub "connection settings" panel rebuilt on the control plane's own
|
||||
* snapshot journal. The heartbeat decides how often every instance is probed; the offline window
|
||||
* decides how long a lost probe is tolerated before the fleet view calls the device offline.
|
||||
*/
|
||||
export interface ConnectionSettings {
|
||||
readonly heartbeatSeconds: number;
|
||||
readonly offlineSeconds: number;
|
||||
readonly authorizationMode: 'auto' | 'manual';
|
||||
}
|
||||
|
||||
export const MIN_HEARTBEAT_SECONDS = 5;
|
||||
export const MAX_HEARTBEAT_SECONDS = 300;
|
||||
export const MAX_OFFLINE_SECONDS = 1_800;
|
||||
|
||||
export const DEFAULT_CONNECTION_SETTINGS: Readonly<ConnectionSettings> = Object.freeze({
|
||||
heartbeatSeconds: 30,
|
||||
offlineSeconds: 90,
|
||||
authorizationMode: 'auto',
|
||||
});
|
||||
|
||||
const SETTING_KEY = 'connection.settings';
|
||||
|
||||
const FIELDS: readonly (keyof ConnectionSettings)[] = [
|
||||
'heartbeatSeconds',
|
||||
'offlineSeconds',
|
||||
'authorizationMode',
|
||||
];
|
||||
|
||||
function safeInteger(value: unknown, name: string): number {
|
||||
if (!Number.isSafeInteger(value))
|
||||
throw new RangeError(`${name} must be a whole number of seconds`);
|
||||
return value as number;
|
||||
}
|
||||
|
||||
export function validateConnectionSettings(value: unknown): ConnectionSettings {
|
||||
const source =
|
||||
value !== null && typeof value === 'object' && !Array.isArray(value)
|
||||
? (value as Record<string, unknown>)
|
||||
: null;
|
||||
if (!source) throw new TypeError('The connection settings payload is invalid');
|
||||
const unknown = Object.keys(source).filter(
|
||||
(key) => !FIELDS.includes(key as keyof ConnectionSettings),
|
||||
);
|
||||
if (unknown.length > 0) throw new TypeError(`Unknown connection setting: ${unknown[0]}`);
|
||||
|
||||
const authorizationMode =
|
||||
source.authorizationMode === undefined
|
||||
? DEFAULT_CONNECTION_SETTINGS.authorizationMode
|
||||
: source.authorizationMode;
|
||||
if (authorizationMode !== 'auto' && authorizationMode !== 'manual')
|
||||
throw new TypeError('authorizationMode must be auto or manual');
|
||||
|
||||
const heartbeatSeconds = safeInteger(
|
||||
source.heartbeatSeconds ?? DEFAULT_CONNECTION_SETTINGS.heartbeatSeconds,
|
||||
'heartbeatSeconds',
|
||||
);
|
||||
if (heartbeatSeconds < MIN_HEARTBEAT_SECONDS || heartbeatSeconds > MAX_HEARTBEAT_SECONDS)
|
||||
throw new RangeError('heartbeatSeconds must be between 5 and 300');
|
||||
|
||||
// An omitted window scales with the heartbeat so a partial payload stays valid; an explicit one
|
||||
// is checked strictly below.
|
||||
const fallbackOffline = Math.min(
|
||||
MAX_OFFLINE_SECONDS,
|
||||
Math.max(DEFAULT_CONNECTION_SETTINGS.offlineSeconds, heartbeatSeconds * 2),
|
||||
);
|
||||
const offlineSeconds = safeInteger(source.offlineSeconds ?? fallbackOffline, 'offlineSeconds');
|
||||
if (offlineSeconds < heartbeatSeconds * 2)
|
||||
throw new RangeError('offlineSeconds must be at least twice the heartbeat interval');
|
||||
if (offlineSeconds > MAX_OFFLINE_SECONDS)
|
||||
throw new RangeError(`offlineSeconds must not exceed ${MAX_OFFLINE_SECONDS}`);
|
||||
|
||||
return { heartbeatSeconds, offlineSeconds, authorizationMode };
|
||||
}
|
||||
|
||||
interface SettingsRow {
|
||||
readonly value_json?: string | null;
|
||||
}
|
||||
|
||||
export class ConnectionSettingsService {
|
||||
readonly #db: Database.Database;
|
||||
readonly #now: () => Date;
|
||||
|
||||
constructor(options: { readonly db: Database.Database; readonly now?: () => Date }) {
|
||||
this.#db = options.db;
|
||||
this.#now = options.now ?? (() => new Date());
|
||||
}
|
||||
|
||||
get(): ConnectionSettings {
|
||||
const row = this.#db
|
||||
.prepare('SELECT value_json FROM app_settings WHERE key = ?')
|
||||
.get(SETTING_KEY) as SettingsRow | undefined;
|
||||
if (!row?.value_json) return { ...DEFAULT_CONNECTION_SETTINGS };
|
||||
try {
|
||||
return validateConnectionSettings(JSON.parse(row.value_json));
|
||||
} catch {
|
||||
// A hand-edited row must never take the heartbeat loop down.
|
||||
return { ...DEFAULT_CONNECTION_SETTINGS };
|
||||
}
|
||||
}
|
||||
|
||||
update(value: unknown): ConnectionSettings {
|
||||
const settings = validateConnectionSettings(value);
|
||||
const now = this.#now().toISOString();
|
||||
this.#db
|
||||
.prepare(
|
||||
`INSERT INTO app_settings (key,value_json,created_at,updated_at)
|
||||
VALUES (?,?,?,?)
|
||||
ON CONFLICT(key) DO UPDATE SET value_json = excluded.value_json,
|
||||
updated_at = excluded.updated_at`,
|
||||
)
|
||||
.run(SETTING_KEY, JSON.stringify(settings), now, now);
|
||||
return settings;
|
||||
}
|
||||
|
||||
get heartbeatMs(): number {
|
||||
return this.get().heartbeatSeconds * 1_000;
|
||||
}
|
||||
|
||||
/** Snapshot lifetime: a probe stays authoritative for one full offline window. */
|
||||
get snapshotTtlMs(): number {
|
||||
return this.get().offlineSeconds * 1_000;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,260 @@
|
||||
import { afterEach, describe, expect, it, vi } from 'vitest';
|
||||
import type { Instance, InstancePage } from '@multi-simadmin/contracts';
|
||||
|
||||
import type { InstanceService } from '../instances/instance-service.js';
|
||||
import type { ConnectionState, ConnectionProbe } from './connection-probe.js';
|
||||
import type { ConnectionSettingsService } from './connection-settings-service.js';
|
||||
import { FleetHeartbeatCoordinator, type FleetStateTransition } from './fleet-heartbeat.js';
|
||||
|
||||
function instance(id: string): Instance {
|
||||
return {
|
||||
id,
|
||||
name: id,
|
||||
origin: `http://10.0.0.1:3000`,
|
||||
authMode: 'none',
|
||||
configRevision: 1,
|
||||
createdAt: '2026-09-05T00:00:00.000Z',
|
||||
updatedAt: '2026-09-05T00:00:00.000Z',
|
||||
} as unknown as Instance;
|
||||
}
|
||||
|
||||
function page(items: readonly Instance[], pageSize: number): InstancePage {
|
||||
return { items: [...items], page: { page: 1, pageSize, total: items.length } };
|
||||
}
|
||||
|
||||
function coordinator(options: {
|
||||
readonly pages: ReadonlyMap<number, readonly Instance[]>;
|
||||
readonly pageSize: number;
|
||||
readonly probe?: (id: string) => Promise<void>;
|
||||
readonly heartbeatMs?: number;
|
||||
readonly concurrency?: number;
|
||||
/** Reachability the snapshot journal already holds when the first beat starts. */
|
||||
readonly before?: ReadonlyMap<string, ConnectionState>;
|
||||
/** Lets a test flip a node to reachable-but-unauthenticated after it answers. */
|
||||
readonly authenticated?: (id: string) => boolean;
|
||||
readonly onTransition?: (transitions: readonly FleetStateTransition[]) => Promise<void>;
|
||||
}) {
|
||||
const listed: string[] = [];
|
||||
const instances = {
|
||||
async list(query: { page?: number; pageSize?: number }) {
|
||||
const pageSize = options.pageSize;
|
||||
const current = query.page ?? 1;
|
||||
listed.push(String(current));
|
||||
return page(options.pages.get(current) ?? [], pageSize);
|
||||
},
|
||||
} as unknown as InstanceService;
|
||||
const probed: string[] = [];
|
||||
const journal = new Map<string, ConnectionState>(options.before ?? []);
|
||||
const probe = {
|
||||
async test(id: string) {
|
||||
probed.push(id);
|
||||
try {
|
||||
await options.probe?.(id);
|
||||
} catch {
|
||||
// A failed probe leaves the previous snapshot in place and lets it expire on its own,
|
||||
// which is exactly what the offline window is for.
|
||||
throw new Error('probe failed');
|
||||
}
|
||||
const authenticated = options.authenticated?.(id) ?? true;
|
||||
journal.set(id, {
|
||||
reachable: true,
|
||||
authenticated,
|
||||
checkedAt: '2026-09-05T00:00:00.000Z',
|
||||
});
|
||||
return {
|
||||
instanceId: id,
|
||||
authenticated,
|
||||
checkedAt: '2026-09-05T00:00:00.000Z',
|
||||
};
|
||||
},
|
||||
reachability: () => journal,
|
||||
} as unknown as ConnectionProbe;
|
||||
const settings = {
|
||||
heartbeatMs: options.heartbeatMs ?? 30_000,
|
||||
} as unknown as ConnectionSettingsService;
|
||||
const beat = new FleetHeartbeatCoordinator({
|
||||
instances,
|
||||
probe,
|
||||
settings,
|
||||
pageSize: options.pageSize,
|
||||
now: () => new Date('2026-09-05T00:00:00.000Z'),
|
||||
...(options.concurrency === undefined ? {} : { concurrency: options.concurrency }),
|
||||
...(options.onTransition === undefined ? {} : { onTransition: options.onTransition }),
|
||||
});
|
||||
return { beat, listed, probed, journal };
|
||||
}
|
||||
|
||||
afterEach(() => {
|
||||
vi.useRealTimers();
|
||||
});
|
||||
|
||||
describe('FleetHeartbeatCoordinator', () => {
|
||||
it('probes every instance once across pages', async () => {
|
||||
const { beat, listed, probed } = coordinator({
|
||||
pageSize: 2,
|
||||
pages: new Map([
|
||||
[1, [instance('a'), instance('b')]],
|
||||
[2, [instance('c')]],
|
||||
]),
|
||||
});
|
||||
await expect(beat.runOnce()).resolves.toEqual({
|
||||
probed: 3,
|
||||
failed: 0,
|
||||
transitions: [],
|
||||
startedAt: '2026-09-05T00:00:00.000Z',
|
||||
finishedAt: '2026-09-05T00:00:00.000Z',
|
||||
});
|
||||
expect(listed).toEqual(['1', '2']);
|
||||
expect([...probed].sort()).toEqual(['a', 'b', 'c']);
|
||||
});
|
||||
|
||||
it('records a baseline on the first beat instead of announcing every node as recovered', async () => {
|
||||
const announced: (readonly FleetStateTransition[])[] = [];
|
||||
const { beat } = coordinator({
|
||||
pageSize: 10,
|
||||
pages: new Map([[1, [instance('a'), instance('b')]]]),
|
||||
onTransition: async (transitions) => {
|
||||
announced.push(transitions);
|
||||
},
|
||||
});
|
||||
await expect(beat.runOnce()).resolves.toMatchObject({ transitions: [] });
|
||||
await expect(beat.runOnce()).resolves.toMatchObject({ transitions: [] });
|
||||
expect(announced).toEqual([]);
|
||||
});
|
||||
|
||||
it('announces a node only once its snapshot has actually expired', async () => {
|
||||
const down: ConnectionState = {
|
||||
reachable: false,
|
||||
authenticated: false,
|
||||
checkedAt: '2026-09-04T00:00:00.000Z',
|
||||
};
|
||||
const up: ConnectionState = {
|
||||
reachable: true,
|
||||
authenticated: true,
|
||||
checkedAt: '2026-09-04T00:00:00.000Z',
|
||||
};
|
||||
const announced: (readonly FleetStateTransition[])[] = [];
|
||||
const { beat, journal } = coordinator({
|
||||
pageSize: 10,
|
||||
concurrency: 1,
|
||||
pages: new Map([[1, [instance('a'), instance('b')]]]),
|
||||
before: new Map([
|
||||
['a', up],
|
||||
['b', down],
|
||||
]),
|
||||
probe: async (id) => {
|
||||
// b never answers; a answers on the first beat and goes quiet afterwards.
|
||||
if (id === 'b' || !journal.get('a')?.reachable) throw new Error('ECONNREFUSED');
|
||||
},
|
||||
onTransition: async (transitions) => {
|
||||
announced.push(transitions);
|
||||
},
|
||||
});
|
||||
|
||||
// a is reachable and b is still down, so nothing moved against the baseline.
|
||||
await expect(beat.runOnce()).resolves.toMatchObject({
|
||||
probed: 1,
|
||||
failed: 1,
|
||||
transitions: [],
|
||||
});
|
||||
|
||||
// a's snapshot ages out. A single lost probe would not be enough on its own; the beat
|
||||
// reports offline only once the offline window the operator configured has passed.
|
||||
journal.set('a', down);
|
||||
await expect(beat.runOnce()).resolves.toMatchObject({
|
||||
transitions: [{ instanceId: 'a', from: 'online', to: 'offline' }],
|
||||
});
|
||||
expect(announced).toHaveLength(1);
|
||||
});
|
||||
|
||||
it('announces an expired device session as a state change', async () => {
|
||||
const announced: (readonly FleetStateTransition[])[] = [];
|
||||
let sessionValid = true;
|
||||
const { beat } = coordinator({
|
||||
pageSize: 10,
|
||||
concurrency: 1,
|
||||
pages: new Map([[1, [instance('a')]]]),
|
||||
authenticated: () => sessionValid,
|
||||
onTransition: async (transitions) => {
|
||||
announced.push(transitions);
|
||||
},
|
||||
});
|
||||
// The first beat only records that the node is healthy.
|
||||
await expect(beat.runOnce()).resolves.toMatchObject({ transitions: [] });
|
||||
sessionValid = false;
|
||||
await expect(beat.runOnce()).resolves.toMatchObject({
|
||||
transitions: [{ instanceId: 'a', from: 'online', to: 'auth-required' }],
|
||||
});
|
||||
expect(announced).toHaveLength(1);
|
||||
});
|
||||
|
||||
it('stops paging on a short page and probes a duplicated id once', async () => {
|
||||
const { beat, listed, probed } = coordinator({
|
||||
pageSize: 2,
|
||||
pages: new Map([[1, [instance('a'), instance('a')]]]),
|
||||
});
|
||||
await beat.runOnce();
|
||||
expect(listed).toEqual(['1', '2']);
|
||||
expect(probed).toEqual(['a']);
|
||||
});
|
||||
|
||||
it('counts a failing device without abandoning the rest of the fleet', async () => {
|
||||
const { beat } = coordinator({
|
||||
pageSize: 10,
|
||||
concurrency: 1,
|
||||
pages: new Map([[1, [instance('a'), instance('b'), instance('c')]]]),
|
||||
probe: async (id) => {
|
||||
if (id === 'b') throw new Error('ECONNREFUSED');
|
||||
},
|
||||
});
|
||||
await expect(beat.runOnce()).resolves.toMatchObject({ probed: 2, failed: 1 });
|
||||
});
|
||||
|
||||
it('shares one pass between concurrent refresh requests', async () => {
|
||||
let release: (() => void) | undefined;
|
||||
const gate = new Promise<void>((resolve) => {
|
||||
release = resolve;
|
||||
});
|
||||
const { beat, probed } = coordinator({
|
||||
pageSize: 10,
|
||||
concurrency: 1,
|
||||
pages: new Map([[1, [instance('a')]]]),
|
||||
probe: () => gate,
|
||||
});
|
||||
const first = beat.runOnce();
|
||||
const second = beat.runOnce();
|
||||
release?.();
|
||||
await Promise.all([first, second]);
|
||||
expect(probed).toEqual(['a']);
|
||||
});
|
||||
|
||||
it('runs a beat on the configured cadence and stops cleanly', async () => {
|
||||
vi.useFakeTimers();
|
||||
const { beat, probed } = coordinator({
|
||||
pageSize: 10,
|
||||
pages: new Map([[1, [instance('a')]]]),
|
||||
heartbeatMs: 5_000,
|
||||
});
|
||||
beat.start();
|
||||
beat.start();
|
||||
await vi.advanceTimersByTimeAsync(5_000);
|
||||
expect(probed).toEqual(['a']);
|
||||
await vi.advanceTimersByTimeAsync(5_000);
|
||||
expect(probed).toEqual(['a', 'a']);
|
||||
beat.stop();
|
||||
await vi.advanceTimersByTimeAsync(60_000);
|
||||
expect(probed).toHaveLength(2);
|
||||
});
|
||||
|
||||
it('refuses an unusable worker or page size', () => {
|
||||
const base = {
|
||||
instances: {} as unknown as InstanceService,
|
||||
probe: {} as unknown as ConnectionProbe,
|
||||
settings: {} as unknown as ConnectionSettingsService,
|
||||
};
|
||||
expect(() => new FleetHeartbeatCoordinator({ ...base, concurrency: 0 })).toThrow(RangeError);
|
||||
expect(() => new FleetHeartbeatCoordinator({ ...base, concurrency: 17 })).toThrow(RangeError);
|
||||
expect(() => new FleetHeartbeatCoordinator({ ...base, pageSize: 0 })).toThrow(RangeError);
|
||||
expect(() => new FleetHeartbeatCoordinator({ ...base, pageSize: 101 })).toThrow(RangeError);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,203 @@
|
||||
import type { ConnectionProbe, ConnectionState } from './connection-probe.js';
|
||||
import type { ConnectionSettingsService } from './connection-settings-service.js';
|
||||
import type { InstanceService } from '../instances/instance-service.js';
|
||||
|
||||
/** Reachability class the console and the notification rules both speak in. */
|
||||
export type FleetObservedState = 'online' | 'auth-required' | 'offline';
|
||||
|
||||
/**
|
||||
* Operator-facing copy shared by the notification templates so every channel reads alike.
|
||||
* `FLEET_STATE_NAMES` describes a steady state, `FLEET_STATE_EVENTS` describes arriving at it.
|
||||
*/
|
||||
export const FLEET_STATE_NAMES: Readonly<Record<FleetObservedState, string>> = Object.freeze({
|
||||
online: '在线',
|
||||
'auth-required': '需要认证',
|
||||
offline: '离线',
|
||||
});
|
||||
|
||||
export const FLEET_STATE_EVENTS: Readonly<Record<FleetObservedState, string>> = Object.freeze({
|
||||
online: '恢复在线',
|
||||
'auth-required': '设备会话已失效',
|
||||
offline: '已离线',
|
||||
});
|
||||
|
||||
export const FLEET_STATE_CODES: Readonly<Record<FleetObservedState, string>> = Object.freeze({
|
||||
online: 'online',
|
||||
'auth-required': 'auth_required',
|
||||
offline: 'offline',
|
||||
});
|
||||
|
||||
export interface FleetStateTransition {
|
||||
readonly instanceId: string;
|
||||
readonly from: FleetObservedState;
|
||||
readonly to: FleetObservedState;
|
||||
}
|
||||
|
||||
export interface FleetHeartbeatSummary {
|
||||
readonly probed: number;
|
||||
readonly failed: number;
|
||||
readonly transitions: readonly FleetStateTransition[];
|
||||
readonly startedAt: string;
|
||||
readonly finishedAt: string;
|
||||
}
|
||||
|
||||
export interface FleetHeartbeatOptions {
|
||||
readonly instances: InstanceService;
|
||||
readonly probe: ConnectionProbe;
|
||||
readonly settings: ConnectionSettingsService;
|
||||
readonly now?: () => Date;
|
||||
/** Devices touched at the same time; the control plane talks to LAN hosts, not the internet. */
|
||||
readonly concurrency?: number;
|
||||
readonly pageSize?: number;
|
||||
/**
|
||||
* Called once per beat with every state change the beat revealed. Reachability is derived
|
||||
* from snapshot expiry, so a single lost probe never fires anything; a node has to stay
|
||||
* down for the whole offline window before it counts as a transition.
|
||||
*/
|
||||
readonly onTransition?: (transitions: readonly FleetStateTransition[]) => Promise<void>;
|
||||
}
|
||||
|
||||
const DEFAULT_CONCURRENCY = 6;
|
||||
const DEFAULT_PAGE_SIZE = 100;
|
||||
const MAX_PAGES = 100;
|
||||
|
||||
function observedState(state: ConnectionState | undefined): FleetObservedState {
|
||||
if (!state?.reachable) return 'offline';
|
||||
return state.authenticated ? 'online' : 'auth-required';
|
||||
}
|
||||
|
||||
/**
|
||||
* Keeps device online state honest without waiting for an operator to open a page: every beat
|
||||
* probes each instance once and lets the snapshot journal carry the result until it expires.
|
||||
*/
|
||||
export class FleetHeartbeatCoordinator {
|
||||
readonly #instances: InstanceService;
|
||||
readonly #probe: ConnectionProbe;
|
||||
readonly #settings: ConnectionSettingsService;
|
||||
readonly #now: () => Date;
|
||||
readonly #concurrency: number;
|
||||
readonly #pageSize: number;
|
||||
readonly #onTransition:
|
||||
| ((transitions: readonly FleetStateTransition[]) => Promise<void>)
|
||||
| undefined;
|
||||
#timer: ReturnType<typeof setTimeout> | undefined;
|
||||
#running: Promise<FleetHeartbeatSummary> | undefined;
|
||||
#stopped = false;
|
||||
/** States seen by the previous beat; undefined until the first beat establishes a baseline. */
|
||||
#previous: ReadonlyMap<string, FleetObservedState> | undefined;
|
||||
|
||||
constructor(options: FleetHeartbeatOptions) {
|
||||
this.#instances = options.instances;
|
||||
this.#probe = options.probe;
|
||||
this.#settings = options.settings;
|
||||
this.#now = options.now ?? (() => new Date());
|
||||
const concurrency = options.concurrency ?? DEFAULT_CONCURRENCY;
|
||||
if (!Number.isSafeInteger(concurrency) || concurrency < 1 || concurrency > 16)
|
||||
throw new RangeError('concurrency must be between 1 and 16');
|
||||
const pageSize = options.pageSize ?? DEFAULT_PAGE_SIZE;
|
||||
if (!Number.isSafeInteger(pageSize) || pageSize < 1 || pageSize > 100)
|
||||
throw new RangeError('pageSize must be between 1 and 100');
|
||||
this.#concurrency = concurrency;
|
||||
this.#pageSize = pageSize;
|
||||
this.#onTransition = options.onTransition;
|
||||
}
|
||||
|
||||
start(): void {
|
||||
this.#stopped = false;
|
||||
if (this.#timer) return;
|
||||
this.#schedule();
|
||||
}
|
||||
|
||||
stop(): void {
|
||||
this.#stopped = true;
|
||||
if (this.#timer) clearTimeout(this.#timer);
|
||||
this.#timer = undefined;
|
||||
}
|
||||
|
||||
/** Runs a beat on demand; concurrent callers share the same pass. */
|
||||
runOnce(): Promise<FleetHeartbeatSummary> {
|
||||
if (this.#running) return this.#running;
|
||||
const pending = this.#beat().finally(() => {
|
||||
if (this.#running === pending) this.#running = undefined;
|
||||
});
|
||||
this.#running = pending;
|
||||
return pending;
|
||||
}
|
||||
|
||||
#schedule(): void {
|
||||
if (this.#stopped) return;
|
||||
const delay = this.#settings.heartbeatMs;
|
||||
this.#timer = setTimeout(() => {
|
||||
this.#timer = undefined;
|
||||
void this.runOnce()
|
||||
.catch(() => undefined)
|
||||
.finally(() => this.#schedule());
|
||||
}, delay);
|
||||
this.#timer.unref?.();
|
||||
}
|
||||
|
||||
async #beat(): Promise<FleetHeartbeatSummary> {
|
||||
const startedAt = this.#now().toISOString();
|
||||
const ids: string[] = [];
|
||||
for (let page = 1; page <= MAX_PAGES; page += 1) {
|
||||
const current = await this.#instances.list({ page, pageSize: this.#pageSize });
|
||||
for (const instance of current.items) if (!ids.includes(instance.id)) ids.push(instance.id);
|
||||
if (current.items.length < this.#pageSize) break;
|
||||
}
|
||||
const before = this.#previous;
|
||||
let probed = 0;
|
||||
let failed = 0;
|
||||
let cursor = 0;
|
||||
const workers = Array.from(
|
||||
{ length: Math.min(this.#concurrency, ids.length) },
|
||||
async (): Promise<void> => {
|
||||
for (;;) {
|
||||
const index = cursor;
|
||||
cursor += 1;
|
||||
if (index >= ids.length) return;
|
||||
try {
|
||||
await this.#probe.test(ids[index] as string);
|
||||
probed += 1;
|
||||
} catch {
|
||||
failed += 1;
|
||||
}
|
||||
}
|
||||
},
|
||||
);
|
||||
await Promise.all(workers);
|
||||
const transitions = this.#diff(before, ids);
|
||||
if (transitions.length > 0) await this.#onTransition?.(transitions);
|
||||
return { probed, failed, transitions, startedAt, finishedAt: this.#now().toISOString() };
|
||||
}
|
||||
|
||||
/** Snapshot of the given instances, including ones that have never been probed. */
|
||||
#states(ids: readonly string[]): ReadonlyMap<string, FleetObservedState> {
|
||||
const reachability = this.#probe.reachability();
|
||||
const states = new Map<string, FleetObservedState>();
|
||||
for (const id of ids) {
|
||||
const state = reachability.get(id);
|
||||
states.set(id, observedState(state));
|
||||
}
|
||||
return states;
|
||||
}
|
||||
|
||||
/**
|
||||
* The first beat only records a baseline. Without one, every instance would start as
|
||||
* "offline" and then announce itself as recovered, so a restart would flood the queue.
|
||||
*/
|
||||
#diff(
|
||||
before: ReadonlyMap<string, FleetObservedState> | undefined,
|
||||
ids: readonly string[],
|
||||
): readonly FleetStateTransition[] {
|
||||
const after = this.#states(ids);
|
||||
this.#previous = after;
|
||||
if (before === undefined) return [];
|
||||
const transitions: FleetStateTransition[] = [];
|
||||
for (const [id, to] of after) {
|
||||
const from = before.get(id);
|
||||
if (from === undefined || from === to) continue;
|
||||
transitions.push({ instanceId: id, from, to });
|
||||
}
|
||||
return transitions;
|
||||
}
|
||||
}
|
||||
@@ -27,6 +27,7 @@ describe('InstanceCredentialResolver', () => {
|
||||
it('resolves only the canonical password reference for the requested existing instance', async () => {
|
||||
let asked: string | undefined;
|
||||
const { db, resolver } = fixture({
|
||||
provider: 'macos-keychain',
|
||||
set: async () => '',
|
||||
get: async (reference) => {
|
||||
asked = reference;
|
||||
@@ -44,6 +45,7 @@ describe('InstanceCredentialResolver', () => {
|
||||
it('does not call SecretStore when no saved credential exists', async () => {
|
||||
let calls = 0;
|
||||
const { resolver } = fixture({
|
||||
provider: 'macos-keychain',
|
||||
set: async () => '',
|
||||
get: async () => {
|
||||
calls += 1;
|
||||
@@ -57,6 +59,7 @@ describe('InstanceCredentialResolver', () => {
|
||||
it('rejects a reference bound to another instance before SecretStore access', async () => {
|
||||
let calls = 0;
|
||||
const { db, resolver } = fixture({
|
||||
provider: 'macos-keychain',
|
||||
set: async () => '',
|
||||
get: async () => {
|
||||
calls += 1;
|
||||
|
||||
@@ -1,9 +1,8 @@
|
||||
import type Database from 'better-sqlite3';
|
||||
import { parseKeychainReference } from '../../infrastructure/secrets/keychain-secret-store.js';
|
||||
import { parseSecretReference } from '../../infrastructure/secrets/secret-reference.js';
|
||||
import type { SecretStore } from '../../infrastructure/secrets/secret-store.js';
|
||||
|
||||
const PURPOSE = 'instance-password';
|
||||
const PROVIDER = 'macos-keychain';
|
||||
export class CredentialResolverError extends Error {
|
||||
constructor(readonly code: 'NOT_FOUND' | 'CREDENTIAL_UNAVAILABLE') {
|
||||
super(code);
|
||||
@@ -24,10 +23,12 @@ export class InstanceCredentialResolver {
|
||||
.prepare(
|
||||
'SELECT external_reference FROM secret_references WHERE instance_id=? AND purpose=? AND provider=?',
|
||||
)
|
||||
.get(instanceId, PURPOSE, PROVIDER) as { external_reference: string } | undefined;
|
||||
.get(instanceId, PURPOSE, this.options.store.provider) as
|
||||
| { external_reference: string }
|
||||
| undefined;
|
||||
if (!row) throw new CredentialResolverError('CREDENTIAL_UNAVAILABLE');
|
||||
try {
|
||||
const parsed = parseKeychainReference(row.external_reference);
|
||||
const parsed = parseSecretReference(row.external_reference);
|
||||
if (parsed.instanceId !== instanceId || parsed.purpose !== PURPOSE || !parsed.slot)
|
||||
throw new Error('invalid binding');
|
||||
const secret = await this.options.store.get(row.external_reference);
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
export interface UpstreamRequest {
|
||||
readonly url: string;
|
||||
readonly method: 'GET' | 'POST';
|
||||
readonly method: 'GET' | 'POST' | 'DELETE';
|
||||
readonly headers: Readonly<Record<string, string>>;
|
||||
/** One-shot secret. Request implementations must not log or persist this field. */
|
||||
readonly secret?: string;
|
||||
@@ -8,6 +8,13 @@ export interface UpstreamRequest {
|
||||
readonly body?: '[REDACTED]';
|
||||
/** Explicit one-shot SMS action payload. Implementations must not log or persist it. */
|
||||
readonly sms?: { readonly phoneNumber: string; readonly content: string };
|
||||
/** Explicit one-shot SMS batch-delete payload. Implementations must not log or persist it. */
|
||||
readonly smsBatchDelete?: { readonly ids: readonly number[] };
|
||||
/**
|
||||
* Explicit one-shot device action payload for an allowlisted device mutation. The gateway
|
||||
* re-validates and canonically serializes it; implementations must not log or persist it.
|
||||
*/
|
||||
readonly deviceAction?: { readonly body: Readonly<Record<string, unknown>> | undefined };
|
||||
}
|
||||
export interface UpstreamResponse {
|
||||
readonly status: number;
|
||||
|
||||
@@ -4,7 +4,8 @@ import { describe, expect, it, vi } from 'vitest';
|
||||
import { EventJournal, formatServerSentEvent, type EventEnvelope } from './event-journal.js';
|
||||
import { migrateDatabase } from '../../infrastructure/database/migrations.js';
|
||||
|
||||
const at = '2026-07-17T12:34:56.789Z';
|
||||
// Journal retention prunes envelopes older than a day, so fixtures stay clock-relative.
|
||||
const at = new Date(Date.now() - 60_000).toISOString();
|
||||
|
||||
function event(kind: EventEnvelope['kind'], id = `${kind}-1`): EventEnvelope {
|
||||
const common = {
|
||||
|
||||
@@ -185,7 +185,6 @@ export class EventJournal {
|
||||
const sequence = Number(result.lastInsertRowid);
|
||||
if (!Number.isSafeInteger(sequence))
|
||||
throw new Error('event journal sequence is not a safe integer');
|
||||
this.#prune();
|
||||
const appended = { sequence, envelope };
|
||||
for (const subscriber of [...this.#subscribers]) subscriber(appended);
|
||||
return appended;
|
||||
@@ -236,7 +235,11 @@ export class EventJournal {
|
||||
};
|
||||
}
|
||||
|
||||
#prune(): void {
|
||||
/**
|
||||
* Maintenance hook for a periodic timer. Pruning per append turned every
|
||||
* insert into an unindexable full-table json_extract scan.
|
||||
*/
|
||||
prune(): void {
|
||||
const cutoff = new Date(Date.now() - this.#retentionMs).toISOString();
|
||||
this.#database
|
||||
.prepare("DELETE FROM event_journal WHERE json_extract(envelope_json, '$.occurredAt') < ?")
|
||||
|
||||
@@ -0,0 +1,245 @@
|
||||
import Database from 'better-sqlite3';
|
||||
import { describe, expect, it } from 'vitest';
|
||||
|
||||
import { migrateDatabase } from '../../infrastructure/database/migrations.js';
|
||||
import { DeviceIdentityError, DeviceIdentityService } from './device-identity-service.js';
|
||||
|
||||
interface Fixture {
|
||||
readonly db: Database.Database;
|
||||
readonly service: DeviceIdentityService;
|
||||
readonly advance: (minutes: number) => void;
|
||||
}
|
||||
|
||||
function fixture(): Fixture {
|
||||
const db = new Database(':memory:');
|
||||
db.pragma('foreign_keys=ON');
|
||||
migrateDatabase(db);
|
||||
let clock = new Date('2026-09-05T02:00:00.000Z');
|
||||
const service = new DeviceIdentityService({ db, now: () => clock });
|
||||
return {
|
||||
db,
|
||||
service,
|
||||
advance: (minutes: number) => {
|
||||
clock = new Date(clock.getTime() + minutes * 60_000);
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
function manualFixture() {
|
||||
const db = new Database(':memory:');
|
||||
db.pragma('foreign_keys=ON');
|
||||
migrateDatabase(db);
|
||||
let clock = new Date('2026-09-05T02:00:00.000Z');
|
||||
const service = new DeviceIdentityService({
|
||||
db,
|
||||
now: () => clock,
|
||||
authorizationMode: () => 'manual',
|
||||
});
|
||||
return {
|
||||
db,
|
||||
service,
|
||||
advance: (minutes: number) => {
|
||||
clock = new Date(clock.getTime() + minutes * 60_000);
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
function register(db: Database.Database, id: string): void {
|
||||
db.prepare(
|
||||
`INSERT INTO instances
|
||||
(id,name,base_url,auth_mode,enabled,config_revision,created_at,updated_at)
|
||||
VALUES (?,?,?,'password',1,1,?,?)`,
|
||||
).run(id, id, `http://${id}.lan:8080`, '2026-09-05T02:00:00.000Z', '2026-09-05T02:00:00.000Z');
|
||||
}
|
||||
|
||||
const device = (overrides: Record<string, unknown> = {}) => ({
|
||||
imei: '860000000000001',
|
||||
manufacturer: 'Quectel',
|
||||
model: 'RM500Q',
|
||||
revision: 'RM500QEAAAR13',
|
||||
...overrides,
|
||||
});
|
||||
|
||||
describe('DeviceIdentityService', () => {
|
||||
it('baselines the first report instead of calling it a conflict', () => {
|
||||
const { db, service } = fixture();
|
||||
register(db, 'a');
|
||||
|
||||
const identity = service.observe('a', { ...device(), origin: 'http://a.lan:8080' });
|
||||
|
||||
expect(identity).toMatchObject({
|
||||
instanceId: 'a',
|
||||
status: 'confirmed',
|
||||
reasons: [],
|
||||
imei: '860000000000001',
|
||||
model: 'RM500Q',
|
||||
origin: 'http://a.lan:8080',
|
||||
});
|
||||
expect(service.isBlocked('a')).toBe(false);
|
||||
});
|
||||
|
||||
it('keeps a device that reports no identity out of the guard', () => {
|
||||
const { db, service } = fixture();
|
||||
register(db, 'a');
|
||||
|
||||
expect(service.observe('a', { agent: 'simadmin-agent 1.9.6' })).toBeUndefined();
|
||||
expect(service.observe('a', { imei: ' ' })).toBeUndefined();
|
||||
expect(service.list()).toEqual([]);
|
||||
});
|
||||
|
||||
it('holds a first report until an operator confirms it in manual authorization mode', () => {
|
||||
const { db, service } = manualFixture();
|
||||
register(db, 'a');
|
||||
|
||||
const identity = service.observe('a', device());
|
||||
|
||||
expect(identity).toMatchObject({
|
||||
instanceId: 'a',
|
||||
status: 'pending',
|
||||
reasons: [],
|
||||
imei: '860000000000001',
|
||||
});
|
||||
expect(service.isBlocked('a')).toBe(true);
|
||||
expect(service.confirm('a')).toMatchObject({ status: 'confirmed', reasons: [] });
|
||||
expect(service.isBlocked('a')).toBe(false);
|
||||
});
|
||||
|
||||
it('holds a node whose hardware no longer matches the record', () => {
|
||||
const { db, service } = fixture();
|
||||
register(db, 'a');
|
||||
service.observe('a', device());
|
||||
|
||||
const swapped = service.observe('a', device({ model: 'RG500Q', imei: '860000000000009' }));
|
||||
|
||||
expect(swapped?.status).toBe('pending');
|
||||
expect(swapped?.reasons).toEqual(['hardware_swapped']);
|
||||
expect(swapped?.changes).toEqual([
|
||||
{ field: 'imei', from: '860000000000001', to: '860000000000009' },
|
||||
{ field: 'model', from: 'RM500Q', to: 'RG500Q' },
|
||||
]);
|
||||
expect(service.isBlocked('a')).toBe(true);
|
||||
});
|
||||
|
||||
it('compares only the fields a partial report actually carries', () => {
|
||||
const { db, service } = fixture();
|
||||
register(db, 'a');
|
||||
service.observe('a', device());
|
||||
|
||||
const partial = service.observe('a', { model: 'RM500Q' });
|
||||
|
||||
expect(partial?.status).toBe('confirmed');
|
||||
expect(partial?.reasons).toEqual([]);
|
||||
// The dropped fields stay on the row: a silent read must not erase what we already knew.
|
||||
expect(partial?.imei).toBe('860000000000001');
|
||||
});
|
||||
|
||||
it('asks both records when one identity is claimed twice', () => {
|
||||
const { db, service } = fixture();
|
||||
register(db, 'a');
|
||||
register(db, 'b');
|
||||
service.observe('a', device());
|
||||
|
||||
const newcomer = service.observe('b', device());
|
||||
|
||||
expect(newcomer?.status).toBe('pending');
|
||||
expect(newcomer?.reasons).toEqual(['imei_claimed']);
|
||||
expect(newcomer?.conflicts).toEqual([{ reason: 'imei_claimed', instanceIds: ['a'] }]);
|
||||
expect(service.get('a')?.status).toBe('pending');
|
||||
expect(service.get('a')?.conflicts).toEqual([{ reason: 'imei_claimed', instanceIds: ['b'] }]);
|
||||
expect(service.summarize()).toEqual({ tracked: 2, pending: ['a', 'b'] });
|
||||
});
|
||||
|
||||
it('accepts the twins that were on screen when the operator confirmed', () => {
|
||||
const { db, service } = fixture();
|
||||
register(db, 'a');
|
||||
register(db, 'b');
|
||||
service.observe('a', device());
|
||||
service.observe('b', device());
|
||||
|
||||
expect(service.confirm('a')).toMatchObject({ status: 'confirmed', reasons: [] });
|
||||
service.observe('a', device());
|
||||
expect(service.get('a')?.status).toBe('confirmed');
|
||||
|
||||
// A third claimant is new information, so the guard comes back down.
|
||||
register(db, 'c');
|
||||
service.observe('c', device());
|
||||
expect(service.get('a')?.status).toBe('pending');
|
||||
// 'b' was on screen when the operator confirmed; only the newcomer is still a question.
|
||||
expect(service.get('a')?.conflicts).toEqual([{ reason: 'imei_claimed', instanceIds: ['c'] }]);
|
||||
});
|
||||
|
||||
it('re-baselines swapped hardware on confirmation', () => {
|
||||
const { db, service } = fixture();
|
||||
register(db, 'a');
|
||||
service.observe('a', device());
|
||||
service.observe('a', device({ model: 'RG500Q' }));
|
||||
|
||||
const confirmed = service.confirm('a');
|
||||
|
||||
expect(confirmed).toMatchObject({ status: 'confirmed', reasons: [], changes: [] });
|
||||
expect(service.isBlocked('a')).toBe(false);
|
||||
service.observe('a', device({ model: 'RG500Q' }));
|
||||
expect(service.get('a')?.status).toBe('confirmed');
|
||||
service.observe('a', device({ model: 'EM120' }));
|
||||
expect(service.get('a')?.reasons).toEqual(['hardware_swapped']);
|
||||
});
|
||||
|
||||
it('sanitizes device-controlled evidence before it is stored', () => {
|
||||
const { db, service } = fixture();
|
||||
register(db, 'a');
|
||||
|
||||
const identity = service.observe('a', {
|
||||
imei: ' 860000000000001\u0000\u001b ',
|
||||
model: 'x'.repeat(200),
|
||||
revision: 42,
|
||||
manufacturer: null,
|
||||
});
|
||||
|
||||
expect(identity?.imei).toBe('860000000000001');
|
||||
expect(identity?.model).toHaveLength(64);
|
||||
expect(identity?.revision).toBe('42');
|
||||
expect(identity?.manufacturer).toBe('');
|
||||
});
|
||||
|
||||
it('reports a missing identity rather than inventing one on confirm', () => {
|
||||
const { db, service } = fixture();
|
||||
register(db, 'a');
|
||||
|
||||
expect(() => service.confirm('a')).toThrow(DeviceIdentityError);
|
||||
try {
|
||||
service.confirm('a');
|
||||
} catch (error) {
|
||||
expect(error).toBeInstanceOf(DeviceIdentityError);
|
||||
expect((error as DeviceIdentityError).code).toBe('NOT_FOUND');
|
||||
}
|
||||
});
|
||||
|
||||
it('forgets the identity with the node it belonged to', () => {
|
||||
const { db, service } = fixture();
|
||||
register(db, 'a');
|
||||
register(db, 'b');
|
||||
service.observe('a', device());
|
||||
service.observe('b', device());
|
||||
|
||||
db.prepare('DELETE FROM instances WHERE id=?').run('b');
|
||||
|
||||
// The row goes with the node, and the survivor is released the next time anything reads it.
|
||||
expect(service.list().map((identity) => identity.instanceId)).toEqual(['a']);
|
||||
expect(service.get('a')).toMatchObject({ status: 'pending' });
|
||||
expect(service.observe('a', device())).toMatchObject({ status: 'confirmed', reasons: [] });
|
||||
});
|
||||
|
||||
it('advances the observation time without touching the confirmation time', () => {
|
||||
const { db, service, advance } = fixture();
|
||||
register(db, 'a');
|
||||
const first = service.observe('a', device());
|
||||
expect(first?.observedAt).toBe('2026-09-05T02:00:00.000Z');
|
||||
|
||||
advance(5);
|
||||
const second = service.observe('a', device());
|
||||
|
||||
expect(second?.observedAt).toBe('2026-09-05T02:05:00.000Z');
|
||||
expect(second?.confirmedAt).toBe('2026-09-05T02:00:00.000Z');
|
||||
expect(second?.fingerprint).toBe(first?.fingerprint);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,461 @@
|
||||
import { createHash } from 'node:crypto';
|
||||
|
||||
import type Database from 'better-sqlite3';
|
||||
|
||||
export type IdentityStatus = 'confirmed' | 'pending';
|
||||
|
||||
/**
|
||||
* `hardware_swapped` is the record that used to answer at this address reporting different
|
||||
* hardware now; `imei_claimed` is a second record answering with the same identity. Neither can
|
||||
* be resolved from here, which is why both stop control traffic until an operator says which
|
||||
* device is the real one.
|
||||
*/
|
||||
export type IdentityReason = 'hardware_swapped' | 'imei_claimed';
|
||||
|
||||
const IDENTITY_FIELDS = ['imei', 'manufacturer', 'model', 'revision'] as const;
|
||||
type IdentityField = (typeof IDENTITY_FIELDS)[number];
|
||||
type IdentityValues = Partial<Record<IdentityField, string>>;
|
||||
|
||||
const MAX_FIELD_LENGTH: Readonly<Record<IdentityField | 'agent' | 'origin', number>> = {
|
||||
imei: 32,
|
||||
manufacturer: 64,
|
||||
model: 64,
|
||||
revision: 64,
|
||||
agent: 64,
|
||||
origin: 256,
|
||||
};
|
||||
|
||||
const CONTROL_CHARACTERS = /[\u0000-\u0008\u000b\u000c\u000e-\u001f\u007f]/gu;
|
||||
|
||||
export interface DeviceIdentityEvidence {
|
||||
readonly imei?: unknown;
|
||||
readonly manufacturer?: unknown;
|
||||
readonly model?: unknown;
|
||||
readonly revision?: unknown;
|
||||
readonly agent?: unknown;
|
||||
readonly origin?: unknown;
|
||||
}
|
||||
|
||||
export interface DeviceIdentityConflict {
|
||||
readonly reason: IdentityReason;
|
||||
readonly instanceIds: readonly string[];
|
||||
}
|
||||
|
||||
export interface DeviceIdentityFieldChange {
|
||||
readonly field: IdentityField;
|
||||
readonly from: string;
|
||||
readonly to: string;
|
||||
}
|
||||
|
||||
export interface DeviceIdentity {
|
||||
readonly instanceId: string;
|
||||
readonly status: IdentityStatus;
|
||||
readonly reasons: readonly IdentityReason[];
|
||||
readonly conflicts: readonly DeviceIdentityConflict[];
|
||||
readonly imei: string;
|
||||
readonly manufacturer: string;
|
||||
readonly model: string;
|
||||
readonly revision: string;
|
||||
readonly agent: string;
|
||||
readonly origin: string;
|
||||
readonly fingerprint: string;
|
||||
readonly confirmedFingerprint: string;
|
||||
readonly changes: readonly DeviceIdentityFieldChange[];
|
||||
readonly observedAt: string;
|
||||
readonly confirmedAt: string;
|
||||
}
|
||||
|
||||
export type DeviceIdentityErrorCode = 'NOT_FOUND' | 'VALIDATION_FAILED';
|
||||
|
||||
export class DeviceIdentityError extends Error {
|
||||
constructor(
|
||||
readonly code: DeviceIdentityErrorCode,
|
||||
message: string,
|
||||
) {
|
||||
super(message);
|
||||
this.name = 'DeviceIdentityError';
|
||||
}
|
||||
}
|
||||
|
||||
export interface DeviceIdentityOptions {
|
||||
readonly db: Database.Database;
|
||||
readonly now?: () => Date;
|
||||
/** Follows the connection setting: manual authorization makes a first report wait. */
|
||||
readonly authorizationMode?: () => 'auto' | 'manual';
|
||||
}
|
||||
|
||||
interface IdentityRow {
|
||||
instance_id: string;
|
||||
status: string;
|
||||
reasons: string;
|
||||
imei: string;
|
||||
manufacturer: string;
|
||||
model: string;
|
||||
revision: string;
|
||||
agent: string;
|
||||
origin: string;
|
||||
fingerprint: string;
|
||||
confirmed_fingerprint: string;
|
||||
confirmed_values_json: string;
|
||||
confirmed_peers_json: string;
|
||||
detail_json: string;
|
||||
observed_at: string;
|
||||
confirmed_at: string;
|
||||
}
|
||||
|
||||
/** Device payloads are untrusted: only a trimmed, control-free, length-capped string is kept. */
|
||||
function text(value: unknown, limit: number): string {
|
||||
if (typeof value !== 'string' && typeof value !== 'number') return '';
|
||||
const raw = typeof value === 'number' ? String(value) : value;
|
||||
return raw.replace(CONTROL_CHARACTERS, '').trim().slice(0, limit);
|
||||
}
|
||||
|
||||
function identityValues(evidence: DeviceIdentityEvidence): IdentityValues {
|
||||
const values: IdentityValues = {};
|
||||
for (const field of IDENTITY_FIELDS) {
|
||||
const value = text(evidence[field], MAX_FIELD_LENGTH[field]);
|
||||
if (value) values[field] = value;
|
||||
}
|
||||
return values;
|
||||
}
|
||||
|
||||
function digest(values: IdentityValues): string {
|
||||
const canonical = IDENTITY_FIELDS.filter((field) => values[field] !== undefined)
|
||||
.map((field) => `${field}=${values[field]}`)
|
||||
.join('\n');
|
||||
return createHash('sha256').update(canonical, 'utf8').digest('hex').slice(0, 16);
|
||||
}
|
||||
|
||||
function stringArray(value: unknown): string[] {
|
||||
return Array.isArray(value)
|
||||
? value.filter((item): item is string => typeof item === 'string')
|
||||
: [];
|
||||
}
|
||||
|
||||
function parseValues(value: string): IdentityValues {
|
||||
try {
|
||||
const parsed: unknown = JSON.parse(value);
|
||||
if (parsed === null || typeof parsed !== 'object' || Array.isArray(parsed)) return {};
|
||||
const record = parsed as Record<string, unknown>;
|
||||
const values: IdentityValues = {};
|
||||
for (const field of IDENTITY_FIELDS) {
|
||||
const entry = text(record[field], MAX_FIELD_LENGTH[field]);
|
||||
if (entry) values[field] = entry;
|
||||
}
|
||||
return values;
|
||||
} catch {
|
||||
return {};
|
||||
}
|
||||
}
|
||||
|
||||
/** The evidence the console shows: who else claims this identity and what changed. */
|
||||
interface IdentityDetail {
|
||||
readonly peers: readonly string[];
|
||||
readonly changes: readonly DeviceIdentityFieldChange[];
|
||||
}
|
||||
|
||||
function parseDetail(value: string): IdentityDetail {
|
||||
let parsed: unknown = {};
|
||||
try {
|
||||
parsed = JSON.parse(value);
|
||||
} catch {
|
||||
return { peers: [], changes: [] };
|
||||
}
|
||||
if (parsed === null || typeof parsed !== 'object' || Array.isArray(parsed))
|
||||
return { peers: [], changes: [] };
|
||||
const record = parsed as Record<string, unknown>;
|
||||
const entries = Array.isArray(record.changes) ? record.changes : [];
|
||||
const changes = entries.filter(
|
||||
(entry): entry is DeviceIdentityFieldChange =>
|
||||
entry !== null &&
|
||||
typeof entry === 'object' &&
|
||||
!Array.isArray(entry) &&
|
||||
IDENTITY_FIELDS.includes((entry as Record<string, unknown>).field as IdentityField) &&
|
||||
typeof (entry as Record<string, unknown>).from === 'string' &&
|
||||
typeof (entry as Record<string, unknown>).to === 'string',
|
||||
);
|
||||
return { peers: stringArray(record.peers), changes };
|
||||
}
|
||||
|
||||
function parseReasons(value: string): IdentityReason[] {
|
||||
const allowed: IdentityReason[] = ['hardware_swapped', 'imei_claimed'];
|
||||
try {
|
||||
const parsed: unknown = JSON.parse(value);
|
||||
if (!Array.isArray(parsed)) return [];
|
||||
return parsed.filter((item): item is IdentityReason =>
|
||||
allowed.includes(item as IdentityReason),
|
||||
);
|
||||
} catch {
|
||||
return [];
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Tracks the hardware behind each registered node and holds the control plane still when that
|
||||
* hardware stops matching the record. Observations come from device reads the console is making
|
||||
* anyway, so a node only gains an identity row once something has actually looked at it.
|
||||
*/
|
||||
export class DeviceIdentityService {
|
||||
readonly #db: Database.Database;
|
||||
readonly #clock: () => Date;
|
||||
readonly #authorizationMode: () => 'auto' | 'manual';
|
||||
readonly #select: Database.Statement;
|
||||
readonly #selectPeers: Database.Statement;
|
||||
readonly #insert: Database.Statement;
|
||||
readonly #update: Database.Statement;
|
||||
|
||||
constructor(options: DeviceIdentityOptions) {
|
||||
this.#db = options.db;
|
||||
this.#clock = options.now ?? (() => new Date());
|
||||
this.#authorizationMode = options.authorizationMode ?? (() => 'auto');
|
||||
this.#select = options.db.prepare('SELECT * FROM device_identities WHERE instance_id=?');
|
||||
this.#selectPeers = options.db.prepare(
|
||||
"SELECT instance_id FROM device_identities WHERE imei=? AND instance_id<>? AND imei<>'' ORDER BY instance_id",
|
||||
);
|
||||
this.#insert = options.db.prepare(
|
||||
`INSERT INTO device_identities (instance_id,status,reasons,imei,manufacturer,model,revision,
|
||||
agent,origin,fingerprint,confirmed_fingerprint,confirmed_values_json,confirmed_peers_json,
|
||||
detail_json,observed_at,confirmed_at,created_at,updated_at)
|
||||
VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)`,
|
||||
);
|
||||
this.#update = options.db.prepare(
|
||||
`UPDATE device_identities SET status=?,reasons=?,imei=?,manufacturer=?,model=?,revision=?,
|
||||
agent=?,origin=?,fingerprint=?,confirmed_fingerprint=?,confirmed_values_json=?,
|
||||
confirmed_peers_json=?,detail_json=?,observed_at=?,confirmed_at=?,updated_at=?
|
||||
WHERE instance_id=?`,
|
||||
);
|
||||
}
|
||||
|
||||
/**
|
||||
* Records what a device just said about itself. Returns undefined when the payload carried no
|
||||
* identity at all, which keeps a device that does not report its IMEI out of the guard rather
|
||||
* than inventing a fingerprint from nothing.
|
||||
*/
|
||||
observe(instanceId: string, evidence: DeviceIdentityEvidence): DeviceIdentity | undefined {
|
||||
if (typeof instanceId !== 'string' || !instanceId) return undefined;
|
||||
const values = identityValues(evidence);
|
||||
const agent = text(evidence.agent, MAX_FIELD_LENGTH.agent);
|
||||
const origin = text(evidence.origin, MAX_FIELD_LENGTH.origin);
|
||||
if (Object.keys(values).length === 0) return undefined;
|
||||
const now = this.#clock().toISOString();
|
||||
const fingerprint = digest(values);
|
||||
const manualFirstReport = this.#authorizationMode() === 'manual';
|
||||
this.#db.transaction(() => {
|
||||
const existing = this.#select.get(instanceId) as IdentityRow | undefined;
|
||||
if (!existing) {
|
||||
this.#insert.run(
|
||||
instanceId,
|
||||
manualFirstReport ? 'pending' : 'confirmed',
|
||||
'[]',
|
||||
values.imei ?? '',
|
||||
values.manufacturer ?? '',
|
||||
values.model ?? '',
|
||||
values.revision ?? '',
|
||||
agent,
|
||||
origin,
|
||||
fingerprint,
|
||||
fingerprint,
|
||||
JSON.stringify(values),
|
||||
'[]',
|
||||
'{"peers":[]}',
|
||||
now,
|
||||
now,
|
||||
now,
|
||||
now,
|
||||
);
|
||||
// A first report is a baseline, but it can still walk into a twin that is already
|
||||
// registered, so the new row goes through the same evaluation as every other read.
|
||||
const inserted = this.#select.get(instanceId) as IdentityRow;
|
||||
this.#write(inserted, values, agent, origin, fingerprint, now, now, manualFirstReport);
|
||||
} else {
|
||||
this.#write(existing, values, agent, origin, fingerprint, now, now);
|
||||
}
|
||||
this.#syncClaims(instanceId, values.imei ?? '', now);
|
||||
})();
|
||||
return this.get(instanceId);
|
||||
}
|
||||
|
||||
list(): readonly DeviceIdentity[] {
|
||||
const rows = this.#db
|
||||
.prepare('SELECT * FROM device_identities ORDER BY status DESC, instance_id')
|
||||
.all() as IdentityRow[];
|
||||
return rows.map((row) => this.#present(row));
|
||||
}
|
||||
|
||||
get(instanceId: string): DeviceIdentity | undefined {
|
||||
const row = this.#select.get(instanceId) as IdentityRow | undefined;
|
||||
return row ? this.#present(row) : undefined;
|
||||
}
|
||||
|
||||
/** Control operations only; reading a device is always allowed, unbinding is the way out. */
|
||||
isBlocked(instanceId: string): boolean {
|
||||
const row = this.#select.get(instanceId) as IdentityRow | undefined;
|
||||
return row?.status === 'pending';
|
||||
}
|
||||
|
||||
summarize(): { readonly tracked: number; readonly pending: readonly string[] } {
|
||||
const rows = this.#db
|
||||
.prepare('SELECT instance_id,status FROM device_identities ORDER BY instance_id')
|
||||
.all() as Array<{ instance_id: string; status: string }>;
|
||||
return {
|
||||
tracked: rows.length,
|
||||
pending: rows.filter((row) => row.status === 'pending').map((row) => row.instance_id),
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* The operator says "this is the device I mean". The current report becomes the baseline and
|
||||
* the other claimants that were visible at that moment are accepted, so a known twin does not
|
||||
* reopen the question on the next read while a third one still does.
|
||||
*/
|
||||
confirm(instanceId: string): DeviceIdentity {
|
||||
const row = this.#select.get(instanceId) as IdentityRow | undefined;
|
||||
if (!row) throw new DeviceIdentityError('NOT_FOUND', 'Device identity was not found');
|
||||
const now = this.#clock().toISOString();
|
||||
const values = identityValues({
|
||||
imei: row.imei,
|
||||
manufacturer: row.manufacturer,
|
||||
model: row.model,
|
||||
revision: row.revision,
|
||||
});
|
||||
const fingerprint = row.fingerprint || digest(values);
|
||||
this.#db.transaction(() => {
|
||||
this.#update.run(
|
||||
'confirmed',
|
||||
'[]',
|
||||
row.imei,
|
||||
row.manufacturer,
|
||||
row.model,
|
||||
row.revision,
|
||||
row.agent,
|
||||
row.origin,
|
||||
fingerprint,
|
||||
fingerprint,
|
||||
JSON.stringify(values),
|
||||
JSON.stringify(this.#peers(instanceId, row.imei)),
|
||||
'{"peers":[]}',
|
||||
row.observed_at,
|
||||
now,
|
||||
now,
|
||||
instanceId,
|
||||
);
|
||||
this.#syncClaims(instanceId, row.imei, now);
|
||||
})();
|
||||
const updated = this.#select.get(instanceId) as IdentityRow | undefined;
|
||||
if (!updated) throw new DeviceIdentityError('NOT_FOUND', 'Device identity was not found');
|
||||
return this.#present(updated);
|
||||
}
|
||||
|
||||
#peers(instanceId: string, imei: string): string[] {
|
||||
if (!imei) return [];
|
||||
return (this.#selectPeers.all(imei, instanceId) as Array<{ instance_id: string }>).map(
|
||||
(row) => row.instance_id,
|
||||
);
|
||||
}
|
||||
|
||||
/** Re-evaluated on every read, so a conflict that walks away stops holding the node. */
|
||||
#write(
|
||||
existing: IdentityRow,
|
||||
values: IdentityValues,
|
||||
agent: string,
|
||||
origin: string,
|
||||
fingerprint: string,
|
||||
observedAt: string,
|
||||
updatedAt: string,
|
||||
forcePending = false,
|
||||
): void {
|
||||
const confirmed = parseValues(existing.confirmed_values_json);
|
||||
const accepted = parseStringArray(existing.confirmed_peers_json);
|
||||
const changes: DeviceIdentityFieldChange[] = [];
|
||||
for (const field of IDENTITY_FIELDS) {
|
||||
const before = confirmed[field];
|
||||
const after = values[field];
|
||||
if (before !== undefined && after !== undefined && before !== after)
|
||||
changes.push({ field, from: before, to: after });
|
||||
}
|
||||
const peers = this.#peers(existing.instance_id, values.imei ?? '');
|
||||
const unaccepted = peers.filter((peer) => !accepted.includes(peer));
|
||||
const reasons: IdentityReason[] = [];
|
||||
if (changes.length > 0) reasons.push('hardware_swapped');
|
||||
if (unaccepted.length > 0) reasons.push('imei_claimed');
|
||||
this.#update.run(
|
||||
reasons.length > 0 || forcePending ? 'pending' : 'confirmed',
|
||||
JSON.stringify(reasons),
|
||||
values.imei ?? existing.imei,
|
||||
values.manufacturer ?? existing.manufacturer,
|
||||
values.model ?? existing.model,
|
||||
values.revision ?? existing.revision,
|
||||
agent || existing.agent,
|
||||
origin || existing.origin,
|
||||
fingerprint,
|
||||
existing.confirmed_fingerprint,
|
||||
existing.confirmed_values_json,
|
||||
existing.confirmed_peers_json,
|
||||
JSON.stringify({ peers: unaccepted, changes }),
|
||||
observedAt,
|
||||
existing.confirmed_at,
|
||||
updatedAt,
|
||||
existing.instance_id,
|
||||
);
|
||||
}
|
||||
|
||||
/**
|
||||
* A twin is told about itself: both records wait for the operator, not just the newcomer. Each
|
||||
* peer is re-read from its own row, so a claim that walks away releases the peer it was holding
|
||||
* and a peer that also drifted keeps its hardware_swapped reason.
|
||||
*/
|
||||
#syncClaims(instanceId: string, imei: string, now: string): void {
|
||||
if (!imei) return;
|
||||
for (const { instance_id: peerId } of this.#selectPeers.all(imei, instanceId) as Array<{
|
||||
instance_id: string;
|
||||
}>) {
|
||||
const peer = this.#select.get(peerId) as IdentityRow | undefined;
|
||||
if (!peer) continue;
|
||||
this.#write(
|
||||
peer,
|
||||
identityValues(peer),
|
||||
peer.agent,
|
||||
peer.origin,
|
||||
peer.fingerprint || digest(identityValues(peer)),
|
||||
peer.observed_at,
|
||||
now,
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
#present(row: IdentityRow): DeviceIdentity {
|
||||
const reasons = parseReasons(row.reasons);
|
||||
const detail = parseDetail(row.detail_json);
|
||||
const conflicts: DeviceIdentityConflict[] = [];
|
||||
if (reasons.includes('hardware_swapped'))
|
||||
conflicts.push({ reason: 'hardware_swapped', instanceIds: [] });
|
||||
if (reasons.includes('imei_claimed'))
|
||||
conflicts.push({ reason: 'imei_claimed', instanceIds: detail.peers });
|
||||
return {
|
||||
instanceId: row.instance_id,
|
||||
status: row.status === 'pending' ? 'pending' : 'confirmed',
|
||||
reasons,
|
||||
conflicts,
|
||||
imei: row.imei,
|
||||
manufacturer: row.manufacturer,
|
||||
model: row.model,
|
||||
revision: row.revision,
|
||||
agent: row.agent,
|
||||
origin: row.origin,
|
||||
fingerprint: row.fingerprint,
|
||||
confirmedFingerprint: row.confirmed_fingerprint,
|
||||
changes: detail.changes,
|
||||
observedAt: row.observed_at,
|
||||
confirmedAt: row.confirmed_at,
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
function parseStringArray(value: string): string[] {
|
||||
try {
|
||||
const parsed: unknown = JSON.parse(value);
|
||||
return Array.isArray(parsed) ? stringArray(parsed) : [];
|
||||
} catch {
|
||||
return [];
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,107 @@
|
||||
import Database from 'better-sqlite3';
|
||||
import { describe, expect, it } from 'vitest';
|
||||
|
||||
import { migrateDatabase } from '../../infrastructure/database/migrations.js';
|
||||
import { SafeUpstreamGateway } from '../../infrastructure/transport/safe-upstream-gateway.js';
|
||||
import {
|
||||
InstanceSessionStore,
|
||||
type UpstreamRequest,
|
||||
} from '../connections/upstream-session-client.js';
|
||||
import type { InstanceService } from '../instances/instance-service.js';
|
||||
import {
|
||||
DEVICE_ACTIONS,
|
||||
type DeviceActionDefinition,
|
||||
type DeviceActionField,
|
||||
} from './device-action-catalog.js';
|
||||
import { DeviceActionService } from './device-action-service.js';
|
||||
|
||||
const ORIGIN = 'http://192.168.1.20:8080';
|
||||
|
||||
/** A value that satisfies the catalog's own validator for every declared field kind. */
|
||||
function sampleValue(field: DeviceActionField): unknown {
|
||||
if (field.id === 'phone_number' || field.id === 'sms_center') return '+15550001111';
|
||||
if (field.id === 'mccmnc') return '00101';
|
||||
if (field.id === 'iccid') return '89882020202220963176';
|
||||
switch (field.kind) {
|
||||
case 'boolean':
|
||||
return true;
|
||||
case 'number':
|
||||
return field.min ?? 1;
|
||||
case 'string-list':
|
||||
return ['1'];
|
||||
case 'choice':
|
||||
return field.choices?.[0]?.value ?? 'auto';
|
||||
default:
|
||||
return 'sample';
|
||||
}
|
||||
}
|
||||
|
||||
function paramsFor(action: DeviceActionDefinition): Record<string, unknown> {
|
||||
const params: Record<string, unknown> = {};
|
||||
for (const field of action.fields ?? []) params[field.id] = sampleValue(field);
|
||||
return params;
|
||||
}
|
||||
|
||||
function harness() {
|
||||
const db = new Database(':memory:');
|
||||
db.pragma('foreign_keys = ON');
|
||||
migrateDatabase(db);
|
||||
db.prepare(
|
||||
`INSERT INTO instances (id,name,base_url,config_revision,created_at,updated_at)
|
||||
VALUES ('node-a','Node A',?,1,?,?)`,
|
||||
).run(ORIGIN, '2026-09-04T00:00:00.000Z', '2026-09-04T00:00:00.000Z');
|
||||
|
||||
const sessions = new InstanceSessionStore();
|
||||
sessions.set('node-a', ORIGIN, 'simadmin_session=opaque');
|
||||
const dispatched: { url: string; method: string; body: string }[] = [];
|
||||
const gateway = new SafeUpstreamGateway({
|
||||
transport: {
|
||||
get: async (url, headers) => {
|
||||
void headers;
|
||||
dispatched.push({ url, method: 'GET', body: '' });
|
||||
return { status: 200, headers: {}, body: '{}' };
|
||||
},
|
||||
post: async (url, headers, body) => {
|
||||
void headers;
|
||||
dispatched.push({ url, method: 'POST', body });
|
||||
return { status: 200, headers: {}, body: '{"status":"success"}' };
|
||||
},
|
||||
delete: async (url, headers) => {
|
||||
void headers;
|
||||
dispatched.push({ url, method: 'DELETE', body: '' });
|
||||
return { status: 200, headers: {}, body: '{"status":"success"}' };
|
||||
},
|
||||
},
|
||||
});
|
||||
const instances = {
|
||||
get: async (id: string) =>
|
||||
id === 'node-a' ? { id: 'node-a', name: 'Node A', origin: ORIGIN } : undefined,
|
||||
} as unknown as InstanceService;
|
||||
const service = new DeviceActionService({
|
||||
instances,
|
||||
sessions,
|
||||
db,
|
||||
request: (request: UpstreamRequest) => gateway.request(request),
|
||||
now: () => new Date('2026-09-04T00:00:00.000Z'),
|
||||
id: () => 'audit-row-1',
|
||||
});
|
||||
return { service, db, dispatched };
|
||||
}
|
||||
|
||||
const context = { actor: 'loopback-control-plane', requestId: 'req-1', confirm: true };
|
||||
|
||||
describe('device action catalog transport parity', () => {
|
||||
it('covers every module the console can render', () => {
|
||||
expect(DEVICE_ACTIONS.length).toBeGreaterThan(40);
|
||||
});
|
||||
|
||||
for (const action of DEVICE_ACTIONS) {
|
||||
it(`dispatches ${action.id} through the pinned allowlist`, async () => {
|
||||
const { service, dispatched } = harness();
|
||||
const result = await service.execute('node-a', action.id, paramsFor(action), context);
|
||||
expect(result.ok).toBe(true);
|
||||
expect(dispatched).toHaveLength(1);
|
||||
expect(dispatched[0]?.url.startsWith(`${ORIGIN}/api/`)).toBe(true);
|
||||
});
|
||||
}
|
||||
});
|
||||
@@ -0,0 +1,946 @@
|
||||
import type { InstanceModuleKey } from './instance-module-catalog.js';
|
||||
|
||||
export type DeviceActionRisk = 'R1' | 'R2' | 'R3';
|
||||
|
||||
export type DeviceActionFieldKind =
|
||||
| 'boolean'
|
||||
| 'number'
|
||||
| 'string'
|
||||
| 'secret'
|
||||
| 'choice'
|
||||
| 'string-list';
|
||||
|
||||
export interface DeviceActionChoice {
|
||||
readonly value: string;
|
||||
readonly label: string;
|
||||
}
|
||||
|
||||
export interface DeviceActionField {
|
||||
/** Device payload key, or the name of a `{placeholder}` in the action path. */
|
||||
readonly id: string;
|
||||
readonly label: string;
|
||||
readonly kind: DeviceActionFieldKind;
|
||||
readonly in?: 'body' | 'path' | 'query';
|
||||
readonly required?: boolean;
|
||||
readonly choices?: readonly DeviceActionChoice[];
|
||||
readonly min?: number;
|
||||
readonly max?: number;
|
||||
readonly maxLength?: number;
|
||||
readonly pattern?: string;
|
||||
readonly hint?: string;
|
||||
}
|
||||
|
||||
export interface DeviceActionDefinition {
|
||||
readonly id: string;
|
||||
readonly module: InstanceModuleKey | 'messages';
|
||||
readonly title: string;
|
||||
readonly description: string;
|
||||
readonly risk: DeviceActionRisk;
|
||||
readonly method: 'POST' | 'DELETE';
|
||||
/** Device path appended to `/api`; may contain `{field}` placeholders filled from path fields. */
|
||||
readonly path: string;
|
||||
/** Constant payload merged before any user-supplied body fields. */
|
||||
readonly fixed?: Readonly<Record<string, unknown>>;
|
||||
/** Sends a literal `{}` document when no field value survives, matching endpoints that still want JSON. */
|
||||
readonly emptyJsonObject?: boolean;
|
||||
readonly fields?: readonly DeviceActionField[];
|
||||
readonly timeoutMs?: number;
|
||||
}
|
||||
|
||||
const BOOL = (id: string, label: string, hint?: string): DeviceActionField => ({
|
||||
id,
|
||||
label,
|
||||
kind: 'boolean',
|
||||
required: true,
|
||||
...(hint ? { hint } : {}),
|
||||
});
|
||||
const TEXT = (
|
||||
id: string,
|
||||
label: string,
|
||||
options: Partial<DeviceActionField> = {},
|
||||
): DeviceActionField => ({ id, label, kind: 'string', required: true, maxLength: 128, ...options });
|
||||
|
||||
const RADIO_MODES: readonly DeviceActionChoice[] = [
|
||||
{ value: 'auto', label: '自动' },
|
||||
{ value: 'lte', label: '仅 LTE' },
|
||||
{ value: 'nr', label: '5G NR' },
|
||||
];
|
||||
const WORK_MODES: readonly DeviceActionChoice[] = [
|
||||
{ value: 'sim', label: '本机 SIM 管理' },
|
||||
{ value: 'sim_overseas', label: '海外卡模式' },
|
||||
{ value: 'esim', label: 'eSIM 模式' },
|
||||
];
|
||||
const APN_PROTOCOLS: readonly DeviceActionChoice[] = [
|
||||
{ value: 'ipv4', label: 'IPv4' },
|
||||
{ value: 'ipv6', label: 'IPv6' },
|
||||
{ value: 'ipv4v6', label: 'IPv4/IPv6' },
|
||||
];
|
||||
const APN_AUTH: readonly DeviceActionChoice[] = [
|
||||
{ value: 'none', label: '不认证' },
|
||||
{ value: 'pap', label: 'PAP' },
|
||||
{ value: 'chap', label: 'CHAP' },
|
||||
];
|
||||
const CELL_RATS: readonly DeviceActionChoice[] = [
|
||||
{ value: 'lte', label: 'LTE' },
|
||||
{ value: 'nr', label: 'NR' },
|
||||
{ value: 'wcdma', label: 'WCDMA' },
|
||||
{ value: 'gsm', label: 'GSM' },
|
||||
];
|
||||
const EMPTY_BODY = {};
|
||||
|
||||
/**
|
||||
* Every device mutation the console is allowed to perform. Paths and payload keys mirror the
|
||||
* SimAdmin device API one for one; anything not listed here cannot be reached from the UI.
|
||||
*/
|
||||
export const DEVICE_ACTIONS: readonly DeviceActionDefinition[] = Object.freeze([
|
||||
{
|
||||
id: 'sim.refresh-details',
|
||||
module: 'sim',
|
||||
title: '刷新 SIM 详情',
|
||||
description: '要求设备重新读取 SIM 卡详细信息并刷新缓存。',
|
||||
risk: 'R1',
|
||||
method: 'POST',
|
||||
path: '/sim/details/refresh',
|
||||
fixed: EMPTY_BODY,
|
||||
timeoutMs: 8_000,
|
||||
},
|
||||
{
|
||||
id: 'sim.cache-phone',
|
||||
module: 'sim',
|
||||
title: '登记本机号码',
|
||||
description: '把本机号码写入设备缓存,供短信与通话界面显示。',
|
||||
risk: 'R1',
|
||||
method: 'POST',
|
||||
path: '/sim/cache',
|
||||
fields: [TEXT('phone_number', '本机号码', { maxLength: 32, pattern: '^\\+?[0-9 ()-]{3,32}$' })],
|
||||
},
|
||||
{
|
||||
id: 'sim.cache-sms-center',
|
||||
module: 'sim',
|
||||
title: '登记短信中心号码',
|
||||
description: '写入短信中心(SMSC)号码。',
|
||||
risk: 'R1',
|
||||
method: 'POST',
|
||||
path: '/sim/cache',
|
||||
fields: [
|
||||
TEXT('sms_center', '短信中心号码', { maxLength: 32, pattern: '^\\+?[0-9 ()-]{3,32}$' }),
|
||||
],
|
||||
},
|
||||
{
|
||||
id: 'apn.save',
|
||||
module: 'sim',
|
||||
title: '保存 APN 配置',
|
||||
description: '修改指定 PDN 上下文的 APN、协议与认证方式。',
|
||||
risk: 'R2',
|
||||
method: 'POST',
|
||||
path: '/apn',
|
||||
fields: [
|
||||
TEXT('context_path', 'PDN 上下文路径', { maxLength: 128 }),
|
||||
{ id: 'apn', label: 'APN 名称', kind: 'string', maxLength: 128 },
|
||||
{ id: 'protocol', label: '协议', kind: 'choice', choices: APN_PROTOCOLS },
|
||||
{ id: 'username', label: '用户名', kind: 'string', maxLength: 128 },
|
||||
{ id: 'password', label: '密码', kind: 'secret', maxLength: 128 },
|
||||
{ id: 'auth_method', label: '认证方式', kind: 'choice', choices: APN_AUTH },
|
||||
],
|
||||
},
|
||||
{
|
||||
id: 'band-lock.apply',
|
||||
module: 'sim',
|
||||
title: '锁定频段',
|
||||
description: '按制式提交允许使用的频段列表,设备只会驻留这些频段。',
|
||||
risk: 'R2',
|
||||
method: 'POST',
|
||||
path: '/band-lock',
|
||||
fields: [
|
||||
{ id: 'lte_fdd_bands', label: 'LTE FDD 频段', kind: 'string-list' },
|
||||
{ id: 'lte_tdd_bands', label: 'LTE TDD 频段', kind: 'string-list' },
|
||||
{ id: 'nr_fdd_bands', label: 'NR FDD 频段', kind: 'string-list' },
|
||||
{ id: 'nr_tdd_bands', label: 'NR TDD 频段', kind: 'string-list' },
|
||||
],
|
||||
},
|
||||
{
|
||||
id: 'band-lock.clear',
|
||||
module: 'sim',
|
||||
title: '解除频段锁定',
|
||||
description: '提交空频段列表,恢复设备默认选网。',
|
||||
risk: 'R2',
|
||||
method: 'POST',
|
||||
path: '/band-lock',
|
||||
fixed: { lte_fdd_bands: [], lte_tdd_bands: [], nr_fdd_bands: [], nr_tdd_bands: [] },
|
||||
},
|
||||
{
|
||||
id: 'cell-lock.apply',
|
||||
module: 'sim',
|
||||
title: '锁定小区',
|
||||
description: '按频点与 PCI 锁定到指定小区,用于信号排查。',
|
||||
risk: 'R2',
|
||||
method: 'POST',
|
||||
path: '/cell-lock',
|
||||
fields: [
|
||||
{ id: 'rat', label: '制式', kind: 'choice', required: true, choices: CELL_RATS },
|
||||
BOOL('enable', '启用小区锁定'),
|
||||
{
|
||||
id: 'arfcn',
|
||||
label: 'ARFCN 频点',
|
||||
kind: 'number',
|
||||
required: true,
|
||||
min: 0,
|
||||
max: 2_684_354_555,
|
||||
},
|
||||
{ id: 'pci', label: 'PCI 物理小区号', kind: 'number', required: true, min: 0, max: 1_007 },
|
||||
],
|
||||
},
|
||||
{
|
||||
id: 'cell-lock.unlock-all',
|
||||
module: 'sim',
|
||||
title: '解除小区锁定',
|
||||
description: '释放全部已锁定的小区。',
|
||||
risk: 'R2',
|
||||
method: 'POST',
|
||||
path: '/cell-lock/unlock-all',
|
||||
fixed: EMPTY_BODY,
|
||||
},
|
||||
{
|
||||
id: 'data.set',
|
||||
module: 'cellular',
|
||||
title: '数据开关',
|
||||
description: '开启或关闭设备的蜂窝数据连接。',
|
||||
risk: 'R1',
|
||||
method: 'POST',
|
||||
path: '/data',
|
||||
fields: [BOOL('active', '启用蜂窝数据')],
|
||||
},
|
||||
{
|
||||
id: 'network.scan',
|
||||
module: 'cellular',
|
||||
title: '扫描运营商',
|
||||
description: '让设备重新扫描可见运营商列表。',
|
||||
risk: 'R1',
|
||||
method: 'POST',
|
||||
path: '/network/operators/scan',
|
||||
fixed: EMPTY_BODY,
|
||||
timeoutMs: 30_000,
|
||||
},
|
||||
{
|
||||
id: 'network.register-manual',
|
||||
module: 'cellular',
|
||||
title: '手动注册网络',
|
||||
description: '按 MCCMNC 强制注册到指定运营商。',
|
||||
risk: 'R2',
|
||||
method: 'POST',
|
||||
path: '/network/register-manual',
|
||||
fields: [TEXT('mccmnc', '运营商 MCCMNC', { maxLength: 6, pattern: '^[0-9]{5,6}$' })],
|
||||
timeoutMs: 30_000,
|
||||
},
|
||||
{
|
||||
id: 'radio-mode.set',
|
||||
module: 'cellular',
|
||||
title: '切换网络模式',
|
||||
description: '切换首选无线制式,切换期间会短暂断网。',
|
||||
risk: 'R2',
|
||||
method: 'POST',
|
||||
path: '/radio-mode',
|
||||
fields: [
|
||||
{ id: 'mode', label: '网络模式', kind: 'choice', required: true, choices: RADIO_MODES },
|
||||
],
|
||||
timeoutMs: 20_000,
|
||||
},
|
||||
{
|
||||
id: 'roaming.set',
|
||||
module: 'cellular',
|
||||
title: '数据漫游开关',
|
||||
description: '允许或禁止在漫游网络上注册。',
|
||||
risk: 'R2',
|
||||
method: 'POST',
|
||||
path: '/roaming',
|
||||
fields: [BOOL('allowed', '允许漫游')],
|
||||
},
|
||||
{
|
||||
id: 'airplane-mode.set',
|
||||
module: 'cellular',
|
||||
title: '飞行模式',
|
||||
description: '开启飞行模式会立即切断全部无线连接。',
|
||||
risk: 'R3',
|
||||
method: 'POST',
|
||||
path: '/airplane-mode',
|
||||
fields: [BOOL('enabled', '开启飞行模式')],
|
||||
},
|
||||
{
|
||||
id: 'cell-monitor.start',
|
||||
module: 'cellular',
|
||||
title: '启动小区监视',
|
||||
description: '开始持续采集服务小区与邻区数据。',
|
||||
risk: 'R1',
|
||||
method: 'POST',
|
||||
path: '/cell-monitor/start',
|
||||
fixed: EMPTY_BODY,
|
||||
},
|
||||
{
|
||||
id: 'cell-monitor.stop',
|
||||
module: 'cellular',
|
||||
title: '停止小区监视',
|
||||
description: '停止后台小区采样,降低设备负载。',
|
||||
risk: 'R1',
|
||||
method: 'POST',
|
||||
path: '/cell-monitor/stop',
|
||||
fixed: EMPTY_BODY,
|
||||
},
|
||||
{
|
||||
id: 'baseband.restart',
|
||||
module: 'cellular',
|
||||
title: '重启基带',
|
||||
description: '重启调制解调器,期间设备会完全离线约一分钟。',
|
||||
risk: 'R3',
|
||||
method: 'POST',
|
||||
path: '/baseband/restart',
|
||||
fixed: EMPTY_BODY,
|
||||
timeoutMs: 60_000,
|
||||
},
|
||||
{
|
||||
id: 'wlan.enabled',
|
||||
module: 'device-network',
|
||||
title: 'WLAN 开关',
|
||||
description: '开启或关闭设备无线网卡。',
|
||||
risk: 'R1',
|
||||
method: 'POST',
|
||||
path: '/device-network/wlan/enabled',
|
||||
fields: [BOOL('enabled', '启用 WLAN')],
|
||||
},
|
||||
{
|
||||
id: 'wlan.scan',
|
||||
module: 'device-network',
|
||||
title: '扫描 WLAN',
|
||||
description: '触发一次无线网络扫描并刷新热点列表。',
|
||||
risk: 'R1',
|
||||
method: 'POST',
|
||||
path: '/device-network/wlan/scan',
|
||||
fixed: EMPTY_BODY,
|
||||
timeoutMs: 30_000,
|
||||
},
|
||||
{
|
||||
id: 'wlan.connect',
|
||||
module: 'device-network',
|
||||
title: '连接 WLAN',
|
||||
description: '加入指定热点,密码留空表示使用已保存的凭据。',
|
||||
risk: 'R2',
|
||||
method: 'POST',
|
||||
path: '/device-network/wlan/connect',
|
||||
fields: [
|
||||
TEXT('ssid', '网络名称(SSID)', { maxLength: 64 }),
|
||||
{ id: 'password', label: '密码', kind: 'secret', maxLength: 128 },
|
||||
{ id: 'auto_join', label: '自动加入', kind: 'boolean' },
|
||||
],
|
||||
timeoutMs: 30_000,
|
||||
},
|
||||
{
|
||||
id: 'wlan.disconnect',
|
||||
module: 'device-network',
|
||||
title: '断开 WLAN',
|
||||
description: '断开当前无线连接,不会删除已保存配置。',
|
||||
risk: 'R1',
|
||||
method: 'POST',
|
||||
path: '/device-network/wlan/disconnect',
|
||||
fixed: EMPTY_BODY,
|
||||
},
|
||||
{
|
||||
id: 'wlan.forget',
|
||||
module: 'device-network',
|
||||
title: '忽略 WLAN 网络',
|
||||
description: '删除已保存的热点配置。',
|
||||
risk: 'R2',
|
||||
method: 'POST',
|
||||
path: '/device-network/wlan/forget',
|
||||
fields: [
|
||||
TEXT('uuid', '配置 UUID', { maxLength: 64 }),
|
||||
TEXT('connection_id', '连接标识', { maxLength: 64 }),
|
||||
],
|
||||
},
|
||||
{
|
||||
id: 'ddns.sync',
|
||||
module: 'device-network',
|
||||
title: '立即同步 DDNS',
|
||||
description: '强制把当前公网地址推送到 DDNS 服务商。',
|
||||
risk: 'R1',
|
||||
method: 'POST',
|
||||
path: '/device-network/ddns/sync',
|
||||
fixed: EMPTY_BODY,
|
||||
timeoutMs: 30_000,
|
||||
},
|
||||
{
|
||||
id: 'ddns.logs-clear',
|
||||
module: 'device-network',
|
||||
title: '清空 DDNS 日志',
|
||||
description: '删除设备上的 DDNS 更新记录。',
|
||||
risk: 'R2',
|
||||
method: 'POST',
|
||||
path: '/device-network/ddns/logs/clear',
|
||||
fixed: EMPTY_BODY,
|
||||
emptyJsonObject: true,
|
||||
},
|
||||
{
|
||||
id: 'esim.enable-profile',
|
||||
module: 'esim',
|
||||
title: '切换 eSIM 配置',
|
||||
description: '启用指定 ICCID 的 Profile,设备会重新注网。',
|
||||
risk: 'R2',
|
||||
method: 'POST',
|
||||
path: '/esim/profiles/{iccid}/enable',
|
||||
fields: [
|
||||
TEXT('iccid', 'ICCID', { in: 'path', maxLength: 32, pattern: '^[A-Za-z0-9_.\\-]{1,32}$' }),
|
||||
],
|
||||
timeoutMs: 60_000,
|
||||
},
|
||||
{
|
||||
id: 'esim.rename-profile',
|
||||
module: 'esim',
|
||||
title: '重命名 eSIM 配置',
|
||||
description: '修改 Profile 的显示名称。',
|
||||
risk: 'R1',
|
||||
method: 'POST',
|
||||
path: '/esim/profiles/{iccid}/rename',
|
||||
fields: [
|
||||
TEXT('iccid', 'ICCID', { in: 'path', maxLength: 32, pattern: '^[A-Za-z0-9_.\\-]{1,32}$' }),
|
||||
TEXT('name', '新名称', { maxLength: 64 }),
|
||||
],
|
||||
timeoutMs: 60_000,
|
||||
},
|
||||
{
|
||||
id: 'esim.delete-profile',
|
||||
module: 'esim',
|
||||
title: '删除 eSIM 配置',
|
||||
description: '从 eUICC 移除 Profile,操作不可撤销。',
|
||||
risk: 'R3',
|
||||
method: 'DELETE',
|
||||
path: '/esim/profiles/{iccid}',
|
||||
fields: [
|
||||
TEXT('iccid', 'ICCID', { in: 'path', maxLength: 32, pattern: '^[A-Za-z0-9_.\\-]{1,32}$' }),
|
||||
],
|
||||
timeoutMs: 60_000,
|
||||
},
|
||||
{
|
||||
id: 'esim.download-profile',
|
||||
module: 'esim',
|
||||
title: '下载 eSIM 配置',
|
||||
description: '通过 SM-DP+ 服务器向 eUICC 写入新的 Profile。',
|
||||
risk: 'R3',
|
||||
method: 'POST',
|
||||
path: '/esim/profiles',
|
||||
fields: [
|
||||
TEXT('smdp', 'SM-DP+ 地址', { maxLength: 128 }),
|
||||
TEXT('matching_id', 'Matching ID', { maxLength: 64 }),
|
||||
{ id: 'confirmation_code', label: '确认码', kind: 'secret', maxLength: 64 },
|
||||
{ id: 'imei', label: 'IMEI', kind: 'string', maxLength: 32 },
|
||||
],
|
||||
timeoutMs: 180_000,
|
||||
},
|
||||
{
|
||||
id: 'esim.lpac-repair',
|
||||
module: 'esim',
|
||||
title: '修复 lpac',
|
||||
description: '重新初始化设备的 lpac 组件。',
|
||||
risk: 'R2',
|
||||
method: 'POST',
|
||||
path: '/esim/lpac/repair',
|
||||
fields: [{ id: 'proxy_prefix', label: '代理前缀', kind: 'string', maxLength: 128 }],
|
||||
timeoutMs: 120_000,
|
||||
},
|
||||
{
|
||||
id: 'call.dial',
|
||||
module: 'calls',
|
||||
title: '拨号',
|
||||
description: '让设备拨打指定号码。',
|
||||
risk: 'R2',
|
||||
method: 'POST',
|
||||
path: '/call/dial',
|
||||
fields: [TEXT('phone_number', '被叫号码', { maxLength: 32, pattern: '^\\+?[0-9 ()-]{3,32}$' })],
|
||||
},
|
||||
{
|
||||
id: 'call.answer',
|
||||
module: 'calls',
|
||||
title: '接听通话',
|
||||
description: '接听指定通话对象。',
|
||||
risk: 'R1',
|
||||
method: 'POST',
|
||||
path: '/call/answer',
|
||||
fields: [TEXT('path', '通话对象路径', { maxLength: 128 })],
|
||||
},
|
||||
{
|
||||
id: 'call.hangup',
|
||||
module: 'calls',
|
||||
title: '挂断通话',
|
||||
description: '挂断指定通话对象。',
|
||||
risk: 'R1',
|
||||
method: 'POST',
|
||||
path: '/call/hangup',
|
||||
fields: [TEXT('path', '通话对象路径', { maxLength: 128 })],
|
||||
},
|
||||
{
|
||||
id: 'call.hangup-all',
|
||||
module: 'calls',
|
||||
title: '挂断全部通话',
|
||||
description: '结束设备上所有进行中的通话。',
|
||||
risk: 'R2',
|
||||
method: 'POST',
|
||||
path: '/call/hangup-all',
|
||||
fixed: EMPTY_BODY,
|
||||
},
|
||||
{
|
||||
id: 'call.waiting',
|
||||
module: 'calls',
|
||||
title: '呼叫等待',
|
||||
description: '开启或关闭网络侧呼叫等待业务。',
|
||||
risk: 'R2',
|
||||
method: 'POST',
|
||||
path: '/call/settings',
|
||||
fixed: { property: 'VoiceCallWaiting' },
|
||||
fields: [
|
||||
{
|
||||
id: 'value',
|
||||
label: '呼叫等待',
|
||||
kind: 'choice',
|
||||
required: true,
|
||||
choices: [
|
||||
{ value: 'enabled', label: '开启' },
|
||||
{ value: 'disabled', label: '关闭' },
|
||||
],
|
||||
},
|
||||
],
|
||||
},
|
||||
{
|
||||
id: 'call.history-clear',
|
||||
module: 'calls',
|
||||
title: '清空通话记录',
|
||||
description: '删除设备上的全部通话历史。',
|
||||
risk: 'R2',
|
||||
method: 'POST',
|
||||
path: '/call/history/clear',
|
||||
fixed: EMPTY_BODY,
|
||||
},
|
||||
{
|
||||
id: 'sms.clear',
|
||||
module: 'messages',
|
||||
title: '清空全部短信',
|
||||
description: '删除设备上保存的所有短信记录。',
|
||||
risk: 'R3',
|
||||
method: 'POST',
|
||||
path: '/sms/clear',
|
||||
fixed: EMPTY_BODY,
|
||||
},
|
||||
{
|
||||
id: 'work-mode.set',
|
||||
module: 'configuration',
|
||||
title: '切换工作模式',
|
||||
description: '切换设备的管理模式,设备会重启相关服务。',
|
||||
risk: 'R3',
|
||||
method: 'POST',
|
||||
path: '/work-mode',
|
||||
fixed: { confirm: true },
|
||||
fields: [
|
||||
{ id: 'mode', label: '工作模式', kind: 'choice', required: true, choices: WORK_MODES },
|
||||
],
|
||||
timeoutMs: 20_000,
|
||||
},
|
||||
{
|
||||
id: 'hub.configure',
|
||||
module: 'configuration',
|
||||
title: '配置设备回连',
|
||||
description: '设置设备回连中心平台的地址与本地兜底策略。',
|
||||
risk: 'R2',
|
||||
method: 'POST',
|
||||
path: '/hub',
|
||||
fields: [
|
||||
BOOL('enabled', '启用回连'),
|
||||
{ id: 'url', label: '回连地址', kind: 'string', maxLength: 256 },
|
||||
{ id: 'local_fallback_enabled', label: '启用本地兜底', kind: 'boolean' },
|
||||
{
|
||||
id: 'local_fallback_timeout_seconds',
|
||||
label: '本地兜底超时(秒)',
|
||||
kind: 'number',
|
||||
min: 10,
|
||||
max: 86_400,
|
||||
},
|
||||
],
|
||||
},
|
||||
{
|
||||
id: 'hub.unbind',
|
||||
module: 'configuration',
|
||||
title: '解除回连绑定',
|
||||
description: '清除设备的中心平台绑定,改由本机管理。',
|
||||
risk: 'R3',
|
||||
method: 'POST',
|
||||
path: '/hub/unbind',
|
||||
fixed: EMPTY_BODY,
|
||||
},
|
||||
{
|
||||
id: 'notifications.test-channel',
|
||||
module: 'notifications',
|
||||
title: '发送测试通知',
|
||||
description: '通过指定渠道发送一条测试消息。',
|
||||
risk: 'R1',
|
||||
method: 'POST',
|
||||
path: '/notifications/test/{channel}',
|
||||
fields: [TEXT('channel', '渠道标识', { in: 'path', maxLength: 64 })],
|
||||
timeoutMs: 30_000,
|
||||
},
|
||||
{
|
||||
id: 'notifications.logs-clear',
|
||||
module: 'notifications',
|
||||
title: '清空通知日志',
|
||||
description: '删除设备上的通知发送记录。',
|
||||
risk: 'R2',
|
||||
method: 'POST',
|
||||
path: '/notifications/logs/clear',
|
||||
fixed: EMPTY_BODY,
|
||||
emptyJsonObject: true,
|
||||
},
|
||||
{
|
||||
id: 'notifications.queue-retry-all',
|
||||
module: 'notifications',
|
||||
title: '重投全部待办通知',
|
||||
description: '让设备立即重试通知队列中所有待发送与失败的任务。',
|
||||
risk: 'R2',
|
||||
method: 'POST',
|
||||
path: '/notifications/queue/retry-all',
|
||||
fixed: EMPTY_BODY,
|
||||
timeoutMs: 30_000,
|
||||
},
|
||||
{
|
||||
id: 'notifications.queue-clear',
|
||||
module: 'notifications',
|
||||
title: '清空通知队列',
|
||||
description: '丢弃设备上尚未发送与已处理的通知队列条目。',
|
||||
risk: 'R2',
|
||||
method: 'POST',
|
||||
path: '/notifications/queue/clear',
|
||||
fixed: EMPTY_BODY,
|
||||
},
|
||||
{
|
||||
id: 'automation.test-task',
|
||||
module: 'automation',
|
||||
title: '试运行自动化任务',
|
||||
description: '立即执行一次指定的设备自动化任务。',
|
||||
risk: 'R1',
|
||||
method: 'POST',
|
||||
path: '/automation/test/{taskId}',
|
||||
fields: [TEXT('taskId', '任务标识', { in: 'path', maxLength: 64 })],
|
||||
timeoutMs: 60_000,
|
||||
},
|
||||
{
|
||||
id: 'automation.logs-clear',
|
||||
module: 'automation',
|
||||
title: '清空自动化日志',
|
||||
description: '删除设备上的自动化执行记录。',
|
||||
risk: 'R2',
|
||||
method: 'POST',
|
||||
path: '/automation/logs/clear',
|
||||
fixed: EMPTY_BODY,
|
||||
emptyJsonObject: true,
|
||||
},
|
||||
{
|
||||
id: 'ota.check-release',
|
||||
module: 'ota',
|
||||
title: '检查最新版本',
|
||||
description: '向发布源查询最新固件版本信息。',
|
||||
risk: 'R1',
|
||||
method: 'POST',
|
||||
path: '/ota/latest-release',
|
||||
fixed: { include_variants: true },
|
||||
fields: [{ id: 'proxy_prefix', label: '加速节点前缀', kind: 'string', maxLength: 256 }],
|
||||
timeoutMs: 60_000,
|
||||
},
|
||||
{
|
||||
id: 'ota.online-prepare',
|
||||
module: 'ota',
|
||||
title: '在线下载更新包',
|
||||
description: '让设备下载并暂存指定的升级包。',
|
||||
risk: 'R2',
|
||||
method: 'POST',
|
||||
path: '/ota/online-prepare',
|
||||
fields: [
|
||||
TEXT('asset_name', '升级包名称', { maxLength: 128 }),
|
||||
{ id: 'proxy_prefix', label: '加速节点前缀', kind: 'string', maxLength: 256 },
|
||||
],
|
||||
timeoutMs: 300_000,
|
||||
},
|
||||
{
|
||||
id: 'ota.apply',
|
||||
module: 'ota',
|
||||
title: '应用更新',
|
||||
description: '安装已暂存的升级包,设备可能自动重启。',
|
||||
risk: 'R3',
|
||||
method: 'POST',
|
||||
path: '/ota/apply',
|
||||
fields: [BOOL('restart_now', '升级后立即重启')],
|
||||
timeoutMs: 300_000,
|
||||
},
|
||||
{
|
||||
id: 'ota.cancel',
|
||||
module: 'ota',
|
||||
title: '取消更新',
|
||||
description: '丢弃已暂存的升级包。',
|
||||
risk: 'R1',
|
||||
method: 'POST',
|
||||
path: '/ota/cancel',
|
||||
fixed: EMPTY_BODY,
|
||||
},
|
||||
{
|
||||
id: 'vowifi.feature',
|
||||
module: 'vowifi',
|
||||
title: 'VoWiFi 功能开关',
|
||||
description: '启用或停用设备的 VoWiFi 能力。',
|
||||
risk: 'R2',
|
||||
method: 'POST',
|
||||
path: '/vowifi/feature',
|
||||
fields: [BOOL('enabled', '启用 VoWiFi')],
|
||||
timeoutMs: 20_000,
|
||||
},
|
||||
{
|
||||
id: 'vowifi.connection',
|
||||
module: 'vowifi',
|
||||
title: 'VoWiFi 连接开关',
|
||||
description: '建立或断开 VoWiFi IMS 连接。',
|
||||
risk: 'R2',
|
||||
method: 'POST',
|
||||
path: '/vowifi/connection',
|
||||
fields: [BOOL('enabled', '保持连接')],
|
||||
timeoutMs: 120_000,
|
||||
},
|
||||
{
|
||||
id: 'vowifi.connect',
|
||||
module: 'vowifi',
|
||||
title: '发起 VoWiFi 注册',
|
||||
description: '立即尝试一次 IMS 注册。',
|
||||
risk: 'R2',
|
||||
method: 'POST',
|
||||
path: '/vowifi/connect',
|
||||
fixed: EMPTY_BODY,
|
||||
timeoutMs: 120_000,
|
||||
},
|
||||
{
|
||||
id: 'backup.export-local',
|
||||
module: 'device-backup',
|
||||
title: '本机生成备份',
|
||||
description: '在设备上生成所选组件的备份文件。',
|
||||
risk: 'R2',
|
||||
method: 'POST',
|
||||
path: '/backup/export-local',
|
||||
fields: [
|
||||
{
|
||||
id: 'components',
|
||||
label: '备份组件',
|
||||
kind: 'string-list',
|
||||
required: true,
|
||||
hint: '留空表示全部',
|
||||
},
|
||||
],
|
||||
timeoutMs: 120_000,
|
||||
},
|
||||
{
|
||||
id: 'backup.data-clear',
|
||||
module: 'device-backup',
|
||||
title: '清除设备数据',
|
||||
description: '按组件清除设备本地数据,操作不可撤销。',
|
||||
risk: 'R3',
|
||||
method: 'POST',
|
||||
path: '/backup/data/clear',
|
||||
fields: [{ id: 'components', label: '要清除的组件', kind: 'string-list', required: true }],
|
||||
timeoutMs: 120_000,
|
||||
},
|
||||
{
|
||||
id: 'backup.delete-file',
|
||||
module: 'device-backup',
|
||||
title: '删除备份文件',
|
||||
description: '删除设备上的一个备份文件。',
|
||||
risk: 'R3',
|
||||
method: 'DELETE',
|
||||
path: '/backup/files/{name}',
|
||||
fields: [TEXT('name', '备份文件名', { in: 'path', maxLength: 128 })],
|
||||
},
|
||||
{
|
||||
id: 'backup.apply-file',
|
||||
module: 'device-backup',
|
||||
title: '恢复备份文件',
|
||||
description: '用设备上的备份文件恢复所选组件。',
|
||||
risk: 'R3',
|
||||
method: 'POST',
|
||||
path: '/backup/files/{name}/apply',
|
||||
fields: [
|
||||
TEXT('name', '备份文件名', { in: 'path', maxLength: 128 }),
|
||||
{
|
||||
id: 'mode',
|
||||
label: '恢复模式',
|
||||
kind: 'choice',
|
||||
in: 'query',
|
||||
required: true,
|
||||
choices: [
|
||||
{ value: 'replace', label: '整体替换' },
|
||||
{ value: 'merge', label: '合并' },
|
||||
],
|
||||
},
|
||||
{ id: 'components', label: '恢复组件', kind: 'string-list', in: 'query', required: true },
|
||||
],
|
||||
timeoutMs: 120_000,
|
||||
},
|
||||
{
|
||||
id: 'network.register-auto',
|
||||
module: 'cellular',
|
||||
title: '自动注册网络',
|
||||
description: '让设备重新向网络发起自动注册,通常用于掉网后恢复驻网。',
|
||||
risk: 'R2',
|
||||
method: 'POST',
|
||||
path: '/network/register-auto',
|
||||
fixed: EMPTY_BODY,
|
||||
timeoutMs: 30_000,
|
||||
},
|
||||
{
|
||||
id: 'call.volume.set',
|
||||
module: 'calls',
|
||||
title: '调整通话音量',
|
||||
description: '设置通话的扬声器与麦克风音量,部分固件未开放该能力。',
|
||||
risk: 'R1',
|
||||
method: 'POST',
|
||||
path: '/call/volume',
|
||||
fields: [
|
||||
{ id: 'speaker_volume', label: '扬声器音量', kind: 'number', min: 0, max: 100 },
|
||||
{ id: 'microphone_volume', label: '麦克风音量', kind: 'number', min: 0, max: 100 },
|
||||
{ id: 'muted', label: '静音', kind: 'boolean' },
|
||||
],
|
||||
},
|
||||
{
|
||||
id: 'call.forwarding.set',
|
||||
module: 'calls',
|
||||
title: '设置呼叫转移',
|
||||
description: '按转移类型登记或清除呼转号码,部分固件未开放该能力。',
|
||||
risk: 'R2',
|
||||
method: 'POST',
|
||||
path: '/call/forwarding',
|
||||
fields: [
|
||||
{
|
||||
id: 'forward_type',
|
||||
label: '转移类型',
|
||||
kind: 'choice',
|
||||
required: true,
|
||||
choices: [
|
||||
{ value: 'unconditional', label: '无条件转移' },
|
||||
{ value: 'busy', label: '遇忙转移' },
|
||||
{ value: 'no_reply', label: '无应答转移' },
|
||||
{ value: 'not_reachable', label: '不可达转移' },
|
||||
],
|
||||
},
|
||||
TEXT('number', '转移目标号码', { maxLength: 32 }),
|
||||
{ id: 'timeout', label: '无应答等待(秒)', kind: 'number', min: 0, max: 300 },
|
||||
],
|
||||
},
|
||||
{
|
||||
id: 'esim.config.save',
|
||||
module: 'esim',
|
||||
title: '保存 eSIM 配置',
|
||||
description: '设置设备侧 lpac 可执行文件路径与自定义 eUICC 可用内存。',
|
||||
risk: 'R2',
|
||||
method: 'POST',
|
||||
path: '/esim/config',
|
||||
fields: [
|
||||
{ id: 'lpac_path', label: 'lpac 路径', kind: 'string', maxLength: 256 },
|
||||
{
|
||||
id: 'custom_memory_total_kb',
|
||||
label: '自定义内存总量(KB)',
|
||||
kind: 'number',
|
||||
min: 1,
|
||||
max: 1_000_000,
|
||||
},
|
||||
],
|
||||
},
|
||||
{
|
||||
id: 'wlan.profile.save',
|
||||
module: 'device-network',
|
||||
title: '保存 WLAN 配置',
|
||||
description: '修改已保存热点的自动加入与 IPv4 获取方式。',
|
||||
risk: 'R2',
|
||||
method: 'POST',
|
||||
path: '/device-network/wlan/profile',
|
||||
fields: [
|
||||
TEXT('connection_id', '连接标识', { maxLength: 64 }),
|
||||
{ id: 'auto_join', label: '自动加入', kind: 'boolean' },
|
||||
{
|
||||
id: 'ipv4_mode',
|
||||
label: 'IPv4 模式',
|
||||
kind: 'choice',
|
||||
choices: [
|
||||
{ value: 'dhcp', label: 'DHCP 自动获取' },
|
||||
{ value: 'manual', label: '手动指定' },
|
||||
],
|
||||
},
|
||||
{ id: 'ipv4_address', label: 'IPv4 地址', kind: 'string', maxLength: 45 },
|
||||
{ id: 'ipv4_prefix', label: 'IPv4 前缀长度', kind: 'number', min: 0, max: 128 },
|
||||
{ id: 'ipv4_gateway', label: 'IPv4 网关', kind: 'string', maxLength: 45 },
|
||||
],
|
||||
},
|
||||
{
|
||||
id: 'auth.settings.save',
|
||||
module: 'configuration',
|
||||
title: '保存安全设置',
|
||||
description: '调整设备后台的密码策略与会话有效期。',
|
||||
risk: 'R2',
|
||||
method: 'POST',
|
||||
path: '/auth/settings',
|
||||
fields: [
|
||||
{ id: 'password_protection_enabled', label: '启用密码保护', kind: 'boolean' },
|
||||
{ id: 'password_min_length', label: '密码最小长度', kind: 'number', min: 1, max: 32 },
|
||||
{ id: 'password_require_letters', label: '密码须含字母', kind: 'boolean' },
|
||||
{ id: 'password_require_digits', label: '密码须含数字', kind: 'boolean' },
|
||||
{ id: 'password_require_symbols', label: '密码须含符号', kind: 'boolean' },
|
||||
{
|
||||
id: 'session_ttl_seconds',
|
||||
label: '会话有效期(秒)',
|
||||
kind: 'number',
|
||||
min: 60,
|
||||
max: 2_592_000,
|
||||
},
|
||||
{
|
||||
id: 'idle_timeout_seconds',
|
||||
label: '空闲超时(秒)',
|
||||
kind: 'number',
|
||||
min: 60,
|
||||
max: 2_592_000,
|
||||
},
|
||||
],
|
||||
},
|
||||
{
|
||||
id: 'auth.password.set',
|
||||
module: 'configuration',
|
||||
title: '修改设备管理密码',
|
||||
description: '设置设备后台的登录密码,修改后需要重新登录设备。',
|
||||
risk: 'R3',
|
||||
method: 'POST',
|
||||
path: '/auth/password',
|
||||
fields: [
|
||||
{ id: 'new_password', label: '新密码', kind: 'secret', required: true, maxLength: 128 },
|
||||
],
|
||||
},
|
||||
]);
|
||||
|
||||
export const DEVICE_ACTION_MODULES: readonly (InstanceModuleKey | 'messages')[] = [
|
||||
'overview',
|
||||
'sim',
|
||||
'cellular',
|
||||
'device-network',
|
||||
'esim',
|
||||
'calls',
|
||||
'messages',
|
||||
'configuration',
|
||||
'device-backup',
|
||||
'notifications',
|
||||
'automation',
|
||||
'ota',
|
||||
'vowifi',
|
||||
];
|
||||
|
||||
export function findDeviceAction(id: string): DeviceActionDefinition | undefined {
|
||||
return DEVICE_ACTIONS.find((action) => action.id === id);
|
||||
}
|
||||
|
||||
export function deviceActionsFor(
|
||||
module: InstanceModuleKey | 'messages',
|
||||
): readonly DeviceActionDefinition[] {
|
||||
return DEVICE_ACTIONS.filter((action) => action.module === module);
|
||||
}
|
||||
@@ -0,0 +1,221 @@
|
||||
import Database from 'better-sqlite3';
|
||||
import { describe, expect, it } from 'vitest';
|
||||
|
||||
import { migrateDatabase } from '../../infrastructure/database/migrations.js';
|
||||
import {
|
||||
InstanceSessionStore,
|
||||
type UpstreamRequest,
|
||||
} from '../connections/upstream-session-client.js';
|
||||
import type { InstanceService } from '../instances/instance-service.js';
|
||||
import { DeviceActionError, DeviceActionService } from './device-action-service.js';
|
||||
|
||||
interface Reply {
|
||||
readonly status: number;
|
||||
readonly body?: unknown;
|
||||
}
|
||||
|
||||
function fixture(
|
||||
replies: readonly Reply[] = [{ status: 200, body: { status: 'success' } }],
|
||||
extra: {
|
||||
readonly ensureSession?: (id: string, origin: string, force?: boolean) => Promise<void>;
|
||||
readonly identities?: { isBlocked(instanceId: string): boolean };
|
||||
} = {},
|
||||
) {
|
||||
const db = new Database(':memory:');
|
||||
db.pragma('foreign_keys = ON');
|
||||
migrateDatabase(db);
|
||||
db.prepare(
|
||||
`INSERT INTO instances (id,name,base_url,config_revision,created_at,updated_at)
|
||||
VALUES ('node-a','Node A','http://node-a.local',1,?,?)`,
|
||||
).run('2026-09-04T00:00:00.000Z', '2026-09-04T00:00:00.000Z');
|
||||
|
||||
const sessions = new InstanceSessionStore();
|
||||
sessions.set('node-a', 'http://node-a.local', 'simadmin_session=opaque');
|
||||
const calls: UpstreamRequest[] = [];
|
||||
let cursor = 0;
|
||||
const instances = {
|
||||
get: async (id: string) =>
|
||||
id === 'node-a' ? { id: 'node-a', name: 'Node A', origin: 'http://node-a.local' } : undefined,
|
||||
} as unknown as InstanceService;
|
||||
const service = new DeviceActionService({
|
||||
instances,
|
||||
sessions,
|
||||
db,
|
||||
request: async (request) => {
|
||||
calls.push(request);
|
||||
const reply = replies[Math.min(cursor, replies.length - 1)] as Reply;
|
||||
cursor += 1;
|
||||
return {
|
||||
status: reply.status,
|
||||
headers: {},
|
||||
body: reply.body === undefined ? '' : JSON.stringify(reply.body),
|
||||
};
|
||||
},
|
||||
...(extra.ensureSession ? { ensureSession: extra.ensureSession } : {}),
|
||||
...(extra.identities ? { identities: extra.identities } : {}),
|
||||
now: () => new Date('2026-09-04T00:00:00.000Z'),
|
||||
id: () => 'audit-row-1',
|
||||
});
|
||||
return { service, db, calls, sessions };
|
||||
}
|
||||
|
||||
const context = { actor: 'loopback-control-plane', requestId: 'req-1', confirm: true };
|
||||
|
||||
describe('DeviceActionService', () => {
|
||||
it('exposes the catalog so the console can render controls without knowing device paths', () => {
|
||||
const { service } = fixture();
|
||||
const actions = service.list();
|
||||
expect(actions.length).toBeGreaterThan(40);
|
||||
expect(actions.every((action) => !('path' in action))).toBe(true);
|
||||
expect(actions.find((action) => action.id === 'band-lock.apply')?.fields).toHaveLength(4);
|
||||
});
|
||||
|
||||
it('dispatches a zero-body action with only an accept header', async () => {
|
||||
const { service, calls } = fixture();
|
||||
const result = await service.execute('node-a', 'cell-lock.unlock-all', {}, context);
|
||||
expect(result.ok).toBe(true);
|
||||
expect(calls[0]).toMatchObject({
|
||||
url: 'http://node-a.local/api/cell-lock/unlock-all',
|
||||
method: 'POST',
|
||||
headers: { accept: 'application/json', cookie: 'simadmin_session=opaque' },
|
||||
deviceAction: { body: undefined },
|
||||
});
|
||||
});
|
||||
|
||||
it('merges fixed payload keys with validated body fields', async () => {
|
||||
const { service, calls } = fixture();
|
||||
await service.execute('node-a', 'call.waiting', { value: 'enabled' }, context);
|
||||
expect(calls[0]?.deviceAction?.body).toEqual({
|
||||
property: 'VoiceCallWaiting',
|
||||
value: 'enabled',
|
||||
});
|
||||
});
|
||||
|
||||
it('substitutes path parameters and builds the restore query in order', async () => {
|
||||
const { service, calls } = fixture();
|
||||
await service.execute(
|
||||
'node-a',
|
||||
'backup.apply-file',
|
||||
{ name: 'backup-2026.tar.gz', mode: 'merge', components: ['instances', 'jobs'] },
|
||||
context,
|
||||
);
|
||||
expect(calls[0]).toMatchObject({
|
||||
url: 'http://node-a.local/api/backup/files/backup-2026.tar.gz/apply?mode=merge&components=instances,jobs',
|
||||
method: 'POST',
|
||||
});
|
||||
});
|
||||
|
||||
it('rejects a risky action without an explicit confirmation', async () => {
|
||||
const { service, calls } = fixture();
|
||||
await expect(
|
||||
service.execute(
|
||||
'node-a',
|
||||
'airplane-mode.set',
|
||||
{ enabled: true },
|
||||
{ actor: 'loopback-control-plane', requestId: 'req-1' },
|
||||
),
|
||||
).rejects.toThrow(DeviceActionError);
|
||||
expect(calls).toHaveLength(0);
|
||||
});
|
||||
|
||||
it('rejects unknown params, out-of-range numbers and values outside a choice set', async () => {
|
||||
const { service, calls } = fixture();
|
||||
await expect(
|
||||
service.execute('node-a', 'data.set', { active: true, extra: 1 }, context),
|
||||
).rejects.toMatchObject({ code: 'VALIDATION_FAILED', fieldId: 'extra' });
|
||||
await expect(
|
||||
service.execute(
|
||||
'node-a',
|
||||
'cell-lock.apply',
|
||||
{ rat: 'lte', enable: true, arfcn: -1, pci: 5 },
|
||||
context,
|
||||
),
|
||||
).rejects.toMatchObject({ code: 'VALIDATION_FAILED', fieldId: 'arfcn' });
|
||||
await expect(
|
||||
service.execute('node-a', 'radio-mode.set', { mode: 'cdma' }, context),
|
||||
).rejects.toMatchObject({ code: 'VALIDATION_FAILED', fieldId: 'mode' });
|
||||
await expect(
|
||||
service.execute('node-a', 'sim.cache-phone', { phone_number: 'abc' }, context),
|
||||
).rejects.toMatchObject({ code: 'VALIDATION_FAILED', fieldId: 'phone_number' });
|
||||
expect(calls).toHaveLength(0);
|
||||
});
|
||||
|
||||
it('refuses a path parameter that could escape the device path', async () => {
|
||||
const { service, calls } = fixture();
|
||||
await expect(
|
||||
service.execute('node-a', 'esim.delete-profile', { iccid: '../../admin' }, context),
|
||||
).rejects.toMatchObject({ code: 'VALIDATION_FAILED', fieldId: 'iccid' });
|
||||
expect(calls).toHaveLength(0);
|
||||
});
|
||||
|
||||
it('refreshes an expired device session once and retries', async () => {
|
||||
const { service, calls, sessions } = fixture(
|
||||
[
|
||||
{ status: 401, body: { status: 'error', msg: 'unauthorized' } },
|
||||
{ status: 200, body: { status: 'success', data: { applied: true } } },
|
||||
],
|
||||
{
|
||||
ensureSession: async () => {
|
||||
sessions.set('node-a', 'http://node-a.local', 'simadmin_session=rotated');
|
||||
},
|
||||
},
|
||||
);
|
||||
const result = await service.execute('node-a', 'ota.cancel', {}, context);
|
||||
expect(calls).toHaveLength(2);
|
||||
expect(result.ok).toBe(true);
|
||||
expect(calls[0]?.headers.cookie).toBe('simadmin_session=opaque');
|
||||
expect(calls.at(-1)?.headers.cookie).toBe('simadmin_session=rotated');
|
||||
});
|
||||
|
||||
it('reports a device failure without pretending it succeeded', async () => {
|
||||
const { service } = fixture([{ status: 200, body: { status: 'error', msg: 'SIM 卡未就绪' } }]);
|
||||
const result = await service.execute('node-a', 'sim.refresh-details', {}, context);
|
||||
expect(result.ok).toBe(false);
|
||||
expect(result.message).toBe('SIM 卡未就绪');
|
||||
});
|
||||
|
||||
it('writes a fully redacted audit record for every dispatch', async () => {
|
||||
const { service, db } = fixture();
|
||||
await service.execute(
|
||||
'node-a',
|
||||
'wlan.connect',
|
||||
{ ssid: 'office', password: 'sup3r-secret', auto_join: true },
|
||||
context,
|
||||
);
|
||||
const row = db
|
||||
.prepare(
|
||||
'SELECT actor,operation_id,risk_level,result_code,parameters_summary_json,duration_ms FROM audit_events WHERE id=?',
|
||||
)
|
||||
.get('audit-row-1') as Record<string, unknown>;
|
||||
expect(row).toMatchObject({
|
||||
actor: 'loopback-control-plane',
|
||||
operation_id: 'wlan.connect',
|
||||
risk_level: 'R2',
|
||||
result_code: 'succeeded',
|
||||
});
|
||||
const summary = JSON.parse(String(row.parameters_summary_json)) as Record<string, unknown>[];
|
||||
expect(summary.map((item) => item.fieldId)).toEqual(['ssid', 'password', 'auto_join']);
|
||||
expect(JSON.stringify(summary)).not.toContain('sup3r-secret');
|
||||
expect(JSON.stringify(summary)).not.toContain('office');
|
||||
expect(summary.every((item) => item.redacted === true)).toBe(true);
|
||||
});
|
||||
|
||||
it('holds every control while the device identity is disputed', async () => {
|
||||
const { service, calls } = fixture(undefined, {
|
||||
identities: { isBlocked: () => true },
|
||||
});
|
||||
await expect(
|
||||
service.execute('node-a', 'cell-lock.unlock-all', {}, context),
|
||||
).rejects.toMatchObject({ code: 'IDENTITY_UNCONFIRMED' });
|
||||
expect(calls).toHaveLength(0);
|
||||
});
|
||||
|
||||
it('still lets an operator release the binding on a disputed device', async () => {
|
||||
const { service, calls } = fixture(undefined, {
|
||||
identities: { isBlocked: () => true },
|
||||
});
|
||||
const result = await service.execute('node-a', 'hub.unbind', {}, context);
|
||||
expect(result.ok).toBe(true);
|
||||
expect(calls).toHaveLength(1);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,458 @@
|
||||
import { randomUUID } from 'node:crypto';
|
||||
|
||||
import type { InstanceService } from './instance-service.js';
|
||||
import {
|
||||
DEVICE_ACTIONS,
|
||||
findDeviceAction,
|
||||
type DeviceActionDefinition,
|
||||
type DeviceActionField,
|
||||
type DeviceActionRisk,
|
||||
} from './device-action-catalog.js';
|
||||
import type { SqliteDatabase } from '../../infrastructure/database/database.js';
|
||||
import type {
|
||||
InstanceSessionStore,
|
||||
UpstreamResponse,
|
||||
UpstreamSessionClientOptions,
|
||||
} from '../connections/upstream-session-client.js';
|
||||
|
||||
export type DeviceActionErrorCode =
|
||||
| 'NOT_FOUND'
|
||||
| 'VALIDATION_FAILED'
|
||||
| 'SESSION_INVALID'
|
||||
| 'UPSTREAM_FAILED'
|
||||
| 'NOT_DISPATCHED'
|
||||
| 'IDENTITY_UNCONFIRMED';
|
||||
|
||||
export class DeviceActionError extends Error {
|
||||
constructor(
|
||||
readonly code: DeviceActionErrorCode,
|
||||
readonly fieldId?: string,
|
||||
) {
|
||||
super(code);
|
||||
this.name = 'DeviceActionError';
|
||||
}
|
||||
}
|
||||
|
||||
export interface DeviceActionDescriptor {
|
||||
readonly id: string;
|
||||
readonly module: DeviceActionDefinition['module'];
|
||||
readonly title: string;
|
||||
readonly description: string;
|
||||
readonly risk: DeviceActionRisk;
|
||||
readonly method: DeviceActionDefinition['method'];
|
||||
readonly fields: readonly DeviceActionField[];
|
||||
}
|
||||
|
||||
export interface DeviceActionResult {
|
||||
readonly actionId: string;
|
||||
readonly title: string;
|
||||
readonly status: number;
|
||||
readonly ok: boolean;
|
||||
readonly data: unknown;
|
||||
readonly message: string | null;
|
||||
readonly durationMs: number;
|
||||
}
|
||||
|
||||
export interface DeviceActionContext {
|
||||
readonly actor: string;
|
||||
readonly requestId: string;
|
||||
/** R2 and R3 actions only dispatch when the caller repeats an explicit confirmation. */
|
||||
readonly confirm?: boolean;
|
||||
}
|
||||
|
||||
const CONTROL_CHARACTERS = /[\u0000-\u0008\u000b\u000c\u000e-\u001f\u007f]/u;
|
||||
const SECRET_KEY =
|
||||
/(password|passwd|secret|token|cookie|authorization|apikey|api_key|privatekey|private_key|session|confirmation_code)/iu;
|
||||
const MAX_RESPONSE_BYTES = 32_768;
|
||||
const MAX_DEPTH = 5;
|
||||
const MAX_ARRAY_ENTRIES = 64;
|
||||
const MAX_STRING_LENGTH = 512;
|
||||
const DEFAULT_TIMEOUT_MS = 15_000;
|
||||
const TIMEOUT_SENTINEL: UpstreamResponse = { status: 0, headers: {}, body: '' };
|
||||
|
||||
function isRecord(value: unknown): value is Record<string, unknown> {
|
||||
return value !== null && typeof value === 'object' && !Array.isArray(value);
|
||||
}
|
||||
|
||||
function sanitize(value: unknown, depth = 0): unknown {
|
||||
if (value === null) return null;
|
||||
switch (typeof value) {
|
||||
case 'string':
|
||||
return value.length > MAX_STRING_LENGTH
|
||||
? `${value.slice(0, MAX_STRING_LENGTH)}...`
|
||||
: CONTROL_CHARACTERS.test(value)
|
||||
? value.replace(CONTROL_CHARACTERS, '')
|
||||
: value;
|
||||
case 'number':
|
||||
return Number.isFinite(value) ? value : null;
|
||||
case 'boolean':
|
||||
return value;
|
||||
default:
|
||||
break;
|
||||
}
|
||||
if (depth >= MAX_DEPTH) return null;
|
||||
if (Array.isArray(value))
|
||||
return value.slice(0, MAX_ARRAY_ENTRIES).map((item) => sanitize(item, depth + 1));
|
||||
if (isRecord(value)) {
|
||||
const output: Record<string, unknown> = {};
|
||||
for (const [key, entry] of Object.entries(value)) {
|
||||
if (SECRET_KEY.test(key)) continue;
|
||||
output[key] = sanitize(entry, depth + 1);
|
||||
}
|
||||
return output;
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
async function withTimeout(
|
||||
operation: Promise<UpstreamResponse>,
|
||||
timeoutMs: number,
|
||||
): Promise<UpstreamResponse> {
|
||||
let timer: ReturnType<typeof setTimeout> | undefined;
|
||||
const guard = new Promise<UpstreamResponse>((resolve) => {
|
||||
timer = setTimeout(() => resolve(TIMEOUT_SENTINEL), timeoutMs);
|
||||
});
|
||||
try {
|
||||
return await Promise.race([operation, guard]);
|
||||
} finally {
|
||||
if (timer) clearTimeout(timer);
|
||||
}
|
||||
}
|
||||
|
||||
function validateBoolean(field: DeviceActionField, raw: unknown): boolean {
|
||||
if (typeof raw !== 'boolean') throw new DeviceActionError('VALIDATION_FAILED', field.id);
|
||||
return raw;
|
||||
}
|
||||
|
||||
function validateNumber(field: DeviceActionField, raw: unknown): number {
|
||||
if (typeof raw !== 'number' || !Number.isSafeInteger(raw))
|
||||
throw new DeviceActionError('VALIDATION_FAILED', field.id);
|
||||
if (field.min !== undefined && raw < field.min)
|
||||
throw new DeviceActionError('VALIDATION_FAILED', field.id);
|
||||
if (field.max !== undefined && raw > field.max)
|
||||
throw new DeviceActionError('VALIDATION_FAILED', field.id);
|
||||
return raw;
|
||||
}
|
||||
|
||||
function validateText(field: DeviceActionField, raw: unknown): string {
|
||||
if (typeof raw !== 'string') throw new DeviceActionError('VALIDATION_FAILED', field.id);
|
||||
const value = raw.trim();
|
||||
if (value.length === 0) {
|
||||
if (field.required) throw new DeviceActionError('VALIDATION_FAILED', field.id);
|
||||
return '';
|
||||
}
|
||||
if (value.length > (field.maxLength ?? 128))
|
||||
throw new DeviceActionError('VALIDATION_FAILED', field.id);
|
||||
if (CONTROL_CHARACTERS.test(value)) throw new DeviceActionError('VALIDATION_FAILED', field.id);
|
||||
if (field.pattern !== undefined && !new RegExp(field.pattern, 'u').test(value))
|
||||
throw new DeviceActionError('VALIDATION_FAILED', field.id);
|
||||
return value;
|
||||
}
|
||||
|
||||
function validateChoice(field: DeviceActionField, raw: unknown): string {
|
||||
const value = validateText(field, raw);
|
||||
if (!value) return '';
|
||||
if (!(field.choices ?? []).some((choice) => choice.value === value))
|
||||
throw new DeviceActionError('VALIDATION_FAILED', field.id);
|
||||
return value;
|
||||
}
|
||||
|
||||
function validateList(field: DeviceActionField, raw: unknown): string[] {
|
||||
if (!Array.isArray(raw) || raw.length > 128)
|
||||
throw new DeviceActionError('VALIDATION_FAILED', field.id);
|
||||
const output: string[] = [];
|
||||
for (const item of raw) {
|
||||
if (typeof item !== 'string' || item.length > 64 || CONTROL_CHARACTERS.test(item))
|
||||
throw new DeviceActionError('VALIDATION_FAILED', field.id);
|
||||
const value = item.trim();
|
||||
if (value) output.push(value);
|
||||
}
|
||||
if (field.required && output.length === 0)
|
||||
throw new DeviceActionError('VALIDATION_FAILED', field.id);
|
||||
return output;
|
||||
}
|
||||
|
||||
/**
|
||||
* The audit reader only accepts fully redacted summaries, so a submitted value is never stored:
|
||||
* the record says which fields were supplied and leaves the content out.
|
||||
*/
|
||||
function summarize(
|
||||
action: DeviceActionDefinition,
|
||||
values: Readonly<Record<string, unknown>>,
|
||||
): readonly { fieldId: string; displayValue: string; redacted: true }[] {
|
||||
return (action.fields ?? [])
|
||||
.filter((field) => values[field.id] !== undefined)
|
||||
.map((field) => ({ fieldId: field.id, displayValue: '[REDACTED]', redacted: true as const }));
|
||||
}
|
||||
|
||||
export interface DeviceActionServiceOptions {
|
||||
readonly instances: InstanceService;
|
||||
readonly sessions: InstanceSessionStore;
|
||||
readonly request: UpstreamSessionClientOptions['request'];
|
||||
readonly db: SqliteDatabase;
|
||||
readonly ensureSession?: (instanceId: string, origin: string, force?: boolean) => Promise<void>;
|
||||
readonly now?: () => Date;
|
||||
readonly id?: () => string;
|
||||
/**
|
||||
* The identity guard. A node whose reported hardware stopped matching its record, or whose
|
||||
* identity is claimed by another record, is not safe to control: the command would land on a
|
||||
* device the operator is not looking at. Unbinding stays available because it is the way out.
|
||||
*/
|
||||
readonly identities?: { isBlocked(instanceId: string): boolean };
|
||||
}
|
||||
|
||||
/**
|
||||
* Runs an allowlisted device mutation on behalf of the console. Every dispatch is validated
|
||||
* against the catalog, audited before the response leaves the process, and re-authorized once
|
||||
* when the device session has expired.
|
||||
*/
|
||||
export class DeviceActionService {
|
||||
private readonly now: () => Date;
|
||||
private readonly id: () => string;
|
||||
|
||||
constructor(private readonly options: DeviceActionServiceOptions) {
|
||||
this.now = options.now ?? (() => new Date());
|
||||
this.id = options.id ?? randomUUID;
|
||||
}
|
||||
|
||||
list(): readonly DeviceActionDescriptor[] {
|
||||
return DEVICE_ACTIONS.map((action) =>
|
||||
Object.freeze({
|
||||
id: action.id,
|
||||
module: action.module,
|
||||
title: action.title,
|
||||
description: action.description,
|
||||
risk: action.risk,
|
||||
method: action.method,
|
||||
fields: Object.freeze(
|
||||
[...(action.fields ?? [])].map((field) => Object.freeze({ ...field })),
|
||||
),
|
||||
}),
|
||||
);
|
||||
}
|
||||
|
||||
async execute(
|
||||
instanceId: string,
|
||||
actionId: string,
|
||||
params: Readonly<Record<string, unknown>>,
|
||||
context: DeviceActionContext,
|
||||
): Promise<DeviceActionResult> {
|
||||
const action = findDeviceAction(actionId);
|
||||
if (!action) throw new DeviceActionError('NOT_FOUND');
|
||||
if (action.risk !== 'R1' && context.confirm !== true)
|
||||
throw new DeviceActionError('VALIDATION_FAILED', 'confirm');
|
||||
const instance = await this.options.instances.get(instanceId);
|
||||
if (!instance) throw new DeviceActionError('NOT_FOUND');
|
||||
const session = this.options.sessions.sessionFor(instanceId);
|
||||
if (session && session.origin !== instance.origin)
|
||||
throw new DeviceActionError('SESSION_INVALID');
|
||||
if (action.id !== 'hub.unbind' && this.options.identities?.isBlocked(instanceId))
|
||||
throw new DeviceActionError('IDENTITY_UNCONFIRMED');
|
||||
|
||||
const values = this.#validate(action, params);
|
||||
const { path, query, body } = this.#assemble(action, values);
|
||||
if (!session && this.options.ensureSession) {
|
||||
try {
|
||||
await this.options.ensureSession(instanceId, instance.origin);
|
||||
} catch {
|
||||
// Passwordless devices accept anonymous mutations; otherwise the probe reports below.
|
||||
}
|
||||
}
|
||||
|
||||
const timeoutMs = action.timeoutMs ?? DEFAULT_TIMEOUT_MS;
|
||||
const target = `${instance.origin}/api${path}${query}`;
|
||||
const send = async (cookie: string | undefined): Promise<UpstreamResponse> => {
|
||||
const headers: Record<string, string> = {};
|
||||
if (body === undefined) headers.accept = 'application/json';
|
||||
else {
|
||||
headers.accept = 'application/json';
|
||||
headers['content-type'] = 'application/json';
|
||||
}
|
||||
if (cookie) headers.cookie = cookie;
|
||||
return withTimeout(
|
||||
this.options.request({
|
||||
url: target,
|
||||
method: action.method,
|
||||
headers,
|
||||
deviceAction: { body },
|
||||
}),
|
||||
timeoutMs,
|
||||
);
|
||||
};
|
||||
|
||||
const startedAt = this.now().getTime();
|
||||
let token = this.options.sessions.sessionFor(instanceId)?.cookie;
|
||||
let response: UpstreamResponse;
|
||||
try {
|
||||
response = await send(token);
|
||||
if ((response.status === 401 || response.status === 403) && this.options.ensureSession) {
|
||||
try {
|
||||
await this.options.ensureSession(instanceId, instance.origin, true);
|
||||
} catch {
|
||||
// Keep the device answer; the result below reports the failure.
|
||||
}
|
||||
const refreshed = this.options.sessions.sessionFor(instanceId)?.cookie;
|
||||
if (refreshed && refreshed !== token) {
|
||||
token = refreshed;
|
||||
response = await send(token);
|
||||
}
|
||||
}
|
||||
} catch {
|
||||
// The pinned transport refused or could not reach the device: nothing left this process.
|
||||
const durationMs = Math.max(0, this.now().getTime() - startedAt);
|
||||
this.#audit(action, instanceId, context, values, 0, false, durationMs);
|
||||
throw new DeviceActionError('UPSTREAM_FAILED');
|
||||
}
|
||||
const durationMs = Math.max(0, this.now().getTime() - startedAt);
|
||||
const parsed = this.#interpret(response);
|
||||
this.#audit(action, instanceId, context, values, response.status, parsed.ok, durationMs);
|
||||
return {
|
||||
actionId: action.id,
|
||||
title: action.title,
|
||||
status: response.status,
|
||||
ok: parsed.ok,
|
||||
data: parsed.data,
|
||||
message: parsed.message,
|
||||
durationMs,
|
||||
};
|
||||
}
|
||||
|
||||
#validate(
|
||||
action: DeviceActionDefinition,
|
||||
params: Readonly<Record<string, unknown>>,
|
||||
): Record<string, unknown> {
|
||||
const values: Record<string, unknown> = {};
|
||||
for (const field of action.fields ?? []) {
|
||||
const raw = params[field.id];
|
||||
if (raw === undefined || raw === null) {
|
||||
if (field.required) throw new DeviceActionError('VALIDATION_FAILED', field.id);
|
||||
continue;
|
||||
}
|
||||
switch (field.kind) {
|
||||
case 'boolean':
|
||||
values[field.id] = validateBoolean(field, raw);
|
||||
break;
|
||||
case 'number':
|
||||
values[field.id] = validateNumber(field, raw);
|
||||
break;
|
||||
case 'string':
|
||||
case 'secret': {
|
||||
const text = validateText(field, raw);
|
||||
if (text) values[field.id] = text;
|
||||
break;
|
||||
}
|
||||
case 'choice': {
|
||||
const choice = validateChoice(field, raw);
|
||||
if (choice) values[field.id] = choice;
|
||||
break;
|
||||
}
|
||||
case 'string-list': {
|
||||
const list = validateList(field, raw);
|
||||
if (list.length > 0 || field.required) values[field.id] = list;
|
||||
break;
|
||||
}
|
||||
default:
|
||||
throw new DeviceActionError('VALIDATION_FAILED', field.id);
|
||||
}
|
||||
}
|
||||
const declared = new Set((action.fields ?? []).map((field) => field.id));
|
||||
for (const key of Object.keys(params))
|
||||
if (!declared.has(key)) throw new DeviceActionError('VALIDATION_FAILED', key);
|
||||
return values;
|
||||
}
|
||||
|
||||
#assemble(
|
||||
action: DeviceActionDefinition,
|
||||
values: Record<string, unknown>,
|
||||
): { path: string; query: string; body: Readonly<Record<string, unknown>> | undefined } {
|
||||
let path = action.path;
|
||||
const queryParts: string[] = [];
|
||||
const body: Record<string, unknown> = { ...(action.fixed ?? {}) };
|
||||
for (const field of action.fields ?? []) {
|
||||
const value = values[field.id];
|
||||
if (value === undefined) continue;
|
||||
const place = field.in ?? 'body';
|
||||
if (place === 'path') {
|
||||
if (!/^[A-Za-z0-9_.\-]{1,128}$/u.test(String(value)))
|
||||
throw new DeviceActionError('VALIDATION_FAILED', field.id);
|
||||
const token = `{${field.id}}`;
|
||||
if (!path.includes(token)) throw new DeviceActionError('VALIDATION_FAILED', field.id);
|
||||
path = path.replace(token, String(value));
|
||||
} else if (place === 'query') {
|
||||
const rendered = Array.isArray(value)
|
||||
? value.map((item) => encodeURIComponent(item)).join(',')
|
||||
: encodeURIComponent(String(value));
|
||||
queryParts.push(`${field.id}=${rendered}`);
|
||||
} else body[field.id] = value;
|
||||
}
|
||||
if (path.includes('{')) throw new DeviceActionError('VALIDATION_FAILED');
|
||||
const query = queryParts.length ? `?${queryParts.join('&')}` : '';
|
||||
const empty = action.emptyJsonObject === true;
|
||||
return {
|
||||
path,
|
||||
query,
|
||||
body: Object.keys(body).length || empty ? body : undefined,
|
||||
};
|
||||
}
|
||||
|
||||
#interpret(response: UpstreamResponse): {
|
||||
ok: boolean;
|
||||
data: unknown;
|
||||
message: string | null;
|
||||
} {
|
||||
if (response.status === 0) return { ok: false, data: null, message: '设备未在限定时间内响应' };
|
||||
if (Buffer.byteLength(response.body, 'utf8') > MAX_RESPONSE_BYTES)
|
||||
return { ok: false, data: null, message: '设备响应超出长度限制' };
|
||||
let root: unknown = null;
|
||||
if (response.body.trim().length > 0) {
|
||||
try {
|
||||
root = JSON.parse(response.body);
|
||||
} catch {
|
||||
root = null;
|
||||
}
|
||||
}
|
||||
const record = isRecord(root) ? root : undefined;
|
||||
const status = typeof record?.status === 'string' ? record.status : undefined;
|
||||
const ok =
|
||||
response.status >= 200 &&
|
||||
response.status < 300 &&
|
||||
(status === undefined || status === 'success' || status === 'ok');
|
||||
const rawMessage =
|
||||
(typeof record?.message === 'string' && record.message) ||
|
||||
(typeof record?.msg === 'string' && record.msg) ||
|
||||
null;
|
||||
const message = rawMessage ? rawMessage.slice(0, MAX_STRING_LENGTH) : null;
|
||||
return { ok, data: sanitize(record?.data ?? record ?? null), message };
|
||||
}
|
||||
|
||||
#audit(
|
||||
action: DeviceActionDefinition,
|
||||
instanceId: string,
|
||||
context: DeviceActionContext,
|
||||
values: Record<string, unknown>,
|
||||
status: number,
|
||||
ok: boolean,
|
||||
durationMs: number,
|
||||
): void {
|
||||
const created_at = this.now().toISOString();
|
||||
this.options.db
|
||||
.prepare(
|
||||
`INSERT INTO audit_events
|
||||
(id,instance_id,job_id,actor,operation_id,risk_level,request_id,parameters_summary_json,body_digest,result_code,duration_ms,created_at)
|
||||
VALUES (?,?,NULL,?,?,?,?,?,?,?,?,?)`,
|
||||
)
|
||||
.run(
|
||||
this.id(),
|
||||
instanceId,
|
||||
context.actor,
|
||||
action.id,
|
||||
action.risk,
|
||||
context.requestId,
|
||||
JSON.stringify(summarize(action, values)),
|
||||
null,
|
||||
ok ? 'succeeded' : 'failed',
|
||||
durationMs,
|
||||
created_at,
|
||||
);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,242 @@
|
||||
import { describe, expect, it } from 'vitest';
|
||||
import type { networkInterfaces } from 'node:os';
|
||||
import type { InstancePage } from '@multi-simadmin/contracts';
|
||||
|
||||
import {
|
||||
DeviceDiscoveryService,
|
||||
DiscoveryError,
|
||||
localRanges,
|
||||
type DiscoveryTransport,
|
||||
} from './device-discovery-service.js';
|
||||
|
||||
const INTERFACES: ReturnType<typeof networkInterfaces> = {
|
||||
en0: [
|
||||
{
|
||||
address: '192.168.1.23',
|
||||
family: 'IPv4',
|
||||
internal: false,
|
||||
netmask: '255.255.255.0',
|
||||
cidr: '192.168.1.23/24',
|
||||
mac: '',
|
||||
},
|
||||
{
|
||||
address: '127.0.0.1',
|
||||
family: 'IPv4',
|
||||
internal: true,
|
||||
netmask: '255.0.0.0',
|
||||
cidr: '127.0.0.1/8',
|
||||
mac: '',
|
||||
},
|
||||
{
|
||||
address: 'fe80::1',
|
||||
family: 'IPv6',
|
||||
internal: false,
|
||||
netmask: '',
|
||||
scopeid: 0,
|
||||
cidr: 'fe80::1/64',
|
||||
mac: '',
|
||||
},
|
||||
],
|
||||
en1: [
|
||||
{
|
||||
address: '10.20.30.40',
|
||||
family: 'IPv4',
|
||||
internal: false,
|
||||
netmask: '255.255.255.0',
|
||||
cidr: '10.20.30.40/24',
|
||||
mac: '',
|
||||
},
|
||||
],
|
||||
en2: [
|
||||
{
|
||||
address: '203.0.113.9',
|
||||
family: 'IPv4',
|
||||
internal: false,
|
||||
netmask: '255.255.255.0',
|
||||
cidr: '203.0.113.9/24',
|
||||
mac: '',
|
||||
},
|
||||
],
|
||||
};
|
||||
|
||||
function page(items: InstancePage['items'], pageSize = 200): InstancePage {
|
||||
return { items, page: { page: 1, pageSize, total: items.length } };
|
||||
}
|
||||
|
||||
const instances = {
|
||||
list: async () =>
|
||||
page([
|
||||
{
|
||||
id: 'inst-known',
|
||||
name: '已知设备',
|
||||
origin: 'http://192.168.1.50:3000',
|
||||
tags: [],
|
||||
groupId: null,
|
||||
revision: 1,
|
||||
capabilityStatus: 'unknown',
|
||||
freshness: 'unknown',
|
||||
credentialConfigured: false,
|
||||
},
|
||||
]),
|
||||
};
|
||||
|
||||
function transportFor(
|
||||
responses: Readonly<Record<string, { status: number; body: string }>>,
|
||||
): DiscoveryTransport & { urls: string[] } {
|
||||
const urls: string[] = [];
|
||||
return {
|
||||
urls,
|
||||
async get(url) {
|
||||
urls.push(url);
|
||||
const hit = responses[url];
|
||||
if (hit) return hit;
|
||||
throw new Error('ECONNREFUSED');
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
const DEVICE_BODY = JSON.stringify({
|
||||
model: 'UFI-003',
|
||||
manufacturer: 'Comtrade',
|
||||
firmware_version: '2.4.1',
|
||||
imei: '490154203237518',
|
||||
phone_number: '13800000000',
|
||||
oversized: 'x'.repeat(200),
|
||||
});
|
||||
|
||||
function service(options: {
|
||||
readonly transport: DiscoveryTransport;
|
||||
readonly now?: () => Date;
|
||||
readonly maxSessions?: number;
|
||||
}) {
|
||||
return new DeviceDiscoveryService({
|
||||
transport: options.transport,
|
||||
instances,
|
||||
interfaces: () => INTERFACES,
|
||||
ports: [3000],
|
||||
concurrency: 8,
|
||||
maxTargets: 600,
|
||||
...(options.now ? { now: options.now } : {}),
|
||||
...(options.maxSessions === undefined ? {} : { maxSessions: options.maxSessions }),
|
||||
});
|
||||
}
|
||||
|
||||
async function settled(
|
||||
discovery: DeviceDiscoveryService,
|
||||
sessionId: string,
|
||||
): Promise<Awaited<ReturnType<DeviceDiscoveryService['status']>>> {
|
||||
for (let attempt = 0; attempt < 200; attempt += 1) {
|
||||
const state = await discovery.status(sessionId);
|
||||
if (state.status === 'completed') return state;
|
||||
await new Promise((resolve) => setTimeout(resolve, 5));
|
||||
}
|
||||
throw new Error('scan did not settle');
|
||||
}
|
||||
|
||||
describe('localRanges', () => {
|
||||
it('keeps only non-internal private IPv4 subnets', () => {
|
||||
expect(localRanges(INTERFACES)).toEqual(['10.20.30.0/24', '192.168.1.0/24']);
|
||||
});
|
||||
});
|
||||
|
||||
describe('DeviceDiscoveryService', () => {
|
||||
it('reports reachable devices with whitelisted identity and known instances', async () => {
|
||||
const transport = transportFor({
|
||||
'http://192.168.1.50:3000/api/health': { status: 200, body: '{"ok":true}' },
|
||||
'http://192.168.1.50:3000/api/device': { status: 200, body: DEVICE_BODY },
|
||||
'http://10.20.30.77:3000/api/health': { status: 401, body: '' },
|
||||
});
|
||||
const discovery = service({ transport });
|
||||
const started = await discovery.start();
|
||||
expect(started.ranges).toEqual(['10.20.30.0/24', '192.168.1.0/24']);
|
||||
expect(started.total).toBe(508);
|
||||
|
||||
const state = await settled(discovery, started.sessionId);
|
||||
expect(state.scanned).toBe(508);
|
||||
expect(state.devices.map((device) => device.origin)).toEqual([
|
||||
'http://10.20.30.77:3000',
|
||||
'http://192.168.1.50:3000',
|
||||
]);
|
||||
const known = state.devices.find((device) => device.origin === 'http://192.168.1.50:3000');
|
||||
expect(known?.knownInstanceId).toBe('inst-known');
|
||||
expect(known?.identity).toEqual({
|
||||
model: 'UFI-003',
|
||||
manufacturer: 'Comtrade',
|
||||
firmware_version: '2.4.1',
|
||||
});
|
||||
const anonymous = state.devices.find((device) => device.origin === 'http://10.20.30.77:3000');
|
||||
expect(anonymous?.httpStatus).toBe(401);
|
||||
expect(anonymous?.knownInstanceId).toBeNull();
|
||||
expect(anonymous?.identity).toEqual({});
|
||||
expect(transport.urls.every((url) => /\/api\/(health|device)$/u.test(url))).toBe(true);
|
||||
});
|
||||
|
||||
it('drops an expired lease for good and closes a live session on demand', async () => {
|
||||
let now = new Date('2026-09-04T10:00:00.000Z');
|
||||
const transport = transportFor({});
|
||||
const discovery = service({ transport, now: () => now });
|
||||
const started = await discovery.start();
|
||||
now = new Date('2026-09-04T11:00:00.000Z');
|
||||
await expect(discovery.renew(started.sessionId)).rejects.toBeInstanceOf(DiscoveryError);
|
||||
// A lease that lapsed is never resurrected, even if the clock comes back.
|
||||
now = new Date('2026-09-04T10:00:30.000Z');
|
||||
await expect(discovery.status(started.sessionId)).rejects.toMatchObject({
|
||||
code: 'NOT_FOUND',
|
||||
});
|
||||
|
||||
const live = await discovery.start();
|
||||
const renewed = await discovery.renew(live.sessionId);
|
||||
expect(renewed.expiresAt).toBe('2026-09-04T10:01:30.000Z');
|
||||
expect((await discovery.status(live.sessionId)).sessionId).toBe(live.sessionId);
|
||||
await discovery.stop(live.sessionId);
|
||||
await expect(discovery.status(live.sessionId)).rejects.toMatchObject({ code: 'NOT_FOUND' });
|
||||
await expect(discovery.stop(live.sessionId)).resolves.toBeUndefined();
|
||||
});
|
||||
|
||||
it('refuses more sessions than the configured ceiling', async () => {
|
||||
const discovery = service({ transport: transportFor({}), maxSessions: 1 });
|
||||
await discovery.start();
|
||||
await expect(discovery.start()).rejects.toMatchObject({ code: 'TOO_MANY_SESSIONS' });
|
||||
});
|
||||
|
||||
it('probes a manually entered address and normalises the origin', async () => {
|
||||
const transport = transportFor({
|
||||
'http://192.168.68.1:3000/api/health': { status: 200, body: '{}' },
|
||||
'http://192.168.68.1:3000/api/device': { status: 200, body: DEVICE_BODY },
|
||||
});
|
||||
const discovery = service({ transport });
|
||||
const probe = await discovery.probe('192.168.68.1:3000');
|
||||
expect(probe).toMatchObject({
|
||||
origin: 'http://192.168.68.1:3000',
|
||||
reachable: true,
|
||||
httpStatus: 200,
|
||||
knownInstanceId: null,
|
||||
});
|
||||
expect(probe.identity.model).toBe('UFI-003');
|
||||
expect('imei' in probe.identity).toBe(false);
|
||||
expect('phone_number' in probe.identity).toBe(false);
|
||||
expect('oversized' in probe.identity).toBe(false);
|
||||
});
|
||||
|
||||
it('marks an unreachable address without failing the request', async () => {
|
||||
const probe = await service({ transport: transportFor({}) }).probe('http://10.0.0.9:3000');
|
||||
expect(probe).toMatchObject({ reachable: false, httpStatus: null, identity: {} });
|
||||
});
|
||||
|
||||
it('rejects addresses that are not a private LAN root URL', async () => {
|
||||
const discovery = service({ transport: transportFor({}) });
|
||||
const rejected = [
|
||||
undefined,
|
||||
'https://example.com',
|
||||
'http://8.8.8.8:3000',
|
||||
'http://admin:p@ss@192.168.1.5:3000',
|
||||
'http://192.168.1.5:3000/api/device',
|
||||
'http://192.168.1.5:3000/?next=x',
|
||||
'ftp://192.168.1.5:3000',
|
||||
'http://192.168.1.5:99999',
|
||||
];
|
||||
for (const value of rejected) {
|
||||
await expect(discovery.probe(value)).rejects.toMatchObject({ code: 'VALIDATION_FAILED' });
|
||||
}
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,392 @@
|
||||
/**
|
||||
* LAN discovery for SimAdmin devices, fused from the Hub "局域网发现 + 地址接入" onboarding flow.
|
||||
*
|
||||
* The scan is deliberately narrow: it only ever dials literal private IPv4 addresses taken from
|
||||
* the host's own network interfaces, on a fixed port set, with one read-only probe path. Nothing
|
||||
* here accepts a caller-supplied host, so the endpoint cannot be turned into an SSRF oracle.
|
||||
*/
|
||||
import { networkInterfaces } from 'node:os';
|
||||
import { randomUUID } from 'node:crypto';
|
||||
import type { InstancePage, InstancePageQuery } from '@multi-simadmin/contracts';
|
||||
|
||||
export interface DiscoveryTransport {
|
||||
get(url: string): Promise<{ readonly status: number; readonly body: string }>;
|
||||
}
|
||||
|
||||
export interface KnownInstanceOrigin {
|
||||
list(query: InstancePageQuery): Promise<InstancePage>;
|
||||
}
|
||||
|
||||
export type DiscoveredDevice = Readonly<{
|
||||
origin: string;
|
||||
address: string;
|
||||
port: number;
|
||||
secure: boolean;
|
||||
httpStatus: number;
|
||||
identity: Readonly<Record<string, string>>;
|
||||
knownInstanceId: string | null;
|
||||
}>;
|
||||
|
||||
export type DiscoverySession = Readonly<{
|
||||
sessionId: string;
|
||||
status: 'scanning' | 'completed';
|
||||
createdAt: string;
|
||||
expiresAt: string;
|
||||
scanned: number;
|
||||
total: number;
|
||||
ranges: readonly string[];
|
||||
devices: readonly DiscoveredDevice[];
|
||||
}>;
|
||||
|
||||
export type DeviceProbe = Readonly<{
|
||||
origin: string;
|
||||
reachable: boolean;
|
||||
httpStatus: number | null;
|
||||
identity: Readonly<Record<string, string>>;
|
||||
knownInstanceId: string | null;
|
||||
}>;
|
||||
|
||||
export type DiscoveryErrorCode = 'VALIDATION_FAILED' | 'NOT_FOUND' | 'TOO_MANY_SESSIONS';
|
||||
|
||||
export class DiscoveryError extends Error {
|
||||
constructor(
|
||||
readonly code: DiscoveryErrorCode,
|
||||
message: string,
|
||||
) {
|
||||
super(message);
|
||||
this.name = 'DiscoveryError';
|
||||
}
|
||||
}
|
||||
|
||||
export interface DeviceDiscoveryServiceOptions {
|
||||
readonly transport: DiscoveryTransport;
|
||||
readonly instances: KnownInstanceOrigin;
|
||||
readonly interfaces?: () => ReturnType<typeof networkInterfaces>;
|
||||
readonly now?: () => Date;
|
||||
readonly idFactory?: () => string;
|
||||
readonly ports?: readonly number[];
|
||||
readonly leaseMs?: number;
|
||||
readonly concurrency?: number;
|
||||
readonly maxTargets?: number;
|
||||
readonly maxSessions?: number;
|
||||
}
|
||||
|
||||
/** SimAdmin serves its console on 3000; 8080/8443 cover the usual reverse-proxy setups. */
|
||||
export const DEFAULT_DISCOVERY_PORTS: readonly number[] = [3000, 8080, 8443];
|
||||
|
||||
const DEFAULT_LEASE_MS = 60_000;
|
||||
const DEFAULT_CONCURRENCY = 48;
|
||||
const DEFAULT_MAX_TARGETS = 1_024;
|
||||
const DEFAULT_MAX_SESSIONS = 4;
|
||||
const HEALTH_PATH = '/api/health';
|
||||
const DEVICE_PATH = '/api/device';
|
||||
/** Only non-identifying fields cross the boundary; IMEI, ICCI and phone numbers never do. */
|
||||
const IDENTITY_FIELDS: readonly string[] = [
|
||||
'model',
|
||||
'manufacturer',
|
||||
'brand',
|
||||
'firmware_version',
|
||||
'os_version',
|
||||
'version',
|
||||
];
|
||||
const MAX_IDENTITY_VALUE_LENGTH = 80;
|
||||
const MAX_IDENTITY_FIELDS = 6;
|
||||
|
||||
type Target = Readonly<{ origin: string; address: string; port: number; secure: boolean }>;
|
||||
|
||||
interface SessionRecord {
|
||||
readonly sessionId: string;
|
||||
readonly createdAt: Date;
|
||||
expiresAt: Date;
|
||||
status: 'scanning' | 'completed';
|
||||
closed: boolean;
|
||||
scanned: number;
|
||||
readonly total: number;
|
||||
readonly ranges: readonly string[];
|
||||
readonly devices: DiscoveredDevice[];
|
||||
}
|
||||
|
||||
function privateV4(address: string): boolean {
|
||||
const parts = address.split('.');
|
||||
if (parts.length !== 4 || parts.some((part) => !/^\d{1,3}$/.test(part))) return false;
|
||||
const octets = parts.map(Number);
|
||||
if (octets.some((part) => part > 255)) return false;
|
||||
const [a, b] = octets as [number, number];
|
||||
return a === 10 || (a === 172 && b >= 16 && b <= 31) || (a === 192 && b === 168);
|
||||
}
|
||||
|
||||
function securePort(port: number): boolean {
|
||||
return port === 443 || port === 8443;
|
||||
}
|
||||
|
||||
function originFor(address: string, port: number): string {
|
||||
return `${securePort(port) ? 'https' : 'http'}://${address}:${port}`;
|
||||
}
|
||||
|
||||
/** Every /24 the host itself sits on, plus an optional caller-supplied /24 in dotted form. */
|
||||
export function localRanges(interfaces: ReturnType<typeof networkInterfaces>): readonly string[] {
|
||||
const ranges = new Set<string>();
|
||||
for (const entries of Object.values(interfaces)) {
|
||||
for (const entry of entries ?? []) {
|
||||
if (entry.internal || entry.family !== 'IPv4') continue;
|
||||
if (!privateV4(entry.address)) continue;
|
||||
const [a, b, c] = entry.address.split('.');
|
||||
if (!a || !b || !c) continue;
|
||||
ranges.add(`${a}.${b}.${c}.0/24`);
|
||||
}
|
||||
}
|
||||
return [...ranges].sort();
|
||||
}
|
||||
|
||||
function expand(ranges: readonly string[], ports: readonly number[], limit: number): Target[] {
|
||||
const targets: Target[] = [];
|
||||
const seen = new Set<string>();
|
||||
for (const range of ranges) {
|
||||
const prefix = range.replace(/\.0\/24$/u, '');
|
||||
for (let host = 1; host <= 254 && targets.length < limit; host += 1) {
|
||||
const address = `${prefix}.${host}`;
|
||||
for (const port of ports) {
|
||||
if (targets.length >= limit) break;
|
||||
const key = `${address}:${port}`;
|
||||
if (seen.has(key)) continue;
|
||||
seen.add(key);
|
||||
targets.push({ origin: originFor(address, port), address, port, secure: securePort(port) });
|
||||
}
|
||||
}
|
||||
}
|
||||
return targets;
|
||||
}
|
||||
|
||||
function identity(payload: unknown): Readonly<Record<string, string>> {
|
||||
if (typeof payload !== 'object' || payload === null) return {};
|
||||
const source = payload as Record<string, unknown>;
|
||||
const result: Record<string, string> = {};
|
||||
for (const field of IDENTITY_FIELDS) {
|
||||
if (Object.keys(result).length >= MAX_IDENTITY_FIELDS) break;
|
||||
const value = source[field];
|
||||
if (typeof value !== 'string' && typeof value !== 'number') continue;
|
||||
const text = String(value).trim();
|
||||
if (!text || text.length > MAX_IDENTITY_VALUE_LENGTH || /[\x00-\x1F\x7F]/u.test(text)) continue;
|
||||
result[field] = text;
|
||||
}
|
||||
return result;
|
||||
}
|
||||
|
||||
function parseJson(body: string): unknown {
|
||||
try {
|
||||
return JSON.parse(body) as unknown;
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
function normalizeDeviceUrl(raw: unknown): Target {
|
||||
if (typeof raw !== 'string')
|
||||
throw new DiscoveryError('VALIDATION_FAILED', '设备地址必须是字符串');
|
||||
const candidate = raw.trim();
|
||||
let url: URL;
|
||||
try {
|
||||
url = new URL(candidate.includes('://') ? candidate : `http://${candidate}`);
|
||||
} catch {
|
||||
throw new DiscoveryError('VALIDATION_FAILED', '设备地址格式不正确');
|
||||
}
|
||||
if (url.protocol !== 'http:' && url.protocol !== 'https:')
|
||||
throw new DiscoveryError('VALIDATION_FAILED', '设备地址必须使用 HTTP 或 HTTPS');
|
||||
if (url.username || url.password || url.search || url.hash)
|
||||
throw new DiscoveryError('VALIDATION_FAILED', '设备地址不能包含凭据、查询或片段');
|
||||
if (url.pathname !== '/' && url.pathname !== '')
|
||||
throw new DiscoveryError('VALIDATION_FAILED', '设备地址只能是根地址');
|
||||
const address = url.hostname.replace(/^\[|\]$/g, '');
|
||||
if (!privateV4(address))
|
||||
throw new DiscoveryError('VALIDATION_FAILED', '设备地址必须是私有局域网 IPv4 地址');
|
||||
const port = url.port ? Number(url.port) : url.protocol === 'https:' ? 443 : 80;
|
||||
if (!Number.isInteger(port) || port < 1 || port > 65_535)
|
||||
throw new DiscoveryError('VALIDATION_FAILED', '设备地址端口不正确');
|
||||
return { origin: url.origin, address, port, secure: url.protocol === 'https:' };
|
||||
}
|
||||
|
||||
export class DeviceDiscoveryService {
|
||||
private readonly sessions = new Map<string, SessionRecord>();
|
||||
private readonly ranges: () => ReturnType<typeof networkInterfaces>;
|
||||
private readonly clock: () => Date;
|
||||
private readonly id: () => string;
|
||||
private readonly ports: readonly number[];
|
||||
private readonly leaseMs: number;
|
||||
private readonly concurrency: number;
|
||||
private readonly maxTargets: number;
|
||||
private readonly maxSessions: number;
|
||||
|
||||
constructor(private readonly options: DeviceDiscoveryServiceOptions) {
|
||||
this.ranges = options.interfaces ?? networkInterfaces;
|
||||
this.clock = options.now ?? (() => new Date());
|
||||
this.id = options.idFactory ?? randomUUID;
|
||||
this.ports = options.ports ?? DEFAULT_DISCOVERY_PORTS;
|
||||
this.leaseMs = options.leaseMs ?? DEFAULT_LEASE_MS;
|
||||
this.concurrency = options.concurrency ?? DEFAULT_CONCURRENCY;
|
||||
this.maxTargets = options.maxTargets ?? DEFAULT_MAX_TARGETS;
|
||||
this.maxSessions = options.maxSessions ?? DEFAULT_MAX_SESSIONS;
|
||||
}
|
||||
|
||||
/** Opens a short-lease scan; the caller renews it while the wizard stays open. */
|
||||
async start(): Promise<DiscoverySession> {
|
||||
this.#evict();
|
||||
if (this.sessions.size >= this.maxSessions)
|
||||
throw new DiscoveryError('TOO_MANY_SESSIONS', '设备发现会话数量已达上限');
|
||||
const ranges = localRanges(this.ranges());
|
||||
const targets = expand(ranges, this.ports, this.maxTargets);
|
||||
const now = this.clock();
|
||||
const record: SessionRecord = {
|
||||
sessionId: this.id(),
|
||||
createdAt: now,
|
||||
expiresAt: new Date(now.getTime() + this.leaseMs),
|
||||
status: 'scanning',
|
||||
closed: false,
|
||||
scanned: 0,
|
||||
total: targets.length,
|
||||
ranges,
|
||||
devices: [],
|
||||
};
|
||||
this.sessions.set(record.sessionId, record);
|
||||
void this.#sweep(record, targets);
|
||||
return this.#view(record);
|
||||
}
|
||||
|
||||
async renew(sessionId: string): Promise<DiscoverySession> {
|
||||
const record = this.#require(sessionId);
|
||||
record.expiresAt = new Date(this.clock().getTime() + this.leaseMs);
|
||||
return this.#view(record);
|
||||
}
|
||||
|
||||
async status(sessionId: string): Promise<DiscoverySession> {
|
||||
return this.#view(this.#require(sessionId));
|
||||
}
|
||||
|
||||
async stop(sessionId: string): Promise<void> {
|
||||
const record = this.#sessions().get(sessionId);
|
||||
if (!record) return;
|
||||
record.closed = true;
|
||||
this.sessions.delete(sessionId);
|
||||
}
|
||||
|
||||
/** Single-address check used by the manual "输入设备地址" step of the wizard. */
|
||||
async probe(rawUrl: unknown): Promise<DeviceProbe> {
|
||||
const target = normalizeDeviceUrl(rawUrl);
|
||||
const known = await this.#knownOrigins();
|
||||
const knownInstanceId = known.get(target.origin) ?? null;
|
||||
const response = await this.options.transport
|
||||
.get(`${target.origin}${HEALTH_PATH}`)
|
||||
.catch(() => null);
|
||||
if (!response)
|
||||
return {
|
||||
origin: target.origin,
|
||||
reachable: false,
|
||||
httpStatus: null,
|
||||
identity: {},
|
||||
knownInstanceId,
|
||||
};
|
||||
return {
|
||||
origin: target.origin,
|
||||
reachable: true,
|
||||
httpStatus: response.status,
|
||||
identity: await this.#identify(target),
|
||||
knownInstanceId,
|
||||
};
|
||||
}
|
||||
|
||||
async #sweep(record: SessionRecord, targets: readonly Target[]): Promise<void> {
|
||||
const known = await this.#knownOrigins().catch(() => new Map<string, string>());
|
||||
let cursor = 0;
|
||||
const worker = async (): Promise<void> => {
|
||||
for (;;) {
|
||||
if (record.closed || this.clock() >= record.expiresAt) return;
|
||||
const index = cursor;
|
||||
cursor += 1;
|
||||
if (index >= targets.length) return;
|
||||
const target = targets[index]!;
|
||||
const device = await this.#inspect(target, known);
|
||||
record.scanned += 1;
|
||||
if (device && !record.closed) record.devices.push(device);
|
||||
}
|
||||
};
|
||||
const lanes = Math.max(1, Math.min(this.concurrency, targets.length || 1));
|
||||
await Promise.all(Array.from({ length: lanes }, () => worker()));
|
||||
if (!record.closed) record.status = 'completed';
|
||||
}
|
||||
|
||||
async #inspect(
|
||||
target: Target,
|
||||
known: ReadonlyMap<string, string>,
|
||||
): Promise<DiscoveredDevice | null> {
|
||||
const response = await this.options.transport
|
||||
.get(`${target.origin}${HEALTH_PATH}`)
|
||||
.catch(() => null);
|
||||
if (!response) return null;
|
||||
return {
|
||||
origin: target.origin,
|
||||
address: target.address,
|
||||
port: target.port,
|
||||
secure: target.secure,
|
||||
httpStatus: response.status,
|
||||
identity: await this.#identify(target),
|
||||
knownInstanceId: known.get(target.origin) ?? null,
|
||||
};
|
||||
}
|
||||
|
||||
async #identify(target: Target): Promise<Readonly<Record<string, string>>> {
|
||||
const response = await this.options.transport
|
||||
.get(`${target.origin}${DEVICE_PATH}`)
|
||||
.catch(() => null);
|
||||
if (!response || response.status >= 400) return {};
|
||||
return identity(parseJson(response.body));
|
||||
}
|
||||
|
||||
async #knownOrigins(): Promise<Map<string, string>> {
|
||||
const map = new Map<string, string>();
|
||||
let page = 1;
|
||||
while (page <= 10) {
|
||||
const result = await this.options.instances.list({ page, pageSize: 200 });
|
||||
for (const instance of result.items) map.set(instance.origin, instance.id);
|
||||
const seen = page * result.page.pageSize;
|
||||
if (seen >= result.page.total || result.items.length === 0) break;
|
||||
page += 1;
|
||||
}
|
||||
return map;
|
||||
}
|
||||
|
||||
#require(sessionId: string): SessionRecord {
|
||||
const record = this.#sessions().get(sessionId);
|
||||
if (!record) throw new DiscoveryError('NOT_FOUND', '设备发现会话不存在或已过期');
|
||||
return record;
|
||||
}
|
||||
|
||||
/**
|
||||
* Leases are dropped lazily: a session lives exactly as long as its lease, so a wizard that
|
||||
* stops renewing simply loses its scan and has to start a new one. Nothing outlives the lease.
|
||||
*/
|
||||
#sessions(): Map<string, SessionRecord> {
|
||||
this.#evict();
|
||||
return this.sessions;
|
||||
}
|
||||
|
||||
#evict(): void {
|
||||
const now = this.clock();
|
||||
for (const [key, record] of this.sessions) {
|
||||
if (record.expiresAt.getTime() <= now.getTime()) this.sessions.delete(key);
|
||||
}
|
||||
}
|
||||
|
||||
#view(record: SessionRecord): DiscoverySession {
|
||||
return {
|
||||
sessionId: record.sessionId,
|
||||
status: record.closed ? 'completed' : record.status,
|
||||
createdAt: record.createdAt.toISOString(),
|
||||
expiresAt: record.expiresAt.toISOString(),
|
||||
scanned: record.scanned,
|
||||
total: record.total,
|
||||
ranges: record.ranges,
|
||||
devices: [...record.devices].sort((left, right) =>
|
||||
left.address.localeCompare(right.address, 'en', { numeric: true }),
|
||||
),
|
||||
};
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,134 @@
|
||||
/**
|
||||
* Read-only catalog of SimAdmin device endpoints, grouped by the console module that renders
|
||||
* them. Every entry is a GET-safe probe: the control plane never mutates a device through this
|
||||
* catalog, so a module read is always safe to retry and safe to run on a schedule.
|
||||
*/
|
||||
export type InstanceModuleKey =
|
||||
| 'overview'
|
||||
| 'sim'
|
||||
| 'cellular'
|
||||
| 'device-network'
|
||||
| 'esim'
|
||||
| 'calls'
|
||||
| 'configuration'
|
||||
| 'device-backup'
|
||||
| 'notifications'
|
||||
| 'automation'
|
||||
| 'ota'
|
||||
| 'vowifi';
|
||||
|
||||
export interface ModuleProbe {
|
||||
/** Stable identifier the console renders; never a device-controlled value. */
|
||||
readonly key: string;
|
||||
/** Path appended to the instance origin, always rooted at the device /api namespace. */
|
||||
readonly path: string;
|
||||
readonly timeoutMs?: number;
|
||||
}
|
||||
|
||||
export const INSTANCE_MODULE_KEYS: readonly InstanceModuleKey[] = [
|
||||
'overview',
|
||||
'sim',
|
||||
'cellular',
|
||||
'device-network',
|
||||
'esim',
|
||||
'calls',
|
||||
'configuration',
|
||||
'device-backup',
|
||||
'notifications',
|
||||
'automation',
|
||||
'ota',
|
||||
'vowifi',
|
||||
];
|
||||
|
||||
export const INSTANCE_MODULE_PROBES: Readonly<Record<InstanceModuleKey, readonly ModuleProbe[]>> = {
|
||||
overview: [
|
||||
{ key: 'device', path: '/device' },
|
||||
{ key: 'sim', path: '/sim' },
|
||||
{ key: 'network', path: '/network' },
|
||||
{ key: 'stats', path: '/stats' },
|
||||
{ key: 'connectivity', path: '/connectivity' },
|
||||
{ key: 'data', path: '/data' },
|
||||
{ key: 'cpu', path: '/stats/cpu' },
|
||||
{ key: 'smsStats', path: '/sms/stats' },
|
||||
{ key: 'networkSpeed', path: '/network/speed' },
|
||||
],
|
||||
sim: [
|
||||
{ key: 'sim', path: '/sim' },
|
||||
{ key: 'apn', path: '/apn' },
|
||||
{ key: 'bandLock', path: '/band-lock' },
|
||||
{ key: 'cellLock', path: '/cell-lock' },
|
||||
],
|
||||
cellular: [
|
||||
{ key: 'network', path: '/network' },
|
||||
{ key: 'signalStrength', path: '/network/signal-strength' },
|
||||
{ key: 'cells', path: '/cells' },
|
||||
{ key: 'cellLocation', path: '/location/cell-info' },
|
||||
{ key: 'operators', path: '/network/operators' },
|
||||
{ key: 'roaming', path: '/roaming' },
|
||||
{ key: 'radioMode', path: '/radio-mode' },
|
||||
{ key: 'airplaneMode', path: '/airplane-mode' },
|
||||
{ key: 'basebandRestart', path: '/baseband/restart/status' },
|
||||
{ key: 'cellMonitor', path: '/cell-monitor/status' },
|
||||
],
|
||||
'device-network': [
|
||||
{ key: 'interfaces', path: '/network/interfaces' },
|
||||
{ key: 'addresses', path: '/network/connection-addresses' },
|
||||
{ key: 'wlanStatus', path: '/device-network/wlan/status' },
|
||||
{ key: 'wlanProfiles', path: '/device-network/wlan/profiles' },
|
||||
{ key: 'ddnsStatus', path: '/device-network/ddns/status' },
|
||||
{ key: 'ddnsConfig', path: '/device-network/ddns/config' },
|
||||
{ key: 'ddnsLogs', path: '/device-network/ddns/logs' },
|
||||
],
|
||||
esim: [
|
||||
{ key: 'euicc', path: '/esim/euicc', timeoutMs: 30_000 },
|
||||
{ key: 'profiles', path: '/esim/profiles?cached=1' },
|
||||
{ key: 'config', path: '/esim/config' },
|
||||
{ key: 'lpacStatus', path: '/esim/lpac/status' },
|
||||
],
|
||||
calls: [
|
||||
{ key: 'calls', path: '/calls' },
|
||||
{ key: 'history', path: '/call/history?limit=50' },
|
||||
{ key: 'settings', path: '/call/settings' },
|
||||
{ key: 'forwarding', path: '/call/forwarding' },
|
||||
{ key: 'volume', path: '/call/volume' },
|
||||
{ key: 'voicemail', path: '/voicemail/status' },
|
||||
{ key: 'ims', path: '/ims/status' },
|
||||
],
|
||||
configuration: [
|
||||
{ key: 'workMode', path: '/work-mode' },
|
||||
{ key: 'authSettings', path: '/auth/settings' },
|
||||
{ key: 'authStatus', path: '/auth/status' },
|
||||
{ key: 'hub', path: '/hub' },
|
||||
],
|
||||
'device-backup': [
|
||||
{ key: 'files', path: '/backup/files' },
|
||||
{ key: 'config', path: '/backup/config' },
|
||||
{ key: 'options', path: '/backup/options' },
|
||||
],
|
||||
notifications: [
|
||||
{ key: 'config', path: '/notifications/config' },
|
||||
{ key: 'queue', path: '/notifications/queue?limit=50' },
|
||||
{ key: 'logs', path: '/notifications/logs?limit=50' },
|
||||
],
|
||||
automation: [
|
||||
{ key: 'config', path: '/automation/config' },
|
||||
{ key: 'logs', path: '/automation/logs' },
|
||||
],
|
||||
ota: [{ key: 'status', path: '/ota/status' }],
|
||||
vowifi: [
|
||||
{ key: 'status', path: '/vowifi/status' },
|
||||
{ key: 'control', path: '/vowifi/control' },
|
||||
{ key: 'profile', path: '/vowifi/profile' },
|
||||
{ key: 'profiles', path: '/vowifi/profiles' },
|
||||
// The aggregate diagnostics feed carries the registration timeline the Hub shows.
|
||||
{ key: 'diagnostics', path: '/vowifi/diagnostics?limit=50', timeoutMs: 30_000 },
|
||||
{ key: 'events', path: '/vowifi/events?limit=50', timeoutMs: 10_000 },
|
||||
{ key: 'smsDeliveries', path: '/vowifi/sms/delivery?limit=20', timeoutMs: 10_000 },
|
||||
{ key: 'soakRuns', path: '/vowifi/soak?limit=20', timeoutMs: 10_000 },
|
||||
{ key: 'restore', path: '/vowifi/esim-restore/status', timeoutMs: 10_000 },
|
||||
],
|
||||
};
|
||||
|
||||
export function isInstanceModuleKey(value: unknown): value is InstanceModuleKey {
|
||||
return typeof value === 'string' && (INSTANCE_MODULE_KEYS as readonly string[]).includes(value);
|
||||
}
|
||||
@@ -0,0 +1,232 @@
|
||||
import { describe, expect, it } from 'vitest';
|
||||
|
||||
import {
|
||||
InstanceSessionStore,
|
||||
type UpstreamRequest,
|
||||
} from '../connections/upstream-session-client.js';
|
||||
import type { InstanceService } from '../instances/instance-service.js';
|
||||
import { InstanceModuleService } from './instance-module-service.js';
|
||||
|
||||
interface Reply {
|
||||
readonly status: number;
|
||||
readonly body?: unknown;
|
||||
}
|
||||
|
||||
function fixture(
|
||||
probes: Readonly<Record<string, Reply>>,
|
||||
extra: {
|
||||
readonly onIdentity?: (
|
||||
instanceId: string,
|
||||
evidence: Readonly<Record<string, unknown>>,
|
||||
origin: string,
|
||||
) => void;
|
||||
} = {},
|
||||
) {
|
||||
const sessions = new InstanceSessionStore();
|
||||
const calls: UpstreamRequest[] = [];
|
||||
const instances = {
|
||||
get: async (id: string) =>
|
||||
id === 'node-a'
|
||||
? {
|
||||
id: 'node-a',
|
||||
name: 'Node A',
|
||||
origin: 'http://node-a.local',
|
||||
tags: [],
|
||||
groupId: null,
|
||||
revision: 1,
|
||||
capabilityStatus: 'unknown',
|
||||
freshness: 'unknown',
|
||||
credentialConfigured: false,
|
||||
}
|
||||
: undefined,
|
||||
} as unknown as InstanceService;
|
||||
const service = new InstanceModuleService({
|
||||
instances,
|
||||
sessions,
|
||||
request: async (request) => {
|
||||
calls.push(request);
|
||||
const path = request.url.slice('http://node-a.local/api'.length);
|
||||
const reply = probes[path] ?? { status: 404 };
|
||||
return {
|
||||
status: reply.status,
|
||||
headers: {},
|
||||
body: reply.body === undefined ? '' : JSON.stringify(reply.body),
|
||||
};
|
||||
},
|
||||
now: () => new Date('2026-09-04T00:00:00.000Z'),
|
||||
...(extra.onIdentity ? { onIdentity: extra.onIdentity } : {}),
|
||||
});
|
||||
return { service, sessions, calls };
|
||||
}
|
||||
|
||||
describe('InstanceModuleService', () => {
|
||||
it('reads every probe for a module and classifies each section', async () => {
|
||||
const { service } = fixture({
|
||||
'/device': { status: 200, body: { model: 'LPAX', android_version: '13' } },
|
||||
'/stats': { status: 200, body: { cpu_percent: 12 } },
|
||||
'/connectivity': { status: 200, body: {} },
|
||||
'/network/speed': {
|
||||
status: 200,
|
||||
body: { interfaces: [{ interface: 'wwan0', rx_bytes_per_sec: 1536 }] },
|
||||
},
|
||||
});
|
||||
const snapshot = await service.read('node-a', 'overview');
|
||||
expect(snapshot.observedAt).toBe('2026-09-04T00:00:00.000Z');
|
||||
const byKey = Object.fromEntries(snapshot.sections.map((section) => [section.key, section]));
|
||||
expect(byKey.device?.state).toBe('ok');
|
||||
expect(byKey.device?.data).toMatchObject({ model: 'LPAX' });
|
||||
expect(byKey.stats?.state).toBe('ok');
|
||||
expect(byKey.connectivity?.state).toBe('empty');
|
||||
expect(byKey.networkSpeed?.state).toBe('ok');
|
||||
expect(byKey.networkSpeed?.data).toMatchObject({
|
||||
interfaces: [{ interface: 'wwan0', rx_bytes_per_sec: 1536 }],
|
||||
});
|
||||
expect(byKey.data?.state).toBe('unsupported');
|
||||
});
|
||||
|
||||
it('reports auth-required when the device rejects an anonymous read', async () => {
|
||||
const { service } = fixture({
|
||||
'/sim': { status: 401 },
|
||||
'/apn': { status: 200, body: { apns: [] } },
|
||||
'/band-lock': { status: 200, body: { locked: false } },
|
||||
'/cell-lock': { status: 200, body: { locked: false } },
|
||||
});
|
||||
const snapshot = await service.read('node-a', 'sim');
|
||||
const sim = snapshot.sections.find((section) => section.key === 'sim');
|
||||
expect(sim?.state).toBe('auth-required');
|
||||
expect(sim?.status).toBe(401);
|
||||
});
|
||||
|
||||
it('retries once with a refreshed session and keeps the authenticated answer', async () => {
|
||||
const sessions = new InstanceSessionStore();
|
||||
let attempts = 0;
|
||||
const service = new InstanceModuleService({
|
||||
instances: {
|
||||
get: async () => ({ id: 'node-a', origin: 'http://node-a.local' }),
|
||||
} as unknown as InstanceService,
|
||||
sessions,
|
||||
request: async () => {
|
||||
attempts += 1;
|
||||
return attempts === 1
|
||||
? { status: 401, headers: {}, body: '' }
|
||||
: { status: 200, headers: {}, body: JSON.stringify({ imei: '123' }) };
|
||||
},
|
||||
ensureSession: async () => {
|
||||
sessions.set('node-a', 'http://node-a.local', 'simadmin_session=abc');
|
||||
},
|
||||
});
|
||||
const snapshot = await service.read('node-a', 'sim');
|
||||
expect(snapshot.authenticated).toBe(true);
|
||||
expect(snapshot.sections.find((section) => section.key === 'sim')?.state).toBe('ok');
|
||||
expect(attempts).toBeGreaterThan(1);
|
||||
});
|
||||
|
||||
it('redacts credential-looking fields and bounds oversized payloads', async () => {
|
||||
const { service } = fixture({
|
||||
'/ota/status': {
|
||||
status: 200,
|
||||
body: {
|
||||
version: '1.2.3',
|
||||
admin_password: 'hunter2',
|
||||
session_token: 'nope',
|
||||
nested: { log: 'x'.repeat(5_000) },
|
||||
},
|
||||
},
|
||||
});
|
||||
const snapshot = await service.read('node-a', 'ota');
|
||||
const raw = JSON.stringify(snapshot);
|
||||
expect(raw).not.toContain('hunter2');
|
||||
expect(raw).not.toContain('nope');
|
||||
const section = snapshot.sections[0];
|
||||
const data = section?.data as { nested?: { log?: string } };
|
||||
expect((data.nested?.log ?? '').length).toBeLessThanOrEqual(2_004);
|
||||
});
|
||||
|
||||
it('turns a hanging probe into a failed section instead of blocking the module', async () => {
|
||||
const hanging = new InstanceModuleService({
|
||||
instances: {
|
||||
get: async () => ({ id: 'node-a', origin: 'http://node-a.local' }),
|
||||
} as unknown as InstanceService,
|
||||
sessions: new InstanceSessionStore(),
|
||||
request: async (request) => {
|
||||
if (request.url.endsWith('/device')) return new Promise<never>(() => {});
|
||||
return { status: 200, headers: {}, body: '{}' };
|
||||
},
|
||||
defaultTimeoutMs: 100,
|
||||
});
|
||||
const snapshot = await hanging.read('node-a', 'overview');
|
||||
expect(snapshot.sections.find((section) => section.key === 'device')?.state).toBe('failed');
|
||||
expect(snapshot.sections.find((section) => section.key === 'stats')?.state).toBe('empty');
|
||||
});
|
||||
|
||||
it('unwraps the device response envelope so panels see the real payload', async () => {
|
||||
const { service } = fixture({
|
||||
'/device': {
|
||||
status: 200,
|
||||
body: { status: 'ok', message: 'Success', data: { model: 'LPAX', imei: '123' } },
|
||||
},
|
||||
'/stats': { status: 200, body: { status: 'ok', message: 'Success', data: {} } },
|
||||
'/connectivity': {
|
||||
status: 200,
|
||||
body: { status: 'error', message: 'Connectivity is not exposed by this backend' },
|
||||
},
|
||||
});
|
||||
const snapshot = await service.read('node-a', 'overview');
|
||||
const byKey = Object.fromEntries(snapshot.sections.map((section) => [section.key, section]));
|
||||
expect(byKey.device?.state).toBe('ok');
|
||||
expect(byKey.device?.data).toEqual({ model: 'LPAX', imei: '123' });
|
||||
expect(byKey.stats?.state).toBe('empty');
|
||||
expect(byKey.connectivity?.state).toBe('unsupported');
|
||||
});
|
||||
|
||||
it('rejects unknown modules and unknown instances', async () => {
|
||||
const { service } = fixture({});
|
||||
await expect(service.read('node-a', 'nope' as 'overview')).rejects.toMatchObject({
|
||||
code: 'VALIDATION_FAILED',
|
||||
});
|
||||
await expect(service.read('missing', 'overview')).rejects.toMatchObject({ code: 'NOT_FOUND' });
|
||||
});
|
||||
|
||||
it('hands the hardware report to the identity guard while reading the overview', async () => {
|
||||
const seen: [string, Record<string, unknown>, string][] = [];
|
||||
const { service } = fixture(
|
||||
{
|
||||
'/device': { status: 200, body: { imei: '860000000000001', model: 'LPAX' } },
|
||||
'/stats': { status: 200, body: {} },
|
||||
'/connectivity': { status: 200, body: {} },
|
||||
},
|
||||
{
|
||||
onIdentity: (instanceId, evidence, origin) => {
|
||||
seen.push([instanceId, evidence as Record<string, unknown>, origin]);
|
||||
},
|
||||
},
|
||||
);
|
||||
await service.read('node-a', 'overview');
|
||||
expect(seen).toEqual([
|
||||
['node-a', { imei: '860000000000001', model: 'LPAX' }, 'http://node-a.local'],
|
||||
]);
|
||||
});
|
||||
|
||||
it('stays quiet when a module other than the overview is read', async () => {
|
||||
const seen: string[] = [];
|
||||
const { service } = fixture(
|
||||
{ '/sim': { status: 200, body: { imei: '860000000000001' } } },
|
||||
{ onIdentity: (instanceId) => void seen.push(instanceId) },
|
||||
);
|
||||
await service.read('node-a', 'sim');
|
||||
expect(seen).toEqual([]);
|
||||
});
|
||||
|
||||
it('never loses an overview the operator asked for because bookkeeping threw', async () => {
|
||||
const { service } = fixture(
|
||||
{ '/device': { status: 200, body: { model: 'LPAX' } } },
|
||||
{
|
||||
onIdentity: () => {
|
||||
throw new Error('guard unavailable');
|
||||
},
|
||||
},
|
||||
);
|
||||
const snapshot = await service.read('node-a', 'overview');
|
||||
expect(snapshot.sections.find((section) => section.key === 'device')?.state).toBe('ok');
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,349 @@
|
||||
import type { InstanceService } from './instance-service.js';
|
||||
import {
|
||||
INSTANCE_MODULE_PROBES,
|
||||
type InstanceModuleKey,
|
||||
type ModuleProbe,
|
||||
} from './instance-module-catalog.js';
|
||||
import type {
|
||||
InstanceSessionStore,
|
||||
UpstreamResponse,
|
||||
UpstreamSessionClientOptions,
|
||||
} from '../connections/upstream-session-client.js';
|
||||
|
||||
export type InstanceModuleErrorCode = 'NOT_FOUND' | 'VALIDATION_FAILED';
|
||||
|
||||
export class InstanceModuleError extends Error {
|
||||
constructor(
|
||||
readonly code: InstanceModuleErrorCode,
|
||||
message: string = code,
|
||||
) {
|
||||
super(message);
|
||||
this.name = 'InstanceModuleError';
|
||||
}
|
||||
}
|
||||
|
||||
export type ModuleSectionState = 'ok' | 'empty' | 'auth-required' | 'unsupported' | 'failed';
|
||||
|
||||
export interface ModuleSection {
|
||||
readonly key: string;
|
||||
readonly path: string;
|
||||
readonly state: ModuleSectionState;
|
||||
readonly status: number | null;
|
||||
readonly data: unknown;
|
||||
}
|
||||
|
||||
export interface ModuleSnapshot {
|
||||
readonly instanceId: string;
|
||||
readonly module: InstanceModuleKey;
|
||||
readonly observedAt: string;
|
||||
readonly authenticated: boolean;
|
||||
readonly sections: readonly ModuleSection[];
|
||||
}
|
||||
|
||||
const MAX_SECTION_BYTES = 64_000;
|
||||
const MAX_SNAPSHOT_BYTES = 220_000;
|
||||
const MAX_DEPTH = 6;
|
||||
const MAX_ARRAY_ENTRIES = 200;
|
||||
const MAX_STRING_LENGTH = 2_000;
|
||||
const CONCURRENCY = 4;
|
||||
const DEFAULT_PROBE_TIMEOUT_MS = 6_000;
|
||||
const CONTROL_CHARACTERS = /[\u0000-\u0008\u000b\u000c\u000e-\u001f\u007f]/u;
|
||||
const SECRET_KEY =
|
||||
/(password|passwd|secret|token|cookie|authorization|apikey|api_key|privatekey|private_key|session)/iu;
|
||||
|
||||
function isRecord(value: unknown): value is Record<string, unknown> {
|
||||
return value !== null && typeof value === 'object' && !Array.isArray(value);
|
||||
}
|
||||
|
||||
/**
|
||||
* Device payloads are untrusted. Depth, breadth and string length are bounded and anything that
|
||||
* looks like a credential is dropped before the snapshot leaves the control plane.
|
||||
*/
|
||||
function sanitize(value: unknown, depth = 0): unknown {
|
||||
if (value === null) return null;
|
||||
switch (typeof value) {
|
||||
case 'string':
|
||||
return value.length > MAX_STRING_LENGTH
|
||||
? `${value.slice(0, MAX_STRING_LENGTH)}...`
|
||||
: CONTROL_CHARACTERS.test(value)
|
||||
? value.replace(CONTROL_CHARACTERS, '')
|
||||
: value;
|
||||
case 'number':
|
||||
return Number.isFinite(value) ? value : null;
|
||||
case 'boolean':
|
||||
return value;
|
||||
default:
|
||||
break;
|
||||
}
|
||||
if (depth >= MAX_DEPTH) return null;
|
||||
if (Array.isArray(value))
|
||||
return value.slice(0, MAX_ARRAY_ENTRIES).map((item) => sanitize(item, depth + 1));
|
||||
if (isRecord(value)) {
|
||||
const output: Record<string, unknown> = {};
|
||||
for (const [key, entry] of Object.entries(value)) {
|
||||
if (SECRET_KEY.test(key)) continue;
|
||||
output[key] = sanitize(entry, depth + 1);
|
||||
}
|
||||
return output;
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
function isEmpty(value: unknown): boolean {
|
||||
if (value === null || value === undefined) return true;
|
||||
if (Array.isArray(value)) return value.length === 0;
|
||||
if (isRecord(value)) return Object.keys(value).length === 0;
|
||||
return false;
|
||||
}
|
||||
|
||||
interface DecodedBody {
|
||||
readonly data: unknown;
|
||||
readonly failed: boolean;
|
||||
readonly envelope: ModuleSectionState | undefined;
|
||||
}
|
||||
|
||||
/**
|
||||
* Device routes answer inside a `{status, message, data}` envelope. The payload the console renders
|
||||
* lives one level deeper, so the envelope is inspected for a capability verdict and then removed.
|
||||
* Error bodies are left whole because the message is the only useful part of them.
|
||||
*/
|
||||
function decodeBody(response: UpstreamResponse): DecodedBody {
|
||||
if (Buffer.byteLength(response.body, 'utf8') > MAX_SECTION_BYTES)
|
||||
return { data: null, failed: true, envelope: undefined };
|
||||
try {
|
||||
const decoded = sanitize(JSON.parse(response.body));
|
||||
const envelope = envelopeState(decoded);
|
||||
if (envelope) return { data: decoded, failed: false, envelope };
|
||||
return { data: unwrapEnvelope(decoded), failed: false, envelope };
|
||||
} catch {
|
||||
return { data: null, failed: true, envelope: undefined };
|
||||
}
|
||||
}
|
||||
|
||||
function unwrapEnvelope(value: unknown): unknown {
|
||||
if (!isRecord(value) || typeof value.status !== 'string' || value.status !== 'ok') return value;
|
||||
return 'data' in value ? value.data : value;
|
||||
}
|
||||
|
||||
/**
|
||||
* Devices answer 200 with an error envelope when a feature is compiled out of the firmware, for
|
||||
* example "Call forwarding is not exposed by ModemManager on this backend". That is a capability
|
||||
* verdict, not a read failure, so it must not render as data or count against the device.
|
||||
*/
|
||||
function envelopeState(data: unknown): ModuleSectionState | undefined {
|
||||
if (!isRecord(data) || data.status !== 'error') return undefined;
|
||||
const message = typeof data.message === 'string' ? data.message.toLocaleLowerCase() : '';
|
||||
if (/not exposed|not supported|unsupported|not implemented|no such|disabled by/u.test(message))
|
||||
return 'unsupported';
|
||||
return 'failed';
|
||||
}
|
||||
|
||||
function classify(status: number, body: DecodedBody): ModuleSectionState {
|
||||
if (status === 401 || status === 403) return 'auth-required';
|
||||
if (status === 404 || status === 405 || status === 501) return 'unsupported';
|
||||
if (body.failed || status < 200 || status >= 300) return 'failed';
|
||||
return body.envelope ?? (isEmpty(body.data) ? 'empty' : 'ok');
|
||||
}
|
||||
|
||||
async function withTimeout<T>(
|
||||
operation: Promise<T>,
|
||||
timeoutMs: number,
|
||||
onTimeout: () => T,
|
||||
): Promise<T> {
|
||||
let timer: ReturnType<typeof setTimeout> | undefined;
|
||||
const guard = new Promise<T>((resolve) => {
|
||||
timer = setTimeout(() => resolve(onTimeout()), timeoutMs);
|
||||
});
|
||||
try {
|
||||
return await Promise.race([operation, guard]);
|
||||
} finally {
|
||||
if (timer) clearTimeout(timer);
|
||||
}
|
||||
}
|
||||
|
||||
const TIMEOUT_SENTINEL: UpstreamResponse = { status: 0, headers: {}, body: '' };
|
||||
|
||||
export interface InstanceModuleServiceOptions {
|
||||
readonly instances: InstanceService;
|
||||
readonly sessions: InstanceSessionStore;
|
||||
readonly request: UpstreamSessionClientOptions['request'];
|
||||
readonly ensureSession?: (instanceId: string, origin: string, force?: boolean) => Promise<void>;
|
||||
readonly now?: () => Date;
|
||||
/** Fallback for catalog entries without an explicit budget; keeps a dead device from stalling. */
|
||||
readonly defaultTimeoutMs?: number;
|
||||
/**
|
||||
* Hears what the device just said about its own hardware. The overview module already reads
|
||||
* /device, so identity is observed by the pages the operator opens rather than by a poller
|
||||
* that costs the LAN extra traffic.
|
||||
*/
|
||||
readonly onIdentity?: (
|
||||
instanceId: string,
|
||||
evidence: Readonly<Record<string, unknown>>,
|
||||
origin: string,
|
||||
) => void;
|
||||
}
|
||||
|
||||
/**
|
||||
* Reads a whole console module from a device in one call. Individual probes fail independently so
|
||||
* a device that only implements part of the API still renders the sections it supports.
|
||||
*/
|
||||
/** Shared across the probes of one read so a device that needs a login is logged in once. */
|
||||
interface ProbeContext {
|
||||
cookie: string | undefined;
|
||||
refreshAttempted: boolean;
|
||||
}
|
||||
|
||||
export class InstanceModuleService {
|
||||
private readonly now: () => Date;
|
||||
private readonly defaultTimeoutMs: number;
|
||||
|
||||
constructor(private readonly options: InstanceModuleServiceOptions) {
|
||||
this.now = options.now ?? (() => new Date());
|
||||
this.defaultTimeoutMs = options.defaultTimeoutMs ?? DEFAULT_PROBE_TIMEOUT_MS;
|
||||
}
|
||||
|
||||
async read(instanceId: string, module: InstanceModuleKey): Promise<ModuleSnapshot> {
|
||||
const probes = INSTANCE_MODULE_PROBES[module];
|
||||
if (!probes) throw new InstanceModuleError('VALIDATION_FAILED', 'Unknown instance module');
|
||||
const instance = await this.options.instances.get(instanceId);
|
||||
if (!instance) throw new InstanceModuleError('NOT_FOUND', 'Instance was not found');
|
||||
|
||||
const session = this.options.sessions.sessionFor(instanceId);
|
||||
if (session && session.origin !== instance.origin)
|
||||
throw new InstanceModuleError('VALIDATION_FAILED', 'Instance session does not match origin');
|
||||
if (!session && this.options.ensureSession) {
|
||||
try {
|
||||
await this.options.ensureSession(instanceId, instance.origin);
|
||||
} catch {
|
||||
// Passwordless devices are readable anonymously; probes report auth-required otherwise.
|
||||
}
|
||||
}
|
||||
|
||||
const context: ProbeContext = {
|
||||
cookie: this.options.sessions.sessionFor(instanceId)?.cookie,
|
||||
refreshAttempted: false,
|
||||
};
|
||||
const sections = await this.#probeAll(instanceId, instance.origin, probes, context);
|
||||
this.#reportIdentity(module, instanceId, instance.origin, sections);
|
||||
return {
|
||||
instanceId,
|
||||
module,
|
||||
observedAt: this.now().toISOString(),
|
||||
authenticated: Boolean(context.cookie),
|
||||
sections: trimToBudget(sections),
|
||||
};
|
||||
}
|
||||
|
||||
#reportIdentity(
|
||||
module: InstanceModuleKey,
|
||||
instanceId: string,
|
||||
origin: string,
|
||||
sections: readonly ModuleSection[],
|
||||
): void {
|
||||
const report = this.options.onIdentity;
|
||||
if (!report || module !== 'overview') return;
|
||||
const device = sections.find((section) => section.key === 'device' && section.state === 'ok');
|
||||
if (!device || !isRecord(device.data)) return;
|
||||
try {
|
||||
report(instanceId, device.data, origin);
|
||||
} catch {
|
||||
// Bookkeeping for the identity guard never breaks a read the operator asked for.
|
||||
}
|
||||
}
|
||||
|
||||
async #probeAll(
|
||||
instanceId: string,
|
||||
origin: string,
|
||||
probes: readonly ModuleProbe[],
|
||||
context: ProbeContext,
|
||||
): Promise<readonly ModuleSection[]> {
|
||||
const results: ModuleSection[] = new Array<ModuleSection>(probes.length);
|
||||
const queue = probes.map((probe, index) => ({ probe, index }));
|
||||
const workers = Array.from({ length: Math.min(CONCURRENCY, queue.length) }, async () => {
|
||||
for (;;) {
|
||||
const next = queue.shift();
|
||||
if (!next) return;
|
||||
results[next.index] = await this.#probeOne(instanceId, origin, next.probe, context);
|
||||
}
|
||||
});
|
||||
await Promise.all(workers);
|
||||
return results;
|
||||
}
|
||||
|
||||
async #probeOne(
|
||||
instanceId: string,
|
||||
origin: string,
|
||||
probe: ModuleProbe,
|
||||
context: ProbeContext,
|
||||
): Promise<ModuleSection> {
|
||||
const timeoutMs = probe.timeoutMs ?? this.defaultTimeoutMs;
|
||||
const send = (token: string | undefined): Promise<UpstreamResponse> =>
|
||||
this.options.request({
|
||||
url: `${origin}/api${probe.path}`,
|
||||
method: 'GET',
|
||||
headers: { accept: 'application/json', ...(token ? { cookie: token } : {}) },
|
||||
});
|
||||
let response: UpstreamResponse;
|
||||
try {
|
||||
response = await withTimeout(send(context.cookie), timeoutMs, () => TIMEOUT_SENTINEL);
|
||||
const rejected = response.status === 401 || response.status === 403;
|
||||
if (rejected && this.options.ensureSession && !context.refreshAttempted) {
|
||||
context.refreshAttempted = true;
|
||||
try {
|
||||
await this.options.ensureSession(instanceId, origin, true);
|
||||
} catch {
|
||||
// Keep the unauthenticated answer; the section reports auth-required below.
|
||||
}
|
||||
const refreshed = this.options.sessions.sessionFor(instanceId)?.cookie;
|
||||
if (refreshed) {
|
||||
context.cookie = refreshed;
|
||||
response = await withTimeout(send(refreshed), timeoutMs, () => TIMEOUT_SENTINEL);
|
||||
}
|
||||
}
|
||||
} catch {
|
||||
return {
|
||||
key: probe.key,
|
||||
path: probe.path,
|
||||
state: 'failed',
|
||||
status: null,
|
||||
data: null,
|
||||
};
|
||||
}
|
||||
const body = decodeBody(response);
|
||||
return {
|
||||
key: probe.key,
|
||||
path: probe.path,
|
||||
state: classify(response.status, body),
|
||||
status: response.status === 0 ? null : response.status,
|
||||
data: body.data,
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
/** Drops the tail of the largest payloads until the whole snapshot fits the response budget. */
|
||||
function trimToBudget(sections: readonly ModuleSection[]): readonly ModuleSection[] {
|
||||
const sizes = sections.map((section) =>
|
||||
section.state === 'ok' || section.state === 'empty'
|
||||
? Buffer.byteLength(JSON.stringify(section.data ?? null), 'utf8')
|
||||
: 0,
|
||||
);
|
||||
let total = sizes.reduce((sum, size) => sum + size, 0);
|
||||
if (total <= MAX_SNAPSHOT_BYTES) return sections;
|
||||
const trimmed = sections.map((section, index) => ({ section, size: sizes[index] ?? 0 }));
|
||||
for (;;) {
|
||||
const largest = trimmed.reduce(
|
||||
(best, item) => (item.size > (best?.size ?? 0) ? item : best),
|
||||
trimmed[0] as { section: ModuleSection; size: number } | undefined,
|
||||
);
|
||||
if (!largest || largest.size === 0 || total <= MAX_SNAPSHOT_BYTES) break;
|
||||
const cut = Math.min(largest.size, total - MAX_SNAPSHOT_BYTES) + 1_024;
|
||||
largest.section = {
|
||||
...largest.section,
|
||||
state: 'ok',
|
||||
data: { truncated: true, note: 'payload exceeded the response budget' },
|
||||
};
|
||||
total -= cut;
|
||||
largest.size = 0;
|
||||
}
|
||||
return trimmed.map((item) => item.section);
|
||||
}
|
||||
@@ -10,6 +10,7 @@ import type { Instance } from '@multi-simadmin/contracts';
|
||||
import { InstanceService, InstanceServiceError } from './instance-service.js';
|
||||
|
||||
class MemorySecrets implements SecretStore {
|
||||
readonly provider = 'macos-keychain';
|
||||
readonly values = new Map<string, string>();
|
||||
readonly deletes: string[] = [];
|
||||
readonly sets: Array<{
|
||||
@@ -109,6 +110,7 @@ describe('InstanceService', () => {
|
||||
name: 'Alpha',
|
||||
origin: 'http://192.168.1.10:8080',
|
||||
tags: ['a', 'z'],
|
||||
groupId: null,
|
||||
revision: 1,
|
||||
capabilityStatus: 'unknown',
|
||||
freshness: 'unknown',
|
||||
|
||||
@@ -10,10 +10,9 @@ import type {
|
||||
SnapshotFreshness,
|
||||
} from '@multi-simadmin/contracts';
|
||||
import type { SecretStore } from '../../infrastructure/secrets/secret-store.js';
|
||||
import { parseKeychainReference } from '../../infrastructure/secrets/keychain-secret-store.js';
|
||||
import { parseSecretReference } from '../../infrastructure/secrets/secret-reference.js';
|
||||
|
||||
const PURPOSE = 'instance-password';
|
||||
const PROVIDER = 'macos-keychain';
|
||||
const MAX_TAGS = 50;
|
||||
const MAX_TAG_LENGTH = 100;
|
||||
|
||||
@@ -53,6 +52,7 @@ interface InstanceRow {
|
||||
base_url: string;
|
||||
config_revision: number;
|
||||
updated_at: string;
|
||||
group_id: string | null;
|
||||
}
|
||||
interface SecretRow {
|
||||
id: string;
|
||||
@@ -161,6 +161,7 @@ export class InstanceService {
|
||||
const name = normalizeName(input.name);
|
||||
const origin = normalizeOrigin(input.origin);
|
||||
const tags = normalizeTags(input.tags);
|
||||
const groupId = this.normalizeGroupId(input.groupId);
|
||||
validatePassword(input.password);
|
||||
const instanceId = this.id();
|
||||
let newSecret: { id: string; external: string } | undefined;
|
||||
@@ -171,9 +172,9 @@ export class InstanceService {
|
||||
this.db.transaction(() => {
|
||||
this.db
|
||||
.prepare(
|
||||
'INSERT INTO instances (id,name,base_url,auth_mode,enabled,config_revision,created_at,updated_at) VALUES (?,?,?, ?,1,1,?,?)',
|
||||
'INSERT INTO instances (id,name,base_url,auth_mode,enabled,config_revision,created_at,updated_at,group_id) VALUES (?,?,?, ?,1,1,?,?,?)',
|
||||
)
|
||||
.run(instanceId, name, origin, newSecret ? 'password' : 'none', now, now);
|
||||
.run(instanceId, name, origin, newSecret ? 'password' : 'none', now, now, groupId);
|
||||
this.replaceTags(instanceId, tags, now);
|
||||
if (newSecret) this.insertReference(newSecret.id, instanceId, newSecret.external, now);
|
||||
})();
|
||||
@@ -217,6 +218,8 @@ export class InstanceService {
|
||||
);
|
||||
if (query.tag !== undefined)
|
||||
values = values.filter(({ value }) => value.tags.includes(query.tag!.trim()));
|
||||
if (query.groupId !== undefined)
|
||||
values = values.filter(({ value }) => value.groupId === query.groupId!.trim());
|
||||
if (query.credentialConfigured !== undefined)
|
||||
values = values.filter(
|
||||
({ value }) => value.credentialConfigured === query.credentialConfigured,
|
||||
@@ -257,6 +260,8 @@ export class InstanceService {
|
||||
const name = patch.name === undefined ? current.name : normalizeName(patch.name);
|
||||
const origin = patch.origin === undefined ? current.base_url : normalizeOrigin(patch.origin);
|
||||
const tags = patch.tags === undefined ? undefined : normalizeTags(patch.tags);
|
||||
const groupId =
|
||||
patch.groupId === undefined ? current.group_id : this.normalizeGroupId(patch.groupId);
|
||||
let committedOldSecret: SecretRow | undefined;
|
||||
let newSecret: { id: string; external: string } | undefined;
|
||||
if (patch.password?.action === 'set')
|
||||
@@ -272,7 +277,7 @@ export class InstanceService {
|
||||
const transactionOldSecret = this.secret(instanceId);
|
||||
const changed = this.db
|
||||
.prepare(
|
||||
'UPDATE instances SET name=?,base_url=?,auth_mode=?,config_revision=config_revision+1,updated_at=? WHERE id=? AND config_revision=?',
|
||||
'UPDATE instances SET name=?,base_url=?,auth_mode=?,config_revision=config_revision+1,updated_at=?,group_id=? WHERE id=? AND config_revision=?',
|
||||
)
|
||||
.run(
|
||||
name,
|
||||
@@ -281,6 +286,7 @@ export class InstanceService {
|
||||
? 'password'
|
||||
: 'none',
|
||||
now,
|
||||
groupId,
|
||||
instanceId,
|
||||
revision,
|
||||
);
|
||||
@@ -401,7 +407,7 @@ export class InstanceService {
|
||||
.prepare(
|
||||
'INSERT INTO secret_references (id,instance_id,purpose,provider,external_reference,created_at,updated_at) VALUES (?,?,?,?,?,?,?)',
|
||||
)
|
||||
.run(id, instanceId, PURPOSE, PROVIDER, external, now, now);
|
||||
.run(id, instanceId, PURPOSE, this.store.provider, external, now, now);
|
||||
}
|
||||
private async storeSecret(
|
||||
instanceId: string,
|
||||
@@ -415,7 +421,7 @@ export class InstanceService {
|
||||
throw new InstanceServiceError('SECRET_STORE_FAILED', 'Could not store instance credential');
|
||||
}
|
||||
try {
|
||||
const parsed = parseKeychainReference(external);
|
||||
const parsed = parseSecretReference(external);
|
||||
if (parsed.instanceId !== instanceId || parsed.purpose !== PURPOSE || parsed.slot !== slot)
|
||||
throw new Error('secret store returned a reference with an invalid binding');
|
||||
} catch {
|
||||
@@ -497,7 +503,7 @@ export class InstanceService {
|
||||
AND secret_cleanup_tasks.purpose=excluded.purpose
|
||||
AND secret_cleanup_tasks.provider=excluded.provider`,
|
||||
)
|
||||
.run(reference, instanceId, PURPOSE, PROVIDER, now, now);
|
||||
.run(reference, instanceId, PURPOSE, this.store.provider, now, now);
|
||||
if (queued.changes !== 1)
|
||||
throw new InstanceServiceError('DATABASE_FAILED', 'Cleanup task could not be persisted');
|
||||
}
|
||||
@@ -522,13 +528,13 @@ export class InstanceService {
|
||||
);
|
||||
}
|
||||
private validateCleanupEntry(entry: CleanupEntry): void {
|
||||
if (entry.purpose !== PURPOSE || entry.provider !== PROVIDER)
|
||||
if (entry.purpose !== PURPOSE || entry.provider !== this.store.provider)
|
||||
throw new InstanceServiceError('DATABASE_FAILED', 'Cleanup task is invalid');
|
||||
this.validateCleanupReference(entry.instanceId, entry.reference);
|
||||
}
|
||||
private validateCleanupReference(instanceId: string, reference: string): void {
|
||||
try {
|
||||
const parsed = parseKeychainReference(reference);
|
||||
const parsed = parseSecretReference(reference);
|
||||
if (
|
||||
parsed.instanceId !== instanceId ||
|
||||
parsed.purpose !== PURPOSE ||
|
||||
@@ -542,13 +548,23 @@ export class InstanceService {
|
||||
private loadRows(where: string, parameters: unknown[]): AggregateRow[] {
|
||||
return this.db
|
||||
.prepare(
|
||||
`SELECT i.id,i.name,i.base_url,i.config_revision,i.updated_at,
|
||||
`SELECT i.id,i.name,i.base_url,i.config_revision,i.updated_at,i.group_id,
|
||||
group_concat(t.tag, char(31)) tags,
|
||||
CASE WHEN EXISTS(SELECT 1 FROM secret_references r WHERE r.instance_id=i.id AND r.purpose=?) THEN 1 ELSE 0 END credential
|
||||
FROM instances i LEFT JOIN instance_tags t ON t.instance_id=i.id ${where} GROUP BY i.id`,
|
||||
)
|
||||
.all(PURPOSE, ...parameters) as AggregateRow[];
|
||||
}
|
||||
|
||||
private normalizeGroupId(value: string | null | undefined): string | null {
|
||||
if (value === undefined || value === null) return null;
|
||||
if (typeof value !== 'string') validation('groupId must be a string or null');
|
||||
const trimmed = value.trim();
|
||||
if (!trimmed) return null;
|
||||
const exists = this.db.prepare('SELECT 1 FROM device_groups WHERE id=?').get(trimmed);
|
||||
if (!exists) throw new InstanceServiceError('NOT_FOUND', 'Device group was not found');
|
||||
return trimmed;
|
||||
}
|
||||
private toInstance(row: AggregateRow): Instance {
|
||||
const capabilityRows = this.db
|
||||
.prepare('SELECT state FROM capabilities WHERE instance_id=?')
|
||||
@@ -577,6 +593,7 @@ export class InstanceService {
|
||||
name: row.name,
|
||||
origin: row.base_url,
|
||||
tags: row.tags ? row.tags.split(String.fromCharCode(31)).sort(codePointCompare) : [],
|
||||
groupId: row.group_id ?? null,
|
||||
revision: row.config_revision,
|
||||
capabilityStatus,
|
||||
freshness,
|
||||
|
||||
@@ -23,6 +23,22 @@ export class JobQueryError extends Error {
|
||||
}
|
||||
}
|
||||
|
||||
export interface ControlPlaneJobSummary {
|
||||
readonly total: number;
|
||||
readonly active: number;
|
||||
readonly succeeded: number;
|
||||
readonly failed: number;
|
||||
readonly attention: number;
|
||||
readonly recent: readonly ControlPlaneJobSummaryItem[];
|
||||
}
|
||||
|
||||
export interface ControlPlaneJobSummaryItem {
|
||||
readonly id: string;
|
||||
readonly operationId: string;
|
||||
readonly status: Job['status'];
|
||||
readonly createdAt: string;
|
||||
}
|
||||
|
||||
interface JobRow {
|
||||
id: unknown;
|
||||
operation_id: unknown;
|
||||
@@ -121,6 +137,55 @@ function safeError(serialized: unknown): ProblemDetails | undefined {
|
||||
export class JobQueryService {
|
||||
constructor(private readonly db: Database.Database) {}
|
||||
|
||||
summary(limit = 5): ControlPlaneJobSummary {
|
||||
if (!Number.isSafeInteger(limit) || limit < 1 || limit > 20) validation('limit is invalid');
|
||||
const totalRow = this.db.prepare('SELECT COUNT(*) AS total FROM jobs').get() as
|
||||
| {
|
||||
total: unknown;
|
||||
}
|
||||
| undefined;
|
||||
if (!totalRow || typeof totalRow.total !== 'number' || !Number.isSafeInteger(totalRow.total))
|
||||
fail('Persisted job count is invalid');
|
||||
const statusRows = this.db
|
||||
.prepare('SELECT status, COUNT(*) AS count FROM jobs GROUP BY status')
|
||||
.all() as Array<{ status: unknown; count: unknown }>;
|
||||
const counts = new Map<string, number>();
|
||||
for (const row of statusRows) {
|
||||
const status = requiredString(row.status, 'job status');
|
||||
if (!(JOB_STATUSES as readonly string[]).includes(status))
|
||||
fail('Persisted job status is invalid');
|
||||
if (!Number.isSafeInteger(row.count) || (row.count as number) < 0)
|
||||
fail('Persisted job status count is invalid');
|
||||
counts.set(status, row.count as number);
|
||||
}
|
||||
const count = (status: Job['status']): number => counts.get(status) ?? 0;
|
||||
const active = count('queued') + count('running') + count('cancelling');
|
||||
const failed = count('failed');
|
||||
const attention = failed + count('partially-succeeded') + count('unknown-result') + active;
|
||||
const recentRows = this.db
|
||||
.prepare(`SELECT ${JOB_COLUMNS} FROM jobs ORDER BY created_at DESC, id ASC LIMIT ?`)
|
||||
.all(limit) as JobRow[];
|
||||
const recent = recentRows.map((row) => {
|
||||
const jobStatus = requiredString(row.status, 'status');
|
||||
if (!(JOB_STATUSES as readonly string[]).includes(jobStatus))
|
||||
fail('Persisted job status is invalid');
|
||||
return freeze({
|
||||
id: requiredString(row.id, 'id'),
|
||||
operationId: requiredString(row.operation_id, 'operation_id'),
|
||||
status: jobStatus as Job['status'],
|
||||
createdAt: timestamp(row.created_at, 'created_at'),
|
||||
});
|
||||
});
|
||||
return freeze({
|
||||
total: totalRow.total,
|
||||
active,
|
||||
succeeded: count('succeeded'),
|
||||
failed,
|
||||
attention,
|
||||
recent: freeze(recent),
|
||||
});
|
||||
}
|
||||
|
||||
get(id: string): Job {
|
||||
if (typeof id !== 'string' || id.length === 0) validation('id must be a non-empty string');
|
||||
const row = this.db.prepare(`SELECT ${JOB_COLUMNS} FROM jobs WHERE id = ?`).get(id) as
|
||||
|
||||
@@ -0,0 +1,108 @@
|
||||
import Database from 'better-sqlite3';
|
||||
import { describe, expect, it } from 'vitest';
|
||||
|
||||
import { migrateDatabase } from '../../infrastructure/database/migrations.js';
|
||||
import { DEFAULT_RECONCILE_GRACE_MS, JobReconcileService } from './job-reconcile-service.js';
|
||||
|
||||
const NOW = new Date('2026-07-20T12:00:00.000Z');
|
||||
const minutesAgo = (minutes: number): string =>
|
||||
new Date(NOW.getTime() - minutes * 60_000).toISOString();
|
||||
|
||||
function setup(): { db: Database.Database; subject: JobReconcileService } {
|
||||
const db = new Database(':memory:');
|
||||
migrateDatabase(db);
|
||||
return {
|
||||
db,
|
||||
subject: new JobReconcileService({ db, now: () => NOW }),
|
||||
};
|
||||
}
|
||||
|
||||
function insertJob(db: Database.Database, id: string, status: string, createdAt: string): void {
|
||||
db.prepare(
|
||||
`INSERT INTO jobs
|
||||
(id, root_job_id, operation_id, risk_level, status, requested_by, request_id,
|
||||
parameters_digest, created_at, updated_at)
|
||||
VALUES (?, ?, 'op.one', 'R2', ?, 'actor', 'request-1', 'digest', ?, ?)`,
|
||||
).run(id, id, status, createdAt, createdAt);
|
||||
db.prepare(
|
||||
`INSERT INTO job_items
|
||||
(id, job_id, instance_id, attempt_number, status, created_at, updated_at)
|
||||
VALUES (?, ?, 'instance-1', 1, ?, ?, ?)`,
|
||||
).run(`${id}-item`, id, status === 'queued' ? 'queued' : 'running', createdAt, createdAt);
|
||||
if (status !== 'queued') {
|
||||
db.prepare(
|
||||
`INSERT INTO job_attempts (id, job_id, status, started_at, created_at)
|
||||
VALUES (?, ?, 'running', ?, ?)`,
|
||||
).run(`${id}-attempt`, id, createdAt, createdAt);
|
||||
}
|
||||
}
|
||||
|
||||
const rows = (db: Database.Database, table: string): Array<Record<string, unknown>> =>
|
||||
db.prepare(`SELECT * FROM ${table}`).all() as Array<Record<string, unknown>>;
|
||||
|
||||
describe('JobReconcileService', () => {
|
||||
it('closes dispatched jobs that can no longer finish', () => {
|
||||
const { db, subject } = setup();
|
||||
insertJob(db, 'job-running', 'running', minutesAgo(60));
|
||||
|
||||
expect(subject.summary().pending).toBe(1);
|
||||
expect(subject.reconcile()).toEqual({ interrupted: 1, pending: 0 });
|
||||
|
||||
expect(rows(db, 'jobs')[0]).toMatchObject({
|
||||
id: 'job-running',
|
||||
status: 'unknown-result',
|
||||
finished_at: NOW.toISOString(),
|
||||
});
|
||||
expect(rows(db, 'job_items')[0]).toMatchObject({
|
||||
status: 'unknown-result',
|
||||
result_code: 'INTERRUPTED',
|
||||
});
|
||||
expect(rows(db, 'job_attempts')[0]).toMatchObject({
|
||||
status: 'unknown-result',
|
||||
finished_at: NOW.toISOString(),
|
||||
});
|
||||
});
|
||||
|
||||
it('cancels queued jobs that never reached a transport', () => {
|
||||
const { db, subject } = setup();
|
||||
insertJob(db, 'job-queued', 'queued', minutesAgo(30));
|
||||
|
||||
expect(subject.reconcile()).toEqual({ interrupted: 1, pending: 0 });
|
||||
expect(rows(db, 'jobs')[0]).toMatchObject({ status: 'cancelled' });
|
||||
expect(rows(db, 'job_items')[0]).toMatchObject({ result_code: 'NEVER_DISPATCHED' });
|
||||
});
|
||||
|
||||
it('leaves jobs inside the grace window alone', () => {
|
||||
const { db, subject } = setup();
|
||||
insertJob(db, 'job-fresh', 'running', minutesAgo(2));
|
||||
insertJob(db, 'job-old', 'cancelling', minutesAgo(DEFAULT_RECONCILE_GRACE_MS + 1));
|
||||
|
||||
expect(subject.reconcile()).toEqual({ interrupted: 1, pending: 1 });
|
||||
const byId = new Map(rows(db, 'jobs').map((row) => [row.id, row.status]));
|
||||
expect(byId.get('job-fresh')).toBe('running');
|
||||
expect(byId.get('job-old')).toBe('unknown-result');
|
||||
});
|
||||
|
||||
it('reports the oldest pending job so the console can warn about it', () => {
|
||||
const { db, subject } = setup();
|
||||
insertJob(db, 'job-newer', 'running', minutesAgo(20));
|
||||
insertJob(db, 'job-older', 'running', minutesAgo(40));
|
||||
insertJob(db, 'job-done', 'succeeded', minutesAgo(90));
|
||||
|
||||
expect(subject.summary()).toEqual({
|
||||
pending: 2,
|
||||
dispatched: 2,
|
||||
queued: 0,
|
||||
oldestCreatedAt: minutesAgo(40),
|
||||
});
|
||||
});
|
||||
|
||||
it('is idempotent and rejects a negative window', () => {
|
||||
const { db, subject } = setup();
|
||||
insertJob(db, 'job-once', 'running', minutesAgo(60));
|
||||
|
||||
expect(subject.reconcile(0).interrupted).toBe(1);
|
||||
expect(subject.reconcile(0)).toEqual({ interrupted: 0, pending: 0 });
|
||||
expect(() => subject.reconcile(-1)).toThrow(RangeError);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,178 @@
|
||||
import type { SqliteDatabase } from '../../infrastructure/database/database.js';
|
||||
|
||||
/**
|
||||
* Reconciles jobs that the console still believes are in flight.
|
||||
*
|
||||
* Every job is executed inside the request that created it, so a row left in `running` after a
|
||||
* crash can never finish on its own. The official Hub exposes the same capability on its command
|
||||
* ledger; here it closes the loop against our own job table instead of a remote queue.
|
||||
*
|
||||
* `queued` rows never reached a transport, so they are cancelled. `running` and `cancelling` rows
|
||||
* may or may not have reached the device, so they become `unknown-result`, which is the only
|
||||
* honest terminal state for an interrupted write.
|
||||
*
|
||||
* The service also owns operator cancellation, which the published contract promises on
|
||||
* `POST /api/v1/jobs/{jobId}/cancel`. Both paths share one rule: a job only ever moves from an
|
||||
* active state to a terminal one, and every write is guarded by the active state, so a cancel
|
||||
* racing an in-flight executor loses cleanly instead of overwriting a real result.
|
||||
*/
|
||||
|
||||
const DISPATCHED_STATUSES = new Set(['running', 'cancelling']);
|
||||
const QUEUED_STATUSES = new Set(['queued']);
|
||||
const ACTIVE_STATUSES = new Set([...DISPATCHED_STATUSES, ...QUEUED_STATUSES]);
|
||||
const ACTIVE_PLACEHOLDERS = '?, ?, ?';
|
||||
|
||||
/** On-demand sweeps leave a margin so a slow batch in another request is never clobbered. */
|
||||
export const DEFAULT_RECONCILE_GRACE_MS = 15 * 60 * 1000;
|
||||
|
||||
export interface JobReconcileResult {
|
||||
/** Jobs closed by this sweep. */
|
||||
readonly interrupted: number;
|
||||
/** Jobs still considered in flight once the sweep finished. */
|
||||
readonly pending: number;
|
||||
}
|
||||
|
||||
export interface JobReconcileSummary {
|
||||
readonly pending: number;
|
||||
readonly dispatched: number;
|
||||
readonly queued: number;
|
||||
readonly oldestCreatedAt: string | null;
|
||||
}
|
||||
|
||||
export interface JobReconcileOptions {
|
||||
readonly db: SqliteDatabase;
|
||||
readonly now?: () => Date;
|
||||
}
|
||||
|
||||
export type JobCancelErrorCode = 'NOT_FOUND' | 'NOT_CANCELLABLE';
|
||||
|
||||
export class JobCancelError extends Error {
|
||||
constructor(
|
||||
readonly code: JobCancelErrorCode,
|
||||
message: string,
|
||||
) {
|
||||
super(message);
|
||||
this.name = 'JobCancelError';
|
||||
}
|
||||
}
|
||||
|
||||
const CANCEL_RESULT_CODE = 'CANCELLED_BY_OPERATOR';
|
||||
|
||||
const isActiveStatus = (status: unknown): status is string =>
|
||||
typeof status === 'string' && ACTIVE_STATUSES.has(status);
|
||||
|
||||
export class JobReconcileService {
|
||||
readonly #db: SqliteDatabase;
|
||||
readonly #now: () => Date;
|
||||
|
||||
constructor(options: JobReconcileOptions) {
|
||||
this.#db = options.db;
|
||||
this.#now = options.now ?? (() => new Date());
|
||||
}
|
||||
|
||||
summary(olderThanMs = 0): JobReconcileSummary {
|
||||
const cutoff = this.#cutoff(olderThanMs);
|
||||
const rows = this.#db
|
||||
.prepare(
|
||||
'SELECT status, created_at FROM jobs WHERE status IN (?,?,?) AND created_at <= ? ORDER BY created_at ASC, id ASC',
|
||||
)
|
||||
.all('queued', 'running', 'cancelling', cutoff) as Array<{
|
||||
status: unknown;
|
||||
created_at: unknown;
|
||||
}>;
|
||||
let dispatched = 0;
|
||||
let queued = 0;
|
||||
for (const row of rows) {
|
||||
if (typeof row.status === 'string' && DISPATCHED_STATUSES.has(row.status)) dispatched += 1;
|
||||
else if (typeof row.status === 'string' && QUEUED_STATUSES.has(row.status)) queued += 1;
|
||||
}
|
||||
const oldest = rows[0]?.created_at;
|
||||
return {
|
||||
pending: rows.length,
|
||||
dispatched,
|
||||
queued,
|
||||
oldestCreatedAt: typeof oldest === 'string' ? oldest : null,
|
||||
};
|
||||
}
|
||||
|
||||
reconcile(olderThanMs = DEFAULT_RECONCILE_GRACE_MS): JobReconcileResult {
|
||||
const cutoff = this.#cutoff(olderThanMs);
|
||||
const closedAt = this.#now().toISOString();
|
||||
const interrupted = this.#db.transaction((): number => {
|
||||
const rows = this.#db
|
||||
.prepare(
|
||||
'SELECT id, status FROM jobs WHERE status IN (?,?,?) AND created_at <= ? ORDER BY created_at ASC, id ASC',
|
||||
)
|
||||
.all('queued', 'running', 'cancelling', cutoff) as Array<{
|
||||
id: unknown;
|
||||
status: unknown;
|
||||
}>;
|
||||
const closeItems = this.#db.prepare(
|
||||
`UPDATE job_items SET status = ?, result_code = ?, finished_at = ?, updated_at = ?
|
||||
WHERE job_id = ? AND status IN ('queued','running','cancelling')`,
|
||||
);
|
||||
const closeAttempts = this.#db.prepare(
|
||||
"UPDATE job_attempts SET status = ?, finished_at = ? WHERE job_id = ? AND status = 'running'",
|
||||
);
|
||||
const closeJob = this.#db.prepare(
|
||||
'UPDATE jobs SET status = ?, finished_at = ?, updated_at = ? WHERE id = ? AND status IN (?,?,?)',
|
||||
);
|
||||
let count = 0;
|
||||
for (const row of rows) {
|
||||
if (typeof row.id !== 'string' || !isActiveStatus(row.status)) continue;
|
||||
const dispatched = DISPATCHED_STATUSES.has(row.status);
|
||||
const finalStatus = dispatched ? 'unknown-result' : 'cancelled';
|
||||
const resultCode = dispatched ? 'INTERRUPTED' : 'NEVER_DISPATCHED';
|
||||
closeItems.run(finalStatus, resultCode, closedAt, closedAt, row.id);
|
||||
closeAttempts.run(finalStatus, closedAt, row.id);
|
||||
closeJob.run(finalStatus, closedAt, closedAt, row.id, 'queued', 'running', 'cancelling');
|
||||
count += 1;
|
||||
}
|
||||
return count;
|
||||
})();
|
||||
return { interrupted, pending: this.summary().pending };
|
||||
}
|
||||
|
||||
#cutoff(olderThanMs: number): string {
|
||||
if (!Number.isFinite(olderThanMs) || olderThanMs < 0) {
|
||||
throw new RangeError('olderThanMs must be a non-negative finite number');
|
||||
}
|
||||
return new Date(this.#now().getTime() - olderThanMs).toISOString();
|
||||
}
|
||||
|
||||
/** Requests cancellation of a job that has not reached a terminal state yet. */
|
||||
cancel(jobId: string): void {
|
||||
if (typeof jobId !== 'string' || jobId.length === 0 || jobId.length > 128) {
|
||||
throw new JobCancelError('NOT_FOUND', 'Job was not found');
|
||||
}
|
||||
const closedAt = this.#now().toISOString();
|
||||
const changed = this.#db.transaction((): boolean => {
|
||||
const row = this.#db.prepare('SELECT status FROM jobs WHERE id = ?').get(jobId) as
|
||||
| { status: unknown }
|
||||
| undefined;
|
||||
if (!row) throw new JobCancelError('NOT_FOUND', 'Job was not found');
|
||||
if (!isActiveStatus(row.status))
|
||||
throw new JobCancelError('NOT_CANCELLABLE', 'Job already reached a terminal state');
|
||||
this.#db
|
||||
.prepare(
|
||||
`UPDATE job_items SET status = 'cancelled', result_code = ?, finished_at = ?, updated_at = ?
|
||||
WHERE job_id = ? AND status IN ('queued', 'running', 'cancelling')`,
|
||||
)
|
||||
.run(CANCEL_RESULT_CODE, closedAt, closedAt, jobId);
|
||||
this.#db
|
||||
.prepare(
|
||||
"UPDATE job_attempts SET status = 'cancelled', finished_at = ? WHERE job_id = ? AND status = 'running'",
|
||||
)
|
||||
.run(closedAt, jobId);
|
||||
const updated = this.#db
|
||||
.prepare(
|
||||
`UPDATE jobs SET status = 'cancelled', finished_at = ?, updated_at = ?
|
||||
WHERE id = ? AND status IN (${ACTIVE_PLACEHOLDERS})`,
|
||||
)
|
||||
.run(closedAt, closedAt, jobId, 'queued', 'running', 'cancelling');
|
||||
return updated.changes === 1;
|
||||
})();
|
||||
if (!changed)
|
||||
throw new JobCancelError('NOT_CANCELLABLE', 'Job already reached a terminal state');
|
||||
}
|
||||
}
|
||||
@@ -20,16 +20,9 @@ function pendingDatabase(): Database.Database {
|
||||
db.pragma('foreign_keys = ON');
|
||||
migrateDatabase(db);
|
||||
const now = '2026-07-16T00:00:00.000Z';
|
||||
db.prepare('INSERT INTO instances VALUES (?,?,?,?,?,?,?,?)').run(
|
||||
'alpha',
|
||||
'Alpha',
|
||||
'https://203.0.113.8',
|
||||
'none',
|
||||
1,
|
||||
3,
|
||||
now,
|
||||
now,
|
||||
);
|
||||
db.prepare(
|
||||
'INSERT INTO instances (id,name,base_url,auth_mode,enabled,config_revision,created_at,updated_at) VALUES (?,?,?,?,?,?,?,?)',
|
||||
).run('alpha', 'Alpha', 'https://203.0.113.8', 'none', 1, 3, now, now);
|
||||
db.prepare('INSERT INTO app_settings VALUES (?,?,?,?)').run(
|
||||
'legacy-import.instance.alpha',
|
||||
JSON.stringify({
|
||||
@@ -46,6 +39,7 @@ function pendingDatabase(): Database.Database {
|
||||
}
|
||||
|
||||
class FakeStore implements SecretStore {
|
||||
readonly provider = 'macos-keychain';
|
||||
readonly sets: { key: SecretKey; value: string }[] = [];
|
||||
readonly deletes: string[] = [];
|
||||
setError?: Error;
|
||||
@@ -233,7 +227,9 @@ describe('activatePendingSecret', () => {
|
||||
migrateDatabase(first);
|
||||
const now = '2026-07-16T00:00:00.000Z';
|
||||
first
|
||||
.prepare('INSERT INTO instances VALUES (?,?,?,?,?,?,?,?)')
|
||||
.prepare(
|
||||
'INSERT INTO instances (id,name,base_url,auth_mode,enabled,config_revision,created_at,updated_at) VALUES (?,?,?,?,?,?,?,?)',
|
||||
)
|
||||
.run('alpha', 'Alpha', 'https://203.0.113.8', 'none', 1, 3, now, now);
|
||||
first
|
||||
.prepare('INSERT INTO app_settings VALUES (?,?,?,?)')
|
||||
|
||||
@@ -4,7 +4,6 @@ import { getManagedDatabaseIdentity } from '../../infrastructure/database/databa
|
||||
import type { SecretStore } from '../../infrastructure/secrets/secret-store.js';
|
||||
|
||||
const PURPOSE = 'instance-password';
|
||||
const PROVIDER = 'macos-keychain';
|
||||
const metadataKey = (instanceId: string): string => `legacy-import.instance.${instanceId}`;
|
||||
const referenceId = (instanceId: string): string => `legacy-secret:${instanceId}:${PURPOSE}`;
|
||||
|
||||
@@ -163,7 +162,7 @@ async function activatePendingSecretExclusive({
|
||||
throw new ActivationError('DATABASE_FAILED', 'Instance changed during secret activation');
|
||||
db.prepare(
|
||||
'INSERT INTO secret_references (id,instance_id,purpose,provider,external_reference,created_at,updated_at) VALUES (?,?,?,?,?,?,?)',
|
||||
).run(id, instanceId, PURPOSE, PROVIDER, externalReference, now, now);
|
||||
).run(id, instanceId, PURPOSE, store.provider, externalReference, now, now);
|
||||
const updated = db
|
||||
.prepare(
|
||||
"UPDATE instances SET auth_mode='password',config_revision=config_revision+1,updated_at=? WHERE id=? AND auth_mode='none' AND config_revision=?",
|
||||
|
||||
@@ -274,26 +274,12 @@ describe('legacy import', () => {
|
||||
const path = await legacyFile(valid);
|
||||
const db = database();
|
||||
const now = new Date().toISOString();
|
||||
db.prepare('INSERT INTO instances VALUES (?,?,?,?,?,?,?,?)').run(
|
||||
'alpha',
|
||||
'Different',
|
||||
'https://203.0.113.99',
|
||||
'none',
|
||||
1,
|
||||
7,
|
||||
now,
|
||||
now,
|
||||
);
|
||||
db.prepare('INSERT INTO instances VALUES (?,?,?,?,?,?,?,?)').run(
|
||||
'owner',
|
||||
'Owner',
|
||||
'http://[2001:4860:4860::8888]:8080',
|
||||
'none',
|
||||
1,
|
||||
1,
|
||||
now,
|
||||
now,
|
||||
);
|
||||
db.prepare(
|
||||
'INSERT INTO instances (id,name,base_url,auth_mode,enabled,config_revision,created_at,updated_at) VALUES (?,?,?,?,?,?,?,?)',
|
||||
).run('alpha', 'Different', 'https://203.0.113.99', 'none', 1, 7, now, now);
|
||||
db.prepare(
|
||||
'INSERT INTO instances (id,name,base_url,auth_mode,enabled,config_revision,created_at,updated_at) VALUES (?,?,?,?,?,?,?,?)',
|
||||
).run('owner', 'Owner', 'http://[2001:4860:4860::8888]:8080', 'none', 1, 1, now, now);
|
||||
const preview = await previewLegacyImport(path, db);
|
||||
expect(preview.instances.map((item) => item.status)).toEqual(['conflict', 'conflict']);
|
||||
expect(preview.counts.conflict).toBe(2);
|
||||
|
||||
@@ -0,0 +1,258 @@
|
||||
import Database from 'better-sqlite3';
|
||||
import { afterEach, beforeEach, describe, expect, it } from 'vitest';
|
||||
|
||||
import { migrateDatabase } from '../../infrastructure/database/migrations.js';
|
||||
import { HubMessageService } from './hub-message-service.js';
|
||||
import type { InstanceMessage } from './instance-message-service.js';
|
||||
|
||||
const now = new Date('2026-09-03T10:00:00.000Z');
|
||||
|
||||
const instanceMessages = new Map<string, readonly InstanceMessage[]>([
|
||||
[
|
||||
'device-1',
|
||||
[
|
||||
{
|
||||
id: '101',
|
||||
direction: 'incoming',
|
||||
phoneNumber: '10086',
|
||||
content: '余额提醒',
|
||||
timestamp: '2026-09-03T09:59:00.000Z',
|
||||
status: 'received',
|
||||
transport: 'modem',
|
||||
},
|
||||
],
|
||||
],
|
||||
]);
|
||||
|
||||
const messageService = {
|
||||
list: async (instanceId: string, query: { limit: number; offset: number }) => ({
|
||||
messages:
|
||||
query.offset >= (instanceMessages.get(instanceId)?.length ?? 0)
|
||||
? []
|
||||
: (instanceMessages.get(instanceId) ?? []).slice(query.offset, query.offset + query.limit),
|
||||
}),
|
||||
send: async () => {
|
||||
const current = [...(instanceMessages.get('device-1') ?? [])];
|
||||
const message: InstanceMessage = {
|
||||
id: '102',
|
||||
direction: 'outgoing',
|
||||
phoneNumber: '10086',
|
||||
content: '已发送',
|
||||
timestamp: now.toISOString(),
|
||||
status: 'sent',
|
||||
transport: 'modem',
|
||||
};
|
||||
instanceMessages.set('device-1', [message, ...current]);
|
||||
return { sent: true as const };
|
||||
},
|
||||
deleteMany: async (items: readonly { instanceId: string; id: string }[]) => ({
|
||||
requested: items.length,
|
||||
deleted: items.length,
|
||||
failed: 0,
|
||||
failures: [],
|
||||
}),
|
||||
};
|
||||
|
||||
const initialInstanceMessages = new Map(instanceMessages);
|
||||
|
||||
describe('HubMessageService', () => {
|
||||
let db: Database.Database;
|
||||
|
||||
beforeEach(() => {
|
||||
db = new Database(':memory:');
|
||||
db.pragma('foreign_keys = ON');
|
||||
migrateDatabase(db);
|
||||
db.prepare(
|
||||
`INSERT INTO instances (id,name,base_url,auth_mode,enabled,config_revision,created_at,updated_at)
|
||||
VALUES ('device-1','Modem A','http://device-1.invalid','password',1,1,?,?)`,
|
||||
).run(now.toISOString(), now.toISOString());
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
instanceMessages.clear();
|
||||
for (const [instanceId, messages] of initialInstanceMessages) {
|
||||
instanceMessages.set(instanceId, [...messages]);
|
||||
}
|
||||
db.close();
|
||||
});
|
||||
|
||||
function service() {
|
||||
return new HubMessageService(db, {
|
||||
instances: {
|
||||
list: async () => ({
|
||||
items: [
|
||||
{
|
||||
id: 'device-1',
|
||||
name: 'Modem A',
|
||||
origin: 'http://device-1.invalid',
|
||||
tags: [],
|
||||
revision: 1,
|
||||
capabilityStatus: 'unknown',
|
||||
freshness: 'unknown',
|
||||
credentialConfigured: true,
|
||||
},
|
||||
],
|
||||
page: { page: 1, pageSize: 100, total: 1 },
|
||||
}),
|
||||
},
|
||||
messages: messageService,
|
||||
now: () => now,
|
||||
});
|
||||
}
|
||||
|
||||
it('persists a full SMS sync and serves subsequent reads from SQLite', async () => {
|
||||
const hub = service();
|
||||
const first = await hub.syncAll();
|
||||
|
||||
expect(first).toMatchObject({ scanned: 1, synced: 1, available: 1 });
|
||||
instanceMessages.set('device-1', []);
|
||||
|
||||
const page = await hub.list({ limit: 10, offset: 0 });
|
||||
expect(page.total).toBe(1);
|
||||
expect(page.items[0]).toMatchObject({
|
||||
instanceId: 'device-1',
|
||||
instanceName: 'Modem A',
|
||||
phoneNumber: '10086',
|
||||
content: '余额提醒',
|
||||
syncedAt: now.toISOString(),
|
||||
});
|
||||
});
|
||||
|
||||
it('records outbound SMS in the central message store after upstream delivery', async () => {
|
||||
const hub = service();
|
||||
await hub.syncAll();
|
||||
|
||||
await hub.send('device-1', { phoneNumber: '10086', content: '已发送' });
|
||||
const page = await hub.list({ limit: 10, offset: 0 });
|
||||
expect(page).toMatchObject({ total: 2 });
|
||||
});
|
||||
|
||||
it('deletes the central copy without requiring another upstream read', async () => {
|
||||
const hub = service();
|
||||
await hub.syncAll();
|
||||
|
||||
const result = await hub.deleteMany([{ instanceId: 'device-1', id: '101' }]);
|
||||
expect(result).toMatchObject({ requested: 1, deleted: 1, failed: 0 });
|
||||
expect(await hub.list({ limit: 10, offset: 0 })).toMatchObject({ total: 0 });
|
||||
});
|
||||
|
||||
describe('conversations', () => {
|
||||
const at = (secondsAgo: number): string =>
|
||||
new Date(now.getTime() - secondsAgo * 1_000).toISOString();
|
||||
|
||||
function message(
|
||||
id: string,
|
||||
direction: InstanceMessage['direction'],
|
||||
phoneNumber: string,
|
||||
content: string,
|
||||
timestamp: string,
|
||||
): InstanceMessage {
|
||||
return {
|
||||
id,
|
||||
direction,
|
||||
phoneNumber,
|
||||
content,
|
||||
timestamp,
|
||||
status: direction,
|
||||
transport: 'modem',
|
||||
};
|
||||
}
|
||||
|
||||
function seed(hub: HubMessageService): void {
|
||||
hub.upsert('device-1', message('a', 'incoming', '10086', '余额提醒', at(300)), now);
|
||||
hub.upsert('device-1', message('b', 'outgoing', '10086', '已查询', at(200)), now);
|
||||
hub.upsert('device-1', message('c', 'incoming', '13900139000', '验证码 1234', at(100)), now);
|
||||
hub.upsert('device-1', message('d', 'incoming', '10086', '流量提醒', at(50)), now);
|
||||
}
|
||||
|
||||
it('groups the archive per node and phone with counts and a newest preview', () => {
|
||||
const hub = service();
|
||||
seed(hub);
|
||||
|
||||
const page = hub.conversations({ limit: 10 });
|
||||
expect(page.totalCount).toBe(2);
|
||||
expect(page.stats).toEqual({ incoming: 3, outgoing: 1, total: 4 });
|
||||
expect(page.items.map((item) => item.phoneNumber)).toEqual(['10086', '13900139000']);
|
||||
expect(page.items[0]).toMatchObject({
|
||||
instanceId: 'device-1',
|
||||
instanceName: 'Modem A',
|
||||
messageCount: 3,
|
||||
incomingCount: 2,
|
||||
lastMessage: { content: '流量提醒', timestamp: at(50) },
|
||||
});
|
||||
expect(page.items[1]).toMatchObject({
|
||||
messageCount: 1,
|
||||
incomingCount: 1,
|
||||
lastMessage: { content: '验证码 1234' },
|
||||
});
|
||||
});
|
||||
|
||||
it('filters conversations by node, phone and search text', () => {
|
||||
const hub = service();
|
||||
seed(hub);
|
||||
|
||||
expect(hub.conversations({ instanceId: 'device-2' }).totalCount).toBe(0);
|
||||
expect(hub.conversations({ phoneNumber: '10086' }).items).toHaveLength(1);
|
||||
const searched = hub.conversations({ search: '验证码' });
|
||||
expect(searched.items.map((item) => item.phoneNumber)).toEqual(['13900139000']);
|
||||
expect(searched.totalCount).toBe(1);
|
||||
});
|
||||
|
||||
it('keeps a thread when only one message matches the direction filter', () => {
|
||||
const hub = service();
|
||||
seed(hub);
|
||||
|
||||
const outgoing = hub.conversations({ direction: 'outgoing' });
|
||||
expect(outgoing.totalCount).toBe(1);
|
||||
expect(outgoing.items[0]).toMatchObject({
|
||||
phoneNumber: '10086',
|
||||
// The thread is still reported at its full size even though only one row matched.
|
||||
messageCount: 3,
|
||||
incomingCount: 2,
|
||||
lastMessage: { direction: 'outgoing', content: '已查询' },
|
||||
});
|
||||
|
||||
const incoming = hub.conversations({ direction: 'incoming' });
|
||||
expect(incoming.totalCount).toBe(2);
|
||||
expect(incoming.items.map((item) => item.phoneNumber)).toEqual(['10086', '13900139000']);
|
||||
expect(incoming.items[0]?.lastMessage).toMatchObject({
|
||||
direction: 'incoming',
|
||||
content: '流量提醒',
|
||||
});
|
||||
expect(incoming.items[1]?.lastMessage).toMatchObject({
|
||||
direction: 'incoming',
|
||||
content: '验证码 1234',
|
||||
});
|
||||
});
|
||||
|
||||
it('reports direction totals for the whole filtered archive, not just the page', () => {
|
||||
const hub = service();
|
||||
seed(hub);
|
||||
|
||||
const page = hub.conversations({ limit: 1 });
|
||||
expect(page.items).toHaveLength(1);
|
||||
expect(page.stats).toEqual({ incoming: 3, outgoing: 1, total: 4 });
|
||||
expect(hub.conversations({ instanceId: 'device-1', search: '10086' }).stats).toEqual({
|
||||
incoming: 2,
|
||||
outgoing: 1,
|
||||
total: 3,
|
||||
});
|
||||
expect(hub.conversations({ instanceId: 'device-2' }).stats).toEqual({
|
||||
incoming: 0,
|
||||
outgoing: 0,
|
||||
total: 0,
|
||||
});
|
||||
});
|
||||
|
||||
it('pages conversations by newest activity', () => {
|
||||
const hub = service();
|
||||
seed(hub);
|
||||
|
||||
const first = hub.conversations({ limit: 1, offset: 0 });
|
||||
const second = hub.conversations({ limit: 1, offset: 1 });
|
||||
expect(first.totalCount).toBe(2);
|
||||
expect(first.items[0]?.phoneNumber).toBe('10086');
|
||||
expect(second.items[0]?.phoneNumber).toBe('13900139000');
|
||||
});
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,505 @@
|
||||
import { randomUUID } from 'node:crypto';
|
||||
|
||||
import type Database from 'better-sqlite3';
|
||||
|
||||
import type { InstancePageQuery } from '@multi-simadmin/contracts';
|
||||
|
||||
import type {
|
||||
DeleteSmsMessageRequest,
|
||||
InstanceMessage,
|
||||
InstanceMessageService,
|
||||
MessageDeleteResult,
|
||||
} from './instance-message-service.js';
|
||||
|
||||
export interface HubMessagePage {
|
||||
readonly items: readonly HubMessage[];
|
||||
readonly total: number;
|
||||
}
|
||||
|
||||
/** SQL literal sets; upstream labels the same direction two different ways. */
|
||||
const INCOMING_DIRECTIONS = "'incoming','received'";
|
||||
const OUTGOING_DIRECTIONS = "'outgoing','sent'";
|
||||
|
||||
export interface HubMessage {
|
||||
readonly id: string;
|
||||
readonly instanceId: string;
|
||||
readonly instanceName: string;
|
||||
readonly direction: string;
|
||||
readonly phoneNumber: string;
|
||||
readonly content: string;
|
||||
readonly timestamp: string;
|
||||
readonly status: string;
|
||||
readonly transport: string;
|
||||
readonly syncedAt: string;
|
||||
}
|
||||
|
||||
export interface HubMessageDevice {
|
||||
readonly id: string;
|
||||
readonly name: string;
|
||||
readonly availability: 'online' | 'unavailable';
|
||||
}
|
||||
|
||||
export interface HubMessageSnapshot {
|
||||
readonly messages: readonly HubMessage[];
|
||||
readonly devices: readonly HubMessageDevice[];
|
||||
readonly total: number;
|
||||
}
|
||||
|
||||
/**
|
||||
* One phone-thread per node, mirroring the Hub conversation model: the archive is
|
||||
* grouped by (instance, phone number) and carries the newest message as a preview.
|
||||
*/
|
||||
export interface HubMessageConversation {
|
||||
readonly instanceId: string;
|
||||
readonly instanceName: string;
|
||||
readonly phoneNumber: string;
|
||||
readonly messageCount: number;
|
||||
readonly incomingCount: number;
|
||||
readonly lastMessage: HubMessage;
|
||||
}
|
||||
|
||||
/** Direction totals for the whole filtered archive, mirroring the Hub SMS counters. */
|
||||
export interface HubMessageDirectionStats {
|
||||
readonly incoming: number;
|
||||
readonly outgoing: number;
|
||||
readonly total: number;
|
||||
}
|
||||
|
||||
export interface HubMessageConversationPage {
|
||||
readonly items: readonly HubMessageConversation[];
|
||||
readonly totalCount: number;
|
||||
readonly stats: HubMessageDirectionStats;
|
||||
}
|
||||
|
||||
export interface HubSyncSummary {
|
||||
readonly scanned: number;
|
||||
readonly available: number;
|
||||
readonly synced: number;
|
||||
readonly unavailable: number;
|
||||
}
|
||||
|
||||
interface HubMessageServiceOptions {
|
||||
readonly instances: {
|
||||
readonly list: (query?: InstancePageQuery) => Promise<{
|
||||
readonly items: readonly { readonly id: string; readonly name: string }[];
|
||||
}>;
|
||||
};
|
||||
readonly messages: Pick<InstanceMessageService, 'list' | 'send' | 'deleteMany'>;
|
||||
readonly onIncomingMessage?: (instanceId: string, message: InstanceMessage) => Promise<void>;
|
||||
readonly now?: () => Date;
|
||||
readonly id?: () => string;
|
||||
readonly maximumMessagesPerDevice?: number;
|
||||
readonly refreshTtlMs?: number;
|
||||
}
|
||||
|
||||
const PAGE_SIZE = 100;
|
||||
const DEFAULT_MAXIMUM = 5_000;
|
||||
const MAX_DEVICES = 5_000;
|
||||
const DEFAULT_REFRESH_TTL_MS = 5_000;
|
||||
|
||||
const isoTimestamp = (value: string, fallback: Date): string => {
|
||||
const time = Date.parse(value);
|
||||
return Number.isFinite(time) ? new Date(time).toISOString() : fallback.toISOString();
|
||||
};
|
||||
|
||||
export class HubMessageService {
|
||||
readonly #db: Database.Database;
|
||||
readonly #options: HubMessageServiceOptions;
|
||||
readonly #availability = new Map<string, boolean>();
|
||||
#lastRefreshAt = Number.NEGATIVE_INFINITY;
|
||||
#inFlight: Promise<HubSyncSummary> | undefined;
|
||||
#lastSummary: HubSyncSummary = { scanned: 0, available: 0, synced: 0, unavailable: 0 };
|
||||
|
||||
constructor(db: Database.Database, options: HubMessageServiceOptions) {
|
||||
this.#db = db;
|
||||
this.#options = options;
|
||||
}
|
||||
|
||||
async #allInstances(): Promise<readonly { readonly id: string; readonly name: string }[]> {
|
||||
const items: { readonly id: string; readonly name: string }[] = [];
|
||||
let page = 1;
|
||||
while (items.length < MAX_DEVICES) {
|
||||
const current = await this.#options.instances.list({ page, pageSize: PAGE_SIZE });
|
||||
items.push(...current.items.slice(0, MAX_DEVICES - items.length));
|
||||
if (current.items.length < PAGE_SIZE) break;
|
||||
page += 1;
|
||||
}
|
||||
return items;
|
||||
}
|
||||
|
||||
async syncAll(): Promise<HubSyncSummary> {
|
||||
const instances = await this.#allInstances();
|
||||
let available = 0;
|
||||
let synced = 0;
|
||||
for (const instance of instances) {
|
||||
try {
|
||||
synced += await this.syncDevice(instance.id);
|
||||
available += 1;
|
||||
} catch {
|
||||
// Preserve the central archive when a device is temporarily unavailable.
|
||||
}
|
||||
}
|
||||
return {
|
||||
scanned: instances.length,
|
||||
available,
|
||||
synced,
|
||||
unavailable: instances.length - available,
|
||||
};
|
||||
}
|
||||
|
||||
async syncDevice(instanceId: string): Promise<number> {
|
||||
try {
|
||||
const stored = await this.#pullDevice(instanceId);
|
||||
this.#availability.set(instanceId, true);
|
||||
return stored;
|
||||
} catch (error) {
|
||||
this.#availability.set(instanceId, false);
|
||||
throw error;
|
||||
}
|
||||
}
|
||||
|
||||
async #pullDevice(instanceId: string): Promise<number> {
|
||||
const observedAt = this.#nowDate();
|
||||
let stored = 0;
|
||||
for (let offset = 0; offset < this.#maximumPerDevice(); offset += PAGE_SIZE) {
|
||||
const result = await this.#options.messages.list(instanceId, {
|
||||
limit: PAGE_SIZE,
|
||||
offset,
|
||||
});
|
||||
for (const message of result.messages) {
|
||||
const inserted = this.upsert(instanceId, message, observedAt);
|
||||
stored += inserted;
|
||||
if (
|
||||
inserted === 1 &&
|
||||
(message.direction === 'incoming' || message.direction === 'received')
|
||||
)
|
||||
await this.#options.onIncomingMessage?.(instanceId, message);
|
||||
}
|
||||
if (result.messages.length < PAGE_SIZE) break;
|
||||
}
|
||||
return stored;
|
||||
}
|
||||
|
||||
upsert(instanceId: string, message: InstanceMessage, observedAt: Date = this.#nowDate()): number {
|
||||
const exists = this.#db
|
||||
.prepare('SELECT 1 FROM sms_messages WHERE instance_id=? AND upstream_id=?')
|
||||
.get(instanceId, message.id);
|
||||
const result = this.#db
|
||||
.prepare(
|
||||
`INSERT INTO sms_messages
|
||||
(id,instance_id,upstream_id,direction,phone_number,content,timestamp,status,transport,synced_at)
|
||||
VALUES (?,?,?,?,?,?,?,?,?,?)
|
||||
ON CONFLICT(instance_id, upstream_id) DO UPDATE SET
|
||||
direction=excluded.direction,
|
||||
phone_number=excluded.phone_number,
|
||||
content=excluded.content,
|
||||
timestamp=excluded.timestamp,
|
||||
status=excluded.status,
|
||||
transport=excluded.transport,
|
||||
synced_at=excluded.synced_at`,
|
||||
)
|
||||
.run(
|
||||
this.#id(),
|
||||
instanceId,
|
||||
message.id,
|
||||
message.direction,
|
||||
message.phoneNumber,
|
||||
message.content,
|
||||
isoTimestamp(message.timestamp, observedAt),
|
||||
message.status,
|
||||
message.transport,
|
||||
observedAt.toISOString(),
|
||||
);
|
||||
return exists ? 0 : result.changes;
|
||||
}
|
||||
|
||||
async list(
|
||||
query: {
|
||||
readonly limit?: number;
|
||||
readonly offset?: number;
|
||||
readonly search?: string;
|
||||
readonly instanceId?: string;
|
||||
readonly phoneNumber?: string;
|
||||
} = {},
|
||||
): Promise<HubMessagePage> {
|
||||
const limit = Math.min(Math.max(query.limit ?? 24, 1), 100);
|
||||
const offset = Math.max(query.offset ?? 0, 0);
|
||||
const filter = this.#filter(query);
|
||||
const where = filter.where;
|
||||
const parameters = [...filter.parameters];
|
||||
const total = Number(
|
||||
(
|
||||
this.#db
|
||||
.prepare(`SELECT COUNT(*) AS count FROM sms_messages${where}`)
|
||||
.get(...parameters) as { count?: number } | undefined
|
||||
)?.count ?? 0,
|
||||
);
|
||||
const rows = this.#db
|
||||
.prepare(
|
||||
`SELECT s.*, COALESCE(i.name, '已移除节点') AS instance_name
|
||||
FROM sms_messages s LEFT JOIN instances i ON i.id = s.instance_id${where}
|
||||
ORDER BY s.timestamp DESC, s.id DESC LIMIT ? OFFSET ?`,
|
||||
)
|
||||
.all(...parameters, limit, offset) as Array<Record<string, unknown>>;
|
||||
return {
|
||||
items: Object.freeze(rows.map((row) => this.#message(row))),
|
||||
total,
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Groups the central archive into phone threads. The newest message of each group
|
||||
* is resolved by the same `timestamp, id` ordering used by `list`, so the preview
|
||||
* always matches the first row a reader sees when opening the conversation.
|
||||
*/
|
||||
conversations(
|
||||
query: {
|
||||
readonly limit?: number;
|
||||
readonly offset?: number;
|
||||
readonly search?: string;
|
||||
readonly instanceId?: string;
|
||||
readonly phoneNumber?: string;
|
||||
readonly direction?: 'incoming' | 'outgoing';
|
||||
} = {},
|
||||
): HubMessageConversationPage {
|
||||
const limit = Math.min(Math.max(query.limit ?? 50, 1), 200);
|
||||
const offset = Math.max(query.offset ?? 0, 0);
|
||||
const filter = this.#filter(query);
|
||||
const scoped = `SELECT id, instance_id, direction, phone_number, content, timestamp,
|
||||
status, transport, synced_at FROM sms_messages${filter.where}`;
|
||||
// A thread only survives a direction filter when it holds at least one matching message,
|
||||
// and the preview resolves inside that subset, while `message_count` stays the full size.
|
||||
const matched =
|
||||
query.direction === 'incoming'
|
||||
? `direction IN (${INCOMING_DIRECTIONS})`
|
||||
: query.direction === 'outgoing'
|
||||
? `direction IN (${OUTGOING_DIRECTIONS})`
|
||||
: '1';
|
||||
const counters = this.#db
|
||||
.prepare(
|
||||
`SELECT COUNT(*) AS total,
|
||||
SUM(CASE WHEN direction IN (${INCOMING_DIRECTIONS}) THEN 1 ELSE 0 END) AS incoming,
|
||||
SUM(CASE WHEN direction IN (${OUTGOING_DIRECTIONS}) THEN 1 ELSE 0 END) AS outgoing
|
||||
FROM (${scoped})`,
|
||||
)
|
||||
.get(...filter.parameters) as Record<string, number | null> | undefined;
|
||||
const stats = Object.freeze({
|
||||
incoming: Number(counters?.incoming ?? 0),
|
||||
outgoing: Number(counters?.outgoing ?? 0),
|
||||
total: Number(counters?.total ?? 0),
|
||||
});
|
||||
// One row per surviving thread, so the outer COUNT is the thread total.
|
||||
const groupedSql = `SELECT instance_id FROM (
|
||||
SELECT *, CASE WHEN ${matched} THEN 1 ELSE 0 END AS matched FROM (${scoped})
|
||||
) GROUP BY instance_id, phone_number HAVING SUM(matched) > 0`;
|
||||
const totalCount = Number(
|
||||
(
|
||||
this.#db
|
||||
.prepare(`SELECT COUNT(*) AS count FROM (${groupedSql})`)
|
||||
.get(...filter.parameters) as { count?: number } | undefined
|
||||
)?.count ?? 0,
|
||||
);
|
||||
const rows = this.#db
|
||||
.prepare(
|
||||
`WITH flagged AS (
|
||||
SELECT *, CASE WHEN ${matched} THEN 1 ELSE 0 END AS matched
|
||||
FROM (${scoped})
|
||||
),
|
||||
grouped AS (
|
||||
SELECT instance_id, phone_number,
|
||||
COUNT(*) AS message_count,
|
||||
SUM(CASE WHEN direction IN (${INCOMING_DIRECTIONS}) THEN 1 ELSE 0 END) AS incoming_count,
|
||||
MAX(CASE WHEN matched = 1 THEN timestamp || '#' || id END) AS latest_key
|
||||
FROM flagged GROUP BY instance_id, phone_number
|
||||
HAVING SUM(matched) > 0
|
||||
)
|
||||
SELECT s.id, s.direction, s.phone_number, s.content, s.timestamp, s.status,
|
||||
s.transport, s.synced_at, g.instance_id, g.message_count, g.incoming_count,
|
||||
COALESCE(i.name, '已移除节点') AS instance_name
|
||||
FROM grouped g
|
||||
JOIN flagged s
|
||||
ON s.instance_id = g.instance_id AND s.phone_number = g.phone_number
|
||||
AND (s.timestamp || '#' || s.id) = g.latest_key
|
||||
LEFT JOIN instances i ON i.id = g.instance_id
|
||||
ORDER BY g.latest_key DESC LIMIT ? OFFSET ?`,
|
||||
)
|
||||
.all(...filter.parameters, limit, offset) as Array<Record<string, unknown>>;
|
||||
return {
|
||||
items: Object.freeze(
|
||||
rows.flatMap((row) => {
|
||||
const messageCount = Number(row.message_count ?? 0);
|
||||
if (!Number.isSafeInteger(messageCount) || messageCount < 1) return [];
|
||||
return [
|
||||
Object.freeze({
|
||||
instanceId: String(row.instance_id),
|
||||
instanceName: String(row.instance_name),
|
||||
phoneNumber: String(row.phone_number),
|
||||
messageCount,
|
||||
incomingCount: Number(row.incoming_count ?? 0),
|
||||
lastMessage: this.#message(row),
|
||||
}),
|
||||
];
|
||||
}),
|
||||
),
|
||||
totalCount,
|
||||
stats,
|
||||
};
|
||||
}
|
||||
|
||||
#filter(
|
||||
query: Readonly<{ search?: string; instanceId?: string; phoneNumber?: string }>,
|
||||
): Readonly<{ where: string; parameters: readonly string[] }> {
|
||||
const conditions: string[] = [];
|
||||
const parameters: string[] = [];
|
||||
if (query.instanceId) {
|
||||
conditions.push('instance_id = ?');
|
||||
parameters.push(query.instanceId);
|
||||
}
|
||||
if (query.phoneNumber) {
|
||||
conditions.push('phone_number = ?');
|
||||
parameters.push(query.phoneNumber);
|
||||
}
|
||||
const search = query.search?.trim();
|
||||
if (search) {
|
||||
conditions.push(
|
||||
`(content LIKE ? ESCAPE '\\' OR phone_number LIKE ? ESCAPE '\\' OR instance_id IN (
|
||||
SELECT id || '|' || name FROM instances WHERE name LIKE ? ESCAPE '\\'
|
||||
))`,
|
||||
);
|
||||
const pattern = `%${search.replaceAll('\\', '\\\\').replaceAll('%', '\\%').replaceAll('_', '\\_')}%`;
|
||||
parameters.push(pattern, pattern, pattern);
|
||||
}
|
||||
return {
|
||||
where: conditions.length > 0 ? ` WHERE ${conditions.join(' AND ')}` : '',
|
||||
parameters,
|
||||
};
|
||||
}
|
||||
|
||||
async snapshot(
|
||||
query: {
|
||||
readonly limit?: number;
|
||||
readonly offset?: number;
|
||||
readonly search?: string;
|
||||
readonly instanceId?: string;
|
||||
readonly phoneNumber?: string;
|
||||
} = {},
|
||||
): Promise<HubMessageSnapshot> {
|
||||
await this.refresh();
|
||||
const [page, instances] = await Promise.all([this.list(query), this.#allInstances()]);
|
||||
return {
|
||||
messages: page.items,
|
||||
devices: instances.map((instance) => ({
|
||||
id: instance.id,
|
||||
name: instance.name,
|
||||
availability: this.#availability.get(instance.id) ? 'online' : 'unavailable',
|
||||
})),
|
||||
total: page.total,
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Pulls every node into the central archive, but only when the last pull is older
|
||||
* than the refresh window; concurrent readers share a single in-flight sync.
|
||||
*/
|
||||
async refresh(force = false): Promise<HubSyncSummary> {
|
||||
const ttl = Math.max(0, this.#options.refreshTtlMs ?? DEFAULT_REFRESH_TTL_MS);
|
||||
const age = this.#nowDate().getTime() - this.#lastRefreshAt;
|
||||
if (!force && this.#lastRefreshAt > Number.NEGATIVE_INFINITY && age < ttl) {
|
||||
return this.#lastSummary;
|
||||
}
|
||||
if (!this.#inFlight) {
|
||||
this.#inFlight = this.syncAll().then((summary) => {
|
||||
this.#lastSummary = summary;
|
||||
this.#lastRefreshAt = this.#nowDate().getTime();
|
||||
return summary;
|
||||
});
|
||||
void this.#inFlight.catch(() => undefined);
|
||||
}
|
||||
try {
|
||||
return await this.#inFlight;
|
||||
} finally {
|
||||
this.#inFlight = undefined;
|
||||
}
|
||||
}
|
||||
|
||||
async send(
|
||||
instanceId: string,
|
||||
input: { readonly phoneNumber: string; readonly content: string },
|
||||
): Promise<{ readonly sent: true }> {
|
||||
const result = await this.#options.messages.send(instanceId, input);
|
||||
try {
|
||||
await this.syncDevice(instanceId);
|
||||
} catch {
|
||||
// Upstream accepted the send; the next sync can repair the central copy.
|
||||
}
|
||||
return result;
|
||||
}
|
||||
|
||||
async deleteMany(input: readonly DeleteSmsMessageRequest[]): Promise<MessageDeleteResult> {
|
||||
const lookup = this.#db.prepare(
|
||||
'SELECT id,upstream_id FROM sms_messages WHERE instance_id=? AND (id=? OR upstream_id=?)',
|
||||
);
|
||||
const resolved = input.map((item) => {
|
||||
const row = lookup.get(item.instanceId, item.id, item.id) as
|
||||
| { id: string; upstream_id: string }
|
||||
| undefined;
|
||||
return {
|
||||
instanceId: item.instanceId,
|
||||
upstreamId: row?.upstream_id ?? item.id,
|
||||
centralId: row?.id ?? item.id,
|
||||
};
|
||||
});
|
||||
const result = await this.#options.messages.deleteMany(
|
||||
resolved.map(({ instanceId, upstreamId }) => ({ instanceId, id: upstreamId })),
|
||||
);
|
||||
const byUpstream = new Map(
|
||||
resolved.map((item) => [`${item.instanceId}|${item.upstreamId}`, item]),
|
||||
);
|
||||
const failed = new Set(result.failures.map((failure) => `${failure.instanceId}|${failure.id}`));
|
||||
const remove = this.#db.prepare(
|
||||
'DELETE FROM sms_messages WHERE instance_id=? AND (id=? OR upstream_id=?)',
|
||||
);
|
||||
this.#db.transaction(() => {
|
||||
for (const item of resolved) {
|
||||
if (failed.has(`${item.instanceId}|${item.upstreamId}`)) continue;
|
||||
remove.run(item.instanceId, item.centralId, item.upstreamId);
|
||||
}
|
||||
})();
|
||||
return Object.freeze({
|
||||
...result,
|
||||
failures: Object.freeze(
|
||||
result.failures.map((failure) => {
|
||||
const item = byUpstream.get(`${failure.instanceId}|${failure.id}`);
|
||||
return Object.freeze({ ...failure, id: item?.centralId ?? failure.id });
|
||||
}),
|
||||
),
|
||||
});
|
||||
}
|
||||
|
||||
#message(row: Record<string, unknown>): HubMessage {
|
||||
return Object.freeze({
|
||||
id: String(row.id),
|
||||
instanceId: String(row.instance_id),
|
||||
instanceName: String(row.instance_name),
|
||||
direction: String(row.direction),
|
||||
phoneNumber: String(row.phone_number),
|
||||
content: String(row.content),
|
||||
timestamp: String(row.timestamp),
|
||||
status: String(row.status),
|
||||
transport: String(row.transport),
|
||||
syncedAt: String(row.synced_at),
|
||||
});
|
||||
}
|
||||
|
||||
#maximumPerDevice(): number {
|
||||
return Math.max(PAGE_SIZE, this.#options.maximumMessagesPerDevice ?? DEFAULT_MAXIMUM);
|
||||
}
|
||||
|
||||
#nowDate(): Date {
|
||||
return this.#options.now?.() ?? new Date();
|
||||
}
|
||||
|
||||
#id(): string {
|
||||
return this.#options.id?.() ?? randomUUID();
|
||||
}
|
||||
}
|
||||
@@ -4,12 +4,19 @@ import {
|
||||
InstanceMessageService,
|
||||
MessageServiceError,
|
||||
parseMessageList,
|
||||
validMessageContent,
|
||||
validMessageId,
|
||||
} from './instance-message-service.js';
|
||||
|
||||
const instance = { id: 'alpha', origin: 'http://192.168.1.10:8080' };
|
||||
const instances = { get: vi.fn(async (id: string) => (id === 'alpha' ? instance : undefined)) };
|
||||
|
||||
describe('InstanceMessageService', () => {
|
||||
it('accepts the automation contract maximum of 2,000 SMS characters', () => {
|
||||
expect(validMessageContent('字'.repeat(2_000))).toBe(true);
|
||||
expect(validMessageContent('字'.repeat(2_001))).toBe(false);
|
||||
});
|
||||
|
||||
it('parses a bounded explicit message allowlist and never exposes pdu or excess fields', () => {
|
||||
const messages = parseMessageList(
|
||||
{
|
||||
@@ -51,6 +58,43 @@ describe('InstanceMessageService', () => {
|
||||
expect(JSON.stringify(messages)).not.toContain('secret');
|
||||
});
|
||||
|
||||
it('accepts the current SimAdmin SMS shape without the removed transport field', () => {
|
||||
expect(
|
||||
parseMessageList(
|
||||
{
|
||||
status: 200,
|
||||
headers: {},
|
||||
body: JSON.stringify({
|
||||
status: 'success',
|
||||
data: {
|
||||
messages: [
|
||||
{
|
||||
id: 42,
|
||||
direction: 'incoming',
|
||||
phone_number: '10086',
|
||||
content: 'current payload',
|
||||
timestamp: '2026-07-29 12:30:00',
|
||||
status: 'received',
|
||||
},
|
||||
],
|
||||
},
|
||||
}),
|
||||
},
|
||||
10,
|
||||
),
|
||||
).toEqual([
|
||||
{
|
||||
id: '42',
|
||||
direction: 'incoming',
|
||||
phoneNumber: '10086',
|
||||
content: 'current payload',
|
||||
timestamp: '2026-07-29 12:30:00',
|
||||
status: 'received',
|
||||
transport: 'modem',
|
||||
},
|
||||
]);
|
||||
});
|
||||
|
||||
it('uses exact owner, bounded query and matching optional session cookie', async () => {
|
||||
const sessions = new InstanceSessionStore();
|
||||
sessions.set('alpha', instance.origin, 'simadmin_session=opaque');
|
||||
@@ -97,6 +141,88 @@ describe('InstanceMessageService', () => {
|
||||
});
|
||||
});
|
||||
|
||||
it('deletes a bounded batch with explicit IDs and reports per-instance failures', async () => {
|
||||
expect(validMessageId('1')).toBe(true);
|
||||
expect(validMessageId('9007199254740991')).toBe(true);
|
||||
expect(validMessageId('9007199254740992')).toBe(false);
|
||||
const request = vi.fn(async (value: unknown) => {
|
||||
const body = value as { readonly smsBatchDelete?: { readonly ids: readonly number[] } };
|
||||
const deleted = body.smsBatchDelete?.ids.length ?? 0;
|
||||
return {
|
||||
status: 200,
|
||||
headers: {},
|
||||
body: JSON.stringify({
|
||||
status: 'success',
|
||||
data: { deleted },
|
||||
}),
|
||||
};
|
||||
});
|
||||
const service = new InstanceMessageService({
|
||||
instances: instances as never,
|
||||
sessions: new InstanceSessionStore(),
|
||||
request,
|
||||
});
|
||||
|
||||
const result = await service.deleteMany([
|
||||
{ instanceId: 'alpha', id: '1' },
|
||||
{ instanceId: 'alpha', id: '2' },
|
||||
{ instanceId: 'missing', id: '3' },
|
||||
]);
|
||||
|
||||
expect(result).toEqual({
|
||||
requested: 3,
|
||||
deleted: 2,
|
||||
failed: 1,
|
||||
failures: [
|
||||
{
|
||||
instanceId: 'missing',
|
||||
instanceName: '',
|
||||
id: '3',
|
||||
code: 'NOT_FOUND',
|
||||
},
|
||||
],
|
||||
});
|
||||
expect(request).toHaveBeenCalledWith({
|
||||
url: 'http://192.168.1.10:8080/api/sms/batch-delete',
|
||||
method: 'POST',
|
||||
headers: {
|
||||
accept: 'application/json',
|
||||
'content-type': 'application/json',
|
||||
},
|
||||
smsBatchDelete: { ids: [1, 2] },
|
||||
});
|
||||
});
|
||||
|
||||
it('fails closed for invalid batch payloads and malformed upstream delete replies', async () => {
|
||||
const request = vi.fn(async () => ({
|
||||
status: 200,
|
||||
headers: {},
|
||||
body: '{"status":"success","data":{"deleted":2}}',
|
||||
}));
|
||||
const service = new InstanceMessageService({
|
||||
instances: instances as never,
|
||||
sessions: new InstanceSessionStore(),
|
||||
request,
|
||||
});
|
||||
await expect(service.deleteMany([])).rejects.toMatchObject({
|
||||
code: 'VALIDATION_FAILED',
|
||||
});
|
||||
await expect(
|
||||
service.deleteMany([{ instanceId: 'alpha', id: 'not-a-number' }]),
|
||||
).rejects.toMatchObject({ code: 'VALIDATION_FAILED' });
|
||||
request.mockResolvedValueOnce({
|
||||
status: 200,
|
||||
headers: {},
|
||||
body: '{"status":"success","data":{"deleted":-1}}',
|
||||
});
|
||||
await expect(service.deleteMany([{ instanceId: 'alpha', id: '1' }])).resolves.toMatchObject({
|
||||
requested: 1,
|
||||
deleted: 0,
|
||||
failed: 1,
|
||||
failures: [{ code: 'UPSTREAM_FAILED' }],
|
||||
});
|
||||
});
|
||||
|
||||
it('fails closed for missing owners, stale sessions and upstream status:error', async () => {
|
||||
const sessions = new InstanceSessionStore();
|
||||
sessions.set('alpha', 'http://192.168.1.99', 'simadmin_session=opaque');
|
||||
|
||||
@@ -24,6 +24,22 @@ export interface SendMessageInput {
|
||||
readonly phoneNumber: string;
|
||||
readonly content: string;
|
||||
}
|
||||
export interface DeleteSmsMessageRequest {
|
||||
readonly instanceId: string;
|
||||
readonly id: string;
|
||||
}
|
||||
export interface MessageDeleteFailure {
|
||||
readonly instanceId: string;
|
||||
readonly instanceName: string;
|
||||
readonly id: string;
|
||||
readonly code: MessageServiceErrorCode;
|
||||
}
|
||||
export interface MessageDeleteResult {
|
||||
readonly requested: number;
|
||||
readonly deleted: number;
|
||||
readonly failed: number;
|
||||
readonly failures: readonly MessageDeleteFailure[];
|
||||
}
|
||||
export type MessageServiceErrorCode =
|
||||
| 'NOT_FOUND'
|
||||
| 'VALIDATION_FAILED'
|
||||
@@ -37,6 +53,7 @@ export class MessageServiceError extends Error {
|
||||
}
|
||||
|
||||
const MAX_RESPONSE_BYTES = 262_144;
|
||||
const MAX_BATCH_DELETE = 500;
|
||||
const DIRECTIONS = new Set<MessageDirection>(['received', 'sent', 'incoming', 'outgoing']);
|
||||
const record = (value: unknown): Record<string, unknown> | undefined =>
|
||||
value !== null && typeof value === 'object' && !Array.isArray(value)
|
||||
@@ -48,6 +65,10 @@ const bounded = (value: unknown, maximum: number): string | undefined =>
|
||||
!/[\u0000-\u0008\u000b\u000c\u000e-\u001f\u007f]/u.test(String(value))
|
||||
? String(value)
|
||||
: undefined;
|
||||
const safeCount = (value: unknown, maximum: number): number | undefined =>
|
||||
typeof value === 'number' && Number.isSafeInteger(value) && value >= 0 && value <= maximum
|
||||
? value
|
||||
: undefined;
|
||||
|
||||
export function validPhoneNumber(value: string): boolean {
|
||||
return value.length >= 3 && value.length <= 32 && /^\+?[0-9][0-9 ()-]*$/u.test(value);
|
||||
@@ -55,12 +76,18 @@ export function validPhoneNumber(value: string): boolean {
|
||||
export function validMessageContent(value: string): boolean {
|
||||
return (
|
||||
value.length >= 1 &&
|
||||
value.length <= 1600 &&
|
||||
Buffer.byteLength(value, 'utf8') <= 6400 &&
|
||||
value.length <= 2000 &&
|
||||
Buffer.byteLength(value, 'utf8') <= 8000 &&
|
||||
!/[\u0000-\u0008\u000b\u000c\u000e-\u001f\u007f]/u.test(value)
|
||||
);
|
||||
}
|
||||
|
||||
export function validMessageId(value: string): boolean {
|
||||
if (!/^\d{1,18}$/u.test(value)) return false;
|
||||
const parsed = Number(value);
|
||||
return Number.isSafeInteger(parsed) && parsed >= 0;
|
||||
}
|
||||
|
||||
export function parseMessageList(
|
||||
response: UpstreamResponse,
|
||||
limit: number,
|
||||
@@ -87,10 +114,12 @@ export function parseMessageList(
|
||||
const value = record(item);
|
||||
const id = bounded(value?.id, 128);
|
||||
const phoneNumber = bounded(value?.phone_number, 32);
|
||||
const content = bounded(value?.content, 1600);
|
||||
const content = bounded(value?.content, 2000);
|
||||
const timestamp = bounded(value?.timestamp, 64);
|
||||
const status = bounded(value?.status, 32);
|
||||
const transport = bounded(value?.transport, 32);
|
||||
// Current SimAdmin SmsMessage has no transport field; older captures sometimes did.
|
||||
// Keep the aggregate contract stable without dropping every current message.
|
||||
const transport = bounded(value?.transport, 32) ?? 'modem';
|
||||
const rawDirection = bounded(value?.direction, 16);
|
||||
if (
|
||||
!id ||
|
||||
@@ -98,8 +127,7 @@ export function parseMessageList(
|
||||
!validPhoneNumber(phoneNumber) ||
|
||||
content === undefined ||
|
||||
timestamp === undefined ||
|
||||
status === undefined ||
|
||||
transport === undefined
|
||||
status === undefined
|
||||
)
|
||||
continue;
|
||||
const direction: MessageDirection =
|
||||
@@ -127,12 +155,37 @@ function parseSendSuccess(response: UpstreamResponse): void {
|
||||
}
|
||||
}
|
||||
|
||||
function parseBatchDeleteSuccess(response: UpstreamResponse, expected: number): number {
|
||||
if (
|
||||
response.status < 200 ||
|
||||
response.status >= 300 ||
|
||||
Buffer.byteLength(response.body, 'utf8') > 32_768
|
||||
)
|
||||
throw new MessageServiceError('UPSTREAM_FAILED');
|
||||
let root: Record<string, unknown> | undefined;
|
||||
try {
|
||||
root = record(JSON.parse(response.body));
|
||||
} catch {
|
||||
throw new MessageServiceError('UPSTREAM_FAILED');
|
||||
}
|
||||
if (root?.status !== 'success' && root?.status !== 'ok')
|
||||
throw new MessageServiceError('UPSTREAM_FAILED');
|
||||
const deleted = safeCount(record(root.data)?.deleted, expected);
|
||||
if (deleted === undefined) throw new MessageServiceError('UPSTREAM_FAILED');
|
||||
return deleted;
|
||||
}
|
||||
|
||||
export class InstanceMessageService {
|
||||
constructor(
|
||||
private readonly options: {
|
||||
readonly instances: InstanceService;
|
||||
readonly sessions: InstanceSessionStore;
|
||||
readonly request: UpstreamSessionClientOptions['request'];
|
||||
readonly ensureSession?: (
|
||||
instanceId: string,
|
||||
origin: string,
|
||||
force?: boolean,
|
||||
) => Promise<void>;
|
||||
},
|
||||
) {}
|
||||
|
||||
@@ -142,7 +195,14 @@ export class InstanceMessageService {
|
||||
const session = this.options.sessions.sessionFor(instanceId);
|
||||
if (session && session.origin !== instance.origin)
|
||||
throw new MessageServiceError('SESSION_INVALID');
|
||||
return { instance, session };
|
||||
if (!session && this.options.ensureSession) {
|
||||
try {
|
||||
await this.options.ensureSession(instanceId, instance.origin);
|
||||
} catch {
|
||||
// No saved credential means this may be a passwordless instance; read anonymously.
|
||||
}
|
||||
}
|
||||
return { instance, session: this.options.sessions.sessionFor(instanceId) };
|
||||
}
|
||||
|
||||
async list(
|
||||
@@ -163,11 +223,17 @@ export class InstanceMessageService {
|
||||
throw new MessageServiceError('VALIDATION_FAILED');
|
||||
const { instance, session } = await this.owner(instanceId);
|
||||
const direction = query.direction ? `&direction=${query.direction}` : '';
|
||||
const response = await this.options.request({
|
||||
url: `${instance.origin}/api/sms/list?limit=${query.limit}&offset=${query.offset}${direction}`,
|
||||
method: 'GET',
|
||||
headers: { accept: 'application/json', ...(session ? { cookie: session.cookie } : {}) },
|
||||
});
|
||||
const request = (cookie?: string) =>
|
||||
this.options.request({
|
||||
url: `${instance.origin}/api/sms/list?limit=${query.limit}&offset=${query.offset}${direction}`,
|
||||
method: 'GET',
|
||||
headers: { accept: 'application/json', ...(cookie ? { cookie } : {}) },
|
||||
});
|
||||
let response = await request(session?.cookie);
|
||||
if ([401, 403].includes(response.status) && this.options.ensureSession) {
|
||||
await this.options.ensureSession(instanceId, instance.origin, true);
|
||||
response = await request(this.options.sessions.sessionFor(instanceId)?.cookie);
|
||||
}
|
||||
return { messages: parseMessageList(response, query.limit) };
|
||||
}
|
||||
|
||||
@@ -175,17 +241,97 @@ export class InstanceMessageService {
|
||||
if (!validPhoneNumber(input.phoneNumber) || !validMessageContent(input.content))
|
||||
throw new MessageServiceError('VALIDATION_FAILED');
|
||||
const { instance, session } = await this.owner(instanceId);
|
||||
const response = await this.options.request({
|
||||
url: `${instance.origin}/api/sms/send`,
|
||||
method: 'POST',
|
||||
headers: {
|
||||
accept: 'application/json',
|
||||
'content-type': 'application/json',
|
||||
...(session ? { cookie: session.cookie } : {}),
|
||||
},
|
||||
sms: { phoneNumber: input.phoneNumber, content: input.content },
|
||||
});
|
||||
const request = (cookie?: string) =>
|
||||
this.options.request({
|
||||
url: `${instance.origin}/api/sms/send`,
|
||||
method: 'POST',
|
||||
headers: {
|
||||
accept: 'application/json',
|
||||
'content-type': 'application/json',
|
||||
...(cookie ? { cookie } : {}),
|
||||
},
|
||||
sms: { phoneNumber: input.phoneNumber, content: input.content },
|
||||
});
|
||||
let response = await request(session?.cookie);
|
||||
if ([401, 403].includes(response.status) && this.options.ensureSession) {
|
||||
await this.options.ensureSession(instanceId, instance.origin, true);
|
||||
response = await request(this.options.sessions.sessionFor(instanceId)?.cookie);
|
||||
}
|
||||
parseSendSuccess(response);
|
||||
return { sent: true };
|
||||
}
|
||||
|
||||
async deleteMany(input: readonly DeleteSmsMessageRequest[]): Promise<MessageDeleteResult> {
|
||||
if (!Array.isArray(input) || input.length < 1 || input.length > MAX_BATCH_DELETE)
|
||||
throw new MessageServiceError('VALIDATION_FAILED');
|
||||
const grouped = new Map<string, Array<{ readonly id: string; readonly numericId: number }>>();
|
||||
const requested = input.length;
|
||||
for (const item of input) {
|
||||
const instanceId = typeof item?.instanceId === 'string' ? item.instanceId : '';
|
||||
const id = typeof item?.id === 'string' ? item.id : '';
|
||||
if (!instanceId || !validMessageId(id)) throw new MessageServiceError('VALIDATION_FAILED');
|
||||
const numericId = Number(id);
|
||||
const current = grouped.get(instanceId);
|
||||
if (current === undefined) grouped.set(instanceId, [{ id, numericId }]);
|
||||
else current.push({ id, numericId });
|
||||
}
|
||||
|
||||
let deleted = 0;
|
||||
const failures: MessageDeleteFailure[] = [];
|
||||
for (const [instanceId, entries] of grouped) {
|
||||
const ids = entries.map((entry) => entry.numericId);
|
||||
let instanceName = '';
|
||||
try {
|
||||
const instance = await this.options.instances.get(instanceId);
|
||||
if (!instance) throw new MessageServiceError('NOT_FOUND');
|
||||
instanceName = instance.name;
|
||||
const session = this.options.sessions.sessionFor(instanceId);
|
||||
if (session && session.origin !== instance.origin)
|
||||
throw new MessageServiceError('SESSION_INVALID');
|
||||
if (!session && this.options.ensureSession) {
|
||||
try {
|
||||
await this.options.ensureSession(instanceId, instance.origin);
|
||||
} catch {
|
||||
// Passwordless instances may still accept anonymous batch deletion.
|
||||
}
|
||||
}
|
||||
const request = (cookie?: string) =>
|
||||
this.options.request({
|
||||
url: `${instance.origin}/api/sms/batch-delete`,
|
||||
method: 'POST',
|
||||
headers: {
|
||||
accept: 'application/json',
|
||||
'content-type': 'application/json',
|
||||
...(cookie ? { cookie } : {}),
|
||||
},
|
||||
smsBatchDelete: { ids },
|
||||
});
|
||||
let response = await request(this.options.sessions.sessionFor(instanceId)?.cookie);
|
||||
if ([401, 403].includes(response.status) && this.options.ensureSession) {
|
||||
await this.options.ensureSession(instanceId, instance.origin, true);
|
||||
response = await request(this.options.sessions.sessionFor(instanceId)?.cookie);
|
||||
}
|
||||
deleted += parseBatchDeleteSuccess(response, entries.length);
|
||||
} catch (error) {
|
||||
const code =
|
||||
error instanceof MessageServiceError ? error.code : ('UPSTREAM_FAILED' as const);
|
||||
for (const entry of entries) {
|
||||
failures.push(
|
||||
Object.freeze({
|
||||
instanceId,
|
||||
instanceName,
|
||||
id: entry.id,
|
||||
code,
|
||||
}),
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
return {
|
||||
requested,
|
||||
deleted,
|
||||
failed: failures.length,
|
||||
failures: Object.freeze(failures),
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
@@ -0,0 +1,190 @@
|
||||
import Database from 'better-sqlite3';
|
||||
|
||||
import { afterEach, beforeEach, describe, expect, it } from 'vitest';
|
||||
|
||||
import { migrateDatabase } from '../../infrastructure/database/migrations.js';
|
||||
import { MessageServiceError } from './instance-message-service.js';
|
||||
import { SmsOutboxService } from './sms-outbox-service.js';
|
||||
|
||||
const START = new Date('2026-09-05T02:00:00.000Z');
|
||||
|
||||
interface Clock {
|
||||
current: Date;
|
||||
advance: (ms: number) => void;
|
||||
}
|
||||
|
||||
function clock(): Clock {
|
||||
const state: Clock = {
|
||||
current: new Date(START),
|
||||
advance: (ms: number) => {
|
||||
state.current = new Date(state.current.getTime() + ms);
|
||||
},
|
||||
};
|
||||
return state;
|
||||
}
|
||||
|
||||
describe('SmsOutboxService', () => {
|
||||
let db: Database.Database;
|
||||
let time: Clock;
|
||||
let offline: Set<string>;
|
||||
let delivered: Array<{ readonly instanceId: string; readonly content: string }>;
|
||||
let sendError: (() => Error) | undefined;
|
||||
|
||||
beforeEach(() => {
|
||||
db = new Database(':memory:');
|
||||
db.pragma('foreign_keys = ON');
|
||||
migrateDatabase(db);
|
||||
for (const [id, name] of [
|
||||
['device-1', 'Modem A'],
|
||||
['device-2', 'Modem B'],
|
||||
]) {
|
||||
db.prepare(
|
||||
`INSERT INTO instances (id,name,base_url,auth_mode,enabled,config_revision,created_at,updated_at)
|
||||
VALUES (?,?,?,'password',1,1,?,?)`,
|
||||
).run(id, name, `http://${id}.invalid`, START.toISOString(), START.toISOString());
|
||||
}
|
||||
time = clock();
|
||||
offline = new Set<string>();
|
||||
delivered = [];
|
||||
sendError = undefined;
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
db.close();
|
||||
});
|
||||
|
||||
function service(overrides: { readonly maxAttempts?: number } = {}): SmsOutboxService {
|
||||
let sequence = 0;
|
||||
return new SmsOutboxService(db, {
|
||||
send: async (instanceId, input) => {
|
||||
if (sendError) throw sendError();
|
||||
delivered.push({ instanceId, content: input.content });
|
||||
return { sent: true };
|
||||
},
|
||||
offlineInstances: () => offline,
|
||||
now: () => time.current,
|
||||
id: () => {
|
||||
sequence += 1;
|
||||
return `queue-${sequence}`;
|
||||
},
|
||||
maxAttempts: overrides.maxAttempts ?? 3,
|
||||
backoffMs: 15_000,
|
||||
});
|
||||
}
|
||||
|
||||
it('delivers straight through while the device is reachable', async () => {
|
||||
const result = await service().submit('device-1', {
|
||||
phoneNumber: '13800138000',
|
||||
content: '在线直发',
|
||||
});
|
||||
expect(result.status).toBe('sent');
|
||||
expect(result.item).toBeNull();
|
||||
expect(delivered).toEqual([{ instanceId: 'device-1', content: '在线直发' }]);
|
||||
expect(service().list().total).toBe(0);
|
||||
});
|
||||
|
||||
it('queues for an offline device and delivers once it returns', async () => {
|
||||
offline.add('device-1');
|
||||
const outbox = service();
|
||||
const result = await outbox.submit('device-1', {
|
||||
phoneNumber: '13800138000',
|
||||
content: '离线排队',
|
||||
});
|
||||
expect(result.status).toBe('queued');
|
||||
expect(result.item?.status).toBe('queued');
|
||||
expect(result.item?.attempts).toBe(0);
|
||||
expect(delivered).toEqual([]);
|
||||
|
||||
expect(await outbox.flush()).toMatchObject({ attempted: 1, delivered: 0, deferred: 1 });
|
||||
expect(outbox.get(result.item?.id ?? '')?.status).toBe('queued');
|
||||
|
||||
offline.delete('device-1');
|
||||
expect(await outbox.flush()).toMatchObject({ attempted: 1, delivered: 1, remaining: 0 });
|
||||
expect(delivered).toEqual([{ instanceId: 'device-1', content: '离线排队' }]);
|
||||
const settled = outbox.get(result.item?.id ?? '');
|
||||
expect(settled?.status).toBe('sent');
|
||||
expect(settled?.sentAt).toBe(time.current.toISOString());
|
||||
});
|
||||
|
||||
it('falls back to the queue when a live send fails', async () => {
|
||||
sendError = () => new MessageServiceError('UPSTREAM_FAILED');
|
||||
const outbox = service();
|
||||
const result = await outbox.submit('device-1', {
|
||||
phoneNumber: '13800138000',
|
||||
content: '节点抖动',
|
||||
});
|
||||
expect(result.status).toBe('queued');
|
||||
expect(result.item?.lastError).toBe('UPSTREAM_FAILED');
|
||||
});
|
||||
|
||||
it('backs off between attempts and gives up at the limit', async () => {
|
||||
sendError = () => new MessageServiceError('UPSTREAM_FAILED');
|
||||
const outbox = service();
|
||||
await outbox.submit('device-1', { phoneNumber: '13800138000', content: '重试退避' });
|
||||
|
||||
expect(await outbox.flush()).toMatchObject({ attempted: 1, deferred: 1 });
|
||||
expect(outbox.list().items[0]).toMatchObject({ attempts: 1, status: 'queued' });
|
||||
// The next attempt is parked behind the backoff window.
|
||||
expect(await outbox.flush()).toMatchObject({ attempted: 0 });
|
||||
|
||||
time.advance(15_000);
|
||||
expect(await outbox.flush()).toMatchObject({ attempted: 1, deferred: 1 });
|
||||
expect(outbox.list().items[0]).toMatchObject({ attempts: 2, status: 'queued' });
|
||||
|
||||
time.advance(30_000);
|
||||
expect(await outbox.flush()).toMatchObject({ attempted: 1, failed: 1, remaining: 0 });
|
||||
const exhausted = outbox.list({ status: 'all' }).items[0];
|
||||
expect(exhausted).toMatchObject({ status: 'failed', attempts: 3 });
|
||||
|
||||
const revived = outbox.retry(exhausted?.id ?? '');
|
||||
expect(revived.status).toBe('queued');
|
||||
expect(revived.maxAttempts).toBeGreaterThanOrEqual(revived.attempts + 1);
|
||||
});
|
||||
|
||||
it('cancels pending work, prunes history, and reconciles an interrupted process', async () => {
|
||||
offline.add('device-1');
|
||||
const outbox = service();
|
||||
const first = await outbox.submit('device-1', {
|
||||
phoneNumber: '13800138000',
|
||||
content: '待取消',
|
||||
});
|
||||
const second = await outbox.submit('device-1', {
|
||||
phoneNumber: '13800138001',
|
||||
content: '待删除',
|
||||
});
|
||||
expect(outbox.summary()).toMatchObject({ queued: 2 });
|
||||
|
||||
expect(outbox.cancel(first.item?.id ?? '').status).toBe('cancelled');
|
||||
expect(() => outbox.cancel(first.item?.id ?? '')).toThrow(MessageServiceError);
|
||||
outbox.remove(second.item?.id ?? '');
|
||||
expect(outbox.list({ status: 'all' }).total).toBe(1);
|
||||
|
||||
time.advance(3_600_000);
|
||||
expect(outbox.prune(0)).toBe(1);
|
||||
expect(outbox.list({ status: 'all' }).total).toBe(0);
|
||||
|
||||
db.prepare(
|
||||
`INSERT INTO sms_outbox (id,instance_id,phone_number,content,status,attempts,max_attempts,
|
||||
available_at,created_at,updated_at)
|
||||
VALUES ('stale','device-1','13800138000','中断','sending',1,3,?,?,?)`,
|
||||
).run(START.toISOString(), START.toISOString(), START.toISOString());
|
||||
expect(outbox.reconcileInterrupted()).toBe(1);
|
||||
expect(outbox.get('stale')).toMatchObject({ status: 'queued', lastError: 'INTERRUPTED' });
|
||||
});
|
||||
|
||||
it('rejects malformed submissions and unknown queue entries', async () => {
|
||||
const outbox = service();
|
||||
await expect(outbox.submit('', { phoneNumber: '13800138000', content: 'x' })).rejects.toThrow(
|
||||
MessageServiceError,
|
||||
);
|
||||
await expect(
|
||||
outbox.submit('device-1', { phoneNumber: 'not-a-number', content: 'x' }),
|
||||
).rejects.toThrow(MessageServiceError);
|
||||
await expect(
|
||||
outbox.submit('device-1', { phoneNumber: '13800138000', content: ' ' }),
|
||||
).rejects.toThrow(MessageServiceError);
|
||||
expect(outbox.get('missing')).toBeUndefined();
|
||||
expect(() => outbox.cancel('missing')).toThrow(MessageServiceError);
|
||||
expect(outbox.list({ status: 'all', limit: 10, offset: 0 }).items).toEqual([]);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,483 @@
|
||||
import { randomUUID } from 'node:crypto';
|
||||
|
||||
import type Database from 'better-sqlite3';
|
||||
|
||||
import { MessageServiceError } from './instance-message-service.js';
|
||||
|
||||
export type SmsOutboxStatus = 'queued' | 'sending' | 'sent' | 'failed' | 'cancelled';
|
||||
|
||||
export interface SmsOutboxItem {
|
||||
readonly id: string;
|
||||
readonly instanceId: string;
|
||||
readonly phoneNumber: string;
|
||||
readonly content: string;
|
||||
readonly status: SmsOutboxStatus;
|
||||
readonly attempts: number;
|
||||
readonly maxAttempts: number;
|
||||
readonly lastError: string | null;
|
||||
readonly availableAt: string;
|
||||
readonly sentAt: string | null;
|
||||
readonly createdAt: string;
|
||||
readonly updatedAt: string;
|
||||
}
|
||||
|
||||
export interface SmsOutboxPage {
|
||||
readonly items: readonly SmsOutboxItem[];
|
||||
readonly total: number;
|
||||
}
|
||||
|
||||
export interface SmsOutboxSummary {
|
||||
readonly queued: number;
|
||||
readonly sending: number;
|
||||
readonly failed: number;
|
||||
readonly sent: number;
|
||||
}
|
||||
|
||||
export interface SmsOutboxFlushResult {
|
||||
readonly attempted: number;
|
||||
readonly delivered: number;
|
||||
readonly deferred: number;
|
||||
readonly failed: number;
|
||||
readonly remaining: number;
|
||||
}
|
||||
|
||||
export interface SmsOutboxSubmitResult {
|
||||
readonly status: 'sent' | 'queued';
|
||||
readonly item: SmsOutboxItem | null;
|
||||
}
|
||||
|
||||
export interface SmsOutboxServiceOptions {
|
||||
readonly send: (
|
||||
instanceId: string,
|
||||
input: { readonly phoneNumber: string; readonly content: string },
|
||||
) => Promise<unknown>;
|
||||
/**
|
||||
* Devices the heartbeat positively marked unreachable. Never-probed devices stay optimistic so
|
||||
* a send still goes straight to the node; only known-dead devices queue.
|
||||
*/
|
||||
readonly offlineInstances: () => ReadonlySet<string>;
|
||||
readonly now?: () => Date;
|
||||
readonly id?: () => string;
|
||||
readonly maxAttempts?: number;
|
||||
readonly backoffMs?: number;
|
||||
readonly maximumBackoffMs?: number;
|
||||
readonly pageSize?: number;
|
||||
}
|
||||
|
||||
const DEFAULT_MAX_ATTEMPTS = 24;
|
||||
const DEFAULT_BACKOFF_MS = 15_000;
|
||||
const DEFAULT_MAXIMUM_BACKOFF_MS = 600_000;
|
||||
const DEFAULT_PAGE_SIZE = 50;
|
||||
const MAX_PAGE_SIZE = 200;
|
||||
const MAX_PHONE_LENGTH = 32;
|
||||
const MAX_CONTENT_LENGTH = 2_000;
|
||||
const MAX_ERROR_LENGTH = 200;
|
||||
const OPEN_STATUSES: readonly SmsOutboxStatus[] = ['queued', 'sending'];
|
||||
|
||||
interface OutboxRow {
|
||||
readonly id: string;
|
||||
readonly instance_id: string;
|
||||
readonly phone_number: string;
|
||||
readonly content: string;
|
||||
readonly status: string;
|
||||
readonly attempts: number;
|
||||
readonly max_attempts: number;
|
||||
readonly last_error: string | null;
|
||||
readonly available_at: string;
|
||||
readonly sent_at: string | null;
|
||||
readonly created_at: string;
|
||||
readonly updated_at: string;
|
||||
}
|
||||
|
||||
function truncate(value: string, maximum: number): string {
|
||||
return value.length <= maximum ? value : `${value.slice(0, maximum - 1)}…`;
|
||||
}
|
||||
|
||||
function describeError(error: unknown): string {
|
||||
if (error instanceof MessageServiceError) return error.code;
|
||||
if (error instanceof Error && error.message.trim() !== '')
|
||||
return truncate(error.message.replace(/\s+/gu, ' '), MAX_ERROR_LENGTH);
|
||||
return 'DELIVERY_FAILED';
|
||||
}
|
||||
|
||||
function validPhoneNumber(value: unknown): value is string {
|
||||
return (
|
||||
typeof value === 'string' &&
|
||||
value.trim().length >= 3 &&
|
||||
value.trim().length <= MAX_PHONE_LENGTH &&
|
||||
/^[+0-9][0-9 ()-]*$/u.test(value.trim())
|
||||
);
|
||||
}
|
||||
|
||||
function validContent(value: unknown): value is string {
|
||||
return (
|
||||
typeof value === 'string' &&
|
||||
value.trim().length > 0 &&
|
||||
value.trim().length <= MAX_CONTENT_LENGTH
|
||||
);
|
||||
}
|
||||
|
||||
/**
|
||||
* Durable outbound SMS queue. The Hub keeps accepting sends while a device is offline and hands
|
||||
* them over once the device returns, so this service owns the wait rather than failing the request.
|
||||
*/
|
||||
export class SmsOutboxService {
|
||||
readonly #db: Database.Database;
|
||||
readonly #options: SmsOutboxServiceOptions;
|
||||
readonly #now: () => Date;
|
||||
readonly #id: () => string;
|
||||
readonly #maxAttempts: number;
|
||||
readonly #backoffMs: number;
|
||||
readonly #maximumBackoffMs: number;
|
||||
readonly #pageSize: number;
|
||||
#flushing: Promise<SmsOutboxFlushResult> | undefined;
|
||||
|
||||
constructor(db: Database.Database, options: SmsOutboxServiceOptions) {
|
||||
const maxAttempts = options.maxAttempts ?? DEFAULT_MAX_ATTEMPTS;
|
||||
if (!Number.isSafeInteger(maxAttempts) || maxAttempts < 1 || maxAttempts > 1_000)
|
||||
throw new RangeError('maxAttempts must be between 1 and 1000');
|
||||
const backoffMs = options.backoffMs ?? DEFAULT_BACKOFF_MS;
|
||||
if (!Number.isSafeInteger(backoffMs) || backoffMs < 1_000 || backoffMs > 600_000)
|
||||
throw new RangeError('backoffMs must be between 1000 and 600000');
|
||||
const maximumBackoffMs = options.maximumBackoffMs ?? DEFAULT_MAXIMUM_BACKOFF_MS;
|
||||
if (!Number.isSafeInteger(maximumBackoffMs) || maximumBackoffMs < backoffMs)
|
||||
throw new RangeError('maximumBackoffMs must be at least backoffMs');
|
||||
const pageSize = options.pageSize ?? DEFAULT_PAGE_SIZE;
|
||||
if (!Number.isSafeInteger(pageSize) || pageSize < 1 || pageSize > MAX_PAGE_SIZE)
|
||||
throw new RangeError(`pageSize must be between 1 and ${MAX_PAGE_SIZE}`);
|
||||
this.#db = db;
|
||||
this.#options = options;
|
||||
this.#now = options.now ?? (() => new Date());
|
||||
this.#id = options.id ?? (() => randomUUID());
|
||||
this.#maxAttempts = maxAttempts;
|
||||
this.#backoffMs = backoffMs;
|
||||
this.#maximumBackoffMs = maximumBackoffMs;
|
||||
this.#pageSize = pageSize;
|
||||
}
|
||||
|
||||
/**
|
||||
* Delivers immediately when the device is reachable and otherwise parks the message in the
|
||||
* queue. A failed live attempt also falls back to the queue so a flapping device never loses
|
||||
* an operator send.
|
||||
*/
|
||||
async submit(
|
||||
instanceId: string,
|
||||
input: { readonly phoneNumber: unknown; readonly content: unknown },
|
||||
): Promise<SmsOutboxSubmitResult> {
|
||||
if (typeof instanceId !== 'string' || instanceId.trim() === '')
|
||||
throw new MessageServiceError('VALIDATION_FAILED');
|
||||
if (!validPhoneNumber(input.phoneNumber) || !validContent(input.content))
|
||||
throw new MessageServiceError('VALIDATION_FAILED');
|
||||
const phoneNumber = input.phoneNumber.trim();
|
||||
const content = input.content.trim();
|
||||
if (!this.#offline().has(instanceId)) {
|
||||
try {
|
||||
await this.#options.send(instanceId, { phoneNumber, content });
|
||||
return { status: 'sent', item: null };
|
||||
} catch (error) {
|
||||
if (error instanceof MessageServiceError && error.code === 'VALIDATION_FAILED') throw error;
|
||||
const item = this.#enqueue(instanceId, phoneNumber, content, describeError(error));
|
||||
return { status: 'queued', item };
|
||||
}
|
||||
}
|
||||
const item = this.#enqueue(instanceId, phoneNumber, content, null);
|
||||
return { status: 'queued', item };
|
||||
}
|
||||
|
||||
list(
|
||||
query: {
|
||||
readonly status?: SmsOutboxStatus | 'open' | 'all';
|
||||
readonly instanceId?: string;
|
||||
readonly limit?: number;
|
||||
readonly offset?: number;
|
||||
} = {},
|
||||
): SmsOutboxPage {
|
||||
const limit = Math.min(Math.max(Math.trunc(query.limit ?? this.#pageSize), 1), MAX_PAGE_SIZE);
|
||||
const offset = Math.max(Math.trunc(query.offset ?? 0), 0);
|
||||
const conditions: string[] = [];
|
||||
const parameters: unknown[] = [];
|
||||
const status = query.status ?? 'open';
|
||||
if (status === 'open') {
|
||||
conditions.push(`status IN (${OPEN_STATUSES.map(() => '?').join(',')})`);
|
||||
parameters.push(...OPEN_STATUSES);
|
||||
} else if (status !== 'all') {
|
||||
conditions.push('status = ?');
|
||||
parameters.push(status);
|
||||
}
|
||||
if (typeof query.instanceId === 'string' && query.instanceId.trim() !== '') {
|
||||
conditions.push('instance_id = ?');
|
||||
parameters.push(query.instanceId.trim());
|
||||
}
|
||||
const where = conditions.length > 0 ? ` WHERE ${conditions.join(' AND ')}` : '';
|
||||
const total = (this.#db
|
||||
.prepare(`SELECT COUNT(*) AS count FROM sms_outbox${where}`)
|
||||
.get(...parameters) ?? { count: 0 }) as { count: number };
|
||||
const rows = this.#db
|
||||
.prepare(
|
||||
`SELECT * FROM sms_outbox${where}
|
||||
ORDER BY CASE status WHEN 'sending' THEN 0 WHEN 'queued' THEN 1 ELSE 2 END,
|
||||
available_at, created_at
|
||||
LIMIT ? OFFSET ?`,
|
||||
)
|
||||
.all(...parameters, limit, offset) as readonly OutboxRow[];
|
||||
return Object.freeze({
|
||||
items: Object.freeze(rows.map((row) => this.#item(row))),
|
||||
total: Number.isSafeInteger(total.count) ? total.count : 0,
|
||||
});
|
||||
}
|
||||
|
||||
summary(): SmsOutboxSummary {
|
||||
const rows = this.#db
|
||||
.prepare('SELECT status, COUNT(*) AS count FROM sms_outbox GROUP BY status')
|
||||
.all() as readonly { readonly status: string; readonly count: number }[];
|
||||
const counts = new Map(rows.map((row) => [row.status, Number(row.count) || 0]));
|
||||
return Object.freeze({
|
||||
queued: counts.get('queued') ?? 0,
|
||||
sending: counts.get('sending') ?? 0,
|
||||
failed: counts.get('failed') ?? 0,
|
||||
sent: counts.get('sent') ?? 0,
|
||||
});
|
||||
}
|
||||
|
||||
get(id: string): SmsOutboxItem | undefined {
|
||||
const row = this.#db.prepare('SELECT * FROM sms_outbox WHERE id=?').get(id) as
|
||||
| OutboxRow
|
||||
| undefined;
|
||||
return row ? this.#item(row) : undefined;
|
||||
}
|
||||
|
||||
cancel(id: string): SmsOutboxItem {
|
||||
const current = this.#require(id);
|
||||
if (!OPEN_STATUSES.includes(current.status)) throw new MessageServiceError('VALIDATION_FAILED');
|
||||
const timestamp = this.#now().toISOString();
|
||||
this.#db
|
||||
.prepare("UPDATE sms_outbox SET status='cancelled',last_error=?,updated_at=? WHERE id=?")
|
||||
.run('CANCELLED_BY_OPERATOR', timestamp, id);
|
||||
return this.#require(id);
|
||||
}
|
||||
|
||||
/** Re-arms a cancelled or exhausted message so the next flush can try it again. */
|
||||
retry(id: string): SmsOutboxItem {
|
||||
const current = this.#require(id);
|
||||
if (current.status === 'sending' || current.status === 'sent')
|
||||
throw new MessageServiceError('VALIDATION_FAILED');
|
||||
const timestamp = this.#now().toISOString();
|
||||
this.#db
|
||||
.prepare(
|
||||
`UPDATE sms_outbox
|
||||
SET status='queued',available_at=?,last_error=NULL,max_attempts=MAX(max_attempts,attempts+1),
|
||||
updated_at=?
|
||||
WHERE id=?`,
|
||||
)
|
||||
.run(timestamp, timestamp, id);
|
||||
return this.#require(id);
|
||||
}
|
||||
|
||||
remove(id: string): void {
|
||||
this.#require(id);
|
||||
this.#db.prepare('DELETE FROM sms_outbox WHERE id=?').run(id);
|
||||
}
|
||||
|
||||
/** Drops delivered and abandoned entries; queued work is always preserved. */
|
||||
prune(olderThanMs: number): number {
|
||||
if (!Number.isSafeInteger(olderThanMs) || olderThanMs < 0)
|
||||
throw new RangeError('olderThanMs must be a non-negative integer');
|
||||
const cutoff = new Date(this.#now().getTime() - olderThanMs).toISOString();
|
||||
const info = this.#db
|
||||
.prepare(
|
||||
`DELETE FROM sms_outbox
|
||||
WHERE status IN ('sent','failed','cancelled') AND updated_at <= ?`,
|
||||
)
|
||||
.run(cutoff);
|
||||
return Number(info.changes ?? 0);
|
||||
}
|
||||
|
||||
/** Startup sweep: nothing can be mid-flight before the first request of a fresh process. */
|
||||
reconcileInterrupted(): number {
|
||||
const timestamp = this.#now().toISOString();
|
||||
const info = this.#db
|
||||
.prepare(
|
||||
`UPDATE sms_outbox
|
||||
SET status='queued',last_error=?,available_at=?,updated_at=? WHERE status='sending'`,
|
||||
)
|
||||
.run('INTERRUPTED', timestamp, timestamp);
|
||||
return Number(info.changes ?? 0);
|
||||
}
|
||||
|
||||
/** Concurrent callers share one pass, the same way the heartbeat collapses overlapping beats. */
|
||||
flush(): Promise<SmsOutboxFlushResult> {
|
||||
if (this.#flushing) return this.#flushing;
|
||||
const pending = this.#flush().finally(() => {
|
||||
if (this.#flushing === pending) this.#flushing = undefined;
|
||||
});
|
||||
this.#flushing = pending;
|
||||
return pending;
|
||||
}
|
||||
|
||||
async #flush(): Promise<SmsOutboxFlushResult> {
|
||||
const claimed = this.#claim();
|
||||
let delivered = 0;
|
||||
let deferred = 0;
|
||||
let failed = 0;
|
||||
const offline = this.#offline();
|
||||
for (const item of claimed) {
|
||||
if (offline.has(item.instanceId)) {
|
||||
// Waiting for the device is not a failed attempt, so keep the item due on the next tick.
|
||||
this.#defer(item.id, 'DEVICE_OFFLINE');
|
||||
deferred += 1;
|
||||
continue;
|
||||
}
|
||||
try {
|
||||
await this.#options.send(item.instanceId, {
|
||||
phoneNumber: item.phoneNumber,
|
||||
content: item.content,
|
||||
});
|
||||
this.#settle(item.id, 'sent', null);
|
||||
delivered += 1;
|
||||
} catch (error) {
|
||||
const reason = describeError(error);
|
||||
if (error instanceof MessageServiceError && error.code === 'VALIDATION_FAILED') {
|
||||
this.#settle(item.id, 'failed', reason);
|
||||
failed += 1;
|
||||
continue;
|
||||
}
|
||||
if (this.#release(item.id, reason) === 'failed') failed += 1;
|
||||
else deferred += 1;
|
||||
}
|
||||
}
|
||||
const summary = this.summary();
|
||||
return Object.freeze({
|
||||
attempted: claimed.length,
|
||||
delivered,
|
||||
deferred,
|
||||
failed,
|
||||
remaining: summary.queued + summary.sending,
|
||||
});
|
||||
}
|
||||
|
||||
#offline(): ReadonlySet<string> {
|
||||
try {
|
||||
return this.#options.offlineInstances();
|
||||
} catch {
|
||||
// A broken reachability read must not lose queued work; treat every device as reachable.
|
||||
return new Set();
|
||||
}
|
||||
}
|
||||
|
||||
#claim(): readonly SmsOutboxItem[] {
|
||||
const now = this.#now();
|
||||
const timestamp = now.toISOString();
|
||||
return this.#db.transaction((): readonly SmsOutboxItem[] => {
|
||||
const rows = this.#db
|
||||
.prepare(
|
||||
`SELECT * FROM sms_outbox
|
||||
WHERE status='queued' AND available_at <= ?
|
||||
ORDER BY available_at, created_at LIMIT ?`,
|
||||
)
|
||||
.all(timestamp, this.#pageSize) as readonly OutboxRow[];
|
||||
const claim = this.#db.prepare(
|
||||
`UPDATE sms_outbox
|
||||
SET status='sending',attempts=attempts+1,last_error=NULL,updated_at=?
|
||||
WHERE id=? AND status='queued'`,
|
||||
);
|
||||
const claimed: SmsOutboxItem[] = [];
|
||||
for (const row of rows) {
|
||||
const info = claim.run(timestamp, row.id);
|
||||
if (Number(info.changes ?? 0) > 0) claimed.push(this.#item(row, 'sending'));
|
||||
}
|
||||
return claimed;
|
||||
})();
|
||||
}
|
||||
|
||||
/** Puts a claimed item back in line; returns 'failed' once the attempt budget is spent. */
|
||||
#release(id: string, reason: string): 'queued' | 'failed' {
|
||||
const current = this.#require(id);
|
||||
const backoff = Math.min(
|
||||
this.#backoffMs * 2 ** Math.max(current.attempts - 1, 0),
|
||||
this.#maximumBackoffMs,
|
||||
);
|
||||
const timestamp = this.#now().toISOString();
|
||||
const availableAt = new Date(this.#now().getTime() + backoff).toISOString();
|
||||
const exhausted = current.attempts >= current.maxAttempts;
|
||||
this.#db
|
||||
.prepare(
|
||||
`UPDATE sms_outbox
|
||||
SET status=?,last_error=?,available_at=?,updated_at=? WHERE id=?`,
|
||||
)
|
||||
.run(exhausted ? 'failed' : 'queued', reason, availableAt, timestamp, id);
|
||||
return exhausted ? 'failed' : 'queued';
|
||||
}
|
||||
|
||||
#defer(id: string, reason: string): void {
|
||||
const timestamp = this.#now().toISOString();
|
||||
this.#db
|
||||
.prepare(
|
||||
"UPDATE sms_outbox SET status='queued',last_error=?,available_at=?,updated_at=? WHERE id=?",
|
||||
)
|
||||
.run(reason, timestamp, timestamp, id);
|
||||
}
|
||||
|
||||
#settle(id: string, status: 'sent' | 'failed', reason: string | null): void {
|
||||
const timestamp = this.#now().toISOString();
|
||||
this.#db
|
||||
.prepare(
|
||||
`UPDATE sms_outbox
|
||||
SET status=?,last_error=?,sent_at=?,updated_at=? WHERE id=?`,
|
||||
)
|
||||
.run(status, reason, status === 'sent' ? timestamp : null, timestamp, id);
|
||||
}
|
||||
|
||||
#enqueue(instanceId: string, phoneNumber: string, content: string, reason: string | null) {
|
||||
const timestamp = this.#now().toISOString();
|
||||
const id = this.#id();
|
||||
this.#db
|
||||
.prepare(
|
||||
`INSERT INTO sms_outbox (id,instance_id,phone_number,content,status,attempts,max_attempts,
|
||||
last_error,available_at,created_at,updated_at)
|
||||
VALUES (?,?,?,?,'queued',0,?,?,?,?,?)`,
|
||||
)
|
||||
.run(
|
||||
id,
|
||||
instanceId,
|
||||
phoneNumber,
|
||||
content,
|
||||
this.#maxAttempts,
|
||||
reason,
|
||||
timestamp,
|
||||
timestamp,
|
||||
timestamp,
|
||||
);
|
||||
return this.#require(id);
|
||||
}
|
||||
|
||||
#require(id: string): SmsOutboxItem {
|
||||
const item = this.get(id);
|
||||
if (!item) throw new MessageServiceError('NOT_FOUND');
|
||||
return item;
|
||||
}
|
||||
|
||||
#item(row: OutboxRow, status?: SmsOutboxStatus): SmsOutboxItem {
|
||||
return Object.freeze({
|
||||
id: String(row.id),
|
||||
instanceId: String(row.instance_id),
|
||||
phoneNumber: String(row.phone_number),
|
||||
content: String(row.content),
|
||||
status: status ?? this.#status(row.status),
|
||||
attempts: Number.isSafeInteger(row.attempts) ? row.attempts : Number(row.attempts) || 0,
|
||||
maxAttempts: Number.isSafeInteger(row.max_attempts)
|
||||
? row.max_attempts
|
||||
: Number(row.max_attempts) || this.#maxAttempts,
|
||||
lastError: typeof row.last_error === 'string' ? row.last_error : null,
|
||||
availableAt: String(row.available_at),
|
||||
sentAt: typeof row.sent_at === 'string' ? row.sent_at : null,
|
||||
createdAt: String(row.created_at),
|
||||
updatedAt: String(row.updated_at),
|
||||
});
|
||||
}
|
||||
|
||||
#status(value: unknown): SmsOutboxStatus {
|
||||
return value === 'sending' || value === 'sent' || value === 'failed' || value === 'cancelled'
|
||||
? value
|
||||
: 'queued';
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,489 @@
|
||||
import Database from 'better-sqlite3';
|
||||
import { afterEach, describe, expect, it } from 'vitest';
|
||||
|
||||
import { migrateDatabase } from '../../infrastructure/database/migrations.js';
|
||||
import type { SecretStore } from '../../infrastructure/secrets/secret-store.js';
|
||||
import {
|
||||
CentralNotificationService,
|
||||
type NotificationDeliverRequest,
|
||||
} from './central-notification-service.js';
|
||||
|
||||
class MemorySecrets implements SecretStore {
|
||||
readonly provider = 'macos-keychain';
|
||||
readonly values = new Map<string, string>();
|
||||
|
||||
async set(key: { instanceId: string; purpose: string; slot?: string }, value: string) {
|
||||
const reference = `memory://notification/${key.instanceId}/${key.purpose}`;
|
||||
this.values.set(reference, value);
|
||||
return reference;
|
||||
}
|
||||
|
||||
async get(reference: string) {
|
||||
return this.values.get(reference);
|
||||
}
|
||||
|
||||
async delete(reference: string) {
|
||||
return this.values.delete(reference);
|
||||
}
|
||||
}
|
||||
|
||||
const directories: Database.Database[] = [];
|
||||
|
||||
afterEach(() => {
|
||||
for (const db of directories.splice(0)) db.close();
|
||||
});
|
||||
|
||||
function fixture(clock?: () => Date) {
|
||||
const db = new Database(':memory:');
|
||||
db.pragma('foreign_keys = ON');
|
||||
migrateDatabase(db);
|
||||
directories.push(db);
|
||||
const store = new MemorySecrets();
|
||||
let sequence = 0;
|
||||
const deliveries: NotificationDeliverRequest[] = [];
|
||||
const service = new CentralNotificationService(db, {
|
||||
store,
|
||||
now: clock ?? (() => new Date('2026-09-03T08:00:00.000Z')),
|
||||
id: () => `id-${++sequence}`,
|
||||
deliver: async (request) => {
|
||||
deliveries.push(request);
|
||||
const endpoint = String(request.config['url'] ?? request.config['server_url'] ?? '');
|
||||
if (endpoint.includes('unavailable')) throw new Error('channel unreachable');
|
||||
return { ok: true };
|
||||
},
|
||||
});
|
||||
return { db, service, store, deliveries };
|
||||
}
|
||||
|
||||
const bark = {
|
||||
name: 'Bark',
|
||||
type: 'bark' as const,
|
||||
enabled: true,
|
||||
config: { server_url: 'https://api.day.app', group: 'Island', device_key: 'device-key' },
|
||||
};
|
||||
|
||||
describe('CentralNotificationService', () => {
|
||||
it('stores channel credentials in the secret store and returns redacted channels', async () => {
|
||||
const { db, service, store } = fixture();
|
||||
const channel = await service.createChannel(bark);
|
||||
|
||||
expect(channel).toMatchObject({
|
||||
name: 'Bark',
|
||||
type: 'bark',
|
||||
enabled: true,
|
||||
config: { server_url: 'https://api.day.app', group: 'Island' },
|
||||
hasSecret: true,
|
||||
secretFields: ['device_key'],
|
||||
});
|
||||
expect(JSON.stringify(channel)).not.toContain('device-key');
|
||||
expect(store.values.get(channel.secretReference ?? '')).toBe(
|
||||
JSON.stringify({ device_key: 'device-key' }),
|
||||
);
|
||||
expect(
|
||||
db
|
||||
.prepare(
|
||||
'SELECT config_json, secret_reference, secret_fields FROM notification_channels WHERE id = ?',
|
||||
)
|
||||
.get(channel.id),
|
||||
).toEqual({
|
||||
config_json: JSON.stringify({
|
||||
server_url: 'https://api.day.app',
|
||||
group: 'Island',
|
||||
sound: '',
|
||||
level: '',
|
||||
icon: '',
|
||||
auto_copy: true,
|
||||
save_history: true,
|
||||
}),
|
||||
secret_reference: channel.secretReference,
|
||||
secret_fields: 'device_key',
|
||||
});
|
||||
});
|
||||
|
||||
it('creates, matches, and updates device-scoped rules without exposing credentials', async () => {
|
||||
const { service } = fixture();
|
||||
const channel = await service.createChannel(bark);
|
||||
const rule = await service.createRule({
|
||||
name: '短信转发',
|
||||
eventType: 'sms',
|
||||
enabled: true,
|
||||
condition: { field: 'content', mode: 'contains', value: '验证码' },
|
||||
scope: { mode: 'tags', tags: ['lab', 'east'], match: 'any' },
|
||||
channelIds: [channel.id],
|
||||
templates: { title: '来自 {{sender}}', body: '{{content}}' },
|
||||
});
|
||||
|
||||
expect(rule).toMatchObject({
|
||||
name: '短信转发',
|
||||
eventType: 'sms',
|
||||
channels: [{ id: channel.id, name: 'Bark', enabled: true }],
|
||||
});
|
||||
expect(
|
||||
service.matchRules('sms', {
|
||||
instanceId: 'instance-1',
|
||||
instanceTags: ['east'],
|
||||
fields: { content: '您的验证码是 1234' },
|
||||
}),
|
||||
).toEqual([rule]);
|
||||
|
||||
const updated = await service.updateRule(rule.id, {
|
||||
name: '验证码转发',
|
||||
scope: { mode: 'devices', instanceIds: ['instance-2'] },
|
||||
});
|
||||
expect(updated.name).toBe('验证码转发');
|
||||
expect(updated.scope).toEqual({ mode: 'devices', instanceIds: ['instance-2'] });
|
||||
});
|
||||
|
||||
it('matches rules for any selected device group', async () => {
|
||||
const { service } = fixture();
|
||||
const channel = await service.createChannel(bark);
|
||||
const rule = await service.createRule({
|
||||
name: '外场转发',
|
||||
eventType: 'system',
|
||||
enabled: true,
|
||||
condition: { field: 'status', mode: 'all' },
|
||||
scope: { mode: 'groups', groups: ['group-field', 'group-lab'] },
|
||||
channelIds: [channel.id],
|
||||
templates: { title: '{{title}}', body: '{{content}}' },
|
||||
});
|
||||
|
||||
expect(rule.scope).toEqual({ mode: 'groups', groups: ['group-field', 'group-lab'] });
|
||||
expect(
|
||||
service.matchRules('system', {
|
||||
instanceId: 'instance-1',
|
||||
instanceGroupIds: ['group-field'],
|
||||
fields: { status: 'offline' },
|
||||
}),
|
||||
).toEqual([rule]);
|
||||
expect(
|
||||
service.matchRules('system', {
|
||||
instanceId: 'instance-2',
|
||||
instanceGroupIds: ['group-office'],
|
||||
fields: { status: 'offline' },
|
||||
}),
|
||||
).toEqual([]);
|
||||
});
|
||||
|
||||
it('tests an enabled channel, records delivery, and never logs the secret', async () => {
|
||||
const { db, service, deliveries } = fixture();
|
||||
const channel = await service.createChannel(bark);
|
||||
const result = await service.testChannel(channel.id, {
|
||||
title: '测试通知',
|
||||
body: '融合控制台通知链路正常',
|
||||
});
|
||||
|
||||
expect(result).toMatchObject({ ok: true, status: 'success' });
|
||||
expect(deliveries).toEqual([
|
||||
{
|
||||
channelId: channel.id,
|
||||
type: 'bark',
|
||||
config: {
|
||||
server_url: 'https://api.day.app',
|
||||
group: 'Island',
|
||||
sound: '',
|
||||
level: '',
|
||||
icon: '',
|
||||
auto_copy: true,
|
||||
save_history: true,
|
||||
device_key: 'device-key',
|
||||
},
|
||||
eventType: 'test',
|
||||
occurredAt: '2026-09-03T08:00:00.000Z',
|
||||
title: '测试通知',
|
||||
body: '融合控制台通知链路正常',
|
||||
},
|
||||
]);
|
||||
const delivery = db.prepare('SELECT * FROM notification_deliveries').get() as Record<
|
||||
string,
|
||||
unknown
|
||||
>;
|
||||
expect(delivery).toMatchObject({
|
||||
event_type: 'test',
|
||||
status: 'success',
|
||||
channel_id: channel.id,
|
||||
});
|
||||
expect(JSON.stringify(delivery)).not.toContain('device-key');
|
||||
expect(service.listLogs().items[0]).toMatchObject({
|
||||
eventType: 'test',
|
||||
status: 'success',
|
||||
channelName: 'Bark',
|
||||
});
|
||||
});
|
||||
|
||||
it('records failed channel tests with a safe delivery detail', async () => {
|
||||
const { service } = fixture();
|
||||
const channel = await service.createChannel({
|
||||
...bark,
|
||||
config: { ...bark.config, server_url: 'https://unavailable.invalid' },
|
||||
});
|
||||
|
||||
await expect(
|
||||
service.testChannel(channel.id, { title: '测试通知', body: 'hello' }),
|
||||
).resolves.toMatchObject({ ok: false, status: 'failed', detail: 'channel unreachable' });
|
||||
expect(service.listLogs().items[0]).toMatchObject({
|
||||
status: 'failed',
|
||||
detail: 'channel unreachable',
|
||||
});
|
||||
});
|
||||
|
||||
describe('delivery log retention', () => {
|
||||
function insertLog(
|
||||
db: Database.Database,
|
||||
channelId: string,
|
||||
id: string,
|
||||
createdAt: string,
|
||||
status: 'success' | 'failed' = 'success',
|
||||
eventType = 'sms',
|
||||
): void {
|
||||
db.prepare(
|
||||
`INSERT INTO notification_deliveries
|
||||
(id,rule_id,channel_id,instance_id,event_type,status,detail,created_at,sent_at)
|
||||
VALUES (?,NULL,?,NULL,?,?,NULL,?,?)`,
|
||||
).run(id, channelId, eventType, status, createdAt, createdAt);
|
||||
}
|
||||
|
||||
it('keeps the Hub retention defaults until they are stored', () => {
|
||||
const { db, service } = fixture();
|
||||
expect(service.logCleanup()).toEqual({
|
||||
retentionDaysEnabled: true,
|
||||
retentionDays: 180,
|
||||
maxEntriesEnabled: false,
|
||||
maxEntries: 10_000,
|
||||
});
|
||||
|
||||
service.updateLogCleanup({ retentionDays: 30, maxEntriesEnabled: true, maxEntries: 500 });
|
||||
|
||||
expect(service.logCleanup()).toEqual({
|
||||
retentionDaysEnabled: true,
|
||||
retentionDays: 30,
|
||||
maxEntriesEnabled: true,
|
||||
maxEntries: 500,
|
||||
});
|
||||
const stored = db
|
||||
.prepare('SELECT value_json FROM app_settings WHERE key = ?')
|
||||
.get('notifications.logs.cleanup') as { value_json: string };
|
||||
expect(JSON.parse(stored.value_json)).toMatchObject({ retentionDays: 30, maxEntries: 500 });
|
||||
});
|
||||
|
||||
it('rejects out-of-range and unknown retention fields', () => {
|
||||
const { service } = fixture();
|
||||
expect(() => service.updateLogCleanup({ retentionDays: 0 })).toThrow();
|
||||
expect(() => service.updateLogCleanup({ maxEntries: 0 })).toThrow();
|
||||
expect(() => service.updateLogCleanup({ retentionDays: 30, extra: true })).toThrow();
|
||||
expect(() => service.updateLogCleanup('180')).toThrow();
|
||||
});
|
||||
|
||||
it('prunes by age and then by the entry ceiling', async () => {
|
||||
const { db, service } = fixture();
|
||||
const channel = await service.createChannel(bark);
|
||||
insertLog(db, channel.id, 'old', '2020-01-01T00:00:00.000Z');
|
||||
insertLog(db, channel.id, 'recent', '2026-09-03T07:00:00.000Z');
|
||||
insertLog(db, channel.id, 'newest', '2026-09-03T07:30:00.000Z');
|
||||
|
||||
service.updateLogCleanup({ retentionDays: 30, maxEntriesEnabled: false });
|
||||
expect(service.pruneLogs()).toBe(1);
|
||||
expect(service.listLogs().items.map((item) => item.id)).toEqual(['newest', 'recent']);
|
||||
|
||||
service.updateLogCleanup({
|
||||
retentionDaysEnabled: false,
|
||||
maxEntriesEnabled: true,
|
||||
maxEntries: 1,
|
||||
});
|
||||
expect(service.pruneLogs()).toBe(1);
|
||||
expect(service.listLogs().items.map((item) => item.id)).toEqual(['newest']);
|
||||
});
|
||||
|
||||
it('filters and clears logs by status, event type and date range', async () => {
|
||||
const { db, service } = fixture();
|
||||
const channel = await service.createChannel(bark);
|
||||
insertLog(db, channel.id, 'a', '2026-09-01T00:00:00.000Z', 'success', 'sms');
|
||||
insertLog(db, channel.id, 'b', '2026-09-02T00:00:00.000Z', 'failed', 'sms');
|
||||
insertLog(db, channel.id, 'c', '2026-09-03T00:00:00.000Z', 'failed', 'system');
|
||||
|
||||
expect(service.listLogs(1, 50, { status: 'failed' }).items.map((item) => item.id)).toEqual([
|
||||
'c',
|
||||
'b',
|
||||
]);
|
||||
expect(service.listLogs(1, 50, { eventType: 'system' }).page.total).toBe(1);
|
||||
expect(
|
||||
service
|
||||
.listLogs(1, 50, { from: '2026-09-02T00:00:00.000Z', to: '2026-09-02T12:00:00.000Z' })
|
||||
.items.map((item) => item.id),
|
||||
).toEqual(['b']);
|
||||
|
||||
expect(service.clearLogs({ status: 'failed', eventType: 'sms' })).toBe(1);
|
||||
expect(service.listLogs().items.map((item) => item.id)).toEqual(['c', 'a']);
|
||||
expect(service.clearLogs('2026-09-01T12:00:00.000Z')).toBe(1);
|
||||
expect(service.listLogs().page.total).toBe(1);
|
||||
expect(() => service.clearLogs({ before: 'not-a-date' })).toThrow();
|
||||
});
|
||||
});
|
||||
|
||||
describe('rate limit and quiet hours', () => {
|
||||
const sms = { fields: { content: '您的验证码是 1234' } };
|
||||
|
||||
async function rule(
|
||||
service: CentralNotificationService,
|
||||
channelId: string,
|
||||
suppression: Record<string, unknown>,
|
||||
) {
|
||||
return service.createRule({
|
||||
name: '抑制规则',
|
||||
eventType: 'sms',
|
||||
enabled: true,
|
||||
condition: { field: 'content', mode: 'all' },
|
||||
scope: { mode: 'all' },
|
||||
channelIds: [channelId],
|
||||
templates: { title: '标题', body: '{{content}}' },
|
||||
...suppression,
|
||||
});
|
||||
}
|
||||
|
||||
it('stores the Hub defaults when a rule omits both suppression blocks', async () => {
|
||||
const { service } = fixture();
|
||||
const channel = await service.createChannel(bark);
|
||||
const created = await rule(service, channel.id, {});
|
||||
|
||||
expect(created.rateLimit).toEqual({ enabled: false, maxMessages: 20, windowSeconds: 60 });
|
||||
expect(created.quietHours).toEqual([]);
|
||||
});
|
||||
|
||||
it('drops events inside a quiet window and records one suppressed log', async () => {
|
||||
// 08:30 Shanghai time sits inside 08:00-09:00.
|
||||
const { service } = fixture(() => new Date('2026-09-03T00:30:00.000Z'));
|
||||
const channel = await service.createChannel(bark);
|
||||
const created = await rule(service, channel.id, {
|
||||
quietHours: [{ start: '08:00', end: '09:00' }],
|
||||
});
|
||||
|
||||
expect(await service.enqueueEvent('sms', sms)).toEqual([]);
|
||||
const log = service.listLogs().items[0];
|
||||
expect(log).toMatchObject({
|
||||
status: 'quiet_hours',
|
||||
eventType: 'sms',
|
||||
ruleId: created.id,
|
||||
channelName: '—',
|
||||
detail: '免打扰时段:08:00-09:00',
|
||||
});
|
||||
expect(log?.channelId).toBeUndefined();
|
||||
expect(service.logSummary().suppressed).toBe(1);
|
||||
|
||||
expect(service.listLogs(1, 50, { status: 'quiet_hours' }).page.total).toBe(1);
|
||||
expect(service.listLogs(1, 50, { status: 'success' }).page.total).toBe(0);
|
||||
});
|
||||
|
||||
it('treats a window that ends before it starts as crossing midnight', async () => {
|
||||
let instant = new Date('2026-09-02T23:30:00.000Z');
|
||||
const { service } = fixture(() => instant);
|
||||
const channel = await service.createChannel(bark);
|
||||
await rule(service, channel.id, { quietHours: [{ start: '22:00', end: '08:00' }] });
|
||||
|
||||
// 07:30 Shanghai, inside the window.
|
||||
expect(await service.enqueueEvent('sms', sms)).toEqual([]);
|
||||
// 09:00 Shanghai, outside the window.
|
||||
instant = new Date('2026-09-03T01:00:00.000Z');
|
||||
expect(await service.enqueueEvent('sms', sms)).toHaveLength(1);
|
||||
});
|
||||
|
||||
it('enforces the rate limit inside the window and releases it afterwards', async () => {
|
||||
let instant = new Date('2026-09-03T00:00:00.000Z');
|
||||
const { service } = fixture(() => instant);
|
||||
const channel = await service.createChannel(bark);
|
||||
await rule(service, channel.id, {
|
||||
rateLimit: { enabled: true, maxMessages: 2, windowSeconds: 60 },
|
||||
});
|
||||
|
||||
expect(await service.enqueueEvent('sms', sms)).toHaveLength(1);
|
||||
expect(await service.enqueueEvent('sms', sms)).toHaveLength(1);
|
||||
expect(await service.enqueueEvent('sms', sms)).toEqual([]);
|
||||
expect(service.logSummary()).toMatchObject({ suppressed: 1, total: 1 });
|
||||
|
||||
instant = new Date('2026-09-03T00:01:01.000Z');
|
||||
expect(await service.enqueueEvent('sms', sms)).toHaveLength(1);
|
||||
});
|
||||
|
||||
it('prefers the quiet window over the rate limit', async () => {
|
||||
const { service } = fixture(() => new Date('2026-09-03T00:30:00.000Z'));
|
||||
const channel = await service.createChannel(bark);
|
||||
await rule(service, channel.id, {
|
||||
rateLimit: { enabled: true, maxMessages: 5, windowSeconds: 60 },
|
||||
quietHours: [{ start: '08:00', end: '09:00' }],
|
||||
});
|
||||
|
||||
expect(await service.enqueueEvent('sms', sms)).toEqual([]);
|
||||
expect(service.listLogs().items[0]?.status).toBe('quiet_hours');
|
||||
});
|
||||
|
||||
it('leaves suppression settings untouched when a patch omits them', async () => {
|
||||
const { service } = fixture();
|
||||
const channel = await service.createChannel(bark);
|
||||
const created = await rule(service, channel.id, {
|
||||
rateLimit: { enabled: true, maxMessages: 3, windowSeconds: 120 },
|
||||
quietHours: [{ start: '23:00', end: '06:00' }],
|
||||
});
|
||||
|
||||
const patched = await service.updateRule(created.id, { name: '改名后的规则' });
|
||||
expect(patched.name).toBe('改名后的规则');
|
||||
expect(patched.rateLimit).toEqual({ enabled: true, maxMessages: 3, windowSeconds: 120 });
|
||||
expect(patched.quietHours).toEqual([{ start: '23:00', end: '06:00' }]);
|
||||
|
||||
const disabled = await service.updateRule(created.id, {
|
||||
rateLimit: { enabled: false, maxMessages: 3, windowSeconds: 120 },
|
||||
quietHours: [],
|
||||
});
|
||||
expect(disabled.rateLimit.enabled).toBe(false);
|
||||
expect(disabled.quietHours).toEqual([]);
|
||||
});
|
||||
|
||||
it('rejects malformed suppression settings', async () => {
|
||||
const { service } = fixture();
|
||||
const channel = await service.createChannel(bark);
|
||||
const base = {
|
||||
name: '抑制规则',
|
||||
eventType: 'sms',
|
||||
enabled: true,
|
||||
condition: { field: 'content', mode: 'all' },
|
||||
scope: { mode: 'all' },
|
||||
channelIds: [channel.id],
|
||||
templates: { title: '标题', body: '正文' },
|
||||
};
|
||||
|
||||
await expect(
|
||||
service.createRule({
|
||||
...base,
|
||||
rateLimit: { enabled: true, maxMessages: 0, windowSeconds: 60 },
|
||||
}),
|
||||
).rejects.toThrow();
|
||||
await expect(
|
||||
service.createRule({
|
||||
...base,
|
||||
rateLimit: { enabled: true, maxMessages: 20, windowSeconds: 86_401 },
|
||||
}),
|
||||
).rejects.toThrow();
|
||||
await expect(
|
||||
service.createRule({
|
||||
...base,
|
||||
rateLimit: { enabled: true, maxMessages: 20, windowSeconds: 60, extra: 1 },
|
||||
}),
|
||||
).rejects.toThrow();
|
||||
await expect(
|
||||
service.createRule({ ...base, quietHours: [{ start: '08:00', end: '08:00' }] }),
|
||||
).rejects.toThrow();
|
||||
await expect(
|
||||
service.createRule({ ...base, quietHours: [{ start: '8:00', end: '09:00' }] }),
|
||||
).rejects.toThrow();
|
||||
await expect(
|
||||
service.createRule({ ...base, quietHours: [{ start: '08:00', end: '09:00', extra: 1 }] }),
|
||||
).rejects.toThrow();
|
||||
await expect(
|
||||
service.createRule({
|
||||
...base,
|
||||
quietHours: Array.from({ length: 9 }, (_, index) => ({
|
||||
start: '00:00',
|
||||
end: index === 8 ? '23:59' : String(index).padStart(2, '0') + ':30',
|
||||
})),
|
||||
}),
|
||||
).rejects.toThrow();
|
||||
});
|
||||
});
|
||||
});
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,305 @@
|
||||
import { createHmac } from 'node:crypto';
|
||||
import { describe, expect, it } from 'vitest';
|
||||
|
||||
import { deliverThroughChannel, type HttpRequest } from './channel-delivery.js';
|
||||
|
||||
const NOW = new Date('2026-09-03T08:00:00.000Z');
|
||||
|
||||
function recorder(response: { status: number; body: string }) {
|
||||
const calls: HttpRequest[] = [];
|
||||
return {
|
||||
calls,
|
||||
request: async (request: HttpRequest) => {
|
||||
calls.push(request);
|
||||
return response;
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
const base = {
|
||||
title: '设备离线',
|
||||
body: 'SIM 卡所在设备已断开',
|
||||
eventType: 'device',
|
||||
occurredAt: NOW.toISOString(),
|
||||
instanceName: 'lab-01',
|
||||
};
|
||||
|
||||
describe('deliverThroughChannel', () => {
|
||||
it('signs the generic webhook with the Hub timestamp scheme', async () => {
|
||||
const sink = recorder({ status: 204, body: '' });
|
||||
const result = await deliverThroughChannel(
|
||||
{
|
||||
...base,
|
||||
type: 'webhook',
|
||||
config: {
|
||||
url: 'https://hook.example.test/notify',
|
||||
http_method: 'post',
|
||||
secret: 'topsecret',
|
||||
headers: 'X-Tenant: acme\nbad header: skip',
|
||||
},
|
||||
},
|
||||
{ request: sink.request, now: () => NOW },
|
||||
);
|
||||
|
||||
expect(result).toEqual({ ok: true });
|
||||
const [request] = sink.calls;
|
||||
expect(request?.url).toBe('https://hook.example.test/notify');
|
||||
expect(request?.method).toBe('POST');
|
||||
const payload = JSON.parse(String(request?.body)) as Record<string, unknown>;
|
||||
expect(payload).toEqual({
|
||||
event: 'device',
|
||||
title: '设备离线',
|
||||
body: 'SIM 卡所在设备已断开',
|
||||
instance: 'lab-01',
|
||||
occurred_at: NOW.toISOString(),
|
||||
});
|
||||
const timestamp = String(request?.headers['x-hub-timestamp']);
|
||||
expect(timestamp).toBe(String(NOW.getTime()));
|
||||
expect(request?.headers['x-hub-signature']).toBe(
|
||||
`sha256=${createHmac('sha256', 'topsecret')
|
||||
.update(`${timestamp}.${String(request?.body)}`)
|
||||
.digest('base64')}`,
|
||||
);
|
||||
expect(request?.headers['x-tenant']).toBe('acme');
|
||||
expect(Object.keys(request?.headers ?? {})).not.toContain('bad header: skip');
|
||||
});
|
||||
|
||||
it('posts Bark options to the device-key path', async () => {
|
||||
const sink = recorder({ status: 200, body: '{"code":200}' });
|
||||
const result = await deliverThroughChannel(
|
||||
{
|
||||
...base,
|
||||
type: 'bark',
|
||||
config: {
|
||||
server_url: 'https://bark.example.test/',
|
||||
device_key: 'abc def',
|
||||
group: 'Island',
|
||||
sound: 'bell',
|
||||
level: 'timeSensitive',
|
||||
icon: 'https://bark.example.test/icon.png',
|
||||
auto_copy: true,
|
||||
save_history: false,
|
||||
},
|
||||
},
|
||||
{ request: sink.request, now: () => NOW },
|
||||
);
|
||||
|
||||
expect(result).toEqual({ ok: true });
|
||||
const [request] = sink.calls;
|
||||
expect(request?.url).toBe('https://bark.example.test/abc%20def');
|
||||
expect(JSON.parse(String(request?.body))).toEqual({
|
||||
title: '设备离线',
|
||||
body: 'SIM 卡所在设备已断开',
|
||||
group: 'Island',
|
||||
autocopy: 1,
|
||||
save: 0,
|
||||
sound: 'bell',
|
||||
level: 'timeSensitive',
|
||||
icon: 'https://bark.example.test/icon.png',
|
||||
});
|
||||
});
|
||||
|
||||
it('treats a Bark error code in a 200 response as a failure', async () => {
|
||||
const sink = recorder({ status: 200, body: '{"code":400,"message":"bad device key"}' });
|
||||
const result = await deliverThroughChannel(
|
||||
{
|
||||
...base,
|
||||
type: 'bark',
|
||||
config: { server_url: 'https://bark.example.test', device_key: 'k' },
|
||||
},
|
||||
{ request: sink.request, now: () => NOW },
|
||||
);
|
||||
expect(result).toEqual({ ok: false, detail: 'code=400 bad device key' });
|
||||
});
|
||||
|
||||
it('signs the DingTalk robot webhook', async () => {
|
||||
const sink = recorder({ status: 200, body: '{"errcode":0}' });
|
||||
await deliverThroughChannel(
|
||||
{
|
||||
...base,
|
||||
type: 'dingtalk_robot',
|
||||
config: {
|
||||
webhook_url: 'https://oapi.dingtalk.com/robot/send',
|
||||
access_token: 'token-1',
|
||||
secret: 'SEC-abc',
|
||||
at_mobiles: '13800000000, 13900000000',
|
||||
at_all: true,
|
||||
},
|
||||
},
|
||||
{ request: sink.request, now: () => NOW },
|
||||
);
|
||||
|
||||
const [request] = sink.calls;
|
||||
const url = new URL(String(request?.url));
|
||||
expect(url.searchParams.get('access_token')).toBe('token-1');
|
||||
const timestamp = String(url.searchParams.get('timestamp'));
|
||||
expect(url.searchParams.get('sign')).toBe(
|
||||
createHmac('sha256', 'SEC-abc').update(`${timestamp}\nSEC-abc`).digest('base64'),
|
||||
);
|
||||
const body = JSON.parse(String(request?.body)) as Record<string, unknown>;
|
||||
expect(body['msgtype']).toBe('text');
|
||||
expect(body['at']).toEqual({ atMobiles: ['13800000000', '13900000000'], isAtAll: true });
|
||||
});
|
||||
|
||||
it('signs the Feishu robot webhook with the key-derived signature', async () => {
|
||||
const sink = recorder({ status: 200, body: '{"code":0}' });
|
||||
await deliverThroughChannel(
|
||||
{
|
||||
...base,
|
||||
type: 'feishu_robot',
|
||||
config: { webhook_url: 'https://open.feishu.cn/open-apis/bot/v2/hook/x', secret: 'fs' },
|
||||
},
|
||||
{ request: sink.request, now: () => NOW },
|
||||
);
|
||||
const body = JSON.parse(String(sink.calls[0]?.body)) as Record<string, unknown>;
|
||||
const timestamp = String(Math.floor(NOW.getTime() / 1000));
|
||||
expect(body['timestamp']).toBe(timestamp);
|
||||
expect(body['sign']).toBe(createHmac('sha256', `${timestamp}\nfs`).update('').digest('base64'));
|
||||
});
|
||||
|
||||
it('exchanges credentials before sending a WeCom application message', async () => {
|
||||
const calls: HttpRequest[] = [];
|
||||
const result = await deliverThroughChannel(
|
||||
{
|
||||
...base,
|
||||
type: 'wecom_app',
|
||||
config: {
|
||||
corp_id: 'wx-corp',
|
||||
agent_id: 1000002,
|
||||
secret: 'app-secret',
|
||||
to_user: 'alice|bob',
|
||||
to_party: '2',
|
||||
safe: true,
|
||||
},
|
||||
},
|
||||
{
|
||||
now: () => NOW,
|
||||
request: async (request) => {
|
||||
calls.push(request);
|
||||
if (request.url.includes('gettoken'))
|
||||
return { status: 200, body: '{"errcode":0,"access_token":"token-9"}' };
|
||||
return { status: 200, body: '{"errcode":0,"errmsg":"ok"}' };
|
||||
},
|
||||
},
|
||||
);
|
||||
|
||||
expect(result).toEqual({ ok: true });
|
||||
expect(calls).toHaveLength(2);
|
||||
expect(new URL(String(calls[0]?.url)).searchParams.get('corpsecret')).toBe('app-secret');
|
||||
const send = calls[1];
|
||||
expect(send?.url).toBe('https://qyapi.weixin.qq.com/cgi-bin/message/send?access_token=token-9');
|
||||
expect(JSON.parse(String(send?.body))).toEqual({
|
||||
touser: 'alice|bob',
|
||||
msgtype: 'text',
|
||||
agentid: 1000002,
|
||||
text: { content: '设备离线\nSIM 卡所在设备已断开\n来源:lab-01\n' + NOW.toISOString() },
|
||||
safe: 1,
|
||||
toparty: '2',
|
||||
});
|
||||
});
|
||||
|
||||
it('reports a WeCom token failure without attempting the send', async () => {
|
||||
const calls: HttpRequest[] = [];
|
||||
const result = await deliverThroughChannel(
|
||||
{
|
||||
...base,
|
||||
type: 'wecom_app',
|
||||
config: { corp_id: 'c', agent_id: 1, secret: 's' },
|
||||
},
|
||||
{
|
||||
now: () => NOW,
|
||||
request: async (request) => {
|
||||
calls.push(request);
|
||||
return { status: 200, body: '{"errcode":40013,"errmsg":"invalid corp"}' };
|
||||
},
|
||||
},
|
||||
);
|
||||
expect(calls).toHaveLength(1);
|
||||
expect(result).toEqual({ ok: false, detail: 'errcode=40013 invalid corp' });
|
||||
});
|
||||
|
||||
it('uses the DingTalk v2 token header for application messages', async () => {
|
||||
const calls: HttpRequest[] = [];
|
||||
const result = await deliverThroughChannel(
|
||||
{
|
||||
...base,
|
||||
type: 'dingtalk_app',
|
||||
config: {
|
||||
app_key: 'key',
|
||||
app_secret: 'shh',
|
||||
robot_code: 'robot',
|
||||
open_conversation_id: 'cid',
|
||||
msg_key: 'sampleMarkdown',
|
||||
},
|
||||
},
|
||||
{
|
||||
now: () => NOW,
|
||||
request: async (request) => {
|
||||
calls.push(request);
|
||||
if (request.url.endsWith('accessToken'))
|
||||
return { status: 200, body: '{"accessToken":"at-1"}' };
|
||||
return { status: 200, body: '{"processQueryKey":"pk"}' };
|
||||
},
|
||||
},
|
||||
);
|
||||
expect(result).toEqual({ ok: true });
|
||||
const send = calls[1];
|
||||
expect(send?.headers['x-acs-dingtalk-access-token']).toBe('at-1');
|
||||
const body = JSON.parse(String(send?.body)) as Record<string, unknown>;
|
||||
expect(body['msgKey']).toBe('sampleMarkdown');
|
||||
expect(JSON.parse(String(body['msgParam']))).toEqual({
|
||||
title: '设备离线',
|
||||
text: '**设备离线**\n\nSIM 卡所在设备已断开\n\n来源:lab-01\n\n' + NOW.toISOString(),
|
||||
});
|
||||
});
|
||||
|
||||
it('routes Server 酱 v3 keys to the ft07 endpoint as a form body', async () => {
|
||||
const sink = recorder({ status: 200, body: '{"code":0}' });
|
||||
await deliverThroughChannel(
|
||||
{ ...base, type: 'serverchan', config: { send_key: 'sct12345@9876', uid: '42' } },
|
||||
{ request: sink.request, now: () => NOW },
|
||||
);
|
||||
const [request] = sink.calls;
|
||||
expect(request?.url).toBe('https://push.ft07.com/send/sct12345%409876.send');
|
||||
expect(request?.headers['content-type']).toBe('application/x-www-form-urlencoded');
|
||||
expect(request?.body).toContain('title=' + encodeURIComponent('设备离线'));
|
||||
expect(request?.body).toContain('uid=42');
|
||||
});
|
||||
|
||||
it('falls back to the legacy Server 酱 host for plain keys', async () => {
|
||||
const sink = recorder({ status: 200, body: '' });
|
||||
await deliverThroughChannel(
|
||||
{ ...base, type: 'serverchan', config: { send_key: 'SCT999' } },
|
||||
{ request: sink.request, now: () => NOW },
|
||||
);
|
||||
expect(sink.calls[0]?.url).toBe('https://sctapi.ftqq.com/SCT999.send');
|
||||
});
|
||||
|
||||
it('rejects non-HTTP endpoints and missing credentials with a readable detail', async () => {
|
||||
await expect(
|
||||
deliverThroughChannel(
|
||||
{ ...base, type: 'webhook', config: { url: 'file:///etc/passwd' } },
|
||||
{ now: () => NOW },
|
||||
),
|
||||
).resolves.toEqual({ ok: false, detail: '回调地址 只支持 http 或 https' });
|
||||
await expect(
|
||||
deliverThroughChannel({ ...base, type: 'telegram', config: { chat_id: '1' } }, {}),
|
||||
).resolves.toEqual({ ok: false, detail: '缺少 Telegram Bot Token' });
|
||||
});
|
||||
|
||||
it('delegates email delivery to the injected SMTP sender', async () => {
|
||||
const seen: string[] = [];
|
||||
const result = await deliverThroughChannel(
|
||||
{ ...base, type: 'email', config: { smtp_host: 'smtp.example.test' } },
|
||||
{
|
||||
now: () => NOW,
|
||||
sendEmail: async (input) => {
|
||||
seen.push(input.type);
|
||||
return { ok: true };
|
||||
},
|
||||
},
|
||||
);
|
||||
expect(result).toEqual({ ok: true });
|
||||
expect(seen).toEqual(['email']);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,558 @@
|
||||
import { createHmac } from 'node:crypto';
|
||||
|
||||
import {
|
||||
notificationChannelSpec,
|
||||
parseKeyValueField,
|
||||
type NotificationChannelConfigValue,
|
||||
type NotificationChannelType,
|
||||
} from '@multi-simadmin/contracts';
|
||||
|
||||
export type ChannelConfig = Readonly<Record<string, NotificationChannelConfigValue>>;
|
||||
|
||||
export interface ChannelDeliveryInput {
|
||||
readonly type: NotificationChannelType;
|
||||
readonly config: ChannelConfig;
|
||||
readonly title: string;
|
||||
readonly body: string;
|
||||
readonly eventType: string;
|
||||
readonly occurredAt: string;
|
||||
readonly instanceName?: string;
|
||||
}
|
||||
|
||||
export interface ChannelDeliveryResult {
|
||||
readonly ok: boolean;
|
||||
readonly detail?: string;
|
||||
}
|
||||
|
||||
export interface HttpRequest {
|
||||
readonly url: string;
|
||||
readonly method: 'GET' | 'POST' | 'PUT' | 'PATCH';
|
||||
readonly headers: Readonly<Record<string, string>>;
|
||||
readonly body?: string;
|
||||
}
|
||||
|
||||
export interface HttpResponse {
|
||||
readonly status: number;
|
||||
readonly body: string;
|
||||
}
|
||||
|
||||
export type HttpRequester = (request: HttpRequest, timeoutMs: number) => Promise<HttpResponse>;
|
||||
|
||||
export interface ChannelDeliveryOptions {
|
||||
readonly request?: HttpRequester | undefined;
|
||||
readonly sendEmail?: (
|
||||
input: ChannelDeliveryInput,
|
||||
timeoutMs: number,
|
||||
) => Promise<ChannelDeliveryResult>;
|
||||
readonly timeoutMs?: number | undefined;
|
||||
readonly now?: (() => Date) | undefined;
|
||||
}
|
||||
|
||||
const DEFAULT_TIMEOUT_MS = 10_000;
|
||||
const MAX_RESPONSE_BYTES = 64 * 1024;
|
||||
|
||||
export class ChannelDeliveryError extends Error {
|
||||
constructor(message: string) {
|
||||
super(message);
|
||||
this.name = 'ChannelDeliveryError';
|
||||
}
|
||||
}
|
||||
|
||||
function stringConfig(config: ChannelConfig, key: string): string {
|
||||
const value = config[key];
|
||||
if (typeof value === 'string') return value.trim();
|
||||
if (typeof value === 'number' && Number.isFinite(value)) return String(value);
|
||||
if (typeof value === 'boolean') return value ? 'true' : 'false';
|
||||
return '';
|
||||
}
|
||||
|
||||
function booleanConfig(config: ChannelConfig, key: string): boolean {
|
||||
const value = config[key];
|
||||
return value === true || value === 'true' || value === 1;
|
||||
}
|
||||
|
||||
function numberConfig(config: ChannelConfig, key: string, fallback: number): number {
|
||||
const value = config[key];
|
||||
if (typeof value === 'number' && Number.isFinite(value)) return value;
|
||||
const parsed = Number(typeof value === 'string' ? value.trim() : '');
|
||||
return Number.isFinite(parsed) && parsed > 0 ? parsed : fallback;
|
||||
}
|
||||
|
||||
function trimSlash(value: string): string {
|
||||
return value.replace(/\/+$/u, '');
|
||||
}
|
||||
|
||||
function httpsUrl(raw: string, label: string): string {
|
||||
let parsed: URL;
|
||||
try {
|
||||
parsed = new URL(raw);
|
||||
} catch {
|
||||
throw new ChannelDeliveryError(`${label} 不是合法地址`);
|
||||
}
|
||||
if (parsed.protocol !== 'https:' && parsed.protocol !== 'http:')
|
||||
throw new ChannelDeliveryError(`${label} 只支持 http 或 https`);
|
||||
return parsed.toString();
|
||||
}
|
||||
|
||||
function withQuery(raw: string, parameters: Readonly<Record<string, string>>): string {
|
||||
const url = new URL(raw);
|
||||
for (const [key, value] of Object.entries(parameters)) {
|
||||
if (value !== '' && !url.searchParams.has(key)) url.searchParams.set(key, value);
|
||||
}
|
||||
return url.toString();
|
||||
}
|
||||
|
||||
function sign(algorithm: string, key: string, data: string): string {
|
||||
return createHmac(algorithm, key).update(data).digest('base64');
|
||||
}
|
||||
|
||||
function json(value: unknown): string {
|
||||
return JSON.stringify(value);
|
||||
}
|
||||
|
||||
function parseBody(raw: string): Record<string, unknown> | undefined {
|
||||
if (raw.length === 0) return undefined;
|
||||
try {
|
||||
const value: unknown = JSON.parse(raw);
|
||||
return typeof value === 'object' && value !== null && !Array.isArray(value)
|
||||
? (value as Record<string, unknown>)
|
||||
: undefined;
|
||||
} catch {
|
||||
return undefined;
|
||||
}
|
||||
}
|
||||
|
||||
/** Turns a service reply into a verdict; most vendors answer 200 even on failure. */
|
||||
function verdict(
|
||||
response: HttpResponse,
|
||||
failurePaths: readonly string[] = ['errcode', 'code'],
|
||||
successValues: Readonly<Record<string, number>> = {},
|
||||
): ChannelDeliveryResult {
|
||||
if (response.status < 200 || response.status >= 300) {
|
||||
return { ok: false, detail: `HTTP ${response.status} ${response.body.slice(0, 200)}`.trim() };
|
||||
}
|
||||
const payload = parseBody(response.body);
|
||||
if (!payload) return { ok: true };
|
||||
for (const path of failurePaths) {
|
||||
const value = payload[path];
|
||||
// Most vendors use errcode 0; Bark reports success as code 200.
|
||||
if (typeof value === 'number' && value !== (successValues[path] ?? 0)) {
|
||||
const message = payload['msg'] ?? payload['errmsg'] ?? payload['message'];
|
||||
return {
|
||||
ok: false,
|
||||
detail: `${path}=${value}${typeof message === 'string' ? ` ${message.slice(0, 180)}` : ''}`,
|
||||
};
|
||||
}
|
||||
if (typeof value === 'boolean' && value === false) {
|
||||
const message = payload['error_description'] ?? payload['description'];
|
||||
return {
|
||||
ok: false,
|
||||
detail: `${path}=false${typeof message === 'string' ? ` ${message.slice(0, 180)}` : ''}`,
|
||||
};
|
||||
}
|
||||
}
|
||||
return { ok: true };
|
||||
}
|
||||
|
||||
async function defaultRequester(request: HttpRequest, timeoutMs: number): Promise<HttpResponse> {
|
||||
const response = await fetch(request.url, {
|
||||
method: request.method,
|
||||
headers: request.headers,
|
||||
...(request.body === undefined ? {} : { body: request.body }),
|
||||
signal: AbortSignal.timeout(timeoutMs),
|
||||
// A signed payload must never be replayed to a redirect target.
|
||||
redirect: 'manual',
|
||||
});
|
||||
const text = (await response.text()).slice(0, MAX_RESPONSE_BYTES);
|
||||
return { status: response.status, body: text };
|
||||
}
|
||||
|
||||
function markdownBody(input: ChannelDeliveryInput): string {
|
||||
const source = input.instanceName ? `\n\n来源:${input.instanceName}` : '';
|
||||
return `**${input.title}**\n\n${input.body}${source}\n\n${input.occurredAt}`;
|
||||
}
|
||||
|
||||
function plainBody(input: ChannelDeliveryInput): string {
|
||||
const source = input.instanceName ? `\n来源:${input.instanceName}` : '';
|
||||
return `${input.title}\n${input.body}${source}\n${input.occurredAt}`;
|
||||
}
|
||||
|
||||
function deliverWebhook(input: ChannelDeliveryInput, now: () => Date): HttpRequest | undefined {
|
||||
const raw = stringConfig(input.config, 'url');
|
||||
if (!raw) throw new ChannelDeliveryError('缺少回调地址');
|
||||
const url = httpsUrl(raw, '回调地址');
|
||||
const method = (stringConfig(input.config, 'http_method') || 'POST').toUpperCase();
|
||||
const payload = json({
|
||||
event: input.eventType,
|
||||
title: input.title,
|
||||
body: input.body,
|
||||
instance: input.instanceName ?? null,
|
||||
occurred_at: input.occurredAt,
|
||||
});
|
||||
const headers: Record<string, string> = { 'content-type': 'application/json' };
|
||||
const secret = stringConfig(input.config, 'secret');
|
||||
if (secret) {
|
||||
const timestamp = String(now().getTime());
|
||||
headers['x-hub-timestamp'] = timestamp;
|
||||
headers['x-hub-signature'] = `sha256=${sign('sha256', secret, `${timestamp}.${payload}`)}`;
|
||||
}
|
||||
for (const [key, value] of Object.entries(parseKeyValueField(input.config['headers']))) {
|
||||
if (/^[A-Za-z0-9!#$%&'*+.^_`|~-]+$/u.test(key)) headers[key.toLowerCase()] = value;
|
||||
}
|
||||
return {
|
||||
url,
|
||||
method: method === 'PUT' || method === 'PATCH' ? (method as 'PUT' | 'PATCH') : 'POST',
|
||||
headers,
|
||||
body: payload,
|
||||
};
|
||||
}
|
||||
|
||||
function deliverBark(input: ChannelDeliveryInput): HttpRequest | undefined {
|
||||
const server = trimSlash(stringConfig(input.config, 'server_url') || 'https://api.day.app');
|
||||
const key = stringConfig(input.config, 'device_key');
|
||||
if (!key) throw new ChannelDeliveryError('缺少 Bark 设备 Key');
|
||||
const body: Record<string, string | number | boolean> = {
|
||||
title: input.title,
|
||||
body: input.body,
|
||||
group: stringConfig(input.config, 'group') || 'SimAdminHub',
|
||||
autocopy: booleanConfig(input.config, 'auto_copy') ? 1 : 0,
|
||||
save: booleanConfig(input.config, 'save_history') ? 1 : 0,
|
||||
};
|
||||
const sound = stringConfig(input.config, 'sound');
|
||||
const level = stringConfig(input.config, 'level');
|
||||
const icon = stringConfig(input.config, 'icon');
|
||||
if (sound) body['sound'] = sound;
|
||||
if (level) body['level'] = level;
|
||||
if (icon) body['icon'] = icon;
|
||||
return {
|
||||
url: httpsUrl(`${server}/${encodeURIComponent(key)}`, 'Bark 服务器地址'),
|
||||
method: 'POST',
|
||||
headers: { 'content-type': 'application/json; charset=utf-8' },
|
||||
body: json(body),
|
||||
};
|
||||
}
|
||||
|
||||
function deliverPushPlus(input: ChannelDeliveryInput): HttpRequest | undefined {
|
||||
const token = stringConfig(input.config, 'token');
|
||||
if (!token) throw new ChannelDeliveryError('缺少 PushPlus Token');
|
||||
const body: Record<string, string> = {
|
||||
token,
|
||||
title: input.title,
|
||||
content: input.body,
|
||||
template: stringConfig(input.config, 'template') || 'txt',
|
||||
};
|
||||
const topic = stringConfig(input.config, 'topic');
|
||||
const channel = stringConfig(input.config, 'channel');
|
||||
const callback = stringConfig(input.config, 'callback_url');
|
||||
if (topic) body['topic'] = topic;
|
||||
if (channel) body['channel'] = channel;
|
||||
if (callback) body['callbackUrl'] = callback;
|
||||
return {
|
||||
url: 'https://www.pushplus.plus/send',
|
||||
method: 'POST',
|
||||
headers: { 'content-type': 'application/json' },
|
||||
body: json(body),
|
||||
};
|
||||
}
|
||||
|
||||
function deliverWeComRobot(input: ChannelDeliveryInput): HttpRequest | undefined {
|
||||
const raw = stringConfig(input.config, 'webhook_url');
|
||||
if (!raw) throw new ChannelDeliveryError('缺少企业微信机器人 Webhook 地址');
|
||||
const key = stringConfig(input.config, 'key');
|
||||
const url = withQuery(httpsUrl(raw, '企业微信 Webhook 地址'), key ? { key } : {});
|
||||
return {
|
||||
url,
|
||||
method: 'POST',
|
||||
headers: { 'content-type': 'application/json' },
|
||||
body: json({ msgtype: 'markdown', markdown: { content: markdownBody(input) } }),
|
||||
};
|
||||
}
|
||||
|
||||
function deliverDingTalkRobot(
|
||||
input: ChannelDeliveryInput,
|
||||
now: () => Date,
|
||||
): HttpRequest | undefined {
|
||||
const raw = stringConfig(input.config, 'webhook_url');
|
||||
if (!raw) throw new ChannelDeliveryError('缺少钉钉机器人 Webhook 地址');
|
||||
const parameters: Record<string, string> = {};
|
||||
const accessToken = stringConfig(input.config, 'access_token');
|
||||
if (accessToken) parameters['access_token'] = accessToken;
|
||||
const secret = stringConfig(input.config, 'secret');
|
||||
if (secret) {
|
||||
const timestamp = String(now().getTime());
|
||||
parameters['timestamp'] = timestamp;
|
||||
// withQuery encodes values already; pre-encoding here would double-escape the signature.
|
||||
parameters['sign'] = sign('sha256', secret, `${timestamp}\n${secret}`);
|
||||
}
|
||||
const mobiles = stringConfig(input.config, 'at_mobiles')
|
||||
.split(/[,,\s]+/u)
|
||||
.map((item) => item.trim())
|
||||
.filter((item) => item !== '')
|
||||
.slice(0, 50);
|
||||
const body: Record<string, unknown> = {
|
||||
msgtype: 'text',
|
||||
text: { content: plainBody(input) },
|
||||
at: { atMobiles: mobiles, isAtAll: booleanConfig(input.config, 'at_all') },
|
||||
};
|
||||
return {
|
||||
url: withQuery(httpsUrl(raw, '钉钉 Webhook 地址'), parameters),
|
||||
method: 'POST',
|
||||
headers: { 'content-type': 'application/json' },
|
||||
body: json(body),
|
||||
};
|
||||
}
|
||||
|
||||
function deliverFeishuRobot(input: ChannelDeliveryInput, now: () => Date): HttpRequest | undefined {
|
||||
const raw = stringConfig(input.config, 'webhook_url');
|
||||
if (!raw) throw new ChannelDeliveryError('缺少飞书机器人 Webhook 地址');
|
||||
const body: Record<string, unknown> = {
|
||||
msg_type: 'text',
|
||||
content: { text: plainBody(input) },
|
||||
};
|
||||
const secret = stringConfig(input.config, 'secret');
|
||||
if (secret) {
|
||||
const timestamp = String(Math.floor(now().getTime() / 1000));
|
||||
body['timestamp'] = timestamp;
|
||||
body['sign'] = sign('sha256', `${timestamp}\n${secret}`, '');
|
||||
}
|
||||
const token = stringConfig(input.config, 'token');
|
||||
const url = withQuery(httpsUrl(raw, '飞书 Webhook 地址'), token ? { token } : {});
|
||||
return { url, method: 'POST', headers: { 'content-type': 'application/json' }, body: json(body) };
|
||||
}
|
||||
|
||||
function deliverTelegram(input: ChannelDeliveryInput): HttpRequest | undefined {
|
||||
const base = trimSlash(stringConfig(input.config, 'api_base_url') || 'https://api.telegram.org');
|
||||
const token = stringConfig(input.config, 'bot_token');
|
||||
const chatId = stringConfig(input.config, 'chat_id');
|
||||
if (!token) throw new ChannelDeliveryError('缺少 Telegram Bot Token');
|
||||
if (!chatId) throw new ChannelDeliveryError('缺少 Telegram Chat ID');
|
||||
const body: Record<string, unknown> = {
|
||||
chat_id: chatId,
|
||||
text: `${input.title}\n\n${input.body}`,
|
||||
disable_web_page_preview: booleanConfig(input.config, 'disable_web_page_preview'),
|
||||
};
|
||||
const parseMode = stringConfig(input.config, 'parse_mode');
|
||||
if (parseMode) body['parse_mode'] = parseMode;
|
||||
return {
|
||||
url: httpsUrl(`${base}/bot${encodeURIComponent(token)}/sendMessage`, 'Telegram 接口地址'),
|
||||
method: 'POST',
|
||||
headers: { 'content-type': 'application/json' },
|
||||
body: json(body),
|
||||
};
|
||||
}
|
||||
|
||||
function deliverServerChan(input: ChannelDeliveryInput): HttpRequest | undefined {
|
||||
const sendKey = stringConfig(input.config, 'send_key');
|
||||
if (!sendKey) throw new ChannelDeliveryError('缺少 Server 酱 SendKey');
|
||||
const endpoint = sendKey.includes('@')
|
||||
? `https://push.ft07.com/send/${encodeURIComponent(sendKey)}.send`
|
||||
: `https://sctapi.ftqq.com/${encodeURIComponent(sendKey)}.send`;
|
||||
const fields: Record<string, string> = { title: input.title, desp: input.body };
|
||||
for (const key of ['uid', 'channel', 'openid'] as const) {
|
||||
const value = stringConfig(input.config, key);
|
||||
if (value) fields[key] = value;
|
||||
}
|
||||
const form = new URLSearchParams(fields).toString();
|
||||
return {
|
||||
url: httpsUrl(endpoint, 'Server 酱 SendKey'),
|
||||
method: 'POST',
|
||||
headers: { 'content-type': 'application/x-www-form-urlencoded' },
|
||||
body: form,
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Builds the outbound request for single-shot channels. Channels that need a
|
||||
* token exchange are handled separately because the second call depends on the
|
||||
* first response.
|
||||
*/
|
||||
function buildRequests(
|
||||
input: ChannelDeliveryInput,
|
||||
now: () => Date,
|
||||
): readonly {
|
||||
readonly request: HttpRequest;
|
||||
readonly failurePaths: readonly string[];
|
||||
readonly successValues?: Readonly<Record<string, number>>;
|
||||
}[] {
|
||||
switch (input.type) {
|
||||
case 'webhook':
|
||||
return [{ request: single(deliverWebhook(input, now)), failurePaths: [] }];
|
||||
case 'bark':
|
||||
return [
|
||||
{
|
||||
request: single(deliverBark(input)),
|
||||
failurePaths: ['code'],
|
||||
successValues: { code: 200 },
|
||||
},
|
||||
];
|
||||
case 'pushplus':
|
||||
return [{ request: single(deliverPushPlus(input)), failurePaths: ['code'] }];
|
||||
case 'wecom_robot':
|
||||
return [{ request: single(deliverWeComRobot(input)), failurePaths: ['errcode'] }];
|
||||
case 'dingtalk_robot':
|
||||
return [{ request: single(deliverDingTalkRobot(input, now)), failurePaths: ['errcode'] }];
|
||||
case 'feishu_robot':
|
||||
return [
|
||||
{
|
||||
request: single(deliverFeishuRobot(input, now)),
|
||||
failurePaths: ['code', 'StatusCode'],
|
||||
},
|
||||
];
|
||||
case 'telegram':
|
||||
return [{ request: single(deliverTelegram(input)), failurePaths: ['ok'] }];
|
||||
case 'serverchan':
|
||||
return [{ request: single(deliverServerChan(input)), failurePaths: ['code'] }];
|
||||
default:
|
||||
return [];
|
||||
}
|
||||
}
|
||||
|
||||
/** A channel adapter either produced a request or explained why it could not. */
|
||||
function single(request: HttpRequest | undefined): HttpRequest {
|
||||
if (!request) throw new ChannelDeliveryError('通道配置不完整');
|
||||
return request;
|
||||
}
|
||||
|
||||
async function deliverWeComApp(
|
||||
input: ChannelDeliveryInput,
|
||||
request: HttpRequester,
|
||||
timeoutMs: number,
|
||||
): Promise<ChannelDeliveryResult> {
|
||||
const base = trimSlash(
|
||||
stringConfig(input.config, 'api_base_url') || 'https://qyapi.weixin.qq.com',
|
||||
);
|
||||
const corpId = stringConfig(input.config, 'corp_id');
|
||||
const secret = stringConfig(input.config, 'secret');
|
||||
const agentId = stringConfig(input.config, 'agent_id');
|
||||
if (!corpId || !secret || !agentId)
|
||||
throw new ChannelDeliveryError('企业微信应用需要企业 ID、应用 Secret 与 AgentId');
|
||||
const tokenUrl = httpsUrl(
|
||||
withQuery(`${base}/cgi-bin/gettoken`, { corpid: corpId, corpsecret: secret }),
|
||||
'企业微信接口地址',
|
||||
);
|
||||
const tokenResponse = await request(
|
||||
{ url: tokenUrl, method: 'GET', headers: { accept: 'application/json' } },
|
||||
timeoutMs,
|
||||
);
|
||||
const tokenPayload = parseBody(tokenResponse.body);
|
||||
const accessToken =
|
||||
tokenResponse.status >= 200 &&
|
||||
tokenResponse.status < 300 &&
|
||||
typeof tokenPayload?.['access_token'] === 'string'
|
||||
? tokenPayload['access_token']
|
||||
: undefined;
|
||||
if (!accessToken) return verdict(tokenResponse, ['errcode']);
|
||||
const sendUrl = httpsUrl(
|
||||
withQuery(`${base}/cgi-bin/message/send`, { access_token: accessToken }),
|
||||
'企业微信接口地址',
|
||||
);
|
||||
const body: Record<string, unknown> = {
|
||||
touser: stringConfig(input.config, 'to_user') || '@all',
|
||||
msgtype: 'text',
|
||||
agentid: numberConfig(input.config, 'agent_id', 0),
|
||||
text: { content: plainBody(input) },
|
||||
safe: booleanConfig(input.config, 'safe') ? 1 : 0,
|
||||
};
|
||||
const toParty = stringConfig(input.config, 'to_party');
|
||||
const toTag = stringConfig(input.config, 'to_tag');
|
||||
if (toParty) body['toparty'] = toParty;
|
||||
if (toTag) body['totag'] = toTag;
|
||||
const response = await request(
|
||||
{
|
||||
url: sendUrl,
|
||||
method: 'POST',
|
||||
headers: { 'content-type': 'application/json' },
|
||||
body: json(body),
|
||||
},
|
||||
timeoutMs,
|
||||
);
|
||||
return verdict(response, ['errcode']);
|
||||
}
|
||||
|
||||
async function deliverDingTalkApp(
|
||||
input: ChannelDeliveryInput,
|
||||
request: HttpRequester,
|
||||
timeoutMs: number,
|
||||
): Promise<ChannelDeliveryResult> {
|
||||
const appKey = stringConfig(input.config, 'app_key');
|
||||
const appSecret = stringConfig(input.config, 'app_secret');
|
||||
const robotCode = stringConfig(input.config, 'robot_code');
|
||||
const conversationId = stringConfig(input.config, 'open_conversation_id');
|
||||
if (!appKey || !appSecret || !robotCode || !conversationId)
|
||||
throw new ChannelDeliveryError('钉钉应用缺少 App Key、App Secret、Robot Code 或会话 ID');
|
||||
const tokenResponse = await request(
|
||||
{
|
||||
url: 'https://api.dingtalk.com/v1.0/oauth2/accessToken',
|
||||
method: 'POST',
|
||||
headers: { 'content-type': 'application/json' },
|
||||
body: json({ appKey, appSecret }),
|
||||
},
|
||||
timeoutMs,
|
||||
);
|
||||
const tokenPayload = parseBody(tokenResponse.body);
|
||||
const accessToken =
|
||||
tokenResponse.status >= 200 &&
|
||||
tokenResponse.status < 300 &&
|
||||
typeof tokenPayload?.['accessToken'] === 'string'
|
||||
? tokenPayload['accessToken']
|
||||
: undefined;
|
||||
if (!accessToken) return verdict(tokenResponse, ['code', 'errcode']);
|
||||
const msgKey = stringConfig(input.config, 'msg_key') || 'sampleText';
|
||||
const msgParam =
|
||||
msgKey === 'sampleMarkdown'
|
||||
? { title: input.title, text: markdownBody(input) }
|
||||
: { content: plainBody(input) };
|
||||
const response = await request(
|
||||
{
|
||||
url: 'https://api.dingtalk.com/v1.0/robot/groupMessages/send',
|
||||
method: 'POST',
|
||||
headers: {
|
||||
'content-type': 'application/json',
|
||||
'x-acs-dingtalk-access-token': accessToken,
|
||||
},
|
||||
body: json({
|
||||
robotCode,
|
||||
openConversationId: conversationId,
|
||||
msgKey,
|
||||
msgParam: json(msgParam),
|
||||
}),
|
||||
},
|
||||
timeoutMs,
|
||||
);
|
||||
return verdict(response, ['code', 'errcode']);
|
||||
}
|
||||
|
||||
export function channelSupportsDelivery(type: NotificationChannelType | string): boolean {
|
||||
return notificationChannelSpec(type) !== undefined;
|
||||
}
|
||||
|
||||
/** Delivers one notification through the channel adapter that matches its type. */
|
||||
export async function deliverThroughChannel(
|
||||
input: ChannelDeliveryInput,
|
||||
options: ChannelDeliveryOptions = {},
|
||||
): Promise<ChannelDeliveryResult> {
|
||||
const request = options.request ?? defaultRequester;
|
||||
const now = options.now ?? (() => new Date());
|
||||
const timeoutMs = options.timeoutMs ?? DEFAULT_TIMEOUT_MS;
|
||||
try {
|
||||
if (input.type === 'email') {
|
||||
if (!options.sendEmail) return { ok: false, detail: '邮件通道未启用 SMTP 发送器' };
|
||||
return await options.sendEmail(input, timeoutMs);
|
||||
}
|
||||
if (input.type === 'wecom_app') return await deliverWeComApp(input, request, timeoutMs);
|
||||
if (input.type === 'dingtalk_app') return await deliverDingTalkApp(input, request, timeoutMs);
|
||||
const steps = buildRequests(input, now);
|
||||
if (steps.length === 0) return { ok: false, detail: `不支持的通道类型:${input.type}` };
|
||||
let result: ChannelDeliveryResult = { ok: true };
|
||||
for (const step of steps) {
|
||||
const response = await request(step.request, timeoutMs);
|
||||
result = verdict(response, step.failurePaths, step.successValues ?? {});
|
||||
if (!result.ok) return result;
|
||||
}
|
||||
return result;
|
||||
} catch (error) {
|
||||
if (error instanceof ChannelDeliveryError) return { ok: false, detail: error.message };
|
||||
const message = error instanceof Error ? error.message : '投递失败';
|
||||
return { ok: false, detail: message.slice(0, 300) };
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,713 @@
|
||||
import type { Instance } from '@multi-simadmin/contracts';
|
||||
import type { InstanceService } from '../instances/instance-service.js';
|
||||
import type {
|
||||
InstanceSessionStore,
|
||||
UpstreamResponse,
|
||||
UpstreamSessionClientOptions,
|
||||
} from '../connections/upstream-session-client.js';
|
||||
|
||||
export type FleetNotificationState = 'ready' | 'unavailable' | 'failed';
|
||||
|
||||
export interface FleetNotificationDevice {
|
||||
readonly id: string;
|
||||
readonly name: string;
|
||||
readonly state: FleetNotificationState;
|
||||
}
|
||||
|
||||
export interface FleetNotificationChannelTypeSummary {
|
||||
readonly type: string;
|
||||
readonly total: number;
|
||||
readonly enabled: number;
|
||||
}
|
||||
|
||||
export interface FleetNotificationConfigSummary {
|
||||
readonly channelCount: number;
|
||||
readonly channelEnabled: number;
|
||||
readonly ruleCount: number;
|
||||
readonly ruleEnabled: number;
|
||||
readonly channelTypes: readonly FleetNotificationChannelTypeSummary[];
|
||||
}
|
||||
|
||||
export interface FleetNotificationLogEntry {
|
||||
readonly id: string;
|
||||
readonly eventType: string;
|
||||
readonly status: string;
|
||||
readonly ruleName?: string;
|
||||
readonly channelName?: string;
|
||||
readonly createdAt: string;
|
||||
}
|
||||
|
||||
export interface FleetNotificationLogSummary {
|
||||
readonly total: number;
|
||||
readonly success: number;
|
||||
readonly failed: number;
|
||||
readonly quietHours: number;
|
||||
readonly unmatched: number;
|
||||
readonly noAvailableChannel: number;
|
||||
readonly other: number;
|
||||
readonly recent: readonly FleetNotificationLogEntry[];
|
||||
}
|
||||
|
||||
export interface FleetNotificationQueueEntry {
|
||||
readonly id: string;
|
||||
readonly instanceId: string;
|
||||
readonly instanceName: string;
|
||||
readonly status: string;
|
||||
readonly eventType: string;
|
||||
readonly ruleName?: string;
|
||||
readonly channelName?: string;
|
||||
readonly createdAt: string;
|
||||
}
|
||||
|
||||
export interface FleetNotificationQueueSummary {
|
||||
readonly total: number;
|
||||
readonly pending: number;
|
||||
readonly scheduled: number;
|
||||
readonly retrying: number;
|
||||
readonly sending: number;
|
||||
readonly failed: number;
|
||||
readonly recent: readonly FleetNotificationQueueEntry[];
|
||||
}
|
||||
|
||||
export interface InstanceNotificationSummary {
|
||||
readonly observedAt: string;
|
||||
readonly deviceCount: number;
|
||||
readonly readyCount: number;
|
||||
readonly unavailableCount: number;
|
||||
readonly devices: readonly FleetNotificationDevice[];
|
||||
readonly config: FleetNotificationConfigSummary;
|
||||
readonly logs: FleetNotificationLogSummary;
|
||||
readonly queue: FleetNotificationQueueSummary;
|
||||
}
|
||||
|
||||
export interface FleetNotificationActionFailure {
|
||||
readonly instanceId: string;
|
||||
readonly instanceName: string;
|
||||
readonly code: NotificationServiceErrorCode;
|
||||
}
|
||||
|
||||
export interface FleetNotificationQueueRetryResult {
|
||||
readonly requested: number;
|
||||
readonly succeeded: number;
|
||||
readonly failed: number;
|
||||
readonly skipped: number;
|
||||
readonly failures: readonly FleetNotificationActionFailure[];
|
||||
}
|
||||
|
||||
export type NotificationServiceErrorCode =
|
||||
| 'NOT_FOUND'
|
||||
| 'VALIDATION_FAILED'
|
||||
| 'UPSTREAM_FAILED'
|
||||
| 'SESSION_INVALID';
|
||||
|
||||
export class NotificationServiceError extends Error {
|
||||
constructor(readonly code: NotificationServiceErrorCode) {
|
||||
super(code);
|
||||
this.name = 'NotificationServiceError';
|
||||
}
|
||||
}
|
||||
|
||||
const MAX_DEVICES = 200;
|
||||
const MAX_RESPONSE_BYTES = 262_144;
|
||||
const MAX_LOGS = 200;
|
||||
const MAX_QUEUE = 500;
|
||||
const MAX_CHANNELS = 80;
|
||||
const MAX_RULES = 200;
|
||||
const MAX_RECENT_LOGS = 20;
|
||||
const MAX_RECENT_QUEUE = 20;
|
||||
const DEFAULT_LOGS_LIMIT = 50;
|
||||
const DEFAULT_QUEUE_LIMIT = 100;
|
||||
const LEGACY_CHANNEL_TYPES = [
|
||||
'webhook',
|
||||
'bark',
|
||||
'pushplus',
|
||||
'wecom_app',
|
||||
'wecom_robot',
|
||||
'dingtalk_robot',
|
||||
'dingtalk_app',
|
||||
'feishu_robot',
|
||||
'telegram',
|
||||
'email',
|
||||
'serverchan',
|
||||
] as const;
|
||||
const LOG_STATUSES = new Set([
|
||||
'success',
|
||||
'failed',
|
||||
'quiet_hours',
|
||||
'unmatched',
|
||||
'no_available_channel',
|
||||
]);
|
||||
const QUEUE_STATUSES = new Set(['pending', 'scheduled', 'retrying', 'sending', 'failed']);
|
||||
|
||||
const record = (value: unknown): Record<string, unknown> | undefined =>
|
||||
value !== null && typeof value === 'object' && !Array.isArray(value)
|
||||
? (value as Record<string, unknown>)
|
||||
: undefined;
|
||||
|
||||
const bounded = (value: unknown, maximum: number): string | undefined =>
|
||||
(typeof value === 'string' || typeof value === 'number') &&
|
||||
String(value).length <= maximum &&
|
||||
!/[\u0000-\u0008\u000b\u000c\u000e-\u001f\u007f]/u.test(String(value))
|
||||
? String(value)
|
||||
: undefined;
|
||||
|
||||
const safeCount = (value: unknown, maximum: number): number | undefined =>
|
||||
(typeof value === 'number' || typeof value === 'string') &&
|
||||
/^\d+$/u.test(String(value)) &&
|
||||
Number.isSafeInteger(Number(value)) &&
|
||||
Number(value) >= 0 &&
|
||||
Number(value) <= maximum
|
||||
? Number(value)
|
||||
: undefined;
|
||||
|
||||
function hasSuccessStatus(body: string): Record<string, unknown> | undefined {
|
||||
let root: Record<string, unknown> | undefined;
|
||||
try {
|
||||
root = record(JSON.parse(body));
|
||||
} catch {
|
||||
return undefined;
|
||||
}
|
||||
return root && (root.status === 'success' || root.status === 'ok' || root.status === 'OK')
|
||||
? root
|
||||
: undefined;
|
||||
}
|
||||
|
||||
function parseApiResponse(response: UpstreamResponse): Record<string, unknown> {
|
||||
if (
|
||||
response.status < 200 ||
|
||||
response.status >= 300 ||
|
||||
Buffer.byteLength(response.body, 'utf8') > MAX_RESPONSE_BYTES
|
||||
)
|
||||
throw new NotificationServiceError('UPSTREAM_FAILED');
|
||||
const root = hasSuccessStatus(response.body);
|
||||
const data = record(root?.data);
|
||||
if (!data) throw new NotificationServiceError('UPSTREAM_FAILED');
|
||||
return data;
|
||||
}
|
||||
|
||||
function channelTypeSummary(
|
||||
current: FleetNotificationChannelTypeSummary | undefined,
|
||||
type: string,
|
||||
enabled: boolean,
|
||||
): FleetNotificationChannelTypeSummary {
|
||||
return {
|
||||
type,
|
||||
total: (current?.total ?? 0) + 1,
|
||||
enabled: (current?.enabled ?? 0) + (enabled ? 1 : 0),
|
||||
};
|
||||
}
|
||||
|
||||
function parseConfig(data: Record<string, unknown>): FleetNotificationConfigSummary {
|
||||
let channelCount = 0;
|
||||
let channelEnabled = 0;
|
||||
const channelTypes = new Map<string, FleetNotificationChannelTypeSummary>();
|
||||
const collect = (type: string, value: unknown): void => {
|
||||
const item = record(value);
|
||||
if (!item || typeof item.enabled !== 'boolean') return;
|
||||
channelCount += 1;
|
||||
if (item.enabled) channelEnabled += 1;
|
||||
channelTypes.set(type, channelTypeSummary(channelTypes.get(type), type, item.enabled));
|
||||
};
|
||||
|
||||
if (Array.isArray(data.channels)) {
|
||||
for (const value of data.channels.slice(0, MAX_CHANNELS)) {
|
||||
const item = record(value);
|
||||
const type = bounded(item?.type, 80);
|
||||
if (!item || !type || typeof item.enabled !== 'boolean') continue;
|
||||
channelCount += 1;
|
||||
if (item.enabled) channelEnabled += 1;
|
||||
channelTypes.set(type, channelTypeSummary(channelTypes.get(type), type, item.enabled));
|
||||
}
|
||||
} else {
|
||||
for (const type of LEGACY_CHANNEL_TYPES) collect(type, data[type]);
|
||||
}
|
||||
|
||||
let ruleCount = 0;
|
||||
let ruleEnabled = 0;
|
||||
if (Array.isArray(data.rules)) {
|
||||
for (const value of data.rules.slice(0, MAX_RULES)) {
|
||||
const item = record(value);
|
||||
if (!item || typeof item.enabled !== 'boolean') continue;
|
||||
ruleCount += 1;
|
||||
if (item.enabled) ruleEnabled += 1;
|
||||
}
|
||||
}
|
||||
|
||||
return {
|
||||
channelCount,
|
||||
channelEnabled,
|
||||
ruleCount,
|
||||
ruleEnabled,
|
||||
channelTypes: Object.freeze([...channelTypes.values()]),
|
||||
};
|
||||
}
|
||||
|
||||
function parseLogEntry(value: unknown): FleetNotificationLogEntry | undefined {
|
||||
const item = record(value);
|
||||
if (!item) return undefined;
|
||||
const id = bounded(item.id, 128);
|
||||
const eventType = bounded(item.event_type, 80);
|
||||
const status = bounded(item.status, 40);
|
||||
const ruleName = bounded(item.rule_name, 160);
|
||||
const channelName = bounded(item.channel_name, 160);
|
||||
const createdAt = bounded(item.created_at, 64);
|
||||
if (!id || !eventType || !status || !createdAt) return undefined;
|
||||
return Object.freeze({
|
||||
id,
|
||||
eventType,
|
||||
status,
|
||||
...(ruleName === undefined ? {} : { ruleName }),
|
||||
...(channelName === undefined ? {} : { channelName }),
|
||||
createdAt,
|
||||
});
|
||||
}
|
||||
|
||||
function parseLogs(data: Record<string, unknown>): FleetNotificationLogSummary {
|
||||
const total = safeCount(data.total, 10_000_000);
|
||||
if (!Array.isArray(data.logs) || data.logs.length > MAX_LOGS || total === undefined)
|
||||
throw new NotificationServiceError('UPSTREAM_FAILED');
|
||||
const recent: FleetNotificationLogEntry[] = [];
|
||||
const counts = {
|
||||
success: 0,
|
||||
failed: 0,
|
||||
quietHours: 0,
|
||||
unmatched: 0,
|
||||
noAvailableChannel: 0,
|
||||
other: 0,
|
||||
};
|
||||
for (const value of data.logs) {
|
||||
const entry = parseLogEntry(value);
|
||||
if (!entry) throw new NotificationServiceError('UPSTREAM_FAILED');
|
||||
if (LOG_STATUSES.has(entry.status)) {
|
||||
if (entry.status === 'success') counts.success += 1;
|
||||
else if (entry.status === 'failed') counts.failed += 1;
|
||||
else if (entry.status === 'quiet_hours') counts.quietHours += 1;
|
||||
else if (entry.status === 'unmatched') counts.unmatched += 1;
|
||||
else counts.noAvailableChannel += 1;
|
||||
} else {
|
||||
counts.other += 1;
|
||||
}
|
||||
if (recent.length < MAX_RECENT_LOGS) recent.push(entry);
|
||||
}
|
||||
return {
|
||||
total,
|
||||
...counts,
|
||||
recent: Object.freeze(recent),
|
||||
};
|
||||
}
|
||||
|
||||
function parseQueueEntry(
|
||||
value: unknown,
|
||||
instanceId: string,
|
||||
instanceName: string,
|
||||
): FleetNotificationQueueEntry | undefined {
|
||||
const item = record(value);
|
||||
if (!item) return undefined;
|
||||
const id = bounded(item.id, 128);
|
||||
const status = bounded(item.status, 32);
|
||||
const eventType = bounded(item.event_type, 80);
|
||||
const ruleName = bounded(item.rule_name, 160);
|
||||
const channelName = bounded(item.channel_name, 160);
|
||||
const createdAt = bounded(item.created_at, 64);
|
||||
if (!id || !status || !eventType || !createdAt) return undefined;
|
||||
return Object.freeze({
|
||||
id,
|
||||
instanceId,
|
||||
instanceName,
|
||||
status,
|
||||
eventType,
|
||||
...(ruleName === undefined ? {} : { ruleName }),
|
||||
...(channelName === undefined ? {} : { channelName }),
|
||||
createdAt,
|
||||
});
|
||||
}
|
||||
|
||||
function parseQueue(
|
||||
data: Record<string, unknown>,
|
||||
instanceId: string,
|
||||
instanceName: string,
|
||||
): FleetNotificationQueueSummary {
|
||||
const total = safeCount(data.total, 10_000_000);
|
||||
if (!Array.isArray(data.items) || data.items.length > MAX_QUEUE || total === undefined)
|
||||
throw new NotificationServiceError('UPSTREAM_FAILED');
|
||||
const recent: FleetNotificationQueueEntry[] = [];
|
||||
const counts = { pending: 0, scheduled: 0, retrying: 0, sending: 0, failed: 0 };
|
||||
for (const value of data.items) {
|
||||
const entry = parseQueueEntry(value, instanceId, instanceName);
|
||||
if (!entry) throw new NotificationServiceError('UPSTREAM_FAILED');
|
||||
if (QUEUE_STATUSES.has(entry.status)) {
|
||||
counts[entry.status as keyof typeof counts] += 1;
|
||||
}
|
||||
if (recent.length < MAX_RECENT_QUEUE) recent.push(entry);
|
||||
}
|
||||
return {
|
||||
total,
|
||||
...counts,
|
||||
recent: Object.freeze(recent),
|
||||
};
|
||||
}
|
||||
|
||||
function aggregateLogs(
|
||||
value: FleetNotificationLogSummary,
|
||||
accumulator: FleetNotificationLogSummary,
|
||||
): FleetNotificationLogSummary {
|
||||
return {
|
||||
total: value.total + accumulator.total,
|
||||
success: value.success + accumulator.success,
|
||||
failed: value.failed + accumulator.failed,
|
||||
quietHours: value.quietHours + accumulator.quietHours,
|
||||
unmatched: value.unmatched + accumulator.unmatched,
|
||||
noAvailableChannel: value.noAvailableChannel + accumulator.noAvailableChannel,
|
||||
other: value.other + accumulator.other,
|
||||
recent: Object.freeze([...value.recent, ...accumulator.recent].slice(0, MAX_RECENT_LOGS)),
|
||||
};
|
||||
}
|
||||
|
||||
function aggregateQueue(
|
||||
value: FleetNotificationQueueSummary,
|
||||
accumulator: FleetNotificationQueueSummary,
|
||||
): FleetNotificationQueueSummary {
|
||||
return {
|
||||
total: value.total + accumulator.total,
|
||||
pending: value.pending + accumulator.pending,
|
||||
scheduled: value.scheduled + accumulator.scheduled,
|
||||
retrying: value.retrying + accumulator.retrying,
|
||||
sending: value.sending + accumulator.sending,
|
||||
failed: value.failed + accumulator.failed,
|
||||
recent: Object.freeze([...value.recent, ...accumulator.recent].slice(0, MAX_RECENT_QUEUE)),
|
||||
};
|
||||
}
|
||||
|
||||
function emptyLogSummary(): FleetNotificationLogSummary {
|
||||
return {
|
||||
total: 0,
|
||||
success: 0,
|
||||
failed: 0,
|
||||
quietHours: 0,
|
||||
unmatched: 0,
|
||||
noAvailableChannel: 0,
|
||||
other: 0,
|
||||
recent: Object.freeze([]),
|
||||
};
|
||||
}
|
||||
|
||||
function emptyQueueSummary(): FleetNotificationQueueSummary {
|
||||
return {
|
||||
total: 0,
|
||||
pending: 0,
|
||||
scheduled: 0,
|
||||
retrying: 0,
|
||||
sending: 0,
|
||||
failed: 0,
|
||||
recent: Object.freeze([]),
|
||||
};
|
||||
}
|
||||
|
||||
interface InstanceReadResult {
|
||||
readonly config: FleetNotificationConfigSummary;
|
||||
readonly logs: FleetNotificationLogSummary;
|
||||
readonly queue: FleetNotificationQueueSummary;
|
||||
}
|
||||
|
||||
type InstanceNotificationResult =
|
||||
| {
|
||||
readonly id: string;
|
||||
readonly name: string;
|
||||
readonly state: 'ready';
|
||||
readonly value: InstanceReadResult;
|
||||
}
|
||||
| {
|
||||
readonly id: string;
|
||||
readonly name: string;
|
||||
readonly state: 'unavailable' | 'failed';
|
||||
};
|
||||
|
||||
export class InstanceNotificationService {
|
||||
constructor(
|
||||
private readonly options: {
|
||||
readonly instances: InstanceService;
|
||||
readonly sessions: InstanceSessionStore;
|
||||
readonly request: UpstreamSessionClientOptions['request'];
|
||||
readonly ensureSession?: (
|
||||
instanceId: string,
|
||||
origin: string,
|
||||
force?: boolean,
|
||||
) => Promise<void>;
|
||||
},
|
||||
) {}
|
||||
|
||||
async summarize(): Promise<InstanceNotificationSummary> {
|
||||
const instances = await this.listInstances();
|
||||
const results: readonly InstanceNotificationResult[] = await Promise.all(
|
||||
instances.map(async (instance) => {
|
||||
try {
|
||||
return {
|
||||
id: instance.id,
|
||||
name: instance.name,
|
||||
state: 'ready' as const,
|
||||
value: await this.readInstance(instance),
|
||||
};
|
||||
} catch (error) {
|
||||
if (error instanceof NotificationServiceError && error.code === 'NOT_FOUND') {
|
||||
return {
|
||||
id: instance.id,
|
||||
name: instance.name,
|
||||
state: 'unavailable' as const,
|
||||
};
|
||||
}
|
||||
return { id: instance.id, name: instance.name, state: 'failed' as const };
|
||||
}
|
||||
}),
|
||||
);
|
||||
|
||||
const ready = results.filter(
|
||||
(result): result is Extract<InstanceNotificationResult, { state: 'ready' }> =>
|
||||
result.state === 'ready',
|
||||
);
|
||||
const config = ready.reduce(
|
||||
(accumulator, result) => {
|
||||
accumulator.channelCount += result.value.config.channelCount;
|
||||
accumulator.channelEnabled += result.value.config.channelEnabled;
|
||||
accumulator.ruleCount += result.value.config.ruleCount;
|
||||
accumulator.ruleEnabled += result.value.config.ruleEnabled;
|
||||
return accumulator;
|
||||
},
|
||||
{ channelCount: 0, channelEnabled: 0, ruleCount: 0, ruleEnabled: 0 },
|
||||
);
|
||||
const channelTypes = new Map<string, FleetNotificationChannelTypeSummary>();
|
||||
for (const result of ready) {
|
||||
for (const typeSummary of result.value.config.channelTypes) {
|
||||
const current = channelTypes.get(typeSummary.type);
|
||||
channelTypes.set(
|
||||
typeSummary.type,
|
||||
channelTypeSummary(current, typeSummary.type, typeSummary.enabled > 0),
|
||||
);
|
||||
}
|
||||
}
|
||||
const logs = ready.reduce(
|
||||
(accumulator, result) => aggregateLogs(result.value.logs, accumulator),
|
||||
emptyLogSummary(),
|
||||
);
|
||||
const queue = ready.reduce(
|
||||
(accumulator, result) => aggregateQueue(result.value.queue, accumulator),
|
||||
emptyQueueSummary(),
|
||||
);
|
||||
return {
|
||||
observedAt: new Date().toISOString(),
|
||||
deviceCount: instances.length,
|
||||
readyCount: ready.length,
|
||||
unavailableCount: results.length - ready.length,
|
||||
devices: Object.freeze(
|
||||
results.map((result) =>
|
||||
Object.freeze({
|
||||
id: result.id,
|
||||
name: result.name,
|
||||
state: result.state,
|
||||
}),
|
||||
),
|
||||
),
|
||||
config: { ...config, channelTypes: Object.freeze([...channelTypes.values()]) },
|
||||
logs,
|
||||
queue,
|
||||
};
|
||||
}
|
||||
|
||||
private async listInstances(): Promise<readonly Instance[]> {
|
||||
const instances: Instance[] = [];
|
||||
let page = 1;
|
||||
while (instances.length < MAX_DEVICES) {
|
||||
const current = await this.options.instances.list({ page, pageSize: 100 });
|
||||
instances.push(...current.items.slice(0, MAX_DEVICES - instances.length));
|
||||
if (current.items.length < 100) break;
|
||||
page += 1;
|
||||
}
|
||||
return instances;
|
||||
}
|
||||
|
||||
private async readInstance(instance: Instance): Promise<InstanceReadResult> {
|
||||
const cookie = await this.ensureOwnerCookie(instance);
|
||||
const [config, logs, queue] = await Promise.all([
|
||||
this.readEndpoint(instance, cookie, '/api/notifications/config').then(parseConfig),
|
||||
this.readEndpoint(
|
||||
instance,
|
||||
cookie,
|
||||
`/api/notifications/logs?limit=${DEFAULT_LOGS_LIMIT}&offset=0`,
|
||||
).then(parseLogs),
|
||||
this.readEndpoint(
|
||||
instance,
|
||||
cookie,
|
||||
`/api/notifications/queue?limit=${DEFAULT_QUEUE_LIMIT}`,
|
||||
).then((data) => parseQueue(data, instance.id, instance.name)),
|
||||
]);
|
||||
return { config, logs, queue };
|
||||
}
|
||||
|
||||
private async ensureOwnerCookie(instance: Instance): Promise<string | undefined> {
|
||||
const current = this.options.sessions.sessionFor(instance.id);
|
||||
if (current && current.origin !== instance.origin)
|
||||
throw new NotificationServiceError('SESSION_INVALID');
|
||||
if (!current && this.options.ensureSession) {
|
||||
try {
|
||||
await this.options.ensureSession(instance.id, instance.origin);
|
||||
} catch {
|
||||
// Passwordless or anonymous reads may still succeed without a session.
|
||||
}
|
||||
}
|
||||
const session = this.options.sessions.sessionFor(instance.id);
|
||||
return session && session.origin === instance.origin ? session.cookie : undefined;
|
||||
}
|
||||
|
||||
private async readEndpoint(
|
||||
instance: Instance,
|
||||
cookie: string | undefined,
|
||||
path: string,
|
||||
): Promise<Record<string, unknown>> {
|
||||
const request = (currentCookie?: string) =>
|
||||
this.options.request({
|
||||
url: `${instance.origin}${path}`,
|
||||
method: 'GET',
|
||||
headers: {
|
||||
accept: 'application/json',
|
||||
...(currentCookie ? { cookie: currentCookie } : {}),
|
||||
},
|
||||
});
|
||||
let response = await request(cookie);
|
||||
if ([401, 403].includes(response.status) && this.options.ensureSession) {
|
||||
await this.options.ensureSession(instance.id, instance.origin, true);
|
||||
response = await request(this.options.sessions.sessionFor(instance.id)?.cookie);
|
||||
}
|
||||
return parseApiResponse(response);
|
||||
}
|
||||
|
||||
async retryAllQueue(): Promise<FleetNotificationQueueRetryResult> {
|
||||
const instances = await this.listInstances();
|
||||
const outcomes = await Promise.all(
|
||||
instances.map(async (instance) => {
|
||||
try {
|
||||
await this.postEndpoint(instance, '/api/notifications/queue/retry-all');
|
||||
return {
|
||||
instanceId: instance.id,
|
||||
instanceName: instance.name,
|
||||
kind: 'succeeded' as const,
|
||||
};
|
||||
} catch (error) {
|
||||
if (error instanceof NotificationServiceError && error.code === 'NOT_FOUND')
|
||||
return {
|
||||
instanceId: instance.id,
|
||||
instanceName: instance.name,
|
||||
kind: 'skipped' as const,
|
||||
};
|
||||
return {
|
||||
instanceId: instance.id,
|
||||
instanceName: instance.name,
|
||||
kind: 'failed' as const,
|
||||
code:
|
||||
error instanceof NotificationServiceError ? error.code : ('UPSTREAM_FAILED' as const),
|
||||
};
|
||||
}
|
||||
}),
|
||||
);
|
||||
const failures = outcomes.filter(
|
||||
(outcome): outcome is Extract<typeof outcome, { readonly kind: 'failed' }> =>
|
||||
outcome.kind === 'failed',
|
||||
);
|
||||
return {
|
||||
requested: instances.length,
|
||||
succeeded: outcomes.filter((outcome) => outcome.kind === 'succeeded').length,
|
||||
failed: failures.length,
|
||||
skipped: outcomes.filter((outcome) => outcome.kind === 'skipped').length,
|
||||
failures: Object.freeze(
|
||||
failures.map((failure) =>
|
||||
Object.freeze({
|
||||
instanceId: failure.instanceId,
|
||||
instanceName: failure.instanceName,
|
||||
code: failure.code,
|
||||
}),
|
||||
),
|
||||
),
|
||||
};
|
||||
}
|
||||
|
||||
async retryQueueItem(instanceId: string, queueId: string): Promise<{ readonly retried: true }> {
|
||||
await this.queueItemAction(instanceId, queueId, 'retry');
|
||||
return { retried: true };
|
||||
}
|
||||
|
||||
async deleteQueueItem(instanceId: string, queueId: string): Promise<{ readonly deleted: true }> {
|
||||
await this.queueItemAction(instanceId, queueId, 'delete');
|
||||
return { deleted: true };
|
||||
}
|
||||
|
||||
private async queueItemAction(
|
||||
instanceId: string,
|
||||
queueId: string,
|
||||
action: 'retry' | 'delete',
|
||||
): Promise<void> {
|
||||
if (!/^[A-Za-z0-9_.-]{1,128}$/u.test(queueId))
|
||||
throw new NotificationServiceError('VALIDATION_FAILED');
|
||||
const instance = await this.options.instances.get(instanceId);
|
||||
if (!instance) throw new NotificationServiceError('NOT_FOUND');
|
||||
try {
|
||||
const path =
|
||||
action === 'retry'
|
||||
? `/api/notifications/queue/${queueId}/retry`
|
||||
: `/api/notifications/queue/${queueId}`;
|
||||
if (action === 'delete') {
|
||||
await this.deleteEndpoint(instance, path);
|
||||
return;
|
||||
}
|
||||
await this.postEndpoint(instance, path);
|
||||
} catch (error) {
|
||||
if (error instanceof NotificationServiceError) throw error;
|
||||
throw new NotificationServiceError('UPSTREAM_FAILED');
|
||||
}
|
||||
}
|
||||
|
||||
private async postEndpoint(instance: Instance, path: string): Promise<void> {
|
||||
const cookie = await this.ensureOwnerCookie(instance);
|
||||
const request = (currentCookie?: string) =>
|
||||
this.options.request({
|
||||
url: `${instance.origin}${path}`,
|
||||
method: 'POST',
|
||||
headers: {
|
||||
accept: 'application/json',
|
||||
...(currentCookie ? { cookie: currentCookie } : {}),
|
||||
},
|
||||
});
|
||||
let response = await request(cookie);
|
||||
if ([401, 403].includes(response.status) && this.options.ensureSession) {
|
||||
await this.options.ensureSession(instance.id, instance.origin, true);
|
||||
response = await request(this.options.sessions.sessionFor(instance.id)?.cookie);
|
||||
}
|
||||
assertActionSuccess(response);
|
||||
}
|
||||
|
||||
private async deleteEndpoint(instance: Instance, path: string): Promise<void> {
|
||||
const cookie = await this.ensureOwnerCookie(instance);
|
||||
const request = (currentCookie?: string) =>
|
||||
this.options.request({
|
||||
url: `${instance.origin}${path}`,
|
||||
method: 'DELETE',
|
||||
headers: {
|
||||
accept: 'application/json',
|
||||
...(currentCookie ? { cookie: currentCookie } : {}),
|
||||
},
|
||||
});
|
||||
let response = await request(cookie);
|
||||
if ([401, 403].includes(response.status) && this.options.ensureSession) {
|
||||
await this.options.ensureSession(instance.id, instance.origin, true);
|
||||
response = await request(this.options.sessions.sessionFor(instance.id)?.cookie);
|
||||
}
|
||||
assertActionSuccess(response);
|
||||
}
|
||||
}
|
||||
|
||||
function assertActionSuccess(response: UpstreamResponse): void {
|
||||
if (
|
||||
response.status < 200 ||
|
||||
response.status >= 300 ||
|
||||
Buffer.byteLength(response.body, 'utf8') > MAX_RESPONSE_BYTES
|
||||
)
|
||||
throw new NotificationServiceError('UPSTREAM_FAILED');
|
||||
if (!hasSuccessStatus(response.body)) throw new NotificationServiceError('UPSTREAM_FAILED');
|
||||
}
|
||||
@@ -0,0 +1,215 @@
|
||||
import { createServer, type Server, type Socket } from 'node:net';
|
||||
import { afterEach, describe, expect, it } from 'vitest';
|
||||
|
||||
import type { ChannelDeliveryInput } from './channel-delivery.js';
|
||||
import { sendEmailNotification } from './smtp-sender.js';
|
||||
|
||||
interface FakePeer {
|
||||
readonly port: number;
|
||||
readonly lines: string[];
|
||||
stop(): Promise<void>;
|
||||
}
|
||||
|
||||
const peers: FakePeer[] = [];
|
||||
|
||||
afterEach(async () => {
|
||||
for (const peer of peers.splice(0)) await peer.stop();
|
||||
});
|
||||
|
||||
/**
|
||||
* Scripted SMTP peer. It answers one reply per command and stays quiet while a
|
||||
* message body streams in, which is how a real server behaves after `354`.
|
||||
*/
|
||||
function fakeServer(script: readonly string[]): Promise<FakePeer> {
|
||||
return new Promise((resolve) => {
|
||||
const lines: string[] = [];
|
||||
const sockets = new Set<Socket>();
|
||||
const server: Server = createServer((socket) => {
|
||||
sockets.add(socket);
|
||||
socket.once('close', () => sockets.delete(socket));
|
||||
let index = 0;
|
||||
let buffered = '';
|
||||
let inData = false;
|
||||
socket.write(`${script[index++] ?? '221 bye'}\r\n`);
|
||||
socket.on('data', (chunk) => {
|
||||
buffered += chunk.toString('utf8');
|
||||
let boundary = buffered.indexOf('\r\n');
|
||||
while (boundary >= 0) {
|
||||
const line = buffered.slice(0, boundary);
|
||||
buffered = buffered.slice(boundary + 2);
|
||||
boundary = buffered.indexOf('\r\n');
|
||||
lines.push(line);
|
||||
if (inData) {
|
||||
if (line === '.') inData = false;
|
||||
else continue;
|
||||
}
|
||||
if (line === 'DATA') inData = true;
|
||||
// A silent peer is deliberate: it lets a test assert the client timeout.
|
||||
const reply = script[index++];
|
||||
if (reply !== undefined) socket.write(`${reply}\r\n`);
|
||||
}
|
||||
});
|
||||
});
|
||||
server.listen(0, '127.0.0.1', () => {
|
||||
const address = server.address();
|
||||
const peer: FakePeer = {
|
||||
port: typeof address === 'object' && address ? address.port : 0,
|
||||
lines,
|
||||
stop: () =>
|
||||
new Promise<void>((done) => {
|
||||
for (const socket of sockets) socket.destroy();
|
||||
server.close(() => done());
|
||||
}),
|
||||
};
|
||||
peers.push(peer);
|
||||
resolve(peer);
|
||||
});
|
||||
});
|
||||
}
|
||||
|
||||
function input(config: ChannelDeliveryInput['config']): ChannelDeliveryInput {
|
||||
return {
|
||||
type: 'email',
|
||||
config,
|
||||
title: '设备离线',
|
||||
body: 'SIM 卡所在设备已断开',
|
||||
eventType: 'device',
|
||||
occurredAt: '2026-09-03T08:00:00.000Z',
|
||||
};
|
||||
}
|
||||
|
||||
// One reply per command: greeting, EHLO, AUTH LOGIN, user, password, MAIL FROM,
|
||||
// two RCPT TO, DATA, the "." terminator, then QUIT.
|
||||
const AUTH_SCRIPT = [
|
||||
'220 smtp ready',
|
||||
'250 ehlo',
|
||||
'334 ' + Buffer.from('Username:').toString('base64'),
|
||||
'334 ' + Buffer.from('Password:').toString('base64'),
|
||||
'235 authenticated',
|
||||
'250 sender ok',
|
||||
'250 recipient ok',
|
||||
'250 recipient ok',
|
||||
'354 send data',
|
||||
'250 message accepted',
|
||||
'221 bye',
|
||||
];
|
||||
|
||||
const OPEN_SCRIPT = [
|
||||
'220 smtp ready',
|
||||
'250 ehlo',
|
||||
'250 sender ok',
|
||||
'250 recipient ok',
|
||||
'354 send data',
|
||||
'250 message accepted',
|
||||
'221 bye',
|
||||
];
|
||||
|
||||
function messageLines(lines: readonly string[]): string[] {
|
||||
const start = lines.indexOf('DATA');
|
||||
return start < 0 ? [] : lines.slice(start + 1);
|
||||
}
|
||||
|
||||
describe('sendEmailNotification', () => {
|
||||
it('runs AUTH LOGIN and submits one UTF-8 message per recipient', async () => {
|
||||
const peer = await fakeServer(AUTH_SCRIPT);
|
||||
const result = await sendEmailNotification(
|
||||
input({
|
||||
smtp_host: '127.0.0.1',
|
||||
smtp_port: peer.port,
|
||||
smtp_security: 'none',
|
||||
username: 'notify@example.test',
|
||||
password: 'p@ss',
|
||||
sender_address: 'notify@example.test',
|
||||
sender_name: 'SimAdmin 控制台',
|
||||
receiver_addresses: 'ops@example.test, admin@example.test',
|
||||
message_format: 'plain',
|
||||
}),
|
||||
2_000,
|
||||
);
|
||||
|
||||
expect(result).toEqual({ ok: true });
|
||||
expect(peer.lines[0]).toMatch(/^EHLO /u);
|
||||
expect(peer.lines[1]).toBe('AUTH LOGIN');
|
||||
expect(peer.lines[2]).toBe(Buffer.from('notify@example.test').toString('base64'));
|
||||
expect(peer.lines[3]).toBe(Buffer.from('p@ss').toString('base64'));
|
||||
expect(peer.lines[4]).toBe('MAIL FROM:<notify@example.test>');
|
||||
expect(peer.lines.filter((line) => line.startsWith('RCPT TO:<'))).toEqual([
|
||||
'RCPT TO:<ops@example.test>',
|
||||
'RCPT TO:<admin@example.test>',
|
||||
]);
|
||||
|
||||
const message = messageLines(peer.lines);
|
||||
expect(message.at(-1)).toBe('.');
|
||||
const headers = message.slice(0, message.indexOf(''));
|
||||
expect(headers.join('\n')).toContain('Subject: =?UTF-8?B?');
|
||||
expect(headers.join('\n')).toContain('Content-Type: text/plain; charset=UTF-8');
|
||||
const payload = Buffer.from(
|
||||
message.slice(message.indexOf('') + 1, message.length - 1).join(''),
|
||||
'base64',
|
||||
).toString('utf8');
|
||||
expect(payload).toBe('设备离线\n\nSIM 卡所在设备已断开');
|
||||
});
|
||||
|
||||
it('switches the MIME subtype when the channel asks for HTML', async () => {
|
||||
const peer = await fakeServer(OPEN_SCRIPT);
|
||||
const result = await sendEmailNotification(
|
||||
input({
|
||||
smtp_host: '127.0.0.1',
|
||||
smtp_port: peer.port,
|
||||
smtp_security: 'none',
|
||||
sender_address: 'a@example.test',
|
||||
receiver_addresses: 'b@example.test',
|
||||
message_format: 'html',
|
||||
}),
|
||||
2_000,
|
||||
);
|
||||
expect(result).toEqual({ ok: true });
|
||||
expect(messageLines(peer.lines).join('\n')).toContain('Content-Type: text/html');
|
||||
});
|
||||
|
||||
it('refuses to dial out when no receiver address is usable', async () => {
|
||||
await expect(
|
||||
sendEmailNotification(
|
||||
input({
|
||||
smtp_host: '127.0.0.1',
|
||||
smtp_port: 1,
|
||||
sender_address: 'notify@example.test',
|
||||
receiver_addresses: 'not-an-address',
|
||||
}),
|
||||
500,
|
||||
),
|
||||
).resolves.toEqual({ ok: false, detail: '缺少有效的收件地址' });
|
||||
});
|
||||
|
||||
it('reports a rejected handshake instead of throwing', async () => {
|
||||
const peer = await fakeServer(['220 smtp ready', '421 service denied']);
|
||||
await expect(
|
||||
sendEmailNotification(
|
||||
input({
|
||||
smtp_host: '127.0.0.1',
|
||||
smtp_port: peer.port,
|
||||
smtp_security: 'none',
|
||||
sender_address: 'a@example.test',
|
||||
receiver_addresses: 'b@example.test',
|
||||
}),
|
||||
2_000,
|
||||
),
|
||||
).resolves.toMatchObject({ ok: false, detail: 'SMTP 返回 421:421 service denied' });
|
||||
});
|
||||
|
||||
it('gives up with a readable detail when the server stops answering', async () => {
|
||||
const peer = await fakeServer(['220 smtp ready']);
|
||||
await expect(
|
||||
sendEmailNotification(
|
||||
input({
|
||||
smtp_host: '127.0.0.1',
|
||||
smtp_port: peer.port,
|
||||
smtp_security: 'none',
|
||||
sender_address: 'a@example.test',
|
||||
receiver_addresses: 'b@example.test',
|
||||
}),
|
||||
300,
|
||||
),
|
||||
).resolves.toMatchObject({ ok: false, detail: 'SMTP 响应超时' });
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,295 @@
|
||||
import { createConnection } from 'node:net';
|
||||
import { connect as connectTls, TLSSocket } from 'node:tls';
|
||||
import { hostname } from 'node:os';
|
||||
|
||||
import type { ChannelDeliveryInput, ChannelDeliveryResult } from './channel-delivery.js';
|
||||
|
||||
type Socket = ReturnType<typeof createConnection> | TLSSocket;
|
||||
|
||||
const CRLF = '\r\n';
|
||||
const MAX_REPLY_BYTES = 64 * 1024;
|
||||
|
||||
function configText(config: ChannelDeliveryInput['config'], key: string): string {
|
||||
const value = config[key];
|
||||
if (typeof value === 'string') return value.trim();
|
||||
if (typeof value === 'number') return String(value);
|
||||
return '';
|
||||
}
|
||||
|
||||
function configNumber(
|
||||
config: ChannelDeliveryInput['config'],
|
||||
key: string,
|
||||
fallback: number,
|
||||
): number {
|
||||
const parsed = Number(configText(config, key));
|
||||
return Number.isSafeInteger(parsed) && parsed > 0 && parsed <= 65_535 ? parsed : fallback;
|
||||
}
|
||||
|
||||
function configFlag(config: ChannelDeliveryInput['config'], key: string): boolean {
|
||||
return config[key] === true || config[key] === 'true' || config[key] === 1;
|
||||
}
|
||||
|
||||
function splitAddresses(raw: string): string[] {
|
||||
return raw
|
||||
.split(/[,;,;\s]+/u)
|
||||
.map((value) => value.trim())
|
||||
.filter((value) => /^[^\s@]+@[^\s@]+$/u.test(value))
|
||||
.slice(0, 50);
|
||||
}
|
||||
|
||||
function encodeAddress(name: string, address: string): string {
|
||||
if (!name) return `<${address}>`;
|
||||
return `${encodeWords(name)} <${address}>`;
|
||||
}
|
||||
|
||||
function encodeWords(value: string): string {
|
||||
return `=?UTF-8?B?${Buffer.from(value, 'utf8').toString('base64')}?=`;
|
||||
}
|
||||
|
||||
/** Header values must not smuggle extra headers through raw newlines. */
|
||||
function foldHeader(value: string): string {
|
||||
return value.replace(/[\r\n\u0000-\u0008\u000B\u000C\u000E-\u001F]/gu, ' ').slice(0, 500);
|
||||
}
|
||||
|
||||
function buildMessage(input: ChannelDeliveryInput, html: boolean, sender: string): string {
|
||||
const headers = [
|
||||
`From: ${encodeAddress(foldHeader(configText(input.config, 'sender_name')), sender)}`,
|
||||
`Subject: ${encodeWords(foldHeader(input.title))}`,
|
||||
'MIME-Version: 1.0',
|
||||
`Date: ${new Date(input.occurredAt).toUTCString()}`,
|
||||
`Content-Type: text/${html ? 'html' : 'plain'}; charset=UTF-8`,
|
||||
'Content-Transfer-Encoding: base64',
|
||||
];
|
||||
const body = html
|
||||
? `<p>${input.title}</p><p>${input.body}</p>`
|
||||
: `${input.title}\n\n${input.body}`;
|
||||
const encoded =
|
||||
Buffer.from(body, 'utf8')
|
||||
.toString('base64')
|
||||
.match(/.{1,76}/gu)
|
||||
?.join(CRLF) ?? '';
|
||||
return `${headers.join(CRLF)}${CRLF}${CRLF}${encoded}${CRLF}`;
|
||||
}
|
||||
|
||||
class SmtpSession {
|
||||
#socket: Socket;
|
||||
#buffer = '';
|
||||
#waiters: (() => void)[] = [];
|
||||
#closed = false;
|
||||
#failure: Error | undefined;
|
||||
|
||||
constructor(socket: Socket) {
|
||||
this.#socket = socket;
|
||||
this.#attach(socket);
|
||||
}
|
||||
|
||||
get socket(): Socket {
|
||||
return this.#socket;
|
||||
}
|
||||
|
||||
#attach(socket: Socket): void {
|
||||
socket.on('data', (chunk: Buffer) => {
|
||||
this.#buffer += chunk.toString('utf8');
|
||||
if (this.#buffer.length > MAX_REPLY_BYTES) this.#fail(new Error('SMTP 响应过大'));
|
||||
this.#wake();
|
||||
});
|
||||
socket.on('error', (error: Error) => this.#fail(error));
|
||||
socket.on('close', () => {
|
||||
this.#closed = true;
|
||||
this.#fail(new Error('SMTP 连接已关闭'));
|
||||
});
|
||||
}
|
||||
|
||||
/** Swaps the transport after STARTTLS without losing the reply reader. */
|
||||
replace(socket: Socket): void {
|
||||
this.#socket.removeAllListeners('data');
|
||||
this.#socket.removeAllListeners('error');
|
||||
this.#socket.removeAllListeners('close');
|
||||
this.#buffer = '';
|
||||
this.#closed = false;
|
||||
this.#socket = socket;
|
||||
this.#attach(socket);
|
||||
}
|
||||
|
||||
#fail(error: Error): void {
|
||||
this.#failure ??= error;
|
||||
this.#wake();
|
||||
}
|
||||
|
||||
#wake(): void {
|
||||
const waiting = this.#waiters.splice(0);
|
||||
for (const resolve of waiting) resolve();
|
||||
}
|
||||
|
||||
write(line: string): void {
|
||||
if (this.#failure) throw this.#failure;
|
||||
this.#socket.write(line);
|
||||
}
|
||||
|
||||
#readReply(): { readonly code: number; readonly text: string } | undefined {
|
||||
const lines = this.#buffer.split(CRLF);
|
||||
const collected: string[] = [];
|
||||
let consumed = 0;
|
||||
for (const line of lines) {
|
||||
if (!/^\d{3}[- ]/u.test(line)) break;
|
||||
collected.push(line);
|
||||
consumed += line.length + CRLF.length;
|
||||
if (line[3] === ' ') {
|
||||
this.#buffer = this.#buffer.slice(consumed);
|
||||
const last = collected[collected.length - 1] ?? '0';
|
||||
return { code: Number(last.slice(0, 3)), text: collected.join('\n') };
|
||||
}
|
||||
}
|
||||
return undefined;
|
||||
}
|
||||
|
||||
async reply(timeoutMs: number): Promise<{ readonly code: number; readonly text: string }> {
|
||||
const deadline = Date.now() + timeoutMs;
|
||||
for (;;) {
|
||||
const parsed = this.#readReply();
|
||||
if (parsed) return parsed;
|
||||
if (this.#failure) throw this.#failure;
|
||||
if (this.#closed) throw new Error('SMTP 连接已关闭');
|
||||
const remaining = deadline - Date.now();
|
||||
if (remaining <= 0) throw new Error('SMTP 响应超时');
|
||||
// The wake-up callback must resolve as well as clear the timer, or a reply
|
||||
// that lands in this window would leave the session waiting forever.
|
||||
await new Promise<void>((resolve) => {
|
||||
const timer = setTimeout(resolve, Math.min(remaining, 100));
|
||||
this.#waiters.push(() => {
|
||||
clearTimeout(timer);
|
||||
resolve();
|
||||
});
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
close(): void {
|
||||
this.#socket.removeAllListeners();
|
||||
this.#socket.destroy();
|
||||
}
|
||||
}
|
||||
|
||||
function expect(reply: { code: number; text: string }, codes: readonly number[]): void {
|
||||
if (!codes.includes(reply.code))
|
||||
throw new Error(`SMTP 返回 ${reply.code}:${reply.text.slice(0, 160)}`);
|
||||
}
|
||||
|
||||
function plainSocket(host: string, port: number, timeoutMs: number): Promise<Socket> {
|
||||
return new Promise((resolve, reject) => {
|
||||
const socket = createConnection({ host, port });
|
||||
const onTimeout = (): void => {
|
||||
socket.destroy();
|
||||
reject(new Error('SMTP 连接超时'));
|
||||
};
|
||||
socket.setTimeout(timeoutMs);
|
||||
socket.once('timeout', onTimeout);
|
||||
socket.once('connect', () => {
|
||||
// The timer only guards the handshake. Every command afterwards carries its own reply
|
||||
// deadline, and leaving an idle timer armed would destroy the session at the same
|
||||
// instant that deadline expires, making the reported detail a coin flip between
|
||||
// "connection closed" and "timed out".
|
||||
socket.removeListener('timeout', onTimeout);
|
||||
socket.setTimeout(0);
|
||||
resolve(socket);
|
||||
});
|
||||
socket.once('error', reject);
|
||||
});
|
||||
}
|
||||
|
||||
function upgradeSocket(socket: Socket, host: string, rejectUnauthorized: boolean): Promise<Socket> {
|
||||
return new Promise((resolve, reject) => {
|
||||
socket.removeAllListeners('data');
|
||||
const upgraded = connectTls({ socket, servername: host, rejectUnauthorized }, () =>
|
||||
resolve(upgraded),
|
||||
);
|
||||
upgraded.once('error', reject);
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* Minimal SMTP client behind the email channel: implicit TLS, STARTTLS or plain,
|
||||
* optional AUTH LOGIN, and one UTF-8 message per delivery.
|
||||
*/
|
||||
export async function sendEmailNotification(
|
||||
input: ChannelDeliveryInput,
|
||||
timeoutMs: number,
|
||||
): Promise<ChannelDeliveryResult> {
|
||||
const host = configText(input.config, 'smtp_host');
|
||||
const sender = configText(input.config, 'sender_address');
|
||||
const receivers = splitAddresses(configText(input.config, 'receiver_addresses'));
|
||||
if (!host) return { ok: false, detail: '缺少 SMTP 地址' };
|
||||
if (!sender) return { ok: false, detail: '缺少发件地址' };
|
||||
// Envelope commands are line-framed; reject anything that could break the frame.
|
||||
if (!/^[^\s@<>,;:"']+@[^\s@<>,;:"']+$/u.test(sender))
|
||||
return { ok: false, detail: '发件地址无效' };
|
||||
if (receivers.length === 0) return { ok: false, detail: '缺少有效的收件地址' };
|
||||
const port = configNumber(input.config, 'smtp_port', 465);
|
||||
const security = configText(input.config, 'smtp_security') || 'implicit_tls';
|
||||
const rejectUnauthorized = !configFlag(input.config, 'allow_insecure_tls');
|
||||
const username = configText(input.config, 'username');
|
||||
const password = configText(input.config, 'password');
|
||||
|
||||
let session: SmtpSession | undefined;
|
||||
try {
|
||||
const initial: Socket =
|
||||
security === 'implicit_tls'
|
||||
? await new Promise<Socket>((resolve, reject) => {
|
||||
const socket = connectTls({
|
||||
host,
|
||||
port,
|
||||
servername: host,
|
||||
rejectUnauthorized,
|
||||
timeout: timeoutMs,
|
||||
});
|
||||
socket.once('secure', () => resolve(socket));
|
||||
socket.once('error', reject);
|
||||
})
|
||||
: await plainSocket(host, port, timeoutMs);
|
||||
session = new SmtpSession(initial);
|
||||
expect(await session.reply(timeoutMs), [220]);
|
||||
const clientName = hostname() || 'localhost';
|
||||
session.write(`EHLO ${clientName}${CRLF}`);
|
||||
expect(await session.reply(timeoutMs), [250]);
|
||||
|
||||
if (security === 'starttls') {
|
||||
session.write(`STARTTLS${CRLF}`);
|
||||
expect(await session.reply(timeoutMs), [220]);
|
||||
session.replace(await upgradeSocket(session.socket, host, rejectUnauthorized));
|
||||
session.write(`EHLO ${clientName}${CRLF}`);
|
||||
expect(await session.reply(timeoutMs), [250]);
|
||||
}
|
||||
|
||||
if (username && password) {
|
||||
session.write(`AUTH LOGIN${CRLF}`);
|
||||
expect(await session.reply(timeoutMs), [334]);
|
||||
session.write(`${Buffer.from(username, 'utf8').toString('base64')}${CRLF}`);
|
||||
expect(await session.reply(timeoutMs), [334]);
|
||||
session.write(`${Buffer.from(password, 'utf8').toString('base64')}${CRLF}`);
|
||||
expect(await session.reply(timeoutMs), [235]);
|
||||
}
|
||||
|
||||
session.write(`MAIL FROM:<${sender}>${CRLF}`);
|
||||
expect(await session.reply(timeoutMs), [250]);
|
||||
for (const receiver of receivers) {
|
||||
session.write(`RCPT TO:<${receiver}>${CRLF}`);
|
||||
expect(await session.reply(timeoutMs), [250, 251]);
|
||||
}
|
||||
session.write(`DATA${CRLF}`);
|
||||
expect(await session.reply(timeoutMs), [354]);
|
||||
const message = buildMessage(
|
||||
input,
|
||||
configText(input.config, 'message_format') === 'html',
|
||||
sender,
|
||||
).replace(/^\./gmu, '..');
|
||||
session.write(`${message}.${CRLF}`);
|
||||
expect(await session.reply(timeoutMs), [250]);
|
||||
session.write(`QUIT${CRLF}`);
|
||||
return { ok: true };
|
||||
} catch (error) {
|
||||
const detail = error instanceof Error ? error.message : '邮件投递失败';
|
||||
return { ok: false, detail: detail.slice(0, 300) };
|
||||
} finally {
|
||||
session?.close();
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,141 @@
|
||||
import Database from 'better-sqlite3';
|
||||
import { afterEach, describe, expect, it } from 'vitest';
|
||||
|
||||
import { DeviceIdentityService } from '../identity/device-identity-service.js';
|
||||
import { migrateDatabase } from '../../infrastructure/database/migrations.js';
|
||||
import { MetricsService } from './metrics-service.js';
|
||||
|
||||
let db: Database.Database | undefined;
|
||||
|
||||
const NOW = '2026-09-05T02:00:00.000Z';
|
||||
|
||||
function fixture() {
|
||||
db = new Database(':memory:');
|
||||
db.pragma('foreign_keys=ON');
|
||||
migrateDatabase(db);
|
||||
const identities = new DeviceIdentityService({ db });
|
||||
const metrics = new MetricsService({
|
||||
db,
|
||||
version: '1.9.6',
|
||||
sources: {
|
||||
connections: {
|
||||
summarize: () => [
|
||||
{
|
||||
instanceId: 'node-a',
|
||||
total: 40,
|
||||
success: 38,
|
||||
failed: 2,
|
||||
averageDurationMs: 12,
|
||||
lastObservedAt: NOW,
|
||||
lastErrorCode: null,
|
||||
availabilityPercent: 95,
|
||||
},
|
||||
],
|
||||
},
|
||||
identities,
|
||||
},
|
||||
now: () => new Date(NOW),
|
||||
});
|
||||
return { db, identities, metrics };
|
||||
}
|
||||
|
||||
function seedNode(database: Database.Database, id: string, name: string, enabled = 1): void {
|
||||
database
|
||||
.prepare(
|
||||
`INSERT INTO instances (id,name,base_url,auth_mode,enabled,config_revision,created_at,updated_at)
|
||||
VALUES (?,?,'http://' || ? || ':8080','password',?,1,?,?)`,
|
||||
)
|
||||
.run(id, name, id, enabled, NOW, NOW);
|
||||
}
|
||||
|
||||
afterEach(() => {
|
||||
db?.close();
|
||||
db = undefined;
|
||||
});
|
||||
|
||||
describe('MetricsService', () => {
|
||||
it('renders the fleet, queue and job gauges with every state present', () => {
|
||||
const { db: database, metrics } = fixture();
|
||||
seedNode(database, 'node-a', '机房 A');
|
||||
seedNode(database, 'node-b', '机房 B', 0);
|
||||
database
|
||||
.prepare(
|
||||
`INSERT INTO sms_outbox (id,instance_id,phone_number,content,status,available_at,created_at,updated_at)
|
||||
VALUES ('s1','node-a','+8613800000000','hi','failed',?,?,?)`,
|
||||
)
|
||||
.run(NOW, NOW, NOW);
|
||||
database
|
||||
.prepare(
|
||||
`INSERT INTO notification_queue (id,event_type,status,payload_json,available_at,created_at,updated_at)
|
||||
VALUES ('q1','sms','pending','{}',?,?,?)`,
|
||||
)
|
||||
.run(NOW, NOW, NOW);
|
||||
database
|
||||
.prepare(
|
||||
`INSERT INTO jobs (id,operation_id,risk_level,status,requested_by,request_id,parameters_digest,created_at,updated_at)
|
||||
VALUES ('j1','node.reboot','R2','running','op','r1','digest',?,?)`,
|
||||
)
|
||||
.run(NOW, NOW);
|
||||
|
||||
const text = metrics.render();
|
||||
expect(text).toContain('multi_simadmin_nodes_total 2');
|
||||
expect(text).toContain('multi_simadmin_node_enabled{node="node-b"} 0');
|
||||
expect(text).toContain('multi_simadmin_node_info{node="node-a"} 1');
|
||||
// Operator-assigned names stay out of the session-free exposition entirely.
|
||||
expect(text).not.toContain('机房 A');
|
||||
expect(text).toContain('multi_simadmin_node_probe_total{node="node-a",outcome="success"} 38');
|
||||
expect(text).toContain('multi_simadmin_node_availability_ratio{node="node-a"} 0.950');
|
||||
expect(text).toContain('multi_simadmin_sms_outbox{status="failed"} 1');
|
||||
expect(text).toContain('multi_simadmin_sms_outbox{status="queued"} 0');
|
||||
expect(text).toContain('multi_simadmin_notification_queue{status="pending"} 1');
|
||||
expect(text).toContain('multi_simadmin_jobs{status="running"} 1');
|
||||
expect(text).toContain('multi_simadmin_jobs{status="succeeded"} 0');
|
||||
expect(text).toContain('multi_simadmin_scheduled_tasks{status="armed"} 0');
|
||||
expect(text).toContain('multi_simadmin_build_info{version="1.9.6"} 1');
|
||||
// A node with no probe history is counted rather than silently missing.
|
||||
expect(text).toContain('multi_simadmin_node_unprobed 1');
|
||||
});
|
||||
|
||||
it('reports the identity guard for each node it has seen', () => {
|
||||
const { db: database, identities, metrics } = fixture();
|
||||
seedNode(database, 'node-a', 'A');
|
||||
seedNode(database, 'node-b', 'B');
|
||||
const report = { imei: '860000000000009', model: 'RM500Q' };
|
||||
identities.observe('node-a', report);
|
||||
identities.observe('node-b', report);
|
||||
|
||||
const text = metrics.render();
|
||||
expect(text).toContain('multi_simadmin_node_identity_pending{node="node-a"} 1');
|
||||
expect(text).toContain('multi_simadmin_node_identity_pending{node="node-b"} 1');
|
||||
expect(text).toContain('multi_simadmin_identity_pending_total 2');
|
||||
});
|
||||
|
||||
it('never emits a node name, so a hostile name cannot forge a series', () => {
|
||||
const { db: database, metrics } = fixture();
|
||||
seedNode(database, 'node-a', 'evil" name\nwith break');
|
||||
|
||||
const text = metrics.render();
|
||||
// Names are excluded from the session-free exposition outright; escaping a
|
||||
// label is not the right tool for untrusted operator input.
|
||||
expect(text).not.toContain('evil"');
|
||||
expect(text.split('\n').filter((line) => line.includes('with break'))).toHaveLength(0);
|
||||
});
|
||||
|
||||
it('emits a valid exposition: help and type precede every series', () => {
|
||||
const { db: database, metrics } = fixture();
|
||||
seedNode(database, 'node-a', 'A');
|
||||
const lines = metrics.render().trimEnd().split('\n');
|
||||
const names = new Set<string>();
|
||||
for (const line of lines) {
|
||||
if (line.startsWith('# ')) {
|
||||
const [, kind, name] = line.split(' ');
|
||||
if (kind === 'TYPE') expect(names.has(name ?? ''), name).toBe(false);
|
||||
continue;
|
||||
}
|
||||
const name = line.slice(0, line.search(/[{ ]/u));
|
||||
expect(line, name).toMatch(/^[a-z_][a-z0-9_]*(\{[^{}]*\})? [-0-9.e+]+$/u);
|
||||
names.add(name);
|
||||
}
|
||||
expect(names.size).toBeGreaterThan(10);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,244 @@
|
||||
import type Database from 'better-sqlite3';
|
||||
|
||||
import type { ConnectionSummary } from '../system/connection-log-service.js';
|
||||
import type { DeviceIdentity } from '../identity/device-identity-service.js';
|
||||
|
||||
/**
|
||||
* Prometheus exposition for the fused control plane.
|
||||
*
|
||||
* The scrape must never reach a device: every value here comes from rows the console already
|
||||
* writes (instances, probes, queues, jobs, identity records). A monitoring system polling every
|
||||
* 15 seconds should not be able to wake a modem.
|
||||
*/
|
||||
|
||||
const PREFIX = 'multi_simadmin';
|
||||
|
||||
export interface MetricsSource {
|
||||
readonly connections: { summarize(): readonly ConnectionSummary[] };
|
||||
readonly identities: { list(): readonly DeviceIdentity[] };
|
||||
}
|
||||
|
||||
export interface MetricsOptions {
|
||||
readonly db: Database.Database;
|
||||
readonly version: string;
|
||||
readonly sources: MetricsSource;
|
||||
readonly now?: () => Date;
|
||||
}
|
||||
|
||||
interface CountRow {
|
||||
readonly status: string;
|
||||
readonly count: number;
|
||||
}
|
||||
|
||||
/** Label values are operator- or device-controlled, so they are escaped the Prometheus way. */
|
||||
function label(value: string): string {
|
||||
return value.replace(/\\/gu, '\\\\').replace(/"/gu, '\\"').replace(/\n/gu, '\\n');
|
||||
}
|
||||
|
||||
const number = (value: unknown): number => {
|
||||
const parsed = typeof value === 'number' ? value : Number(value);
|
||||
return Number.isFinite(parsed) ? parsed : 0;
|
||||
};
|
||||
|
||||
function sample(name: string, help: string, type: 'gauge' | 'counter', lines: readonly string[]) {
|
||||
return [`# HELP ${name} ${help}`, `# TYPE ${name} ${type}`, ...lines];
|
||||
}
|
||||
|
||||
export class MetricsService {
|
||||
readonly #db: Database.Database;
|
||||
readonly #version: string;
|
||||
readonly #sources: MetricsSource;
|
||||
readonly #clock: () => Date;
|
||||
|
||||
constructor(options: MetricsOptions) {
|
||||
this.#db = options.db;
|
||||
this.#version = options.version;
|
||||
this.#sources = options.sources;
|
||||
this.#clock = options.now ?? (() => new Date());
|
||||
}
|
||||
|
||||
render(): string {
|
||||
const nodes = this.#db
|
||||
.prepare('SELECT id,name,enabled FROM instances ORDER BY name, id')
|
||||
.all() as Array<{
|
||||
id: string;
|
||||
name: string;
|
||||
enabled: number;
|
||||
}>;
|
||||
const ids = new Set(nodes.map((node) => node.id));
|
||||
const lines: string[] = [];
|
||||
|
||||
lines.push(
|
||||
...sample(`${PREFIX}_build_info`, 'Build identity of the control plane.', 'gauge', [
|
||||
`${PREFIX}_build_info{version="${label(this.#version)}"} 1`,
|
||||
]),
|
||||
...sample(`${PREFIX}_scrape_timestamp_seconds`, 'Wall-clock time of this scrape.', 'gauge', [
|
||||
`${PREFIX}_scrape_timestamp_seconds ${Math.floor(this.#clock().getTime() / 1000)}`,
|
||||
]),
|
||||
);
|
||||
|
||||
// Operator-assigned names can leak hostnames/locations to unauthenticated
|
||||
// scrapers; the series intentionally stays opaque to node ids.
|
||||
const info = nodes.map((node) => `${PREFIX}_node_info{node="${label(node.id)}"} 1`);
|
||||
lines.push(
|
||||
...sample(`${PREFIX}_node_info`, 'Registered nodes and their addresses.', 'gauge', info),
|
||||
...sample(`${PREFIX}_nodes_total`, 'Number of registered nodes.', 'gauge', [
|
||||
`${PREFIX}_nodes_total ${nodes.length}`,
|
||||
]),
|
||||
...sample(
|
||||
`${PREFIX}_node_enabled`,
|
||||
'Whether the node is enabled (1) or paused (0).',
|
||||
'gauge',
|
||||
nodes.map(
|
||||
(node) => `${PREFIX}_node_enabled{node="${label(node.id)}"} ${node.enabled ? 1 : 0}`,
|
||||
),
|
||||
),
|
||||
);
|
||||
|
||||
const probes = this.#sources.connections.summarize();
|
||||
lines.push(
|
||||
...sample(
|
||||
`${PREFIX}_node_probe_total`,
|
||||
'Health probes recorded in the retained window, by outcome.',
|
||||
'counter',
|
||||
probes.flatMap((probe) => [
|
||||
`${PREFIX}_node_probe_total{node="${label(probe.instanceId)}",outcome="success"} ${probe.success}`,
|
||||
`${PREFIX}_node_probe_total{node="${label(probe.instanceId)}",outcome="failed"} ${probe.failed}`,
|
||||
]),
|
||||
),
|
||||
...sample(
|
||||
`${PREFIX}_node_availability_ratio`,
|
||||
'Share of successful probes in the retained window.',
|
||||
'gauge',
|
||||
probes.map(
|
||||
(probe) =>
|
||||
`${PREFIX}_node_availability_ratio{node="${label(probe.instanceId)}"} ${(probe.availabilityPercent / 100).toFixed(3)}`,
|
||||
),
|
||||
),
|
||||
...sample(
|
||||
`${PREFIX}_node_probe_duration_ms`,
|
||||
'Average probe latency in the retained window.',
|
||||
'gauge',
|
||||
probes.map(
|
||||
(probe) =>
|
||||
`${PREFIX}_node_probe_duration_ms{node="${label(probe.instanceId)}"} ${number(probe.averageDurationMs)}`,
|
||||
),
|
||||
),
|
||||
);
|
||||
|
||||
const identities = this.#sources.identities.list();
|
||||
const pending = identities.filter((identity) => identity.status === 'pending');
|
||||
lines.push(
|
||||
...sample(
|
||||
`${PREFIX}_node_identity_tracked`,
|
||||
'Whether the node has ever reported its hardware identity.',
|
||||
'gauge',
|
||||
identities.map(
|
||||
(identity) => `${PREFIX}_node_identity_tracked{node="${label(identity.instanceId)}"} 1`,
|
||||
),
|
||||
),
|
||||
...sample(
|
||||
`${PREFIX}_node_identity_pending`,
|
||||
'1 while the identity guard holds the node and control actions are refused.',
|
||||
'gauge',
|
||||
identities.map(
|
||||
(identity) =>
|
||||
`${PREFIX}_node_identity_pending{node="${label(identity.instanceId)}"} ${identity.status === 'pending' ? 1 : 0}`,
|
||||
),
|
||||
),
|
||||
...sample(
|
||||
`${PREFIX}_identity_pending_total`,
|
||||
'Nodes waiting for an operator to confirm the device behind them.',
|
||||
'gauge',
|
||||
[`${PREFIX}_identity_pending_total ${pending.length}`],
|
||||
),
|
||||
);
|
||||
|
||||
lines.push(
|
||||
...this.#statusMetric(
|
||||
`${PREFIX}_sms_outbox`,
|
||||
'Queued cross-node SMS sends, by state.',
|
||||
'SELECT status, COUNT(*) AS count FROM sms_outbox GROUP BY status',
|
||||
['queued', 'sending', 'sent', 'failed', 'cancelled'],
|
||||
),
|
||||
...this.#statusMetric(
|
||||
`${PREFIX}_notification_queue`,
|
||||
'Central notification queue, by state.',
|
||||
'SELECT status, COUNT(*) AS count FROM notification_queue GROUP BY status',
|
||||
['pending', 'sending', 'succeeded', 'failed', 'cancelled'],
|
||||
),
|
||||
...this.#statusMetric(
|
||||
`${PREFIX}_jobs`,
|
||||
'Fleet operations, by state.',
|
||||
'SELECT status, COUNT(*) AS count FROM jobs GROUP BY status',
|
||||
[
|
||||
'queued',
|
||||
'running',
|
||||
'cancelling',
|
||||
'succeeded',
|
||||
'partially-succeeded',
|
||||
'failed',
|
||||
'cancelled',
|
||||
'unknown-result',
|
||||
],
|
||||
),
|
||||
...this.#statusMetric(
|
||||
`${PREFIX}_scheduled_tasks`,
|
||||
'Automation tasks, by state.',
|
||||
`SELECT CASE WHEN enabled=1 THEN 'armed' ELSE 'paused' END AS status,
|
||||
COUNT(*) AS count FROM scheduled_tasks GROUP BY enabled`,
|
||||
['armed', 'paused'],
|
||||
),
|
||||
);
|
||||
|
||||
const groups = this.#count('SELECT COUNT(*) AS count FROM device_groups');
|
||||
const tags = this.#count('SELECT COUNT(*) AS count FROM tag_registry');
|
||||
const messages = this.#count('SELECT COUNT(*) AS count FROM sms_messages');
|
||||
const probed = new Set(probes.map((probe) => probe.instanceId));
|
||||
const unprobed = [...ids].filter((id) => !probed.has(id)).length;
|
||||
lines.push(
|
||||
...sample(`${PREFIX}_device_groups`, 'Node groups.', 'gauge', [
|
||||
`${PREFIX}_device_groups ${groups}`,
|
||||
]),
|
||||
...sample(`${PREFIX}_device_tags`, 'Labels in the tag registry.', 'gauge', [
|
||||
`${PREFIX}_device_tags ${tags}`,
|
||||
]),
|
||||
...sample(
|
||||
`${PREFIX}_sms_messages`,
|
||||
'SMS rows synchronised into the central message centre.',
|
||||
'gauge',
|
||||
[`${PREFIX}_sms_messages ${messages}`],
|
||||
),
|
||||
...sample(
|
||||
`${PREFIX}_node_unprobed`,
|
||||
'Registered nodes without a health probe in the retained window.',
|
||||
'gauge',
|
||||
[`${PREFIX}_node_unprobed ${unprobed}`],
|
||||
),
|
||||
);
|
||||
|
||||
return `${lines.join('\n')}\n`;
|
||||
}
|
||||
|
||||
#count(sql: string): number {
|
||||
const row = this.#db.prepare(sql).get() as { count?: number } | undefined;
|
||||
return number(row?.count ?? 0);
|
||||
}
|
||||
|
||||
/** Every state is emitted, including the empty ones, so a query never sees a series vanish. */
|
||||
#statusMetric(
|
||||
name: string,
|
||||
help: string,
|
||||
sql: string,
|
||||
states: readonly string[],
|
||||
): readonly string[] {
|
||||
const rows = this.#db.prepare(sql).all() as readonly CountRow[];
|
||||
const counts = new Map(rows.map((row) => [String(row.status), number(row.count)]));
|
||||
return sample(
|
||||
name,
|
||||
help,
|
||||
'gauge',
|
||||
states.map((state) => `${name}{status="${label(state)}"} ${counts.get(state) ?? 0}`),
|
||||
);
|
||||
}
|
||||
}
|
||||
@@ -6,11 +6,13 @@ import { migrateDatabase } from '../../infrastructure/database/migrations.js';
|
||||
import type { SecretStore } from '../../infrastructure/secrets/secret-store.js';
|
||||
import {
|
||||
DELETE_INSTANCE_PARAMETER_SCHEMA_ID,
|
||||
type BindingRelease,
|
||||
DeleteInstanceOperation,
|
||||
DeleteInstanceOperationError,
|
||||
} from './delete-instance-operation.js';
|
||||
|
||||
class Store implements SecretStore {
|
||||
readonly provider = 'macos-keychain';
|
||||
async set() {
|
||||
return '';
|
||||
}
|
||||
@@ -26,7 +28,9 @@ afterEach(() => {
|
||||
for (const database of databases.splice(0)) database.close();
|
||||
});
|
||||
|
||||
function fixture() {
|
||||
function fixture(
|
||||
releaseBinding?: (instanceId: string, requestId: string) => Promise<BindingRelease>,
|
||||
) {
|
||||
const db = new Database(':memory:');
|
||||
db.pragma('foreign_keys=ON');
|
||||
migrateDatabase(db);
|
||||
@@ -43,6 +47,7 @@ function fixture() {
|
||||
db,
|
||||
instances,
|
||||
now,
|
||||
...(releaseBinding === undefined ? {} : { releaseBinding }),
|
||||
idFactory: () => `operation-${++sequence}`,
|
||||
tokenFactory: () => Buffer.alloc(32, 7).toString('base64url'),
|
||||
});
|
||||
@@ -396,4 +401,90 @@ describe('DeleteInstanceOperation', () => {
|
||||
).rejects.toMatchObject({ code: 'CONFIRMATION_INVALID' });
|
||||
expect(db.prepare('SELECT COUNT(*) count FROM jobs').get()).toEqual({ count: 0 });
|
||||
});
|
||||
|
||||
it('keeps the node when the device refuses to release its binding', async () => {
|
||||
const calls: string[] = [];
|
||||
const { instances, operation } = fixture(async (instanceId, requestId) => {
|
||||
calls.push(`${instanceId}:${requestId}`);
|
||||
return { status: 'failed' };
|
||||
});
|
||||
const instance = await instances.create({ name: 'A', origin: 'http://192.168.1.10' });
|
||||
const prepared = await operation.prepare({
|
||||
operationId: 'deleteInstance',
|
||||
targets: [{ instanceId: instance.id, revision: 1 }],
|
||||
parameters: { parameterSchemaId: DELETE_INSTANCE_PARAMETER_SCHEMA_ID, fields: [] },
|
||||
});
|
||||
const job = await operation.executeDelete({
|
||||
instanceId: instance.id,
|
||||
revision: 1,
|
||||
preparationId: prepared.id,
|
||||
confirmationToken: prepared.confirmationToken,
|
||||
actor: 'loopback-control-plane',
|
||||
requestId: 'request-refused',
|
||||
});
|
||||
expect(calls).toEqual([`${instance.id}:request-refused`]);
|
||||
expect(job.status).toBe('failed');
|
||||
expect(job.items).toEqual([
|
||||
{
|
||||
id: expect.any(String),
|
||||
targetId: instance.id,
|
||||
state: 'failed',
|
||||
error: {
|
||||
type: 'about:blank',
|
||||
title: 'Job item failed',
|
||||
status: 502,
|
||||
detail: 'The job item did not complete successfully.',
|
||||
code: 'UNBIND_FAILED',
|
||||
requestId: 'request-refused',
|
||||
},
|
||||
},
|
||||
]);
|
||||
expect(await instances.get(instance.id)).toBeDefined();
|
||||
});
|
||||
|
||||
it('forgets a node that never answered, exactly as an offline device is dropped', async () => {
|
||||
const { instances, operation } = fixture(async () => ({ status: 'skipped' }));
|
||||
const instance = await instances.create({ name: 'A', origin: 'http://192.168.1.10' });
|
||||
const prepared = await operation.prepare({
|
||||
operationId: 'deleteInstance',
|
||||
targets: [{ instanceId: instance.id, revision: 1 }],
|
||||
parameters: { parameterSchemaId: DELETE_INSTANCE_PARAMETER_SCHEMA_ID, fields: [] },
|
||||
});
|
||||
const job = await operation.executeDelete({
|
||||
instanceId: instance.id,
|
||||
revision: 1,
|
||||
preparationId: prepared.id,
|
||||
confirmationToken: prepared.confirmationToken,
|
||||
actor: 'loopback-control-plane',
|
||||
requestId: 'request-offline',
|
||||
});
|
||||
expect(job.status).toBe('succeeded');
|
||||
expect(job.items[0]).not.toHaveProperty('error');
|
||||
expect(await instances.get(instance.id)).toBeUndefined();
|
||||
});
|
||||
|
||||
it('treats a transport that throws while the node was online as a refusal', async () => {
|
||||
const { db, instances, operation } = fixture(async () => {
|
||||
throw new Error('ECONNREFUSED');
|
||||
});
|
||||
const instance = await instances.create({ name: 'A', origin: 'http://192.168.1.10' });
|
||||
const prepared = await operation.prepare({
|
||||
operationId: 'deleteInstance',
|
||||
targets: [{ instanceId: instance.id, revision: 1 }],
|
||||
parameters: { parameterSchemaId: DELETE_INSTANCE_PARAMETER_SCHEMA_ID, fields: [] },
|
||||
});
|
||||
const job = await operation.executeDelete({
|
||||
instanceId: instance.id,
|
||||
revision: 1,
|
||||
preparationId: prepared.id,
|
||||
confirmationToken: prepared.confirmationToken,
|
||||
actor: 'loopback-control-plane',
|
||||
requestId: 'request-thrown',
|
||||
});
|
||||
expect(job.items[0]).toMatchObject({ state: 'failed', error: { code: 'UNBIND_FAILED' } });
|
||||
expect(db.prepare('SELECT result_code FROM job_items').get()).toEqual({
|
||||
result_code: 'UNBIND_FAILED',
|
||||
});
|
||||
expect(await instances.get(instance.id)).toBeDefined();
|
||||
});
|
||||
});
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
import type Database from 'better-sqlite3';
|
||||
import type { EventEnvelope } from '../events/event-journal.js';
|
||||
import { createHash, randomBytes, randomUUID, timingSafeEqual } from 'node:crypto';
|
||||
import type { Job, Preparation, PrepareOperationRequest } from '@multi-simadmin/contracts';
|
||||
import { InstanceService, InstanceServiceError } from '../instances/instance-service.js';
|
||||
@@ -6,6 +7,8 @@ import { InstanceService, InstanceServiceError } from '../instances/instance-ser
|
||||
export const DELETE_INSTANCE_PARAMETER_SCHEMA_ID = 'deleteInstance.parameters.v1' as const;
|
||||
const OPERATION_ID = 'deleteInstance';
|
||||
const ACTOR = 'loopback-control-plane';
|
||||
/** Job item code for "the device still holds its central binding", surfaced to the console. */
|
||||
export const BINDING_REFUSAL_CODE = 'UNBIND_FAILED' as const;
|
||||
const TTL_MS = 5 * 60 * 1000;
|
||||
const PARAMETERS_DIGEST = createHash('sha256')
|
||||
.update(JSON.stringify({ parameterSchemaId: DELETE_INSTANCE_PARAMETER_SCHEMA_ID, fields: [] }))
|
||||
@@ -53,12 +56,28 @@ interface ExecuteInput {
|
||||
readonly actor: string;
|
||||
readonly requestId: string;
|
||||
}
|
||||
|
||||
/**
|
||||
* Answer to "release the central binding on this device before we forget it".
|
||||
* `skipped` means the node never answered a heartbeat, so there is nobody to notify and the
|
||||
* console deletes its own record; the device re-binds if it ever comes back. `failed` keeps the
|
||||
* record, because a device that refused or timed out must not be silently orphaned.
|
||||
*/
|
||||
export type BindingRelease =
|
||||
| { readonly status: 'released' }
|
||||
| { readonly status: 'skipped' }
|
||||
| { readonly status: 'failed' };
|
||||
|
||||
interface Options {
|
||||
readonly db: Database.Database;
|
||||
readonly instances: InstanceService;
|
||||
/** Optional so a control plane without a device transport can still delete. */
|
||||
readonly releaseBinding?: (instanceId: string, requestId: string) => Promise<BindingRelease>;
|
||||
readonly now?: () => Date;
|
||||
readonly idFactory?: () => string;
|
||||
readonly tokenFactory?: () => string;
|
||||
/** Receives job/instance envelopes so the SSE journal can invalidate fleet views. */
|
||||
readonly emit?: (envelope: EventEnvelope) => void;
|
||||
}
|
||||
|
||||
const digest = (value: string): string => createHash('sha256').update(value, 'utf8').digest('hex');
|
||||
@@ -73,16 +92,20 @@ const validRevision = (value: unknown): value is number =>
|
||||
export class DeleteInstanceOperation {
|
||||
private readonly db: Database.Database;
|
||||
private readonly instances: InstanceService;
|
||||
private readonly releaseBinding: Options['releaseBinding'];
|
||||
private readonly clock: () => Date;
|
||||
private readonly id: () => string;
|
||||
private readonly token: () => string;
|
||||
private readonly emit: Options['emit'];
|
||||
|
||||
constructor(options: Options) {
|
||||
this.db = options.db;
|
||||
this.instances = options.instances;
|
||||
this.releaseBinding = options.releaseBinding;
|
||||
this.clock = options.now ?? (() => new Date());
|
||||
this.id = options.idFactory ?? randomUUID;
|
||||
this.token = options.tokenFactory ?? (() => randomBytes(32).toString('base64url'));
|
||||
this.emit = options.emit;
|
||||
}
|
||||
|
||||
async prepare(input: PrepareOperationRequest, requestId = this.id()): Promise<Preparation> {
|
||||
@@ -294,6 +317,15 @@ export class DeleteInstanceOperation {
|
||||
'Confirmation could not be accepted',
|
||||
);
|
||||
|
||||
// An online device is told to release its central binding first, and a refusal keeps the
|
||||
// record so the operator can retry instead of losing sight of a device that is still
|
||||
// reporting to a hub. Only a node that cannot be reached at all is forgotten locally.
|
||||
const release = await this.releaseBindingPort(input.instanceId, input.requestId);
|
||||
if (release?.status === 'failed') {
|
||||
this.closeAsFailed(ids, input.requestId, BINDING_REFUSAL_CODE);
|
||||
this.#emitJob(ids.job, input.requestId);
|
||||
return this.job(ids.job);
|
||||
}
|
||||
try {
|
||||
await this.instances.delete(input.instanceId, input.revision, { allowJobHistory: true });
|
||||
const finished = this.clock().toISOString();
|
||||
@@ -316,6 +348,16 @@ export class DeleteInstanceOperation {
|
||||
)
|
||||
.run(finished, finished, ids.job);
|
||||
})();
|
||||
// Events only after the terminal rows are committed, so a subscriber that
|
||||
// refetches on the notification sees the finished job, not 'running'.
|
||||
this.#emit({
|
||||
kind: 'instance',
|
||||
id: this.id(),
|
||||
occurredAt: this.clock().toISOString(),
|
||||
requestId: input.requestId,
|
||||
instanceId: input.instanceId,
|
||||
});
|
||||
this.#emitJob(ids.job, input.requestId);
|
||||
} catch (error) {
|
||||
const finished = this.clock().toISOString();
|
||||
const state = error instanceof InstanceServiceError ? 'failed' : 'unknown-result';
|
||||
@@ -335,17 +377,85 @@ export class DeleteInstanceOperation {
|
||||
.prepare('UPDATE jobs SET status=?,finished_at=?,updated_at=? WHERE id=?')
|
||||
.run(state, finished, finished, ids.job);
|
||||
})();
|
||||
this.#emitJob(ids.job, input.requestId);
|
||||
}
|
||||
return this.job(ids.job);
|
||||
}
|
||||
|
||||
/** A transport that throws while the node was reachable counts as a refusal, never a pass. */
|
||||
private async releaseBindingPort(
|
||||
instanceId: string,
|
||||
requestId: string,
|
||||
): Promise<BindingRelease | undefined> {
|
||||
if (!this.releaseBinding) return undefined;
|
||||
try {
|
||||
return await this.releaseBinding(instanceId, requestId);
|
||||
} catch {
|
||||
return { status: 'failed' };
|
||||
}
|
||||
}
|
||||
|
||||
/** Closes the job without touching the instance row, so the device stays listed and retryable. */
|
||||
#emit(envelope: EventEnvelope): void {
|
||||
if (!this.emit) return;
|
||||
try {
|
||||
this.emit(envelope);
|
||||
} catch {
|
||||
// A journal failure must never fail the operation itself.
|
||||
}
|
||||
}
|
||||
|
||||
#emitJob(jobId: string, requestId: string): void {
|
||||
this.#emit({
|
||||
kind: 'job',
|
||||
id: this.id(),
|
||||
occurredAt: this.clock().toISOString(),
|
||||
requestId,
|
||||
jobId,
|
||||
});
|
||||
}
|
||||
|
||||
private closeAsFailed(
|
||||
ids: { job: string; item: string; attempt: string },
|
||||
requestId: string,
|
||||
resultCode: string,
|
||||
): void {
|
||||
const finished = this.clock().toISOString();
|
||||
this.db.transaction(() => {
|
||||
this.db
|
||||
.prepare(
|
||||
`UPDATE job_items
|
||||
SET status='failed',result_code=?,error_json=?,finished_at=?,updated_at=?
|
||||
WHERE id=? AND status='running'`,
|
||||
)
|
||||
.run(
|
||||
resultCode,
|
||||
JSON.stringify({ status: 502, code: resultCode, requestId }),
|
||||
finished,
|
||||
finished,
|
||||
ids.item,
|
||||
);
|
||||
this.db
|
||||
.prepare(
|
||||
"UPDATE job_attempts SET status='failed',finished_at=? WHERE id=? AND status='running'",
|
||||
)
|
||||
.run(finished, ids.attempt);
|
||||
this.db
|
||||
.prepare(
|
||||
"UPDATE jobs SET status='failed',finished_at=?,updated_at=? WHERE id=? AND status='running'",
|
||||
)
|
||||
.run(finished, finished, ids.job);
|
||||
})();
|
||||
}
|
||||
|
||||
reconcileInterruptedJobs(): number {
|
||||
const finished = this.clock().toISOString();
|
||||
return this.db.transaction(() => {
|
||||
const jobs = this.db
|
||||
.prepare("SELECT id FROM jobs WHERE operation_id=? AND status='running'")
|
||||
.all(OPERATION_ID) as Array<{ id: string }>;
|
||||
for (const { id } of jobs) {
|
||||
.prepare("SELECT id, request_id FROM jobs WHERE operation_id=? AND status='running'")
|
||||
.all(OPERATION_ID) as Array<{ id: string; request_id: string }>;
|
||||
for (const { id, request_id: requestId } of jobs) {
|
||||
this.#emitJob(id, requestId);
|
||||
this.db
|
||||
.prepare(
|
||||
"UPDATE job_items SET status='unknown-result',result_code='INTERRUPTED',finished_at=?,updated_at=? WHERE job_id=? AND status='running'",
|
||||
@@ -369,7 +479,7 @@ export class DeleteInstanceOperation {
|
||||
private job(id: string): Job {
|
||||
const row = this.db
|
||||
.prepare(
|
||||
'SELECT operation_id,status,root_job_id,retry_of_job_id,created_at FROM jobs WHERE id=?',
|
||||
'SELECT operation_id,status,root_job_id,retry_of_job_id,created_at,request_id FROM jobs WHERE id=?',
|
||||
)
|
||||
.get(id) as {
|
||||
operation_id: string;
|
||||
@@ -377,13 +487,17 @@ export class DeleteInstanceOperation {
|
||||
root_job_id: string;
|
||||
retry_of_job_id: string | null;
|
||||
created_at: string;
|
||||
request_id: string;
|
||||
};
|
||||
const items = this.db
|
||||
.prepare('SELECT id,instance_id,status FROM job_items WHERE job_id=? ORDER BY created_at,id')
|
||||
.prepare(
|
||||
'SELECT id,instance_id,status,result_code FROM job_items WHERE job_id=? ORDER BY created_at,id',
|
||||
)
|
||||
.all(id) as Array<{
|
||||
id: string;
|
||||
instance_id: string;
|
||||
status: 'succeeded' | 'failed' | 'unknown-result';
|
||||
result_code: string | null;
|
||||
}>;
|
||||
const attempts = this.db
|
||||
.prepare(
|
||||
@@ -401,7 +515,25 @@ export class DeleteInstanceOperation {
|
||||
status: row.status,
|
||||
...(row.retry_of_job_id ? { retryOfJobId: row.retry_of_job_id } : {}),
|
||||
rootJobId: row.root_job_id,
|
||||
items: items.map((item) => ({ id: item.id, targetId: item.instance_id, state: item.status })),
|
||||
items: items.map((item) => ({
|
||||
id: item.id,
|
||||
targetId: item.instance_id,
|
||||
state: item.status,
|
||||
// The console reads this code straight out of the execute response to explain why the
|
||||
// node it just confirmed is still in the list.
|
||||
...(item.result_code === BINDING_REFUSAL_CODE
|
||||
? {
|
||||
error: {
|
||||
type: 'about:blank',
|
||||
title: 'Job item failed',
|
||||
status: 502,
|
||||
detail: 'The job item did not complete successfully.',
|
||||
code: BINDING_REFUSAL_CODE,
|
||||
requestId: row.request_id,
|
||||
},
|
||||
}
|
||||
: {}),
|
||||
})),
|
||||
attempts: attempts.map((attempt) => ({
|
||||
id: attempt.id,
|
||||
state: attempt.status,
|
||||
|
||||
@@ -130,6 +130,7 @@ describe('SecureOperationExecution generic R2 slice', () => {
|
||||
query: '',
|
||||
contentType: '',
|
||||
body: undefined,
|
||||
instanceId: 'i-1',
|
||||
});
|
||||
await expectCode(
|
||||
execution.execute(
|
||||
@@ -244,10 +245,11 @@ describe('SecureOperationExecution R3 restart slice', () => {
|
||||
query: '',
|
||||
contentType: '',
|
||||
body: undefined,
|
||||
instanceId: 'i-1',
|
||||
});
|
||||
expect(db.prepare('SELECT risk_level FROM jobs WHERE id=?').get(job.id)).toEqual({
|
||||
risk_level: 'R3',
|
||||
});
|
||||
expect(
|
||||
db.prepare('SELECT risk_level FROM jobs WHERE id=?').get(job.id),
|
||||
).toEqual({ risk_level: 'R3' });
|
||||
});
|
||||
|
||||
it('prepares and executes system reboot only with fixed delay_seconds=3', async () => {
|
||||
@@ -283,6 +285,7 @@ describe('SecureOperationExecution R3 restart slice', () => {
|
||||
query: '',
|
||||
contentType: 'application/json',
|
||||
body: '{"delay_seconds":3}',
|
||||
instanceId: 'i-1',
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
@@ -1,5 +1,6 @@
|
||||
import type Database from 'better-sqlite3';
|
||||
import { createHash, randomBytes, randomUUID, timingSafeEqual } from 'node:crypto';
|
||||
import type { EventEnvelope } from '../events/event-journal.js';
|
||||
import type {
|
||||
ExecuteOperationRequest,
|
||||
Job,
|
||||
@@ -37,6 +38,15 @@ const EXECUTABLE_OPERATIONS: Readonly<Record<string, SecureOperationDescriptor>>
|
||||
requestContentType: 'none',
|
||||
parameterSchemaId: 'simadmin.58e2204.postServiceRestart.parameters.v1',
|
||||
}),
|
||||
postBasebandRestart: Object.freeze({
|
||||
operationId: 'postBasebandRestart',
|
||||
title: 'Restart Baseband',
|
||||
riskLevel: 'R3',
|
||||
method: 'POST',
|
||||
pathTemplate: '/api/baseband/restart',
|
||||
requestContentType: 'none',
|
||||
parameterSchemaId: 'simadmin.58e2204.postBasebandRestart.parameters.v1',
|
||||
}),
|
||||
postSystemReboot: Object.freeze({
|
||||
operationId: 'postSystemReboot',
|
||||
title: 'Reboot System',
|
||||
@@ -56,7 +66,11 @@ export const secureOperationRegistry: SecureOperationRegistry = {
|
||||
},
|
||||
};
|
||||
|
||||
const ZERO_BODY_OPERATIONS = new Set(['postNetworkRegisterAuto', 'postServiceRestart']);
|
||||
const ZERO_BODY_OPERATIONS = new Set([
|
||||
'postNetworkRegisterAuto',
|
||||
'postServiceRestart',
|
||||
'postBasebandRestart',
|
||||
]);
|
||||
const SYSTEM_REBOOT_OPERATION = 'postSystemReboot';
|
||||
const SYSTEM_REBOOT_DELAY_SECONDS = 3;
|
||||
const ACTOR = 'loopback-control-plane';
|
||||
@@ -71,6 +85,8 @@ export interface SafeOperationTransportRequest {
|
||||
readonly contentType: string;
|
||||
/** Serialized JSON body for audited JSON operations; undefined for zero-body POSTs. */
|
||||
readonly body: string | undefined;
|
||||
/** Bound target used only for origin-scoped session attachment; never logged. */
|
||||
readonly instanceId: string;
|
||||
}
|
||||
export interface SafeOperationTransport {
|
||||
request(request: SafeOperationTransportRequest): Promise<{ readonly status: number }>;
|
||||
@@ -106,6 +122,8 @@ interface Options {
|
||||
readonly idFactory?: () => string;
|
||||
readonly tokenFactory?: () => string;
|
||||
readonly nonceFactory?: () => string;
|
||||
/** Receives job terminal transitions so the SSE journal has a live producer. */
|
||||
readonly emit?: (envelope: EventEnvelope) => void;
|
||||
}
|
||||
interface PreparationRow {
|
||||
operation_id: string;
|
||||
@@ -199,7 +217,8 @@ export class SecureOperationExecution {
|
||||
let contentType = '';
|
||||
let serializedBody: string | undefined;
|
||||
if (ZERO_BODY_OPERATIONS.has(descriptor.operationId)) {
|
||||
if (descriptor.requestContentType !== 'none' || parameters.fields.length !== 0) this.validation();
|
||||
if (descriptor.requestContentType !== 'none' || parameters.fields.length !== 0)
|
||||
this.validation();
|
||||
} else if (descriptor.operationId === SYSTEM_REBOOT_OPERATION) {
|
||||
if (descriptor.requestContentType !== 'application/json' || parameters.fields.length !== 1)
|
||||
this.validation();
|
||||
@@ -412,6 +431,7 @@ export class SecureOperationExecution {
|
||||
bound.operation_id === SYSTEM_REBOOT_OPERATION
|
||||
? JSON.stringify({ delay_seconds: SYSTEM_REBOOT_DELAY_SECONDS })
|
||||
: undefined,
|
||||
instanceId: bound.target_instance_id,
|
||||
});
|
||||
state = response.status >= 200 && response.status < 300 ? 'succeeded' : 'failed';
|
||||
code = state === 'succeeded' ? 'UPSTREAM_SUCCEEDED' : 'UPSTREAM_REJECTED';
|
||||
@@ -425,18 +445,40 @@ export class SecureOperationExecution {
|
||||
}
|
||||
}
|
||||
this.finish(ids, state, code);
|
||||
this.#emitTerminal(ids.job, state, requestId);
|
||||
return this.job(ids.job);
|
||||
}
|
||||
|
||||
/**
|
||||
* Retention for consumed/expired/invalidated preparations. Every prepare
|
||||
* (manual or a scheduled retry attempt) inserts a row, so without this the
|
||||
* table grows one row per attempt forever.
|
||||
*/
|
||||
pruneTerminalPreparations(olderThanMs: number): number {
|
||||
if (!Number.isSafeInteger(olderThanMs) || olderThanMs < 0)
|
||||
throw new RangeError('olderThanMs must be a non-negative integer');
|
||||
const cutoff = new Date(this.clock().getTime() - olderThanMs).toISOString();
|
||||
return Number(
|
||||
this.options.db
|
||||
.prepare(
|
||||
"DELETE FROM operation_preparations WHERE status IN ('consumed','expired','invalidated') AND updated_at <= ?",
|
||||
)
|
||||
.run(cutoff).changes,
|
||||
);
|
||||
}
|
||||
|
||||
reconcileInterruptedJobs(): number {
|
||||
const now = this.clock().toISOString();
|
||||
return this.options.db.transaction(() => {
|
||||
const jobs = this.options.db
|
||||
.prepare(
|
||||
"SELECT id FROM jobs WHERE operation_id IN ('postNetworkRegisterAuto','postServiceRestart','postSystemReboot') AND risk_level IN ('R2','R3') AND status='running'",
|
||||
"SELECT id FROM jobs WHERE operation_id IN ('postNetworkRegisterAuto','postServiceRestart','postBasebandRestart','postSystemReboot') AND risk_level IN ('R2','R3') AND status='running'",
|
||||
)
|
||||
.all() as Array<{ id: string }>;
|
||||
for (const row of jobs) this.finishByJob(row.id, now, 'unknown-result', 'INTERRUPTED');
|
||||
for (const row of jobs) {
|
||||
this.finishByJob(row.id, now, 'unknown-result', 'INTERRUPTED');
|
||||
this.#emitTerminal(row.id, 'unknown-result', now);
|
||||
}
|
||||
return jobs.length;
|
||||
})();
|
||||
}
|
||||
@@ -462,6 +504,21 @@ export class SecureOperationExecution {
|
||||
.run(state, now, now, ids.job);
|
||||
})();
|
||||
}
|
||||
#emitTerminal(jobId: string, state: string, requestId: string): void {
|
||||
if (!this.options.emit) return;
|
||||
try {
|
||||
this.options.emit({
|
||||
kind: 'job',
|
||||
id: this.id(),
|
||||
occurredAt: this.clock().toISOString(),
|
||||
requestId,
|
||||
jobId,
|
||||
});
|
||||
} catch {
|
||||
// A journal failure must never fail the operation itself.
|
||||
}
|
||||
}
|
||||
|
||||
private finishByJob(jobId: string, now: string, state: string, code: string): void {
|
||||
this.options.db
|
||||
.prepare(
|
||||
|
||||
@@ -0,0 +1,118 @@
|
||||
import Database from 'better-sqlite3';
|
||||
import { describe, expect, it } from 'vitest';
|
||||
|
||||
import { migrateDatabase } from '../../infrastructure/database/migrations.js';
|
||||
import {
|
||||
DeviceOrganizationError,
|
||||
DeviceOrganizationService,
|
||||
} from './device-organization-service.js';
|
||||
|
||||
function fixture(): { db: Database.Database; service: DeviceOrganizationService } {
|
||||
const db = new Database(':memory:');
|
||||
db.pragma('foreign_keys=ON');
|
||||
migrateDatabase(db);
|
||||
let counter = 0;
|
||||
const service = new DeviceOrganizationService({
|
||||
db,
|
||||
idFactory: () => `group-${++counter}`,
|
||||
now: () => new Date('2026-09-03T10:00:00.000Z'),
|
||||
});
|
||||
return { db, service };
|
||||
}
|
||||
|
||||
function codeOf(operation: () => unknown): string {
|
||||
try {
|
||||
operation();
|
||||
} catch (error) {
|
||||
return error instanceof DeviceOrganizationError ? error.code : 'UNEXPECTED';
|
||||
}
|
||||
return 'NO_ERROR';
|
||||
}
|
||||
|
||||
function insertInstance(db: Database.Database, id: string, name: string, groupId: string | null) {
|
||||
db.prepare(
|
||||
`INSERT INTO instances
|
||||
(id,name,base_url,auth_mode,enabled,config_revision,created_at,updated_at,group_id)
|
||||
VALUES (?,?,?,'password',1,1,'2026-09-03T09:00:00.000Z','2026-09-03T09:00:00.000Z',?)`,
|
||||
).run(id, name, `http://${id}:8080`, groupId);
|
||||
}
|
||||
|
||||
describe('DeviceOrganizationService', () => {
|
||||
it('creates, renames, and counts group members', () => {
|
||||
const { db, service } = fixture();
|
||||
const created = service.createGroup({ name: '总部', description: '办公区' });
|
||||
expect(created).toEqual({
|
||||
id: 'group-1',
|
||||
name: '总部',
|
||||
description: '办公区',
|
||||
deviceCount: 0,
|
||||
createdAt: '2026-09-03T10:00:00.000Z',
|
||||
updatedAt: '2026-09-03T10:00:00.000Z',
|
||||
});
|
||||
insertInstance(db, 'i-1', 'Alpha', 'group-1');
|
||||
expect(service.listGroups()[0]?.deviceCount).toBe(1);
|
||||
expect(service.updateGroup('group-1', { name: '上海总部' }).name).toBe('上海总部');
|
||||
expect(codeOf(() => service.createGroup({ name: '上海总部' }))).toBe('DUPLICATE_GROUP');
|
||||
service.deleteGroup('group-1');
|
||||
expect(service.listGroups()).toEqual([]);
|
||||
expect(
|
||||
(
|
||||
db.prepare('SELECT group_id FROM instances WHERE id=?').get('i-1') as {
|
||||
group_id: string | null;
|
||||
}
|
||||
).group_id,
|
||||
).toBeNull();
|
||||
});
|
||||
|
||||
it('reports missing groups instead of silently succeeding', () => {
|
||||
const { service } = fixture();
|
||||
expect(codeOf(() => service.updateGroup('nope', { name: 'x' }))).toBe('GROUP_NOT_FOUND');
|
||||
expect(codeOf(() => service.deleteGroup('nope'))).toBe('GROUP_NOT_FOUND');
|
||||
});
|
||||
|
||||
it('synchronizes the tag registry with device tags and keeps unused entries', () => {
|
||||
const { db, service } = fixture();
|
||||
insertInstance(db, 'i-1', 'Alpha', null);
|
||||
db.prepare('INSERT INTO instance_tags (instance_id,tag,created_at) VALUES (?,?,?)').run(
|
||||
'i-1',
|
||||
'office',
|
||||
'2026-09-03T09:30:00.000Z',
|
||||
);
|
||||
service.synchronizeTags();
|
||||
expect(service.listTags()).toEqual([
|
||||
{
|
||||
tag: 'office',
|
||||
color: '',
|
||||
deviceCount: 1,
|
||||
createdAt: '2026-09-03T09:30:00.000Z',
|
||||
updatedAt: '2026-09-03T10:00:00.000Z',
|
||||
},
|
||||
]);
|
||||
const tagged = service.createTag({ tag: 'spare', color: 'coral' });
|
||||
expect(tagged.color).toBe('coral');
|
||||
service.synchronizeTags();
|
||||
expect(service.listTags().map((tag) => tag.tag)).toEqual(['office', 'spare']);
|
||||
});
|
||||
|
||||
it('detaches a deleted tag from every device', () => {
|
||||
const { db, service } = fixture();
|
||||
insertInstance(db, 'i-1', 'Alpha', null);
|
||||
db.prepare('INSERT INTO instance_tags (instance_id,tag,created_at) VALUES (?,?,?)').run(
|
||||
'i-1',
|
||||
'lab',
|
||||
'2026-09-03T09:30:00.000Z',
|
||||
);
|
||||
service.synchronizeTags();
|
||||
service.deleteTag('lab');
|
||||
expect(service.listTags()).toEqual([]);
|
||||
expect(db.prepare('SELECT COUNT(*) count FROM instance_tags').get()).toEqual({ count: 0 });
|
||||
expect(codeOf(() => service.deleteTag('lab'))).toBe('TAG_NOT_FOUND');
|
||||
});
|
||||
|
||||
it('rejects blank and oversized names', () => {
|
||||
const { service } = fixture();
|
||||
expect(codeOf(() => service.createGroup({ name: ' ' }))).toBe('VALIDATION_FAILED');
|
||||
expect(codeOf(() => service.createGroup({ name: 'x'.repeat(101) }))).toBe('VALIDATION_FAILED');
|
||||
expect(codeOf(() => service.createTag({ tag: '' }))).toBe('VALIDATION_FAILED');
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,239 @@
|
||||
import { randomUUID } from 'node:crypto';
|
||||
|
||||
import type Database from 'better-sqlite3';
|
||||
|
||||
import type {
|
||||
DeviceGroup,
|
||||
DeviceGroupInput,
|
||||
DeviceGroupPatch,
|
||||
DeviceTag,
|
||||
DeviceTagInput,
|
||||
DeviceTagPatch,
|
||||
} from '@multi-simadmin/contracts';
|
||||
|
||||
export type OrganizationErrorCode =
|
||||
| 'VALIDATION_FAILED'
|
||||
| 'GROUP_NOT_FOUND'
|
||||
| 'TAG_NOT_FOUND'
|
||||
| 'DUPLICATE_GROUP'
|
||||
| 'DATABASE_FAILED';
|
||||
|
||||
export class DeviceOrganizationError extends Error {
|
||||
constructor(
|
||||
readonly code: OrganizationErrorCode,
|
||||
message: string,
|
||||
) {
|
||||
super(message);
|
||||
this.name = 'DeviceOrganizationError';
|
||||
}
|
||||
}
|
||||
|
||||
interface GroupRow {
|
||||
id: string;
|
||||
name: string;
|
||||
description: string;
|
||||
device_count: number;
|
||||
created_at: string;
|
||||
updated_at: string;
|
||||
}
|
||||
|
||||
interface TagRow {
|
||||
tag: string;
|
||||
color: string;
|
||||
device_count: number;
|
||||
created_at: string;
|
||||
updated_at: string;
|
||||
}
|
||||
|
||||
export interface DeviceOrganizationOptions {
|
||||
readonly db: Database.Database;
|
||||
readonly idFactory?: () => string;
|
||||
readonly now?: () => Date;
|
||||
}
|
||||
|
||||
const MAX_GROUPS = 200;
|
||||
const MAX_TAGS = 500;
|
||||
|
||||
function invalid(message: string): never {
|
||||
throw new DeviceOrganizationError('VALIDATION_FAILED', message);
|
||||
}
|
||||
|
||||
export class DeviceOrganizationService {
|
||||
readonly #db: Database.Database;
|
||||
readonly #id: () => string;
|
||||
readonly #clock: () => Date;
|
||||
|
||||
constructor(options: DeviceOrganizationOptions) {
|
||||
this.#db = options.db;
|
||||
this.#id = options.idFactory ?? randomUUID;
|
||||
this.#clock = options.now ?? (() => new Date());
|
||||
}
|
||||
|
||||
listGroups(): readonly DeviceGroup[] {
|
||||
const rows = this.#db
|
||||
.prepare(
|
||||
`SELECT g.id,g.name,g.description,g.created_at,g.updated_at,
|
||||
(SELECT COUNT(*) FROM instances i WHERE i.group_id=g.id) device_count
|
||||
FROM device_groups g ORDER BY g.name COLLATE NOCASE ASC`,
|
||||
)
|
||||
.all() as GroupRow[];
|
||||
return Object.freeze(
|
||||
rows.map((row) =>
|
||||
Object.freeze({
|
||||
id: row.id,
|
||||
name: row.name,
|
||||
description: row.description,
|
||||
deviceCount: row.device_count,
|
||||
createdAt: row.created_at,
|
||||
updatedAt: row.updated_at,
|
||||
}),
|
||||
),
|
||||
);
|
||||
}
|
||||
|
||||
getGroup(groupId: string): DeviceGroup | undefined {
|
||||
return this.listGroups().find((group) => group.id === groupId);
|
||||
}
|
||||
|
||||
createGroup(input: DeviceGroupInput): DeviceGroup {
|
||||
const name = requireText(input.name, '分组名称');
|
||||
if (
|
||||
(this.#db.prepare('SELECT COUNT(*) count FROM device_groups').get() as { count: number })
|
||||
.count >= MAX_GROUPS
|
||||
)
|
||||
invalid('分组数量已达上限');
|
||||
const description = optionalText(input.description ?? '', '分组说明', 500);
|
||||
const now = this.#clock().toISOString();
|
||||
const id = this.#id();
|
||||
try {
|
||||
this.#db
|
||||
.prepare(
|
||||
'INSERT INTO device_groups (id,name,description,created_at,updated_at) VALUES (?,?,?,?,?)',
|
||||
)
|
||||
.run(id, name, description, now, now);
|
||||
} catch (error) {
|
||||
if (isUnique(error)) throw new DeviceOrganizationError('DUPLICATE_GROUP', '分组名称已存在');
|
||||
throw new DeviceOrganizationError('DATABASE_FAILED', '无法创建分组');
|
||||
}
|
||||
return this.getGroup(id)!;
|
||||
}
|
||||
|
||||
updateGroup(groupId: string, patch: DeviceGroupPatch): DeviceGroup {
|
||||
const current = this.#db
|
||||
.prepare('SELECT id,name,description FROM device_groups WHERE id=?')
|
||||
.get(groupId) as { id: string; name: string; description: string } | undefined;
|
||||
if (!current) throw new DeviceOrganizationError('GROUP_NOT_FOUND', '分组不存在');
|
||||
const name = patch.name === undefined ? current.name : requireText(patch.name, '分组名称');
|
||||
const description =
|
||||
patch.description === undefined
|
||||
? current.description
|
||||
: optionalText(patch.description, '分组说明', 500);
|
||||
try {
|
||||
this.#db
|
||||
.prepare('UPDATE device_groups SET name=?,description=?,updated_at=? WHERE id=?')
|
||||
.run(name, description, this.#clock().toISOString(), groupId);
|
||||
} catch (error) {
|
||||
if (isUnique(error)) throw new DeviceOrganizationError('DUPLICATE_GROUP', '分组名称已存在');
|
||||
throw new DeviceOrganizationError('DATABASE_FAILED', '无法更新分组');
|
||||
}
|
||||
return this.getGroup(groupId)!;
|
||||
}
|
||||
|
||||
deleteGroup(groupId: string): void {
|
||||
const changed = this.#db.prepare('DELETE FROM device_groups WHERE id=?').run(groupId).changes;
|
||||
if (changed === 0) throw new DeviceOrganizationError('GROUP_NOT_FOUND', '分组不存在');
|
||||
}
|
||||
|
||||
listTags(): readonly DeviceTag[] {
|
||||
const rows = this.#db
|
||||
.prepare(
|
||||
`SELECT r.tag,r.color,r.created_at,r.updated_at,
|
||||
(SELECT COUNT(*) FROM instance_tags t WHERE t.tag=r.tag) device_count
|
||||
FROM tag_registry r ORDER BY r.tag COLLATE NOCASE ASC`,
|
||||
)
|
||||
.all() as TagRow[];
|
||||
return Object.freeze(
|
||||
rows.map((row) =>
|
||||
Object.freeze({
|
||||
tag: row.tag,
|
||||
color: row.color,
|
||||
deviceCount: row.device_count,
|
||||
createdAt: row.created_at,
|
||||
updatedAt: row.updated_at,
|
||||
}),
|
||||
),
|
||||
);
|
||||
}
|
||||
|
||||
createTag(input: DeviceTagInput): DeviceTag {
|
||||
const tag = requireText(input.tag, '标签');
|
||||
if (
|
||||
(this.#db.prepare('SELECT COUNT(*) count FROM tag_registry').get() as { count: number })
|
||||
.count >= MAX_TAGS
|
||||
)
|
||||
invalid('标签数量已达上限');
|
||||
const color = optionalText(input.color ?? '', '标签颜色', 32);
|
||||
const now = this.#clock().toISOString();
|
||||
try {
|
||||
this.#db
|
||||
.prepare('INSERT INTO tag_registry (tag,color,created_at,updated_at) VALUES (?,?,?,?)')
|
||||
.run(tag, color, now, now);
|
||||
} catch (error) {
|
||||
if (isUnique(error)) invalid('标签已存在');
|
||||
throw new DeviceOrganizationError('DATABASE_FAILED', '无法创建标签');
|
||||
}
|
||||
return this.listTags().find((entry) => entry.tag === tag)!;
|
||||
}
|
||||
|
||||
updateTag(tag: string, patch: DeviceTagPatch): DeviceTag {
|
||||
const existing = this.#db.prepare('SELECT tag FROM tag_registry WHERE tag=?').get(tag);
|
||||
if (!existing) throw new DeviceOrganizationError('TAG_NOT_FOUND', '标签不存在');
|
||||
const color = optionalText(patch.color ?? '', '标签颜色', 32);
|
||||
this.#db
|
||||
.prepare('UPDATE tag_registry SET color=?,updated_at=? WHERE tag=?')
|
||||
.run(color, this.#clock().toISOString(), tag);
|
||||
return this.listTags().find((entry) => entry.tag === tag)!;
|
||||
}
|
||||
|
||||
/** Removing a tag also detaches it from every device that carried it. */
|
||||
deleteTag(tag: string): void {
|
||||
const changed = this.#db.prepare('DELETE FROM tag_registry WHERE tag=?').run(tag).changes;
|
||||
if (changed === 0) throw new DeviceOrganizationError('TAG_NOT_FOUND', '标签不存在');
|
||||
this.#db.prepare('DELETE FROM instance_tags WHERE tag=?').run(tag);
|
||||
}
|
||||
|
||||
/**
|
||||
* Instances may carry free-form tags that were never registered. Adopt them so the
|
||||
* palette stays complete; unused registry entries are deliberate, so nothing is pruned.
|
||||
*/
|
||||
synchronizeTags(): void {
|
||||
const now = this.#clock().toISOString();
|
||||
this.#db
|
||||
.prepare(
|
||||
`INSERT INTO tag_registry (tag,color,created_at,updated_at)
|
||||
SELECT tag,'',MIN(created_at),? FROM instance_tags
|
||||
WHERE tag NOT IN (SELECT tag FROM tag_registry)
|
||||
GROUP BY tag`,
|
||||
)
|
||||
.run(now);
|
||||
}
|
||||
}
|
||||
|
||||
function requireText(value: unknown, field: string): string {
|
||||
if (typeof value !== 'string') invalid(`${field}必须是字符串`);
|
||||
const trimmed = value.trim();
|
||||
if (!trimmed) invalid(`${field}不能为空`);
|
||||
if (trimmed.length > 100) invalid(`${field}最多 100 个字符`);
|
||||
return trimmed;
|
||||
}
|
||||
|
||||
function optionalText(value: unknown, field: string, maximum: number): string {
|
||||
if (typeof value !== 'string') invalid(`${field}必须是字符串`);
|
||||
const trimmed = value.trim();
|
||||
if (trimmed.length > maximum) invalid(`${field}最多 ${maximum} 个字符`);
|
||||
return trimmed;
|
||||
}
|
||||
|
||||
function isUnique(error: unknown): boolean {
|
||||
return error instanceof Error && /UNIQUE constraint failed/i.test(error.message);
|
||||
}
|
||||
@@ -1,6 +1,16 @@
|
||||
import Database from 'better-sqlite3';
|
||||
import { describe, expect, it } from 'vitest';
|
||||
import { migrateDatabase } from '../../infrastructure/database/migrations.js';
|
||||
|
||||
import { InstanceResourceService, parseSim, parseStats } from './instance-resource-service.js';
|
||||
import {
|
||||
InstanceResourceService,
|
||||
parseDevice,
|
||||
parseHealth,
|
||||
parseNetwork,
|
||||
parseSim,
|
||||
parseStats,
|
||||
parseSignalStrength,
|
||||
} from './instance-resource-service.js';
|
||||
|
||||
const response = (body: unknown) => ({
|
||||
status: 200,
|
||||
@@ -27,6 +37,42 @@ describe('instance resource allowlist parsing', () => {
|
||||
).toEqual({ cpuPercent: 23.4, memoryPercent: 67.8, maxTemperatureCelsius: 52.6 });
|
||||
});
|
||||
|
||||
it('falls back to version metadata exposed by stats on older SimAdmin builds', () => {
|
||||
expect(
|
||||
parseStats(
|
||||
response({
|
||||
data: {
|
||||
cpu_load: { load_percent: 10 },
|
||||
system: { app_version: '1.8.7', architecture: 'aarch64' },
|
||||
},
|
||||
}),
|
||||
),
|
||||
).toMatchObject({ cpuPercent: 10, version: '1.8.7', platform: 'aarch64' });
|
||||
});
|
||||
|
||||
it('reads device uptime from the system block and rejects absurd values', () => {
|
||||
expect(
|
||||
parseStats(
|
||||
response({
|
||||
data: {
|
||||
system: { uptime_seconds: 93_784, boot_id: 'secret-boot' },
|
||||
},
|
||||
}),
|
||||
),
|
||||
).toEqual({ uptimeSeconds: 93_784 });
|
||||
expect(parseStats(response({ data: { uptime_seconds: 4_000_000_000 } }))).toEqual({});
|
||||
expect(parseStats(response({ data: { uptime_seconds: -5 } }))).toEqual({});
|
||||
expect(parseStats(response({ data: { uptime_seconds: '93784' } }))).toEqual({});
|
||||
});
|
||||
|
||||
it('extracts the upstream SimAdmin version from the health endpoint', () => {
|
||||
expect(
|
||||
parseHealth(
|
||||
response({ status: 'ok', version: '1.9.4', platform: 'linux-aarch64', secret: 'drop' }),
|
||||
),
|
||||
).toEqual({ version: '1.9.4', platform: 'linux-aarch64' });
|
||||
});
|
||||
|
||||
it('extracts, validates, deduplicates and bounds phone numbers only', () => {
|
||||
expect(
|
||||
parseSim(
|
||||
@@ -38,10 +84,242 @@ describe('instance resource allowlist parsing', () => {
|
||||
},
|
||||
}),
|
||||
),
|
||||
).toEqual({ phoneNumbers: ['+86 138-0000-0000'] });
|
||||
).toEqual({ simPresent: true, phoneNumbers: ['+86 138-0000-0000'] });
|
||||
});
|
||||
|
||||
it('extracts device link state without exposing hardware identity', () => {
|
||||
expect(
|
||||
parseDevice(response({ data: { imei: 'secret', online: true, powered: true } })),
|
||||
).toEqual({ hardwareOnline: true, controlOnline: true });
|
||||
expect(parseDevice(response({ data: { online: false, powered: true } }))).toEqual({
|
||||
hardwareOnline: true,
|
||||
controlOnline: false,
|
||||
});
|
||||
expect(parseDevice(response({ data: { online: true, powered: false } }))).toEqual({
|
||||
hardwareOnline: false,
|
||||
controlOnline: true,
|
||||
});
|
||||
});
|
||||
|
||||
it('extracts carrier, registration, access technology and signal percentage', () => {
|
||||
expect(
|
||||
parseNetwork(
|
||||
response({
|
||||
data: {
|
||||
operator_name: 'China Mobile',
|
||||
registration_status: 'registered_home',
|
||||
technology_preference: 'LTE',
|
||||
mcc: 460,
|
||||
mnc: 0,
|
||||
secret: 'drop',
|
||||
},
|
||||
}),
|
||||
),
|
||||
).toEqual({
|
||||
carrier: '中国移动',
|
||||
cellularRegistration: 'registered_home',
|
||||
accessTechnology: 'LTE',
|
||||
cellularOnline: true,
|
||||
});
|
||||
// An unassigned or foreign PLMN keeps whatever the device reported.
|
||||
expect(
|
||||
parseNetwork(
|
||||
response({
|
||||
data: {
|
||||
operator_name: 'Telekom.de',
|
||||
registration_status: 'registered_roaming',
|
||||
mcc: 262,
|
||||
mnc: 1,
|
||||
},
|
||||
}),
|
||||
),
|
||||
).toEqual({
|
||||
carrier: 'Telekom.de',
|
||||
cellularRegistration: 'registered_roaming',
|
||||
cellularOnline: true,
|
||||
});
|
||||
expect(parseSignalStrength(response({ data: { strength: 80 } }))).toEqual({
|
||||
signalPercent: 80,
|
||||
});
|
||||
});
|
||||
|
||||
it('marks SIM presence while retaining phone-number privacy rules', () => {
|
||||
expect(
|
||||
parseSim(
|
||||
response({
|
||||
data: {
|
||||
phone_numbers: ['+8613800000000'],
|
||||
imsi: 'drop',
|
||||
iccid: 'drop',
|
||||
},
|
||||
}),
|
||||
),
|
||||
).toEqual({ simPresent: true, phoneNumbers: ['+8613800000000'] });
|
||||
});
|
||||
|
||||
describe('persisted resource snapshots', () => {
|
||||
it('serves the first overview after a restart from status_snapshots with zero upstream calls', async () => {
|
||||
const db = new Database(':memory:');
|
||||
migrateDatabase(db);
|
||||
// status_snapshots carries an FK to instances; seed the node it belongs to.
|
||||
db.prepare(
|
||||
"INSERT INTO instances (id,name,base_url,created_at,updated_at) VALUES ('device-1','device-1','http://192.168.3.55:3000','2026-01-01T00:00:00.000Z','2026-01-01T00:00:00.000Z')",
|
||||
).run();
|
||||
let clock = 1_000;
|
||||
let livePasses = 0;
|
||||
const build = () => {
|
||||
const requests: Array<{ url: string }> = [];
|
||||
const service = new InstanceResourceService({
|
||||
instances: { get: async () => ({ origin: 'http://192.168.3.55:3000' }) } as never,
|
||||
sessions: { sessionFor: () => undefined } as never,
|
||||
request: async (request: { url: string }) => {
|
||||
requests.push(request);
|
||||
return request.url.endsWith('/api/device')
|
||||
? response({ data: { online: true } })
|
||||
: { status: 503, headers: {}, body: '' };
|
||||
},
|
||||
db,
|
||||
now: () => clock,
|
||||
} as never);
|
||||
return { service, requests };
|
||||
};
|
||||
|
||||
const first = build();
|
||||
await first.service.get('device-1');
|
||||
livePasses = first.requests.length;
|
||||
expect(livePasses).toBe(6);
|
||||
|
||||
// A brand-new service (process restart) reads the persisted snapshot.
|
||||
const second = build();
|
||||
const resources = await second.service.get('device-1');
|
||||
expect(second.requests).toHaveLength(0);
|
||||
expect(resources).toEqual({ controlOnline: true });
|
||||
|
||||
// Past the freshness window the probe goes live again and repersists.
|
||||
clock += 121_000;
|
||||
await second.service.get('device-1');
|
||||
expect(second.requests.length).toBe(livePasses);
|
||||
db.close();
|
||||
});
|
||||
});
|
||||
|
||||
describe('overview cache', () => {
|
||||
const minimal = (cacheTtlMs?: number) => {
|
||||
const requests: Array<{ url: string }> = [];
|
||||
const service = new InstanceResourceService({
|
||||
instances: { get: async () => ({ origin: 'http://192.168.3.55:3000' }) } as never,
|
||||
sessions: { sessionFor: () => undefined } as never,
|
||||
request: async (request: { url: string }) => {
|
||||
requests.push(request);
|
||||
return request.url.endsWith('/api/device')
|
||||
? response({ data: { online: true } })
|
||||
: { status: 503, headers: {}, body: '' };
|
||||
},
|
||||
...(cacheTtlMs === undefined ? {} : { cacheTtlMs }),
|
||||
} as never);
|
||||
return { service, requests };
|
||||
};
|
||||
|
||||
it('serves repeat overview reads from the TTL cache and refreshes after expiry', async () => {
|
||||
let clock = 1_000;
|
||||
const { service, requests } = minimal();
|
||||
(service as unknown as { options: { now?: () => number } }).options.now = () => clock;
|
||||
|
||||
await service.get('device-1');
|
||||
const firstPass = requests.length;
|
||||
await service.get('device-1');
|
||||
expect(requests.length).toBe(firstPass);
|
||||
clock += 31_000;
|
||||
await service.get('device-1');
|
||||
expect(requests.length).toBeGreaterThan(firstPass);
|
||||
});
|
||||
|
||||
it('coalesces concurrent overview reads into one upstream pass', async () => {
|
||||
const { service, requests } = minimal();
|
||||
await Promise.all([
|
||||
service.get('device-1'),
|
||||
service.get('device-1'),
|
||||
service.get('device-1'),
|
||||
]);
|
||||
expect(requests).toHaveLength(6);
|
||||
});
|
||||
|
||||
it('probes live again when a detail view forces a refresh', async () => {
|
||||
const clock = 5_000;
|
||||
const { service, requests } = minimal();
|
||||
(service as unknown as { options: { now?: () => number } }).options.now = () => clock;
|
||||
|
||||
await service.get('device-1');
|
||||
const afterOverview = requests.length;
|
||||
await service.get('device-1', { force: true });
|
||||
expect(requests.length).toBe(afterOverview + 6);
|
||||
});
|
||||
|
||||
it('keeps reads uncached when the TTL is disabled', async () => {
|
||||
const { service, requests } = minimal(0);
|
||||
await service.get('device-1');
|
||||
await service.get('device-1');
|
||||
expect(requests).toHaveLength(12);
|
||||
});
|
||||
});
|
||||
|
||||
it('probes passwordless instances without manufacturing a cookie', async () => {
|
||||
const requests: Array<{ url: string; headers: Readonly<Record<string, string>> }> = [];
|
||||
const service = new InstanceResourceService({
|
||||
instances: { get: async () => ({ origin: 'http://192.168.3.55:3000' }) } as never,
|
||||
sessions: { sessionFor: () => undefined } as never,
|
||||
request: async (request) => {
|
||||
requests.push(request);
|
||||
return response(
|
||||
request.url.endsWith('/api/stats')
|
||||
? { data: { cpu_load: { load_percent: 12 }, memory: { used_percent: 34 } } }
|
||||
: request.url.endsWith('/api/sim')
|
||||
? { data: { phone_numbers: ['13800000000'] } }
|
||||
: request.url.endsWith('/api/device')
|
||||
? { data: { online: true, powered: true } }
|
||||
: request.url.endsWith('/api/network')
|
||||
? {
|
||||
data: {
|
||||
operator_name: 'China Mobile',
|
||||
registration_status: 'registered_home',
|
||||
technology_preference: 'LTE',
|
||||
},
|
||||
}
|
||||
: request.url.endsWith('/api/network/signal-strength')
|
||||
? { data: { strength: 72 } }
|
||||
: { status: 'ok', version: '2.0.1', platform: 'linux' },
|
||||
);
|
||||
},
|
||||
});
|
||||
await expect(service.get('device-1')).resolves.toEqual({
|
||||
cpuPercent: 12,
|
||||
memoryPercent: 34,
|
||||
phoneNumbers: ['13800000000'],
|
||||
simPresent: true,
|
||||
version: '2.0.1',
|
||||
platform: 'linux',
|
||||
hardwareOnline: true,
|
||||
controlOnline: true,
|
||||
carrier: 'China Mobile',
|
||||
cellularRegistration: 'registered_home',
|
||||
accessTechnology: 'LTE',
|
||||
cellularOnline: true,
|
||||
signalPercent: 72,
|
||||
});
|
||||
expect(requests.map((request) => new URL(request.url).pathname)).toEqual([
|
||||
'/api/stats',
|
||||
'/api/sim',
|
||||
'/api/health',
|
||||
'/api/device',
|
||||
'/api/network',
|
||||
'/api/network/signal-strength',
|
||||
]);
|
||||
expect(requests.map((request) => request.headers)).toEqual(
|
||||
Array.from({ length: 6 }, () => ({ accept: 'application/json' })),
|
||||
);
|
||||
});
|
||||
|
||||
it('keeps partial resource behavior when device telemetry endpoints are unavailable', async () => {
|
||||
const requests: Array<{ headers: Readonly<Record<string, string>> }> = [];
|
||||
const service = new InstanceResourceService({
|
||||
instances: { get: async () => ({ origin: 'http://192.168.3.55:3000' }) } as never,
|
||||
@@ -51,7 +329,9 @@ describe('instance resource allowlist parsing', () => {
|
||||
return response(
|
||||
request.url.endsWith('/api/stats')
|
||||
? { data: { cpu_load: { load_percent: 12 }, memory: { used_percent: 34 } } }
|
||||
: { data: { phone_numbers: ['13800000000'] } },
|
||||
: request.url.endsWith('/api/sim')
|
||||
? { data: { phone_numbers: ['13800000000'] } }
|
||||
: { status: 'ok', version: '2.0.1', platform: 'linux' },
|
||||
);
|
||||
},
|
||||
});
|
||||
@@ -59,10 +339,12 @@ describe('instance resource allowlist parsing', () => {
|
||||
cpuPercent: 12,
|
||||
memoryPercent: 34,
|
||||
phoneNumbers: ['13800000000'],
|
||||
simPresent: true,
|
||||
version: '2.0.1',
|
||||
platform: 'linux',
|
||||
});
|
||||
expect(requests.map((request) => request.headers)).toEqual([
|
||||
{ accept: 'application/json' },
|
||||
{ accept: 'application/json' },
|
||||
...Array.from({ length: 6 }, () => ({ accept: 'application/json' })),
|
||||
]);
|
||||
});
|
||||
|
||||
|
||||
@@ -1,3 +1,7 @@
|
||||
import { lookupOperator } from '@multi-simadmin/contracts';
|
||||
import { randomUUID } from 'node:crypto';
|
||||
import type Database from 'better-sqlite3';
|
||||
|
||||
import type { InstanceService } from '../instances/instance-service.js';
|
||||
import type {
|
||||
InstanceSessionStore,
|
||||
@@ -10,6 +14,18 @@ export interface InstanceResources {
|
||||
readonly memoryPercent?: number;
|
||||
readonly maxTemperatureCelsius?: number;
|
||||
readonly phoneNumbers?: readonly string[];
|
||||
readonly version?: string;
|
||||
readonly platform?: string;
|
||||
readonly hardwareOnline?: boolean;
|
||||
readonly controlOnline?: boolean;
|
||||
readonly simPresent?: boolean;
|
||||
readonly carrier?: string;
|
||||
readonly cellularRegistration?: string;
|
||||
readonly accessTechnology?: string;
|
||||
readonly cellularOnline?: boolean;
|
||||
readonly signalPercent?: number;
|
||||
/** Device uptime in whole seconds, reported by the node stats module. */
|
||||
readonly uptimeSeconds?: number;
|
||||
}
|
||||
|
||||
const MAX_BODY_BYTES = 32_768;
|
||||
@@ -32,6 +48,39 @@ function data(response: UpstreamResponse): Record<string, unknown> | undefined {
|
||||
}
|
||||
}
|
||||
|
||||
const safeText = (value: unknown, maximum = 128): string | undefined =>
|
||||
typeof value === 'string' &&
|
||||
value.length > 0 &&
|
||||
value.length <= maximum &&
|
||||
!/[\u0000-\u001f\u007f]/u.test(value)
|
||||
? value
|
||||
: undefined;
|
||||
|
||||
const safeCode = (value: unknown, maximum = 64): string | undefined =>
|
||||
typeof value === 'string' &&
|
||||
value.length > 0 &&
|
||||
value.length <= maximum &&
|
||||
/^[A-Za-z0-9_-]+$/u.test(value)
|
||||
? value
|
||||
: undefined;
|
||||
|
||||
export function parseHealth(response: UpstreamResponse): InstanceResources {
|
||||
if (response.status < 200 || response.status >= 300) return {};
|
||||
if (Buffer.byteLength(response.body, 'utf8') > MAX_BODY_BYTES) return {};
|
||||
try {
|
||||
const root = record(JSON.parse(response.body));
|
||||
if (!root) return {};
|
||||
const version = safeText(root.version);
|
||||
const platform = safeText(root.platform);
|
||||
return {
|
||||
...(version ? { version } : {}),
|
||||
...(platform ? { platform } : {}),
|
||||
};
|
||||
} catch {
|
||||
return {};
|
||||
}
|
||||
}
|
||||
|
||||
export function parseStats(response: UpstreamResponse): InstanceResources {
|
||||
const value = data(response);
|
||||
if (!value) return {};
|
||||
@@ -43,55 +92,267 @@ export function parseStats(response: UpstreamResponse): InstanceResources {
|
||||
.filter((item): item is number => item !== undefined)
|
||||
: [];
|
||||
const maxTemperatureCelsius = temperatures.length > 0 ? Math.max(...temperatures) : undefined;
|
||||
const system = record(value.system);
|
||||
const version =
|
||||
safeText(value.version) ??
|
||||
safeText(value.current_version) ??
|
||||
safeText(system?.version) ??
|
||||
safeText(system?.app_version);
|
||||
const platform =
|
||||
safeText(value.platform) ?? safeText(system?.platform) ?? safeText(system?.architecture);
|
||||
const uptimeSeconds =
|
||||
ranged(system?.uptime_seconds, 0, 680_400_000) ?? ranged(value.uptime_seconds, 0, 680_400_000);
|
||||
return {
|
||||
...(cpuPercent === undefined ? {} : { cpuPercent }),
|
||||
...(memoryPercent === undefined ? {} : { memoryPercent }),
|
||||
...(maxTemperatureCelsius === undefined ? {} : { maxTemperatureCelsius }),
|
||||
...(version ? { version } : {}),
|
||||
...(platform ? { platform } : {}),
|
||||
...(uptimeSeconds === undefined ? {} : { uptimeSeconds: Math.floor(uptimeSeconds) }),
|
||||
};
|
||||
}
|
||||
|
||||
export function parseSim(response: UpstreamResponse): InstanceResources {
|
||||
const value = data(response);
|
||||
if (!value || !Array.isArray(value.phone_numbers)) return {};
|
||||
const phoneNumbers = value.phone_numbers.filter(
|
||||
if (!value) return {};
|
||||
const numbers = Array.isArray(value.phone_numbers) ? value.phone_numbers : [];
|
||||
const phoneNumbers = numbers.filter(
|
||||
(item): item is string =>
|
||||
typeof item === 'string' &&
|
||||
item.length > 0 &&
|
||||
item.length <= 32 &&
|
||||
/^[+0-9 ()-]+$/.test(item),
|
||||
);
|
||||
return phoneNumbers.length > 0 ? { phoneNumbers: [...new Set(phoneNumbers)].slice(0, 16) } : {};
|
||||
return {
|
||||
simPresent: true,
|
||||
...(phoneNumbers.length > 0 ? { phoneNumbers: [...new Set(phoneNumbers)].slice(0, 16) } : {}),
|
||||
};
|
||||
}
|
||||
|
||||
export function parseDevice(response: UpstreamResponse): InstanceResources {
|
||||
const value = data(response);
|
||||
if (!value) return {};
|
||||
const hardwareOnline = typeof value.powered === 'boolean' ? value.powered : undefined;
|
||||
const controlOnline = typeof value.online === 'boolean' ? value.online : undefined;
|
||||
return {
|
||||
...(hardwareOnline === undefined ? {} : { hardwareOnline }),
|
||||
...(controlOnline === undefined ? {} : { controlOnline }),
|
||||
};
|
||||
}
|
||||
|
||||
export function parseNetwork(response: UpstreamResponse): InstanceResources {
|
||||
const value = data(response);
|
||||
if (!value) return {};
|
||||
const reported = safeText(value.operator_name, 64);
|
||||
// Devices spell the same network every which way ("China Mobile", "CHN-CMCC", "46000"); the
|
||||
// shared PLMN registry settles it, exactly like the Hub does.
|
||||
const known = lookupOperator(value.mcc, value.mnc);
|
||||
const carrier = known?.name ?? reported;
|
||||
const registration = safeCode(value.registration_status);
|
||||
const accessTechnology = safeText(value.technology_preference, 32);
|
||||
const cellularOnline =
|
||||
registration === undefined ? undefined : registration.startsWith('registered');
|
||||
return {
|
||||
...(carrier ? { carrier } : {}),
|
||||
...(registration ? { cellularRegistration: registration } : {}),
|
||||
...(accessTechnology ? { accessTechnology } : {}),
|
||||
...(cellularOnline === undefined ? {} : { cellularOnline }),
|
||||
};
|
||||
}
|
||||
|
||||
export function parseSignalStrength(response: UpstreamResponse): InstanceResources {
|
||||
const signalPercent = ranged(data(response)?.strength, 0, 100);
|
||||
return signalPercent === undefined ? {} : { signalPercent };
|
||||
}
|
||||
|
||||
const DEFAULT_CACHE_TTL_MS = 30_000;
|
||||
const DEFAULT_SNAPSHOT_FRESH_MS = 120_000;
|
||||
const RESOURCE_SNAPSHOT_CATEGORY = 'resources';
|
||||
|
||||
export class InstanceResourceService {
|
||||
readonly #cache = new Map<
|
||||
string,
|
||||
{ readonly resources: InstanceResources; readonly fetchedAt: number }
|
||||
>();
|
||||
readonly #inflight = new Map<string, Promise<InstanceResources>>();
|
||||
|
||||
constructor(
|
||||
private readonly options: {
|
||||
readonly instances: InstanceService;
|
||||
readonly sessions: InstanceSessionStore;
|
||||
readonly request: UpstreamSessionClientOptions['request'];
|
||||
readonly ensureSession?: (
|
||||
instanceId: string,
|
||||
origin: string,
|
||||
force?: boolean,
|
||||
) => Promise<void>;
|
||||
/** Fleet overview fan-out reads through this cache; 0 disables it. */
|
||||
readonly cacheTtlMs?: number;
|
||||
/**
|
||||
* When set, every live fetch is persisted into status_snapshots
|
||||
* (category 'resources') and a snapshot younger than snapshotFreshMs is
|
||||
* served before any upstream call, so the first overview after a restart
|
||||
* costs zero device requests.
|
||||
*/
|
||||
readonly db?: Database.Database;
|
||||
readonly snapshotFreshMs?: number;
|
||||
readonly now?: () => number;
|
||||
},
|
||||
) {}
|
||||
|
||||
async get(instanceId: string): Promise<InstanceResources> {
|
||||
/**
|
||||
* Fleet overview fans out one read per registered node; without a shared TTL
|
||||
* that is six live upstream requests per device on every page render (plus a
|
||||
* re-login storm when sessions expired). Detail views pass `force` to probe
|
||||
* the selected device immediately.
|
||||
*/
|
||||
async get(
|
||||
instanceId: string,
|
||||
{ force = false }: { readonly force?: boolean } = {},
|
||||
): Promise<InstanceResources> {
|
||||
const ttl = this.options.cacheTtlMs ?? DEFAULT_CACHE_TTL_MS;
|
||||
const now = this.options.now?.() ?? Date.now();
|
||||
if (ttl > 0 && !force) {
|
||||
const cached = this.#cache.get(instanceId);
|
||||
if (cached && now - cached.fetchedAt < ttl) return cached.resources;
|
||||
const persisted = this.#readSnapshot(instanceId, now);
|
||||
if (persisted) {
|
||||
this.#cache.set(instanceId, { resources: persisted, fetchedAt: now });
|
||||
return persisted;
|
||||
}
|
||||
const inflight = this.#inflight.get(instanceId);
|
||||
if (inflight) return inflight;
|
||||
const pending = this.#fetch(instanceId)
|
||||
.then((resources) => {
|
||||
const fetchedAt = this.options.now?.() ?? now;
|
||||
this.#cache.set(instanceId, { resources, fetchedAt });
|
||||
this.#writeSnapshot(instanceId, resources, fetchedAt);
|
||||
return resources;
|
||||
})
|
||||
.finally(() => {
|
||||
this.#inflight.delete(instanceId);
|
||||
});
|
||||
this.#inflight.set(instanceId, pending);
|
||||
return pending;
|
||||
}
|
||||
const resources = await this.#fetch(instanceId);
|
||||
if (ttl > 0) {
|
||||
this.#cache.set(instanceId, { resources, fetchedAt: now });
|
||||
this.#writeSnapshot(instanceId, resources, now);
|
||||
}
|
||||
return resources;
|
||||
}
|
||||
|
||||
#readSnapshot(instanceId: string, now: number): InstanceResources | undefined {
|
||||
const db = this.options.db;
|
||||
if (!db) return undefined;
|
||||
const freshMs = this.options.snapshotFreshMs ?? DEFAULT_SNAPSHOT_FRESH_MS;
|
||||
try {
|
||||
const row = db
|
||||
.prepare(
|
||||
'SELECT payload_json, observed_at FROM status_snapshots WHERE instance_id=? AND category=?',
|
||||
)
|
||||
.get(instanceId, RESOURCE_SNAPSHOT_CATEGORY) as
|
||||
| { payload_json: string; observed_at: string }
|
||||
| undefined;
|
||||
if (!row) return undefined;
|
||||
const age = now - Date.parse(row.observed_at);
|
||||
if (!Number.isFinite(age) || age < 0 || age >= freshMs) return undefined;
|
||||
const payload: unknown = JSON.parse(row.payload_json);
|
||||
if (typeof payload !== 'object' || payload === null || Array.isArray(payload))
|
||||
return undefined;
|
||||
return payload as InstanceResources;
|
||||
} catch {
|
||||
return undefined;
|
||||
}
|
||||
}
|
||||
|
||||
#writeSnapshot(instanceId: string, resources: InstanceResources, observedAt: number): void {
|
||||
const db = this.options.db;
|
||||
if (!db) return;
|
||||
const freshMs = this.options.snapshotFreshMs ?? DEFAULT_SNAPSHOT_FRESH_MS;
|
||||
const observed = new Date(observedAt).toISOString();
|
||||
const expiresAt = new Date(observedAt + freshMs).toISOString();
|
||||
try {
|
||||
db.prepare(
|
||||
`INSERT INTO status_snapshots (id, instance_id, category, state, payload_json, observed_at, expires_at, created_at)
|
||||
VALUES (?, ?, ?, 'fresh', ?, ?, ?, ?)
|
||||
ON CONFLICT(instance_id, category) DO UPDATE SET
|
||||
id=excluded.id, state=excluded.state, payload_json=excluded.payload_json,
|
||||
observed_at=excluded.observed_at, expires_at=excluded.expires_at, created_at=excluded.created_at`,
|
||||
).run(
|
||||
randomUUID(),
|
||||
instanceId,
|
||||
RESOURCE_SNAPSHOT_CATEGORY,
|
||||
JSON.stringify(resources),
|
||||
observed,
|
||||
expiresAt,
|
||||
observed,
|
||||
);
|
||||
} catch {
|
||||
// Persistence is an optimization; a failed write must not fail the read.
|
||||
}
|
||||
}
|
||||
|
||||
async #fetch(instanceId: string): Promise<InstanceResources> {
|
||||
const [instance, session] = await Promise.all([
|
||||
this.options.instances.get(instanceId),
|
||||
Promise.resolve(this.options.sessions.sessionFor(instanceId)),
|
||||
]);
|
||||
if (!instance) return {};
|
||||
if (session && session.origin !== instance.origin) return {};
|
||||
const get = (path: '/api/stats' | '/api/sim') =>
|
||||
this.options.request({
|
||||
url: `${instance.origin}${path}`,
|
||||
method: 'GET',
|
||||
headers: {
|
||||
accept: 'application/json',
|
||||
...(session ? { cookie: session.cookie } : {}),
|
||||
},
|
||||
});
|
||||
const [stats, sim] = await Promise.allSettled([get('/api/stats'), get('/api/sim')]);
|
||||
if (!session && this.options.ensureSession) {
|
||||
try {
|
||||
await this.options.ensureSession(instanceId, instance.origin);
|
||||
} catch {
|
||||
// Passwordless and temporarily unavailable credentials still permit anonymous reads.
|
||||
}
|
||||
}
|
||||
const paths = [
|
||||
'/api/stats',
|
||||
'/api/sim',
|
||||
'/api/health',
|
||||
'/api/device',
|
||||
'/api/network',
|
||||
'/api/network/signal-strength',
|
||||
] as const;
|
||||
const readAll = () => {
|
||||
const activeSession = this.options.sessions.sessionFor(instanceId);
|
||||
return Promise.allSettled(
|
||||
paths.map((path) =>
|
||||
this.options.request({
|
||||
url: `${instance.origin}${path}`,
|
||||
method: 'GET',
|
||||
headers: {
|
||||
accept: 'application/json',
|
||||
...(activeSession?.origin === instance.origin
|
||||
? { cookie: activeSession.cookie }
|
||||
: {}),
|
||||
},
|
||||
}),
|
||||
),
|
||||
);
|
||||
};
|
||||
let results = await readAll();
|
||||
const unauthorized = results.some(
|
||||
(result) => result.status === 'fulfilled' && [401, 403].includes(result.value.status),
|
||||
);
|
||||
if (unauthorized && this.options.ensureSession) {
|
||||
try {
|
||||
await this.options.ensureSession(instanceId, instance.origin, true);
|
||||
results = await readAll();
|
||||
} catch {
|
||||
// Return the safe partial result when re-authentication is unavailable.
|
||||
}
|
||||
}
|
||||
const [stats, sim, health, device, network, signal] = results;
|
||||
return {
|
||||
...(stats.status === 'fulfilled' ? parseStats(stats.value) : {}),
|
||||
...(sim.status === 'fulfilled' ? parseSim(sim.value) : {}),
|
||||
...(stats?.status === 'fulfilled' ? parseStats(stats.value) : {}),
|
||||
...(sim?.status === 'fulfilled' ? parseSim(sim.value) : {}),
|
||||
...(health?.status === 'fulfilled' ? parseHealth(health.value) : {}),
|
||||
...(device?.status === 'fulfilled' ? parseDevice(device.value) : {}),
|
||||
...(network?.status === 'fulfilled' ? parseNetwork(network.value) : {}),
|
||||
...(signal?.status === 'fulfilled' ? parseSignalStrength(signal.value) : {}),
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
@@ -47,6 +47,18 @@ export interface StatusSnapshotServiceOptions {
|
||||
readonly maxStaleMs: number;
|
||||
readonly now?: () => Date;
|
||||
readonly idFactory?: () => string;
|
||||
/** Observes every probe outcome, including ones fenced from persistence by a revision change. */
|
||||
readonly onProbe?: (result: HealthProbeResult) => void;
|
||||
}
|
||||
|
||||
export interface HealthProbeResult {
|
||||
readonly instanceId: string;
|
||||
readonly outcome: 'success' | 'stale' | 'failed' | 'unsupported';
|
||||
readonly state: HealthSnapshotState;
|
||||
readonly errorCode: string | null;
|
||||
readonly httpStatus: number | null;
|
||||
readonly durationMs: number;
|
||||
readonly observedAt: string;
|
||||
}
|
||||
|
||||
export class StatusSnapshotError extends Error {
|
||||
@@ -193,6 +205,7 @@ export class StatusSnapshotService {
|
||||
readonly #maxStaleMs: number;
|
||||
readonly #now: () => Date;
|
||||
readonly #id: () => string;
|
||||
readonly #onProbe: ((result: HealthProbeResult) => void) | undefined;
|
||||
readonly #latestGeneration = new Map<string, number>();
|
||||
#nextGeneration = 0;
|
||||
|
||||
@@ -210,6 +223,7 @@ export class StatusSnapshotService {
|
||||
this.#maxStaleMs = options.maxStaleMs;
|
||||
this.#now = options.now ?? (() => new Date());
|
||||
this.#id = options.idFactory ?? randomUUID;
|
||||
this.#onProbe = options.onProbe;
|
||||
}
|
||||
|
||||
async refreshHealth(instanceId: string): Promise<HealthSnapshot> {
|
||||
@@ -247,6 +261,7 @@ export class StatusSnapshotService {
|
||||
);
|
||||
const { snapshot } = classified;
|
||||
|
||||
this.#observe(instanceId, snapshot, classified.persisted);
|
||||
if (this.#latestGeneration.get(instanceId) === generation) {
|
||||
if (
|
||||
isValidOwner(instanceId, currentOwner) &&
|
||||
@@ -260,6 +275,35 @@ export class StatusSnapshotService {
|
||||
return Object.freeze({ ...snapshot, payload: Object.freeze({ ...snapshot.payload }) });
|
||||
}
|
||||
|
||||
/** A failed connection log write must never take a health probe down with it. */
|
||||
#observe(instanceId: string, snapshot: HealthSnapshot, persisted: PersistedHealthEnvelope): void {
|
||||
if (!this.#onProbe) return;
|
||||
const status = persisted.httpStatus;
|
||||
const outcome: HealthProbeResult['outcome'] =
|
||||
persisted.errorCode === 'UNSUPPORTED'
|
||||
? 'unsupported'
|
||||
: snapshot.state === 'fresh'
|
||||
? 'success'
|
||||
: snapshot.state === 'stale'
|
||||
? 'stale'
|
||||
: status !== null && status >= 200 && status < 300
|
||||
? 'success'
|
||||
: 'failed';
|
||||
try {
|
||||
this.#onProbe({
|
||||
instanceId,
|
||||
outcome,
|
||||
state: snapshot.state,
|
||||
errorCode: persisted.errorCode,
|
||||
httpStatus: status,
|
||||
durationMs: persisted.durationMs,
|
||||
observedAt: snapshot.observedAt,
|
||||
});
|
||||
} catch {
|
||||
// Diagnostics are advisory; the snapshot result stands on its own.
|
||||
}
|
||||
}
|
||||
|
||||
#previous(instanceId: string): SnapshotRow | undefined {
|
||||
return this.#db
|
||||
.prepare(
|
||||
|
||||
@@ -0,0 +1,270 @@
|
||||
import Database from 'better-sqlite3';
|
||||
import { mkdtemp, readFile, readdir, rm, writeFile } from 'node:fs/promises';
|
||||
import { join } from 'node:path';
|
||||
import { tmpdir } from 'node:os';
|
||||
import { afterEach, describe, expect, it } from 'vitest';
|
||||
|
||||
import { migrateDatabase } from '../../infrastructure/database/migrations.js';
|
||||
import {
|
||||
BACKUP_COMPONENT_KEYS,
|
||||
ComponentBackupError,
|
||||
ComponentBackupService,
|
||||
} from './component-backup-service.js';
|
||||
|
||||
const roots: string[] = [];
|
||||
afterEach(async () => {
|
||||
await Promise.all(roots.splice(0).map((root) => rm(root, { recursive: true, force: true })));
|
||||
});
|
||||
|
||||
function database(): Database.Database {
|
||||
const db = new Database(':memory:');
|
||||
db.pragma('foreign_keys = ON');
|
||||
migrateDatabase(db);
|
||||
return db;
|
||||
}
|
||||
|
||||
async function directory(): Promise<string> {
|
||||
const root = await mkdtemp(join(tmpdir(), 'component-backup-'));
|
||||
roots.push(root);
|
||||
return root;
|
||||
}
|
||||
|
||||
function seed(db: Database.Database, tag: string): void {
|
||||
db.prepare('INSERT INTO device_groups (id,name,created_at,updated_at) VALUES (?,?,?,?)').run(
|
||||
`group-${tag}`,
|
||||
`机房 ${tag}`,
|
||||
'2026-01-01T00:00:00.000Z',
|
||||
'2026-01-01T00:00:00.000Z',
|
||||
);
|
||||
db.prepare(
|
||||
`INSERT INTO instances (id,name,base_url,auth_mode,enabled,config_revision,created_at,updated_at)
|
||||
VALUES (?,?,?,?,?,?,?,?)`,
|
||||
).run(
|
||||
`node-${tag}`,
|
||||
`节点 ${tag}`,
|
||||
`http://192.168.1.${tag}:8080`,
|
||||
'password',
|
||||
1,
|
||||
1,
|
||||
'2026-01-01T00:00:00.000Z',
|
||||
'2026-01-01T00:00:00.000Z',
|
||||
);
|
||||
db.prepare('INSERT INTO instance_tags (instance_id,tag,created_at) VALUES (?,?,?)').run(
|
||||
`node-${tag}`,
|
||||
`tag-${tag}`,
|
||||
'2026-01-01T00:00:00.000Z',
|
||||
);
|
||||
db.prepare('INSERT INTO tag_registry (tag,created_at,updated_at) VALUES (?,?,?)').run(
|
||||
`tag-${tag}`,
|
||||
'2026-01-01T00:00:00.000Z',
|
||||
'2026-01-01T00:00:00.000Z',
|
||||
);
|
||||
db.prepare(
|
||||
`INSERT INTO audit_events
|
||||
(id,actor,operation_id,risk_level,request_id,parameters_summary_json,result_code,duration_ms,created_at)
|
||||
VALUES (?,?,?,?,?,?,?,?,?)`,
|
||||
).run(
|
||||
`audit-${tag}`,
|
||||
'operator',
|
||||
'probe',
|
||||
'R0',
|
||||
`${tag}-req`,
|
||||
'[]',
|
||||
'succeeded',
|
||||
1,
|
||||
'2026-01-01T00:00:00.000Z',
|
||||
);
|
||||
db.prepare(
|
||||
`INSERT INTO jobs
|
||||
(id,root_job_id,operation_id,risk_level,status,requested_by,request_id,parameters_digest,created_at,updated_at)
|
||||
VALUES (?,?,?,?,'succeeded',?,?,?,?,?)`,
|
||||
).run(
|
||||
`job-${tag}`,
|
||||
`job-${tag}`,
|
||||
'probe',
|
||||
'R0',
|
||||
'operator',
|
||||
`${tag}-req`,
|
||||
'digest',
|
||||
'2026-01-01T00:00:00.000Z',
|
||||
'2026-01-01T00:00:00.000Z',
|
||||
);
|
||||
db.prepare(
|
||||
`INSERT INTO job_items
|
||||
(id,job_id,instance_id,attempt_number,status,created_at,updated_at)
|
||||
VALUES (?,?,?,1,'succeeded',?,?)`,
|
||||
).run(
|
||||
`item-${tag}`,
|
||||
`job-${tag}`,
|
||||
`node-${tag}`,
|
||||
'2026-01-01T00:00:00.000Z',
|
||||
'2026-01-01T00:00:00.000Z',
|
||||
);
|
||||
db.prepare(
|
||||
`INSERT INTO job_attempts (id,job_id,status,started_at,created_at)
|
||||
VALUES (?,?,?,?,?)`,
|
||||
).run(
|
||||
`attempt-${tag}`,
|
||||
`job-${tag}`,
|
||||
'succeeded',
|
||||
'2026-01-01T00:00:00.000Z',
|
||||
'2026-01-01T00:00:00.000Z',
|
||||
);
|
||||
}
|
||||
|
||||
describe('ComponentBackupService', () => {
|
||||
it('lists every catalog unit with a live row count', async () => {
|
||||
const db = database();
|
||||
seed(db, '1');
|
||||
const service = new ComponentBackupService(db, {
|
||||
version: 'test',
|
||||
backupDirectory: await directory(),
|
||||
});
|
||||
const catalog = service.catalog();
|
||||
expect(catalog.map((entry) => entry.key)).toEqual([...BACKUP_COMPONENT_KEYS]);
|
||||
const devices = catalog.find((entry) => entry.key === 'devices');
|
||||
expect(devices?.rows).toBe(4);
|
||||
expect(devices?.label).toBe('设备与分组');
|
||||
});
|
||||
|
||||
it('creates a component backup that the list endpoint can verify', async () => {
|
||||
const db = database();
|
||||
seed(db, '1');
|
||||
const backupDirectory = await directory();
|
||||
const service = new ComponentBackupService(db, { version: 'test', backupDirectory });
|
||||
const created = await service.create(['devices', 'audit'], '升级前快照');
|
||||
expect(created.integrity).toBe('ok');
|
||||
expect(created.note).toBe('升级前快照');
|
||||
expect(created.components.map((entry) => entry.key)).toEqual(['devices', 'audit']);
|
||||
expect((await readdir(backupDirectory)).length).toBe(1);
|
||||
|
||||
const listed = await service.list();
|
||||
expect(listed).toHaveLength(1);
|
||||
expect(listed[0]?.filename).toBe(created.filename);
|
||||
expect(listed[0]?.integrity).toBe('ok');
|
||||
expect(listed[0]?.compatible).toBe(true);
|
||||
expect(listed[0]?.components.find((entry) => entry.key === 'audit')?.rows).toBe(1);
|
||||
});
|
||||
|
||||
it('merges a selected component into a fresh database and leaves the rest alone', async () => {
|
||||
const source = database();
|
||||
seed(source, '1');
|
||||
seed(source, '2');
|
||||
const backupDirectory = await directory();
|
||||
const service = new ComponentBackupService(source, { version: 'test', backupDirectory });
|
||||
const created = await service.create(['devices', 'audit']);
|
||||
|
||||
const target = database();
|
||||
seed(target, '9');
|
||||
const restoreService = new ComponentBackupService(target, {
|
||||
version: 'test',
|
||||
backupDirectory,
|
||||
});
|
||||
const result = await restoreService.restore(created.filename, ['devices']);
|
||||
expect(result.written).toEqual({ devices: 8 });
|
||||
// The state right before the merge is snapshotted first, so the restore stays reversible.
|
||||
expect(result.safetyBackup).not.toBe(created.filename);
|
||||
const safety = await restoreService.preview(result.safetyBackup);
|
||||
expect(safety.note).toBe(`恢复前自动备份:${created.filename}`);
|
||||
expect(safety.automatic).toBe(true);
|
||||
expect(safety.components.map((component) => component.key)).toEqual(['devices']);
|
||||
const instances = target.prepare('SELECT id FROM instances ORDER BY id').all() as {
|
||||
id: string;
|
||||
}[];
|
||||
expect(instances.map((entry) => entry.id)).toEqual(['node-1', 'node-2', 'node-9']);
|
||||
// The audit component was not selected, so the target keeps only its own row.
|
||||
const audits = target.prepare('SELECT id FROM audit_events ORDER BY id').all() as {
|
||||
id: string;
|
||||
}[];
|
||||
expect(audits.map((entry) => entry.id)).toEqual(['audit-9']);
|
||||
});
|
||||
|
||||
it('refuses a tampered archive', async () => {
|
||||
const db = database();
|
||||
seed(db, '1');
|
||||
const backupDirectory = await directory();
|
||||
const service = new ComponentBackupService(db, { version: 'test', backupDirectory });
|
||||
const created = await service.create(['audit']);
|
||||
const path = join(backupDirectory, created.filename);
|
||||
const document = JSON.parse(await readFile(path, 'utf8')) as {
|
||||
components: { audit: { audit_events: { rows: unknown[][] } } };
|
||||
};
|
||||
const row = document.components.audit.audit_events.rows[0];
|
||||
if (!row) throw new Error('expected a serialized audit row');
|
||||
row[3] = 'R3';
|
||||
await writeFile(path, JSON.stringify(document), 'utf8');
|
||||
|
||||
await expect(service.preview(created.filename)).rejects.toThrowError(ComponentBackupError);
|
||||
await expect(service.restore(created.filename, ['audit'])).rejects.toMatchObject({
|
||||
code: 'INTEGRITY_FAILED',
|
||||
});
|
||||
});
|
||||
|
||||
it('rejects unknown component keys and unsafe filenames', async () => {
|
||||
const db = database();
|
||||
const service = new ComponentBackupService(db, {
|
||||
version: 'test',
|
||||
backupDirectory: await directory(),
|
||||
});
|
||||
await expect(service.create(['nope' as never])).rejects.toMatchObject({
|
||||
code: 'VALIDATION_FAILED',
|
||||
});
|
||||
await expect(service.preview('../secret.json')).rejects.toMatchObject({
|
||||
code: 'VALIDATION_FAILED',
|
||||
});
|
||||
await expect(service.preview('multi-simadmin-components-missing.json')).rejects.toMatchObject({
|
||||
code: 'NOT_FOUND',
|
||||
});
|
||||
});
|
||||
|
||||
it('runs one scheduled backup per period and prunes to the retained count', async () => {
|
||||
const db = database();
|
||||
seed(db, '1');
|
||||
const backupDirectory = await directory();
|
||||
let clock = new Date('2026-03-01T03:35:00.000Z');
|
||||
const service = new ComponentBackupService(db, {
|
||||
version: 'test',
|
||||
backupDirectory,
|
||||
now: () => clock,
|
||||
});
|
||||
service.updateAutoSettings({
|
||||
enabled: true,
|
||||
components: ['devices'],
|
||||
timeOfDay: '02:00',
|
||||
weekday: -1,
|
||||
maximumCount: 2,
|
||||
});
|
||||
expect(await service.runDueAutoBackups()).toMatchObject({ automatic: true });
|
||||
expect(await service.runDueAutoBackups()).toBeNull();
|
||||
|
||||
clock = new Date('2026-03-02T03:31:00.000Z');
|
||||
expect(await service.runDueAutoBackups()).not.toBeNull();
|
||||
clock = new Date('2026-03-03T03:31:00.000Z');
|
||||
expect(await service.runDueAutoBackups()).not.toBeNull();
|
||||
const files = (await readdir(backupDirectory)).filter((name) => name.includes('-auto-'));
|
||||
expect(files).toHaveLength(2);
|
||||
expect(service.autoSettings()).toMatchObject({ enabled: true, maximumCount: 2, weekday: -1 });
|
||||
expect(service.autoSettings().lastRunAt).toBe('2026-03-03T03:31:00.000Z');
|
||||
});
|
||||
|
||||
it('keeps the auto settings readable when the stored payload is damaged', async () => {
|
||||
const db = database();
|
||||
db.prepare(
|
||||
'INSERT INTO app_settings (key,value_json,created_at,updated_at) VALUES (?,?,?,?)',
|
||||
).run(
|
||||
'system.component_backup.auto',
|
||||
'{not json',
|
||||
'2026-01-01T00:00:00.000Z',
|
||||
'2026-01-01T00:00:00.000Z',
|
||||
);
|
||||
const service = new ComponentBackupService(db, {
|
||||
version: 'test',
|
||||
backupDirectory: await directory(),
|
||||
});
|
||||
expect(service.autoSettings()).toMatchObject({
|
||||
enabled: false,
|
||||
timeOfDay: '02:00',
|
||||
components: ['devices', 'notifications', 'automation'],
|
||||
});
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,649 @@
|
||||
import { createHash } from 'node:crypto';
|
||||
import { mkdir, readFile, readdir, rm, stat, writeFile } from 'node:fs/promises';
|
||||
import { basename, join } from 'node:path';
|
||||
|
||||
import type { SqliteDatabase } from '../../infrastructure/database/database.js';
|
||||
|
||||
/**
|
||||
* Backup units the console can export and merge back. Each unit is closed under its own foreign
|
||||
* keys, so restoring one never leaves a dangling reference behind.
|
||||
*/
|
||||
export type BackupComponentKey =
|
||||
| 'devices'
|
||||
| 'notifications'
|
||||
| 'notificationRecords'
|
||||
| 'automation'
|
||||
| 'automationRecords'
|
||||
| 'jobs'
|
||||
| 'sms'
|
||||
| 'settings'
|
||||
| 'audit';
|
||||
|
||||
interface ComponentSpec {
|
||||
readonly label: string;
|
||||
readonly description: string;
|
||||
/** Parent tables first so an upsert never writes a child before its owner exists. */
|
||||
readonly tables: readonly string[];
|
||||
}
|
||||
|
||||
export const BACKUP_COMPONENTS: Readonly<Record<BackupComponentKey, ComponentSpec>> = Object.freeze(
|
||||
{
|
||||
devices: {
|
||||
label: '设备与分组',
|
||||
description: '节点地址、分组、标签与能力快照;密钥材料不会被写入备份文件。',
|
||||
tables: ['device_groups', 'tag_registry', 'instances', 'instance_tags', 'capabilities'],
|
||||
},
|
||||
notifications: {
|
||||
label: '通知配置',
|
||||
description: '通知渠道与转发规则。',
|
||||
tables: ['notification_channels', 'notification_rules'],
|
||||
},
|
||||
notificationRecords: {
|
||||
label: '通知记录',
|
||||
description: '待办队列与历史投递结果。',
|
||||
tables: ['notification_queue', 'notification_deliveries'],
|
||||
},
|
||||
automation: {
|
||||
label: '自动化配置',
|
||||
description: '定时任务定义与执行计划。',
|
||||
tables: ['scheduled_tasks'],
|
||||
},
|
||||
automationRecords: {
|
||||
label: '自动化记录',
|
||||
description: '每次定时执行的状态与结果。',
|
||||
tables: ['scheduled_runs'],
|
||||
},
|
||||
jobs: {
|
||||
label: '命令历史',
|
||||
description: '集中下发的设备操作命令与逐设备执行明细。',
|
||||
tables: ['jobs', 'job_items', 'job_attempts'],
|
||||
},
|
||||
sms: {
|
||||
label: '短信记录',
|
||||
description: '集中保存的跨设备短信正文与会话。',
|
||||
tables: ['sms_messages'],
|
||||
},
|
||||
settings: {
|
||||
label: '系统设置',
|
||||
description: '保留策略、备份计划与控制台安全配置。',
|
||||
tables: ['app_settings', 'console_auth_config'],
|
||||
},
|
||||
audit: {
|
||||
label: '操作审计',
|
||||
description: '设备操作审计账本。',
|
||||
tables: ['audit_events'],
|
||||
},
|
||||
} as const,
|
||||
);
|
||||
|
||||
export const BACKUP_COMPONENT_KEYS = Object.freeze(
|
||||
Object.keys(BACKUP_COMPONENTS) as BackupComponentKey[],
|
||||
);
|
||||
|
||||
const FORMAT = 'multi-simadmin-component-backup';
|
||||
const FORMAT_VERSION = 1;
|
||||
const AUTO_SETTINGS_KEY = 'system.component_backup.auto';
|
||||
const MAX_NOTE_LENGTH = 120;
|
||||
const MAX_FILE_BYTES = 256 * 1024 * 1024;
|
||||
const BACKUP_FILENAME = /^multi-simadmin-components-(?:auto-)?[A-Za-z0-9._-]{1,120}\.json$/u;
|
||||
|
||||
interface TableSnapshot {
|
||||
readonly columns: readonly string[];
|
||||
readonly rows: readonly (readonly unknown[])[];
|
||||
}
|
||||
|
||||
interface BackupDocument {
|
||||
readonly format: string;
|
||||
readonly formatVersion: number;
|
||||
readonly appVersion: string;
|
||||
readonly createdAt: string;
|
||||
readonly note: string;
|
||||
readonly components: Readonly<Partial<Record<BackupComponentKey, Record<string, TableSnapshot>>>>;
|
||||
readonly digest: string;
|
||||
}
|
||||
|
||||
export interface BackupComponentSummary {
|
||||
readonly key: BackupComponentKey;
|
||||
readonly label: string;
|
||||
readonly description: string;
|
||||
readonly rows: number;
|
||||
}
|
||||
|
||||
export interface BackupSummary {
|
||||
readonly filename: string;
|
||||
readonly createdAt: string;
|
||||
readonly sizeBytes: number;
|
||||
readonly appVersion: string;
|
||||
readonly formatVersion: number;
|
||||
readonly note: string;
|
||||
readonly automatic: boolean;
|
||||
readonly integrity: 'ok' | 'failed';
|
||||
readonly compatible: boolean;
|
||||
readonly components: readonly BackupComponentSummary[];
|
||||
}
|
||||
|
||||
export interface AutoBackupSettings {
|
||||
readonly enabled: boolean;
|
||||
readonly components: readonly BackupComponentKey[];
|
||||
readonly timeOfDay: string;
|
||||
readonly weekday: number;
|
||||
readonly maximumCount: number;
|
||||
readonly lastRunAt: string | null;
|
||||
}
|
||||
|
||||
export interface RestoreResult {
|
||||
readonly written: Partial<Record<BackupComponentKey, number>>;
|
||||
/** Filename of the snapshot taken just before the merge, so a restore is always reversible. */
|
||||
readonly safetyBackup: string;
|
||||
}
|
||||
|
||||
/** Download metadata without parsing an already validated archive again. */
|
||||
export interface ComponentBackupFile {
|
||||
readonly filename: string;
|
||||
readonly path: string;
|
||||
readonly sizeBytes: number;
|
||||
readonly createdAt: string;
|
||||
}
|
||||
|
||||
export class ComponentBackupError extends Error {
|
||||
constructor(
|
||||
readonly code:
|
||||
| 'VALIDATION_FAILED'
|
||||
| 'NOT_FOUND'
|
||||
| 'INTEGRITY_FAILED'
|
||||
| 'INCOMPATIBLE'
|
||||
| 'TOO_LARGE',
|
||||
) {
|
||||
super(code);
|
||||
this.name = 'ComponentBackupError';
|
||||
}
|
||||
}
|
||||
|
||||
const isRecord = (value: unknown): value is Record<string, unknown> =>
|
||||
typeof value === 'object' && value !== null && !Array.isArray(value);
|
||||
|
||||
function digestOf(value: unknown): string {
|
||||
return createHash('sha256').update(JSON.stringify(value), 'utf8').digest('hex');
|
||||
}
|
||||
|
||||
function tableExists(db: SqliteDatabase, table: string): boolean {
|
||||
return (
|
||||
(db.prepare("SELECT name FROM sqlite_master WHERE type = 'table' AND name = ?").get(table) as
|
||||
| { name: string }
|
||||
| undefined) !== undefined
|
||||
);
|
||||
}
|
||||
|
||||
function primaryKeyOf(db: SqliteDatabase, table: string): readonly string[] {
|
||||
const info = db.prepare(`PRAGMA table_info(${table})`).all() as { name: string; pk: number }[];
|
||||
return info.filter((column) => column.pk > 0).map((column) => column.name);
|
||||
}
|
||||
|
||||
function normalizeComponents(value: unknown): BackupComponentKey[] {
|
||||
if (!Array.isArray(value) || value.length === 0)
|
||||
throw new ComponentBackupError('VALIDATION_FAILED');
|
||||
const seen = new Set<BackupComponentKey>();
|
||||
for (const entry of value) {
|
||||
if (typeof entry !== 'string' || !(entry in BACKUP_COMPONENTS))
|
||||
throw new ComponentBackupError('VALIDATION_FAILED');
|
||||
seen.add(entry as BackupComponentKey);
|
||||
}
|
||||
// Emit in catalog order so the serialized document, and therefore its digest, is stable.
|
||||
return BACKUP_COMPONENT_KEYS.filter((key) => seen.has(key));
|
||||
}
|
||||
|
||||
function normalizeNote(value: unknown): string {
|
||||
if (value === undefined || value === null) return '';
|
||||
if (typeof value !== 'string') throw new ComponentBackupError('VALIDATION_FAILED');
|
||||
const trimmed = value.trim();
|
||||
if (trimmed.length > MAX_NOTE_LENGTH) throw new ComponentBackupError('VALIDATION_FAILED');
|
||||
return trimmed;
|
||||
}
|
||||
|
||||
function normalizeTimeOfDay(value: unknown): string {
|
||||
if (typeof value !== 'string' || !/^([01]\d|2[0-3]):[0-5]\d$/u.test(value))
|
||||
throw new ComponentBackupError('VALIDATION_FAILED');
|
||||
return value;
|
||||
}
|
||||
|
||||
export interface ComponentBackupOptions {
|
||||
readonly version: string;
|
||||
readonly backupDirectory: string;
|
||||
readonly now?: () => Date;
|
||||
}
|
||||
|
||||
/** Component-scoped export and merge, mirroring the Hub backup centre. */
|
||||
export class ComponentBackupService {
|
||||
readonly #db: SqliteDatabase;
|
||||
readonly #version: string;
|
||||
readonly #directory: string;
|
||||
readonly #now: () => Date;
|
||||
|
||||
constructor(db: SqliteDatabase, options: ComponentBackupOptions) {
|
||||
this.#db = db;
|
||||
this.#version = options.version;
|
||||
this.#directory = options.backupDirectory;
|
||||
this.#now = options.now ?? (() => new Date());
|
||||
}
|
||||
|
||||
/** Live row counts for every unit the console knows how to back up. */
|
||||
catalog(): readonly BackupComponentSummary[] {
|
||||
return BACKUP_COMPONENT_KEYS.map((key) => ({
|
||||
key,
|
||||
label: BACKUP_COMPONENTS[key].label,
|
||||
description: BACKUP_COMPONENTS[key].description,
|
||||
rows: this.#countComponent(key),
|
||||
}));
|
||||
}
|
||||
|
||||
async create(
|
||||
components: readonly BackupComponentKey[],
|
||||
note: unknown = '',
|
||||
automatic = false,
|
||||
): Promise<BackupSummary> {
|
||||
const keys = normalizeComponents(components);
|
||||
const comment = normalizeNote(note);
|
||||
const createdAt = this.#now().toISOString();
|
||||
const payload: Record<string, Record<string, TableSnapshot>> = {};
|
||||
for (const key of keys) {
|
||||
const tables: Record<string, TableSnapshot> = {};
|
||||
for (const table of BACKUP_COMPONENTS[key].tables) {
|
||||
const snapshot = this.#snapshot(table);
|
||||
if (snapshot) tables[table] = snapshot;
|
||||
}
|
||||
payload[key] = tables;
|
||||
}
|
||||
const body = { format: FORMAT, formatVersion: FORMAT_VERSION, createdAt, components: payload };
|
||||
const document: BackupDocument = {
|
||||
format: FORMAT,
|
||||
formatVersion: FORMAT_VERSION,
|
||||
appVersion: this.#version,
|
||||
createdAt,
|
||||
note: comment,
|
||||
components: payload,
|
||||
digest: digestOf(body),
|
||||
};
|
||||
const serialized = JSON.stringify(document);
|
||||
if (Buffer.byteLength(serialized, 'utf8') > MAX_FILE_BYTES)
|
||||
throw new ComponentBackupError('TOO_LARGE');
|
||||
await mkdir(this.#directory, { recursive: true });
|
||||
const filename = `multi-simadmin-components-${automatic ? 'auto-' : ''}${createdAt.replaceAll(/[:.]/gu, '-')}.json`;
|
||||
await writeFile(join(this.#directory, filename), serialized, { mode: 0o600 });
|
||||
return this.#summarize(filename, document, Buffer.byteLength(serialized, 'utf8'));
|
||||
}
|
||||
|
||||
async list(): Promise<readonly BackupSummary[]> {
|
||||
await mkdir(this.#directory, { recursive: true });
|
||||
const names = (await readdir(this.#directory))
|
||||
.filter((name) => BACKUP_FILENAME.test(name))
|
||||
.sort()
|
||||
.reverse();
|
||||
const items: BackupSummary[] = [];
|
||||
for (const filename of names) {
|
||||
const parsed = await this.#read(filename).catch(() => undefined);
|
||||
const details = await stat(join(this.#directory, filename)).catch(() => undefined);
|
||||
if (!details?.isFile()) continue;
|
||||
items.push(
|
||||
parsed
|
||||
? await this.#summarize(filename, parsed.document, details.size)
|
||||
: {
|
||||
filename,
|
||||
createdAt: details.mtime.toISOString(),
|
||||
sizeBytes: details.size,
|
||||
appVersion: '',
|
||||
formatVersion: 0,
|
||||
note: '',
|
||||
automatic: filename.includes('-auto-'),
|
||||
integrity: 'failed',
|
||||
compatible: false,
|
||||
components: [],
|
||||
},
|
||||
);
|
||||
}
|
||||
return items;
|
||||
}
|
||||
|
||||
async preview(filename: string): Promise<BackupSummary> {
|
||||
const { document, sizeBytes } = await this.#read(this.#safeName(filename));
|
||||
return this.#summarize(this.#safeName(filename), document, sizeBytes);
|
||||
}
|
||||
|
||||
/** Merges the selected units back in; untouched components and rows stay exactly as they were. */
|
||||
async restore(
|
||||
filename: string,
|
||||
components: readonly BackupComponentKey[],
|
||||
): Promise<RestoreResult> {
|
||||
const safe = this.#safeName(filename);
|
||||
const keys = normalizeComponents(components);
|
||||
const { document } = await this.#read(safe);
|
||||
if (document.formatVersion > FORMAT_VERSION) throw new ComponentBackupError('INCOMPATIBLE');
|
||||
// Snapshot the units that are about to be overwritten first: a restore must always be
|
||||
// reversible, and the state right before the merge is the only thing worth returning to.
|
||||
const safety = await this.create(keys, `恢复前自动备份:${safe}`, true);
|
||||
const written: Partial<Record<BackupComponentKey, number>> = {};
|
||||
const restore = this.#db.transaction(() => {
|
||||
for (const key of keys) {
|
||||
const tables = isRecord(document.components[key])
|
||||
? (document.components[key] as Record<string, TableSnapshot>)
|
||||
: {};
|
||||
let count = 0;
|
||||
for (const table of BACKUP_COMPONENTS[key].tables) {
|
||||
const snapshot = tables[table];
|
||||
if (!snapshot || !Array.isArray(snapshot.columns) || !Array.isArray(snapshot.rows))
|
||||
continue;
|
||||
count += this.#merge(table, snapshot);
|
||||
}
|
||||
written[key] = count;
|
||||
}
|
||||
});
|
||||
restore();
|
||||
return { written, safetyBackup: safety.filename };
|
||||
}
|
||||
|
||||
/** Where the backup files live, shown so an operator can mirror the folder off-console. */
|
||||
directory(): string {
|
||||
return this.#directory;
|
||||
}
|
||||
|
||||
async remove(filename: string): Promise<{ filename: string }> {
|
||||
await rm(join(this.#directory, this.#safeName(filename)), { force: true });
|
||||
return { filename: this.#safeName(filename) };
|
||||
}
|
||||
|
||||
/** Resolves a listed archive to a path that stays inside the backup directory. */
|
||||
async backupFile(filename: string): Promise<ComponentBackupFile | undefined> {
|
||||
const safe = this.#safeName(filename);
|
||||
const path = join(this.#directory, safe);
|
||||
const details = await stat(path).catch((error: NodeJS.ErrnoException) => {
|
||||
if (error.code === 'ENOENT') return undefined;
|
||||
throw error;
|
||||
});
|
||||
if (!details?.isFile()) return undefined;
|
||||
return {
|
||||
filename: safe,
|
||||
path,
|
||||
sizeBytes: details.size,
|
||||
createdAt: details.mtime.toISOString(),
|
||||
};
|
||||
}
|
||||
|
||||
autoSettings(): AutoBackupSettings {
|
||||
return this.#readAuto();
|
||||
}
|
||||
|
||||
updateAutoSettings(value: unknown): AutoBackupSettings {
|
||||
const source = isRecord(value) ? value : {};
|
||||
if (typeof source.enabled !== 'boolean') throw new ComponentBackupError('VALIDATION_FAILED');
|
||||
const weekday = source.weekday;
|
||||
if (
|
||||
typeof weekday !== 'number' ||
|
||||
!Number.isSafeInteger(weekday) ||
|
||||
weekday < -1 ||
|
||||
weekday > 6
|
||||
)
|
||||
throw new ComponentBackupError('VALIDATION_FAILED');
|
||||
const maximumCount = source.maximumCount;
|
||||
if (
|
||||
typeof maximumCount !== 'number' ||
|
||||
!Number.isSafeInteger(maximumCount) ||
|
||||
maximumCount < 1 ||
|
||||
maximumCount > 500
|
||||
)
|
||||
throw new ComponentBackupError('VALIDATION_FAILED');
|
||||
const settings: AutoBackupSettings = {
|
||||
enabled: source.enabled,
|
||||
components: normalizeComponents(source.components),
|
||||
timeOfDay: normalizeTimeOfDay(source.timeOfDay),
|
||||
weekday,
|
||||
maximumCount,
|
||||
lastRunAt: this.#readAuto().lastRunAt,
|
||||
};
|
||||
const now = this.#now().toISOString();
|
||||
this.#db
|
||||
.prepare(
|
||||
`INSERT INTO app_settings (key, value_json, created_at, updated_at)
|
||||
VALUES (?, ?, ?, ?)
|
||||
ON CONFLICT(key) DO UPDATE SET value_json = excluded.value_json, updated_at = excluded.updated_at`,
|
||||
)
|
||||
.run(AUTO_SETTINGS_KEY, JSON.stringify({ ...settings, lastRunAt: undefined }), now, now);
|
||||
return settings;
|
||||
}
|
||||
|
||||
/** Creates the scheduled backup when the current period has not run yet. */
|
||||
async runDueAutoBackups(): Promise<BackupSummary | null> {
|
||||
const settings = this.#readAuto();
|
||||
if (!settings.enabled || settings.components.length === 0) return null;
|
||||
const dueAt = this.#mostRecentOccurrence(settings.timeOfDay, settings.weekday);
|
||||
if (settings.lastRunAt && Date.parse(settings.lastRunAt) >= dueAt.getTime()) return null;
|
||||
const summary = await this.create(settings.components, '定时自动备份', true);
|
||||
const stamped = this.#now().toISOString();
|
||||
this.#db
|
||||
.prepare(
|
||||
`INSERT INTO app_settings (key, value_json, created_at, updated_at)
|
||||
VALUES (?, ?, ?, ?)
|
||||
ON CONFLICT(key) DO UPDATE SET value_json = excluded.value_json, updated_at = excluded.updated_at`,
|
||||
)
|
||||
.run(
|
||||
AUTO_SETTINGS_KEY,
|
||||
JSON.stringify({ ...settings, lastRunAt: stamped }),
|
||||
stamped,
|
||||
stamped,
|
||||
);
|
||||
await this.#prune(settings.maximumCount);
|
||||
return summary;
|
||||
}
|
||||
|
||||
#countComponent(key: BackupComponentKey): number {
|
||||
let total = 0;
|
||||
for (const table of BACKUP_COMPONENTS[key].tables) {
|
||||
if (!tableExists(this.#db, table)) continue;
|
||||
const row = this.#db.prepare(`SELECT COUNT(*) AS count FROM ${table}`).get() as
|
||||
| { count: number }
|
||||
| undefined;
|
||||
total += Number(row?.count ?? 0);
|
||||
}
|
||||
return total;
|
||||
}
|
||||
|
||||
#snapshot(table: string): TableSnapshot | undefined {
|
||||
if (!tableExists(this.#db, table)) return undefined;
|
||||
const info = this.#db.prepare(`PRAGMA table_info(${table})`).all() as { name: string }[];
|
||||
const columns = info.map((column) => column.name);
|
||||
const rows = this.#db.prepare(`SELECT * FROM ${table} ORDER BY rowid`).all() as Record<
|
||||
string,
|
||||
unknown
|
||||
>[];
|
||||
return {
|
||||
columns,
|
||||
rows: rows.map((row) => columns.map((column) => row[column] ?? null)),
|
||||
};
|
||||
}
|
||||
|
||||
/** Upserts one table; a conflict updates every non-key column instead of replacing the row. */
|
||||
#merge(table: string, snapshot: TableSnapshot): number {
|
||||
if (!tableExists(this.#db, table)) return 0;
|
||||
const columns = snapshot.columns.filter((column) => typeof column === 'string');
|
||||
if (columns.length === 0) return 0;
|
||||
const keys = primaryKeyOf(this.#db, table);
|
||||
const placeholders = columns.map(() => '?').join(', ');
|
||||
const quoted = columns.map((column) => `"${column.replaceAll('"', '""')}"`).join(', ');
|
||||
let sql = `INSERT INTO ${table} (${quoted}) VALUES (${placeholders})`;
|
||||
const updates = columns.filter((column) => !keys.includes(column));
|
||||
if (keys.length > 0 && updates.length > 0)
|
||||
sql += ` ON CONFLICT(${keys.map((key) => `"${key}"`).join(', ')}) DO UPDATE SET ${updates
|
||||
.map((column) => `"${column}" = excluded."${column}"`)
|
||||
.join(', ')}`;
|
||||
else if (keys.length > 0)
|
||||
sql += ` ON CONFLICT(${keys.map((key) => `"${key}"`).join(', ')}) DO NOTHING`;
|
||||
const statement = this.#db.prepare(sql);
|
||||
let written = 0;
|
||||
for (const row of snapshot.rows) {
|
||||
if (!Array.isArray(row) || row.length !== columns.length) continue;
|
||||
statement.run(...row.map((value) => (typeof value === 'boolean' ? Number(value) : value)));
|
||||
written += 1;
|
||||
}
|
||||
return written;
|
||||
}
|
||||
|
||||
#safeName(filename: string): string {
|
||||
if (!BACKUP_FILENAME.test(filename) || basename(filename) !== filename)
|
||||
throw new ComponentBackupError('VALIDATION_FAILED');
|
||||
return filename;
|
||||
}
|
||||
|
||||
async #read(filename: string): Promise<{ document: BackupDocument; sizeBytes: number }> {
|
||||
const path = join(this.#directory, filename);
|
||||
const content = await readFile(path).catch((error: NodeJS.ErrnoException) => {
|
||||
if (error.code === 'ENOENT') throw new ComponentBackupError('NOT_FOUND');
|
||||
throw error;
|
||||
});
|
||||
if (content.byteLength > MAX_FILE_BYTES) throw new ComponentBackupError('TOO_LARGE');
|
||||
let parsed: unknown;
|
||||
try {
|
||||
parsed = JSON.parse(content.toString('utf8'));
|
||||
} catch {
|
||||
throw new ComponentBackupError('INTEGRITY_FAILED');
|
||||
}
|
||||
const document = this.#validate(parsed);
|
||||
return { document, sizeBytes: content.byteLength };
|
||||
}
|
||||
|
||||
#validate(value: unknown): BackupDocument {
|
||||
if (!isRecord(value) || value.format !== FORMAT)
|
||||
throw new ComponentBackupError('INTEGRITY_FAILED');
|
||||
const formatVersion = Number(value.formatVersion);
|
||||
const components = isRecord(value.components) ? value.components : {};
|
||||
const document: BackupDocument = {
|
||||
format: FORMAT,
|
||||
formatVersion: Number.isSafeInteger(formatVersion) ? formatVersion : 0,
|
||||
appVersion: typeof value.appVersion === 'string' ? value.appVersion : '',
|
||||
createdAt: typeof value.createdAt === 'string' ? value.createdAt : '',
|
||||
note: typeof value.note === 'string' ? value.note : '',
|
||||
components: components as BackupDocument['components'],
|
||||
digest: typeof value.digest === 'string' ? value.digest : '',
|
||||
};
|
||||
const expected = digestOf({
|
||||
format: document.format,
|
||||
formatVersion: document.formatVersion,
|
||||
createdAt: document.createdAt,
|
||||
components: document.components,
|
||||
});
|
||||
if (document.digest !== expected) throw new ComponentBackupError('INTEGRITY_FAILED');
|
||||
return document;
|
||||
}
|
||||
|
||||
async #summarize(
|
||||
filename: string,
|
||||
document: BackupDocument,
|
||||
sizeBytes: number,
|
||||
): Promise<BackupSummary> {
|
||||
const components: BackupComponentSummary[] = [];
|
||||
for (const key of BACKUP_COMPONENT_KEYS) {
|
||||
const tables = isRecord(document.components[key])
|
||||
? (document.components[key] as Record<string, TableSnapshot>)
|
||||
: undefined;
|
||||
if (!tables) continue;
|
||||
let rows = 0;
|
||||
for (const snapshot of Object.values(tables))
|
||||
rows += Array.isArray(snapshot?.rows) ? snapshot.rows.length : 0;
|
||||
components.push({
|
||||
key,
|
||||
label: BACKUP_COMPONENTS[key].label,
|
||||
description: BACKUP_COMPONENTS[key].description,
|
||||
rows,
|
||||
});
|
||||
}
|
||||
return {
|
||||
filename,
|
||||
createdAt: document.createdAt,
|
||||
sizeBytes,
|
||||
appVersion: document.appVersion,
|
||||
formatVersion: document.formatVersion,
|
||||
note: document.note,
|
||||
automatic: filename.includes('-auto-'),
|
||||
integrity: 'ok',
|
||||
compatible: document.formatVersion <= FORMAT_VERSION,
|
||||
components,
|
||||
};
|
||||
}
|
||||
|
||||
#readAuto(): AutoBackupSettings {
|
||||
const row = this.#db
|
||||
.prepare('SELECT value_json FROM app_settings WHERE key = ?')
|
||||
.get(AUTO_SETTINGS_KEY) as { value_json: string } | undefined;
|
||||
let stored: Record<string, unknown> | undefined;
|
||||
try {
|
||||
const parsed: unknown = row ? JSON.parse(row.value_json) : undefined;
|
||||
stored = isRecord(parsed) ? parsed : undefined;
|
||||
} catch {
|
||||
stored = undefined;
|
||||
}
|
||||
const fallback = {
|
||||
enabled: false,
|
||||
components: ['devices', 'notifications', 'automation'] as BackupComponentKey[],
|
||||
timeOfDay: '02:00',
|
||||
weekday: -1,
|
||||
maximumCount: 10,
|
||||
};
|
||||
const source: Record<string, unknown> = { ...fallback, ...(stored ?? {}) };
|
||||
let components: BackupComponentKey[] = fallback.components;
|
||||
try {
|
||||
components = normalizeComponents(source.components);
|
||||
} catch {
|
||||
components = fallback.components;
|
||||
}
|
||||
let timeOfDay = fallback.timeOfDay;
|
||||
try {
|
||||
timeOfDay = normalizeTimeOfDay(source.timeOfDay);
|
||||
} catch {
|
||||
timeOfDay = fallback.timeOfDay;
|
||||
}
|
||||
const weekday =
|
||||
typeof source.weekday === 'number' && Number.isSafeInteger(source.weekday)
|
||||
? Math.min(6, Math.max(-1, source.weekday))
|
||||
: fallback.weekday;
|
||||
const maximumCount =
|
||||
typeof source.maximumCount === 'number' && Number.isSafeInteger(source.maximumCount)
|
||||
? Math.min(500, Math.max(1, source.maximumCount))
|
||||
: fallback.maximumCount;
|
||||
return {
|
||||
enabled: source.enabled === true,
|
||||
components,
|
||||
timeOfDay,
|
||||
weekday,
|
||||
maximumCount,
|
||||
lastRunAt: typeof source.lastRunAt === 'string' ? source.lastRunAt : null,
|
||||
};
|
||||
}
|
||||
|
||||
#mostRecentOccurrence(timeOfDay: string, weekday: number): Date {
|
||||
const [hour, minute] = timeOfDay.split(':').map((part) => Number(part));
|
||||
const now = this.#now();
|
||||
const candidate = new Date(now);
|
||||
candidate.setHours(hour ?? 0, minute ?? 0, 0, 0);
|
||||
if (candidate.getTime() > now.getTime()) candidate.setDate(candidate.getDate() - 1);
|
||||
if (weekday >= 0) {
|
||||
let guard = 0;
|
||||
while (candidate.getDay() !== weekday && guard < 8) {
|
||||
candidate.setDate(candidate.getDate() - 1);
|
||||
guard += 1;
|
||||
}
|
||||
}
|
||||
return candidate;
|
||||
}
|
||||
|
||||
/** Keeps the newest `limit` scheduled backups; a limit of zero leaves manual files alone. */
|
||||
async #prune(limit: number): Promise<number> {
|
||||
if (limit <= 0) return 0;
|
||||
const names = (await readdir(this.#directory))
|
||||
.filter((name) => name.startsWith('multi-simadmin-components-auto-'))
|
||||
.sort()
|
||||
.reverse();
|
||||
let removed = 0;
|
||||
for (const filename of names.slice(limit)) {
|
||||
await rm(join(this.#directory, filename), { force: true });
|
||||
removed += 1;
|
||||
}
|
||||
return removed;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,179 @@
|
||||
import Database from 'better-sqlite3';
|
||||
import { afterEach, describe, expect, it } from 'vitest';
|
||||
|
||||
import { migrateDatabase } from '../../infrastructure/database/migrations.js';
|
||||
import { ConnectionLogService } from './connection-log-service.js';
|
||||
|
||||
const databases: Database.Database[] = [];
|
||||
|
||||
function fixture(maximumRows?: number) {
|
||||
const db = new Database(':memory:');
|
||||
db.pragma('foreign_keys = ON');
|
||||
migrateDatabase(db);
|
||||
databases.push(db);
|
||||
db.prepare(
|
||||
`INSERT INTO instances (id,name,base_url,enabled,config_revision,created_at,updated_at)
|
||||
VALUES ('node-a','Node A','http://node-a.local',1,1,'2026-09-01T00:00:00.000Z','2026-09-01T00:00:00.000Z'),
|
||||
('node-b','Node B','http://node-b.local',1,1,'2026-09-01T00:00:00.000Z','2026-09-01T00:00:00.000Z')`,
|
||||
).run();
|
||||
let counter = 0;
|
||||
const service = new ConnectionLogService({
|
||||
db,
|
||||
idFactory: () => `log-${++counter}`,
|
||||
...(maximumRows === undefined ? {} : { maximumRows }),
|
||||
});
|
||||
return { db, service };
|
||||
}
|
||||
|
||||
afterEach(() => {
|
||||
for (const db of databases.splice(0)) db.close();
|
||||
});
|
||||
|
||||
describe('ConnectionLogService', () => {
|
||||
it('records probes and lists them newest first with paging metadata', () => {
|
||||
const { service } = fixture();
|
||||
service.record({
|
||||
instanceId: 'node-a',
|
||||
outcome: 'success',
|
||||
state: 'fresh',
|
||||
errorCode: null,
|
||||
httpStatus: 200,
|
||||
durationMs: 12,
|
||||
observedAt: '2026-09-02T00:00:00.000Z',
|
||||
});
|
||||
service.record({
|
||||
instanceId: 'node-b',
|
||||
outcome: 'failed',
|
||||
state: 'unknown',
|
||||
errorCode: 'ECONNREFUSED',
|
||||
httpStatus: null,
|
||||
durationMs: 30,
|
||||
observedAt: '2026-09-03T00:00:00.000Z',
|
||||
});
|
||||
|
||||
const page = service.list({ pageSize: 1 });
|
||||
expect(page.page).toEqual({ page: 1, pageSize: 1, total: 2 });
|
||||
expect(page.items.map((item) => item.instanceId)).toEqual(['node-b']);
|
||||
expect(page.items[0]).toMatchObject({ errorCode: 'ECONNREFUSED', outcome: 'failed' });
|
||||
});
|
||||
|
||||
it('filters by outcome and free-text search', () => {
|
||||
const { service } = fixture();
|
||||
service.record({
|
||||
instanceId: 'node-a',
|
||||
outcome: 'stale',
|
||||
state: 'stale',
|
||||
errorCode: 'AUTH_REQUIRED',
|
||||
httpStatus: 401,
|
||||
durationMs: 8,
|
||||
observedAt: '2026-09-02T00:00:00.000Z',
|
||||
});
|
||||
service.record({
|
||||
instanceId: 'node-a',
|
||||
outcome: 'success',
|
||||
state: 'fresh',
|
||||
errorCode: null,
|
||||
httpStatus: 200,
|
||||
durationMs: 9,
|
||||
observedAt: '2026-09-02T01:00:00.000Z',
|
||||
});
|
||||
expect(service.list({ outcome: 'stale' }).page.total).toBe(1);
|
||||
expect(service.list({ search: 'auth' }).page.total).toBe(1);
|
||||
expect(service.list({ search: 'node-a', outcome: 'success' }).page.total).toBe(1);
|
||||
});
|
||||
|
||||
it('summarises availability per instance', () => {
|
||||
const { service } = fixture();
|
||||
for (const [index, outcome] of (
|
||||
['success', 'success', 'success', 'failed'] as const
|
||||
).entries()) {
|
||||
service.record({
|
||||
instanceId: 'node-a',
|
||||
outcome,
|
||||
state: outcome === 'success' ? 'fresh' : 'unknown',
|
||||
errorCode: outcome === 'success' ? null : 'UPSTREAM_TIMEOUT',
|
||||
httpStatus: outcome === 'success' ? 200 : null,
|
||||
durationMs: 10 + index,
|
||||
observedAt: `2026-09-0${index + 1}T00:00:00.000Z`,
|
||||
});
|
||||
}
|
||||
const [summary] = service.summarize(['node-a']);
|
||||
expect(summary).toMatchObject({
|
||||
instanceId: 'node-a',
|
||||
total: 4,
|
||||
success: 3,
|
||||
failed: 1,
|
||||
availabilityPercent: 75,
|
||||
lastErrorCode: 'UPSTREAM_TIMEOUT',
|
||||
lastObservedAt: '2026-09-04T00:00:00.000Z',
|
||||
});
|
||||
expect(service.summarize()).toHaveLength(1);
|
||||
});
|
||||
|
||||
it('drops the oldest rows once the rolling cap is reached', () => {
|
||||
const { db, service } = fixture(100);
|
||||
for (let index = 0; index < 105; index += 1) {
|
||||
service.record({
|
||||
instanceId: 'node-a',
|
||||
outcome: 'success',
|
||||
state: 'fresh',
|
||||
errorCode: null,
|
||||
httpStatus: 200,
|
||||
durationMs: index,
|
||||
observedAt: new Date(Date.UTC(2026, 0, 1, 0, 0, index)).toISOString(),
|
||||
});
|
||||
}
|
||||
const total = (
|
||||
db.prepare('SELECT COUNT(*) AS count FROM connection_logs').get() as {
|
||||
count: number;
|
||||
}
|
||||
).count;
|
||||
expect(total).toBe(100);
|
||||
expect(service.list({ pageSize: 100 }).items.at(-1)).toMatchObject({ durationMs: 5 });
|
||||
});
|
||||
|
||||
it('ignores malformed drafts instead of throwing', () => {
|
||||
const { service } = fixture();
|
||||
expect(
|
||||
service.record({
|
||||
instanceId: '',
|
||||
outcome: 'success',
|
||||
state: 'fresh',
|
||||
durationMs: 1,
|
||||
observedAt: '2026-09-02T00:00:00.000Z',
|
||||
}),
|
||||
).toBeUndefined();
|
||||
expect(
|
||||
service.record({
|
||||
instanceId: 'node-a',
|
||||
outcome: 'nope' as never,
|
||||
state: 'fresh',
|
||||
durationMs: 1,
|
||||
observedAt: '2026-09-02T00:00:00.000Z',
|
||||
}),
|
||||
).toBeUndefined();
|
||||
expect(service.list().page.total).toBe(0);
|
||||
});
|
||||
|
||||
it('prunes by cutoff and refuses an unbounded delete', () => {
|
||||
const { service } = fixture();
|
||||
service.record({
|
||||
instanceId: 'node-a',
|
||||
outcome: 'success',
|
||||
state: 'fresh',
|
||||
durationMs: 1,
|
||||
observedAt: '2026-01-01T00:00:00.000Z',
|
||||
});
|
||||
service.record({
|
||||
instanceId: 'node-a',
|
||||
outcome: 'success',
|
||||
state: 'fresh',
|
||||
durationMs: 1,
|
||||
observedAt: '2026-09-01T00:00:00.000Z',
|
||||
});
|
||||
expect(() => service.prune({})).toThrow(/before or instanceId is required/u);
|
||||
expect(service.prune({ before: '2026-06-01T00:00:00.000Z' })).toBe(1);
|
||||
expect(service.list().page.total).toBe(1);
|
||||
expect(service.prune({ instanceId: 'node-a' })).toBe(1);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,305 @@
|
||||
import { randomUUID } from 'node:crypto';
|
||||
|
||||
import type Database from 'better-sqlite3';
|
||||
|
||||
export type ConnectionOutcome = 'success' | 'stale' | 'failed' | 'unsupported';
|
||||
|
||||
/** Mirrors the status_snapshots state domain, which also covers an expired snapshot. */
|
||||
export type ProbeState = 'fresh' | 'stale' | 'expired' | 'unknown';
|
||||
|
||||
const OUTCOMES: readonly ConnectionOutcome[] = ['success', 'stale', 'failed', 'unsupported'];
|
||||
const STATES: readonly ProbeState[] = ['fresh', 'stale', 'expired', 'unknown'];
|
||||
|
||||
export interface ConnectionLogEntry {
|
||||
readonly id: string;
|
||||
readonly instanceId: string;
|
||||
readonly outcome: ConnectionOutcome;
|
||||
readonly state: ProbeState;
|
||||
readonly errorCode: string | null;
|
||||
readonly httpStatus: number | null;
|
||||
readonly durationMs: number;
|
||||
readonly observedAt: string;
|
||||
}
|
||||
|
||||
/** A probe outcome as produced by the status snapshot service on every health refresh. */
|
||||
export interface ConnectionLogDraft {
|
||||
readonly instanceId: string;
|
||||
readonly outcome: ConnectionOutcome;
|
||||
readonly state: ProbeState;
|
||||
readonly errorCode?: string | null;
|
||||
readonly httpStatus?: number | null;
|
||||
readonly durationMs: number;
|
||||
readonly observedAt: string;
|
||||
}
|
||||
|
||||
export interface ConnectionLogQuery {
|
||||
readonly page?: number;
|
||||
readonly pageSize?: number;
|
||||
readonly instanceId?: string | undefined;
|
||||
readonly outcome?: ConnectionOutcome | undefined;
|
||||
readonly search?: string | undefined;
|
||||
readonly from?: string | undefined;
|
||||
readonly to?: string | undefined;
|
||||
}
|
||||
|
||||
export interface ConnectionLogPage {
|
||||
readonly items: readonly ConnectionLogEntry[];
|
||||
readonly page: { readonly page: number; readonly pageSize: number; readonly total: number };
|
||||
}
|
||||
|
||||
export interface ConnectionSummary {
|
||||
readonly instanceId: string;
|
||||
readonly total: number;
|
||||
readonly success: number;
|
||||
readonly failed: number;
|
||||
readonly averageDurationMs: number;
|
||||
readonly lastObservedAt: string | null;
|
||||
readonly lastErrorCode: string | null;
|
||||
readonly availabilityPercent: number;
|
||||
}
|
||||
|
||||
export type ConnectionLogErrorCode = 'VALIDATION_FAILED';
|
||||
|
||||
export class ConnectionLogError extends Error {
|
||||
constructor(
|
||||
readonly code: ConnectionLogErrorCode,
|
||||
message: string,
|
||||
) {
|
||||
super(message);
|
||||
this.name = 'ConnectionLogError';
|
||||
}
|
||||
}
|
||||
|
||||
export interface ConnectionLogOptions {
|
||||
readonly db: Database.Database;
|
||||
readonly idFactory?: () => string;
|
||||
/** Rolling cap so the journal can never outgrow the rest of the control-plane database. */
|
||||
readonly maximumRows?: number;
|
||||
}
|
||||
|
||||
const DEFAULT_MAXIMUM_ROWS = 20_000;
|
||||
const MAX_PAGE_SIZE = 200;
|
||||
|
||||
function invalid(message: string): never {
|
||||
throw new ConnectionLogError('VALIDATION_FAILED', message);
|
||||
}
|
||||
|
||||
function timestamp(value: string): number {
|
||||
const parsed = Date.parse(value);
|
||||
return Number.isFinite(parsed) ? parsed : Number.NaN;
|
||||
}
|
||||
|
||||
export class ConnectionLogService {
|
||||
readonly #db: Database.Database;
|
||||
readonly #id: () => string;
|
||||
readonly #maximumRows: number;
|
||||
|
||||
constructor(options: ConnectionLogOptions) {
|
||||
this.#db = options.db;
|
||||
this.#id = options.idFactory ?? randomUUID;
|
||||
this.#maximumRows = Math.max(100, options.maximumRows ?? DEFAULT_MAXIMUM_ROWS);
|
||||
}
|
||||
|
||||
/** Best-effort write: a failed log insert must never break a health probe. */
|
||||
record(draft: ConnectionLogDraft): ConnectionLogEntry | undefined {
|
||||
if (typeof draft.instanceId !== 'string' || draft.instanceId.length === 0) return undefined;
|
||||
if (!OUTCOMES.includes(draft.outcome) || !STATES.includes(draft.state)) return undefined;
|
||||
const observedAt =
|
||||
typeof draft.observedAt === 'string' && Number.isFinite(timestamp(draft.observedAt))
|
||||
? draft.observedAt
|
||||
: new Date().toISOString();
|
||||
const entry: ConnectionLogEntry = {
|
||||
id: this.#id(),
|
||||
instanceId: draft.instanceId,
|
||||
outcome: draft.outcome,
|
||||
state: draft.state,
|
||||
errorCode: typeof draft.errorCode === 'string' ? draft.errorCode.slice(0, 64) : null,
|
||||
httpStatus:
|
||||
Number.isSafeInteger(draft.httpStatus) && (draft.httpStatus ?? 0) >= 100
|
||||
? (draft.httpStatus as number)
|
||||
: null,
|
||||
durationMs: Number.isSafeInteger(draft.durationMs)
|
||||
? Math.max(0, Math.min(Number.MAX_SAFE_INTEGER, draft.durationMs))
|
||||
: 0,
|
||||
observedAt,
|
||||
};
|
||||
try {
|
||||
this.#db
|
||||
.prepare(
|
||||
`INSERT INTO connection_logs
|
||||
(id,instance_id,outcome,state,error_code,http_status,duration_ms,observed_at)
|
||||
VALUES (@id,@instanceId,@outcome,@state,@errorCode,@httpStatus,@durationMs,@observedAt)`,
|
||||
)
|
||||
.run(entry);
|
||||
this.#enforceCap();
|
||||
} catch {
|
||||
return undefined;
|
||||
}
|
||||
return entry;
|
||||
}
|
||||
|
||||
list(query: ConnectionLogQuery = {}): ConnectionLogPage {
|
||||
const page = query.page ?? 1;
|
||||
const pageSize = query.pageSize ?? 50;
|
||||
if (!Number.isSafeInteger(page) || page < 1) invalid('page must be a positive integer');
|
||||
if (!Number.isSafeInteger(pageSize) || pageSize < 1 || pageSize > MAX_PAGE_SIZE)
|
||||
invalid(`pageSize must be between 1 and ${MAX_PAGE_SIZE}`);
|
||||
if (query.outcome !== undefined && !OUTCOMES.includes(query.outcome))
|
||||
invalid('outcome is invalid');
|
||||
if (query.instanceId !== undefined && (query.instanceId.trim() === '' || !query.instanceId))
|
||||
invalid('instanceId is invalid');
|
||||
for (const key of ['from', 'to'] as const) {
|
||||
const value = query[key];
|
||||
if (value !== undefined && !Number.isFinite(timestamp(value))) invalid(`${key} is invalid`);
|
||||
}
|
||||
if (query.search !== undefined && query.search.length > 200) invalid('search is too long');
|
||||
|
||||
const where: string[] = [];
|
||||
const parameters: Record<string, unknown> = {};
|
||||
if (query.instanceId) {
|
||||
where.push('instance_id = @instanceId');
|
||||
parameters.instanceId = query.instanceId;
|
||||
}
|
||||
if (query.outcome) {
|
||||
where.push('outcome = @outcome');
|
||||
parameters.outcome = query.outcome;
|
||||
}
|
||||
if (query.from) {
|
||||
where.push('observed_at >= @from');
|
||||
parameters.from = new Date(timestamp(query.from)).toISOString();
|
||||
}
|
||||
if (query.to) {
|
||||
where.push('observed_at <= @to');
|
||||
parameters.to = new Date(timestamp(query.to)).toISOString();
|
||||
}
|
||||
if (query.search) {
|
||||
const needle = query.search.trim().toLowerCase();
|
||||
if (needle !== '') {
|
||||
where.push(
|
||||
`(lower(instance_id) LIKE @search OR lower(COALESCE(error_code,'')) LIKE @search
|
||||
OR lower(outcome) LIKE @search OR lower(state) LIKE @search)`,
|
||||
);
|
||||
parameters.search = `%${needle}%`;
|
||||
}
|
||||
}
|
||||
const clause = where.length > 0 ? ` WHERE ${where.join(' AND ')}` : '';
|
||||
const total = (
|
||||
this.#db
|
||||
.prepare(`SELECT COUNT(*) AS count FROM connection_logs${clause}`)
|
||||
.get(parameters) as { count: number }
|
||||
).count;
|
||||
const rows = this.#db
|
||||
.prepare(
|
||||
`SELECT id,instance_id,outcome,state,error_code,http_status,duration_ms,observed_at
|
||||
FROM connection_logs${clause}
|
||||
ORDER BY observed_at DESC, rowid DESC LIMIT @limit OFFSET @offset`,
|
||||
)
|
||||
.all({ ...parameters, limit: pageSize, offset: (page - 1) * pageSize }) as Array<
|
||||
Record<string, unknown>
|
||||
>;
|
||||
return Object.freeze({
|
||||
items: Object.freeze(rows.map((row) => this.#entry(row))),
|
||||
page: Object.freeze({ page, pageSize, total }),
|
||||
});
|
||||
}
|
||||
|
||||
/** Availability rollup over the retained window, newest probe first. */
|
||||
summarize(instanceIds?: readonly string[]): readonly ConnectionSummary[] {
|
||||
const ids = instanceIds ? [...instanceIds] : undefined;
|
||||
// better-sqlite3 cannot expand an array binding, so the IN list is built positionally.
|
||||
const placeholders = ids ? ids.map((_id, index) => `@id${index}`).join(', ') : '';
|
||||
const clause = ids ? ` WHERE instance_id IN (${placeholders})` : '';
|
||||
const parameters: Record<string, unknown> = {};
|
||||
if (ids) ids.forEach((id, index) => (parameters[`id${index}`] = id));
|
||||
const rows = this.#db
|
||||
.prepare(
|
||||
`SELECT instance_id,
|
||||
COUNT(*) AS total,
|
||||
SUM(CASE WHEN outcome = 'success' THEN 1 ELSE 0 END) AS success,
|
||||
SUM(CASE WHEN outcome IN ('failed','unsupported') THEN 1 ELSE 0 END) AS failed,
|
||||
AVG(duration_ms) AS average_duration,
|
||||
MAX(observed_at) AS last_observed_at
|
||||
FROM connection_logs${clause}
|
||||
GROUP BY instance_id`,
|
||||
)
|
||||
.all(parameters) as Array<Record<string, unknown>>;
|
||||
const summaries = rows.map((row) => {
|
||||
const instanceId = String(row.instance_id ?? '');
|
||||
const total = Number(row.total ?? 0);
|
||||
const success = Number(row.success ?? 0);
|
||||
const lastObservedAt = row.last_observed_at === null ? null : String(row.last_observed_at);
|
||||
const last = this.#db
|
||||
.prepare(
|
||||
`SELECT error_code FROM connection_logs
|
||||
WHERE instance_id = ? ORDER BY observed_at DESC, rowid DESC LIMIT 1`,
|
||||
)
|
||||
.get(instanceId) as { error_code: string | null } | undefined;
|
||||
return Object.freeze({
|
||||
instanceId,
|
||||
total,
|
||||
success,
|
||||
failed: Number(row.failed ?? 0),
|
||||
averageDurationMs: Math.round(Number(row.average_duration ?? 0)),
|
||||
lastObservedAt,
|
||||
lastErrorCode: last?.error_code ?? null,
|
||||
availabilityPercent: total > 0 ? Math.round((success / total) * 1000) / 10 : 0,
|
||||
}) satisfies ConnectionSummary;
|
||||
});
|
||||
return Object.freeze(summaries);
|
||||
}
|
||||
|
||||
prune(input: {
|
||||
readonly before?: string | undefined;
|
||||
readonly instanceId?: string | undefined;
|
||||
}): number {
|
||||
const where: string[] = [];
|
||||
const parameters: Record<string, unknown> = {};
|
||||
if (input.before) {
|
||||
const cutoff = timestamp(input.before);
|
||||
if (!Number.isFinite(cutoff)) invalid('before is invalid');
|
||||
where.push('observed_at < @before');
|
||||
parameters.before = new Date(cutoff).toISOString();
|
||||
}
|
||||
if (input.instanceId) {
|
||||
where.push('instance_id = @instanceId');
|
||||
parameters.instanceId = input.instanceId;
|
||||
}
|
||||
if (where.length === 0) invalid('before or instanceId is required');
|
||||
return Number(
|
||||
this.#db.prepare(`DELETE FROM connection_logs WHERE ${where.join(' AND ')}`).run(parameters)
|
||||
.changes,
|
||||
);
|
||||
}
|
||||
|
||||
#enforceCap(): void {
|
||||
const total = (
|
||||
this.#db.prepare('SELECT COUNT(*) AS count FROM connection_logs').get() as {
|
||||
count: number;
|
||||
}
|
||||
).count;
|
||||
if (total <= this.#maximumRows) return;
|
||||
this.#db
|
||||
.prepare(
|
||||
`DELETE FROM connection_logs WHERE rowid IN (
|
||||
SELECT rowid FROM connection_logs ORDER BY observed_at DESC, rowid DESC
|
||||
LIMIT -1 OFFSET @keep
|
||||
)`,
|
||||
)
|
||||
.run({ keep: this.#maximumRows });
|
||||
}
|
||||
|
||||
#entry(row: Record<string, unknown>): ConnectionLogEntry {
|
||||
return Object.freeze({
|
||||
id: String(row.id),
|
||||
instanceId: String(row.instance_id),
|
||||
outcome: row.outcome as ConnectionOutcome,
|
||||
state: row.state as ProbeState,
|
||||
errorCode:
|
||||
row.error_code === null || row.error_code === undefined ? null : String(row.error_code),
|
||||
httpStatus:
|
||||
row.http_status === null || row.http_status === undefined ? null : Number(row.http_status),
|
||||
durationMs: Number(row.duration_ms ?? 0),
|
||||
observedAt: String(row.observed_at),
|
||||
});
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,543 @@
|
||||
import Database from 'better-sqlite3';
|
||||
import { afterEach, describe, expect, it, vi } from 'vitest';
|
||||
|
||||
import { migrateDatabase } from '../../infrastructure/database/migrations.js';
|
||||
import {
|
||||
ConsoleUpdateService,
|
||||
UpdateError,
|
||||
createLaunchdRestartLauncher,
|
||||
createPlatformRestartLauncher,
|
||||
createSystemdRestartLauncher,
|
||||
parseManifestVersion,
|
||||
parseShortStat,
|
||||
type UpdateCommandResult,
|
||||
type UpdateCommandRunner,
|
||||
} from './console-update-service.js';
|
||||
|
||||
const LOCAL = 'aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa';
|
||||
const REMOTE = 'bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb';
|
||||
|
||||
const dbs: Database.Database[] = [];
|
||||
afterEach(() => {
|
||||
for (const db of dbs.splice(0)) db.close();
|
||||
});
|
||||
|
||||
interface FakeGit {
|
||||
readonly calls: string[];
|
||||
runner: UpdateCommandRunner;
|
||||
head: string;
|
||||
remoteHead: string;
|
||||
dirty: string;
|
||||
fastForwardable: boolean;
|
||||
/** Whether HEAD already carries the remote commit: true means the checkout is ahead. */
|
||||
containsRemote: boolean;
|
||||
/** Command prefixes that should fail, e.g. `git` or `git merge-base`. */
|
||||
failing: Set<string>;
|
||||
}
|
||||
|
||||
/** Matches a `command arg ...` key against the configured failing prefixes. */
|
||||
function fails(patterns: ReadonlySet<string>, key: string): boolean {
|
||||
const tokens = key.split(' ');
|
||||
for (const pattern of patterns) {
|
||||
const wanted = pattern.split(' ');
|
||||
if (wanted.length <= tokens.length && wanted.every((token, index) => tokens[index] === token))
|
||||
return true;
|
||||
}
|
||||
return false;
|
||||
}
|
||||
|
||||
function fakeGit(overrides: Partial<Pick<FakeGit, 'head' | 'remoteHead'>> = {}): FakeGit {
|
||||
const state: FakeGit = {
|
||||
calls: [],
|
||||
head: overrides.head ?? LOCAL,
|
||||
remoteHead: overrides.remoteHead ?? REMOTE,
|
||||
dirty: '',
|
||||
fastForwardable: true,
|
||||
containsRemote: false,
|
||||
failing: new Set<string>(),
|
||||
runner: { run: async () => ({ code: 0, stdout: '', stderr: '' }) },
|
||||
};
|
||||
const run = async (command: string, args: readonly string[]): Promise<UpdateCommandResult> => {
|
||||
const key = [command, ...args].join(' ');
|
||||
state.calls.push(key);
|
||||
if (fails(state.failing, key)) return { code: 1, stdout: '', stderr: `${key} failed` };
|
||||
if (command === 'git') {
|
||||
if (args[0] === 'rev-parse' && args[1] === 'HEAD')
|
||||
return { code: 0, stdout: `${state.head}\n`, stderr: '' };
|
||||
if (args[0] === 'rev-parse' && args[1] === '--abbrev-ref')
|
||||
return { code: 0, stdout: 'main\n', stderr: '' };
|
||||
if (args[0] === 'config')
|
||||
return { code: 0, stdout: 'https://example.test/repo.git\n', stderr: '' };
|
||||
if (args[0] === 'ls-remote')
|
||||
return { code: 0, stdout: `${state.remoteHead}\trefs/heads/main\n`, stderr: '' };
|
||||
if (args[0] === 'rev-parse' && args[1] === 'FETCH_HEAD')
|
||||
return { code: 0, stdout: `${state.remoteHead}\n`, stderr: '' };
|
||||
if (args[0] === 'merge-base') {
|
||||
// `HEAD FETCH_HEAD` asks whether we can fast-forward; `<sha> HEAD` asks whether the
|
||||
// checkout already contains the remote commit.
|
||||
const fastForward = args[3] === 'FETCH_HEAD';
|
||||
const ok = fastForward ? state.fastForwardable : state.containsRemote;
|
||||
return { code: ok ? 0 : 1, stdout: '', stderr: '' };
|
||||
}
|
||||
if (args[0] === 'rev-list') return { code: 0, stdout: '3\n', stderr: '' };
|
||||
if (args[0] === 'diff')
|
||||
return {
|
||||
code: 0,
|
||||
stdout: ' 2 files changed, 34 insertions(+), 6 deletions(-)\n',
|
||||
stderr: '',
|
||||
};
|
||||
if (args[0] === 'show')
|
||||
return { code: 0, stdout: '{"name":"multi-simadmin","version":"0.9.0"}', stderr: '' };
|
||||
if (args[0] === 'status') return { code: 0, stdout: state.dirty, stderr: '' };
|
||||
return { code: 0, stdout: '', stderr: '' };
|
||||
}
|
||||
return { code: 0, stdout: '', stderr: '' };
|
||||
};
|
||||
state.runner = { run };
|
||||
return state;
|
||||
}
|
||||
|
||||
function service(
|
||||
input: {
|
||||
readonly git?: FakeGit;
|
||||
readonly launcher?: { supported: boolean; restart: () => Promise<void> };
|
||||
readonly restartDelayMs?: number;
|
||||
readonly preInstallBackup?: (targetCommit: string) => Promise<string>;
|
||||
} = {},
|
||||
) {
|
||||
const db = new Database(':memory:');
|
||||
migrateDatabase(db);
|
||||
dbs.push(db);
|
||||
const git = input.git ?? fakeGit();
|
||||
const serviceOptions = {
|
||||
db,
|
||||
version: '0.1.0',
|
||||
runner: git.runner,
|
||||
now: () => new Date('2026-09-06T00:00:00.000Z'),
|
||||
launcher: input.launcher ?? { supported: false, async restart() {} },
|
||||
...(input.preInstallBackup ? { preInstallBackup: input.preInstallBackup } : {}),
|
||||
...(input.restartDelayMs === undefined ? {} : { restartDelayMs: input.restartDelayMs }),
|
||||
} as const;
|
||||
return { db, git, updates: new ConsoleUpdateService(serviceOptions) };
|
||||
}
|
||||
|
||||
describe('update helpers', () => {
|
||||
it('adds insertions and deletions into one change count', () => {
|
||||
expect(parseShortStat(' 2 files changed, 34 insertions(+), 6 deletions(-)')).toEqual({
|
||||
files: 2,
|
||||
lines: 40,
|
||||
});
|
||||
});
|
||||
|
||||
it('survives an empty or malformed shortstat', () => {
|
||||
expect(parseShortStat('')).toEqual({ files: 0, lines: 0 });
|
||||
expect(parseShortStat('1 file changed, 2 insertions(+)')).toEqual({ files: 1, lines: 2 });
|
||||
});
|
||||
|
||||
it('reads a version out of a manifest and ignores broken JSON', () => {
|
||||
expect(parseManifestVersion('{"version":"1.2.3"}')).toBe('1.2.3');
|
||||
expect(parseManifestVersion('not json')).toBe('');
|
||||
expect(parseManifestVersion('{"version":42}')).toBe('');
|
||||
});
|
||||
});
|
||||
|
||||
describe('ConsoleUpdateService', () => {
|
||||
it('starts idle but ready to install on a git checkout', async () => {
|
||||
const { updates } = service();
|
||||
const status = await updates.status();
|
||||
expect(status.phase).toBe('idle');
|
||||
expect(status.deploymentMode).toBe('git');
|
||||
expect(status.installSupported).toBe(true);
|
||||
expect(status.restartSupported).toBe(false);
|
||||
expect(status.release).toBeNull();
|
||||
});
|
||||
|
||||
it('reports the manual deployment mode when git cannot resolve HEAD', async () => {
|
||||
const git = fakeGit();
|
||||
git.failing.add('git');
|
||||
const { updates } = service({ git });
|
||||
const status = await updates.status();
|
||||
expect(status.deploymentMode).toBe('manual');
|
||||
expect(status.installSupported).toBe(false);
|
||||
expect(status.message).toContain('手动更新');
|
||||
});
|
||||
|
||||
it('says the console is current when the remote matches HEAD', async () => {
|
||||
const git = fakeGit({ remoteHead: LOCAL });
|
||||
const { updates } = service({ git });
|
||||
const status = await updates.check();
|
||||
expect(status.phase).toBe('up_to_date');
|
||||
expect(status.release?.status).toBe('latest');
|
||||
expect(status.message).toContain('已是最新版本');
|
||||
});
|
||||
|
||||
it('offers a download when the remote moved ahead', async () => {
|
||||
const git = fakeGit();
|
||||
const { updates } = service({ git });
|
||||
const status = await updates.check();
|
||||
expect(status.phase).toBe('update_available');
|
||||
expect(status.release?.latestCommit).toBe(REMOTE);
|
||||
expect(status.release?.latestVersion).toBe('');
|
||||
});
|
||||
|
||||
it('says the console is ahead when the checkout already carries the remote commit', async () => {
|
||||
const git = fakeGit();
|
||||
git.containsRemote = true;
|
||||
const { updates } = service({ git });
|
||||
const status = await updates.check();
|
||||
expect(status.phase).toBe('up_to_date');
|
||||
expect(status.release?.status).toBe('latest');
|
||||
expect(status.release?.ahead).toBe(3);
|
||||
expect(status.message).toContain('领先更新源');
|
||||
expect(status.release?.summary).toContain('全部提交');
|
||||
// The download button stays closed because nothing was queued.
|
||||
await expect(updates.install()).rejects.toMatchObject({ code: 'STATE' });
|
||||
});
|
||||
|
||||
it('treats an unfetchable remote commit as an update rather than a local lead', async () => {
|
||||
const git = fakeGit();
|
||||
git.failing.add('git merge-base');
|
||||
const { updates } = service({ git });
|
||||
const status = await updates.check();
|
||||
expect(status.phase).toBe('update_available');
|
||||
expect(status.release?.ahead).toBe(0);
|
||||
});
|
||||
|
||||
it('refuses to check a detached HEAD', async () => {
|
||||
const git = fakeGit();
|
||||
const original = git.runner.run.bind(git.runner);
|
||||
git.runner = {
|
||||
run: async (command, args, options) =>
|
||||
command === 'git' && args[0] === 'rev-parse' && args[1] === '--abbrev-ref'
|
||||
? { code: 0, stdout: 'HEAD\n', stderr: '' }
|
||||
: original(command, args, options),
|
||||
};
|
||||
const { updates } = service({ git });
|
||||
await expect(updates.check()).rejects.toMatchObject({ code: 'STATE' });
|
||||
});
|
||||
|
||||
it('fetches, verifies and describes the candidate', async () => {
|
||||
const git = fakeGit();
|
||||
const { updates } = service({ git });
|
||||
await updates.check();
|
||||
const status = await updates.download();
|
||||
expect(status.phase).toBe('ready');
|
||||
expect(status.release?.ahead).toBe(3);
|
||||
expect(status.release?.changedFiles).toBe(2);
|
||||
expect(status.release?.changedLines).toBe(40);
|
||||
expect(status.release?.latestVersion).toBe('0.9.0');
|
||||
expect(status.message).toContain('3 个提交');
|
||||
expect(git.calls.some((call) => call.startsWith('git fetch'))).toBe(true);
|
||||
});
|
||||
|
||||
it('refuses to download before a check', async () => {
|
||||
const { updates } = service();
|
||||
await expect(updates.download()).rejects.toBeInstanceOf(UpdateError);
|
||||
});
|
||||
|
||||
it('keeps the code untouched when the branch cannot fast-forward', async () => {
|
||||
const git = fakeGit();
|
||||
git.fastForwardable = false;
|
||||
const { updates } = service({ git });
|
||||
await updates.check();
|
||||
const status = await updates.download();
|
||||
expect(status.phase).toBe('failed');
|
||||
expect(status.error).toContain('快进');
|
||||
expect(git.calls.some((call) => call.startsWith('git merge --ff-only'))).toBe(false);
|
||||
});
|
||||
|
||||
it('installs a verified candidate and queues the restart', async () => {
|
||||
const git = fakeGit();
|
||||
const { updates } = service({ git });
|
||||
await updates.check();
|
||||
await updates.download();
|
||||
const status = await updates.install();
|
||||
expect(status.phase).toBe('install_queued');
|
||||
expect(git.calls).toContain(`git merge --ff-only ${REMOTE}`);
|
||||
expect(git.calls).toContain('corepack pnpm install --frozen-lockfile');
|
||||
expect(git.calls).toContain('corepack pnpm --filter @multi-simadmin/web build');
|
||||
});
|
||||
|
||||
it('will not install over uncommitted work', async () => {
|
||||
const git = fakeGit();
|
||||
git.dirty = ' M apps/api/src/control-plane.ts\n';
|
||||
const { updates } = service({ git });
|
||||
await updates.check();
|
||||
await updates.download();
|
||||
await expect(updates.install()).rejects.toMatchObject({ code: 'STATE' });
|
||||
expect(git.calls.some((call) => call.startsWith('git merge --ff-only'))).toBe(false);
|
||||
});
|
||||
|
||||
it('rolls the checkout back when the build fails', async () => {
|
||||
const git = fakeGit();
|
||||
const original = git.runner.run.bind(git.runner);
|
||||
git.runner = {
|
||||
run: async (command, args, options) =>
|
||||
command === 'corepack' && args.includes('build')
|
||||
? { code: 2, stdout: '', stderr: 'vite exited 1' }
|
||||
: original(command, args, options),
|
||||
};
|
||||
const { updates } = service({ git });
|
||||
await updates.check();
|
||||
await updates.download();
|
||||
await expect(updates.install()).rejects.toMatchObject({ code: 'COMMAND' });
|
||||
const status = await updates.status();
|
||||
expect(status.phase).toBe('rolled_back');
|
||||
expect(status.error).toContain('回滚');
|
||||
expect(git.calls).toContain(`git reset --hard ${LOCAL}`);
|
||||
});
|
||||
|
||||
it('snapshots the configuration before the checkout moves', async () => {
|
||||
const snapshots: string[] = [];
|
||||
const { git, updates } = service({
|
||||
preInstallBackup: async (target) => {
|
||||
snapshots.push(target);
|
||||
return 'multi-simadmin-components-auto-preupdate.json';
|
||||
},
|
||||
});
|
||||
await updates.check();
|
||||
await updates.download();
|
||||
const status = await updates.install();
|
||||
expect(snapshots).toEqual([REMOTE]);
|
||||
expect(status.phase).toBe('install_queued');
|
||||
// The snapshot has to land before the merge, otherwise there is nothing to return to.
|
||||
expect(git.calls.some((call) => call.startsWith('git merge --ff-only'))).toBe(true);
|
||||
});
|
||||
|
||||
it('aborts the install when the safety snapshot cannot be written', async () => {
|
||||
const { git, updates } = service({
|
||||
preInstallBackup: async () => {
|
||||
throw new Error('disk full');
|
||||
},
|
||||
});
|
||||
await updates.check();
|
||||
await updates.download();
|
||||
await expect(updates.install()).rejects.toMatchObject({ code: 'COMMAND' });
|
||||
const status = await updates.status();
|
||||
expect(status.phase).toBe('failed');
|
||||
expect(status.message).toContain('更新前备份失败');
|
||||
expect(git.calls.some((call) => call.startsWith('git merge --ff-only'))).toBe(false);
|
||||
});
|
||||
|
||||
it('requires a verified package before installing', async () => {
|
||||
const { updates } = service();
|
||||
await updates.check();
|
||||
await expect(updates.install()).rejects.toMatchObject({ code: 'STATE' });
|
||||
});
|
||||
|
||||
it('reports 501 when nothing supervises the process', async () => {
|
||||
const { updates } = service();
|
||||
await expect(updates.restart()).rejects.toMatchObject({
|
||||
code: 'NOT_SUPPORTED',
|
||||
statusCode: 501,
|
||||
});
|
||||
});
|
||||
|
||||
it('hands a queued install to the supervisor', async () => {
|
||||
const restart = vi.fn(async () => undefined);
|
||||
const { updates } = service({
|
||||
launcher: { supported: true, restart },
|
||||
restartDelayMs: 0,
|
||||
});
|
||||
await updates.check();
|
||||
await updates.download();
|
||||
await updates.install();
|
||||
const status = await updates.restart();
|
||||
expect(status.phase).toBe('restarting');
|
||||
await new Promise((resolve) => setTimeout(resolve, 10));
|
||||
expect(restart).toHaveBeenCalledTimes(1);
|
||||
});
|
||||
|
||||
it('confirms its own update once the restarted process sees the new commit', async () => {
|
||||
const before = fakeGit();
|
||||
const first = service({
|
||||
git: before,
|
||||
launcher: { supported: true, restart: async () => undefined },
|
||||
restartDelayMs: 0,
|
||||
});
|
||||
await first.updates.check();
|
||||
await first.updates.download();
|
||||
await first.updates.install();
|
||||
await first.updates.restart();
|
||||
await new Promise((resolve) => setTimeout(resolve, 10));
|
||||
|
||||
// Same database, new process: HEAD moved to the target commit during the restart.
|
||||
const after = fakeGit({ head: REMOTE });
|
||||
const reopened = new ConsoleUpdateService({
|
||||
db: first.db,
|
||||
version: '0.9.0',
|
||||
runner: after.runner,
|
||||
now: () => new Date('2026-09-06T00:05:00.000Z'),
|
||||
});
|
||||
const status = await reopened.status();
|
||||
expect(status.phase).toBe('up_to_date');
|
||||
expect(status.message).toContain('完成重启');
|
||||
});
|
||||
|
||||
it('marks a restart that came back on the old commit as failed', async () => {
|
||||
const git = fakeGit();
|
||||
const { updates, db } = service({
|
||||
git,
|
||||
launcher: { supported: true, restart: async () => {} },
|
||||
restartDelayMs: 0,
|
||||
});
|
||||
await updates.check();
|
||||
await updates.download();
|
||||
await updates.install();
|
||||
await updates.restart();
|
||||
const reopened = new ConsoleUpdateService({
|
||||
db,
|
||||
version: '0.1.0',
|
||||
runner: git.runner,
|
||||
now: () => new Date('2026-09-06T00:05:00.000Z'),
|
||||
});
|
||||
const status = await reopened.status();
|
||||
expect(status.phase).toBe('failed');
|
||||
expect(status.error).toContain('目标版本');
|
||||
});
|
||||
|
||||
it('rejects a second command while one is running', async () => {
|
||||
const git = fakeGit();
|
||||
let release: () => void = () => {};
|
||||
const gate = new Promise<void>((resolve) => {
|
||||
release = resolve;
|
||||
});
|
||||
const original = git.runner.run.bind(git.runner);
|
||||
git.runner = {
|
||||
run: async (command, args, options) => {
|
||||
if (command === 'git' && args[0] === 'ls-remote') await gate;
|
||||
return original(command, args, options);
|
||||
},
|
||||
};
|
||||
const { updates } = service({ git });
|
||||
const first = updates.check();
|
||||
await expect(updates.check()).rejects.toMatchObject({ code: 'BUSY' });
|
||||
release();
|
||||
await first;
|
||||
});
|
||||
|
||||
it('keeps a corrupt stored state readable', async () => {
|
||||
const { db, updates } = service();
|
||||
db.prepare(
|
||||
`INSERT INTO app_settings (key,value_json,created_at,updated_at)
|
||||
VALUES ('console.update','{not json',?,?)`,
|
||||
).run('2026-09-06T00:00:00.000Z', '2026-09-06T00:00:00.000Z');
|
||||
const status = await updates.status();
|
||||
expect(status.phase).toBe('idle');
|
||||
});
|
||||
});
|
||||
|
||||
describe('createLaunchdRestartLauncher', () => {
|
||||
it('is unsupported without a launchd label', () => {
|
||||
const launcher = createLaunchdRestartLauncher({
|
||||
env: {},
|
||||
runner: { run: async () => ({ code: 0, stdout: '', stderr: '' }) },
|
||||
uid: () => 501,
|
||||
});
|
||||
expect(launcher.supported).toBe(false);
|
||||
});
|
||||
|
||||
it('kickstarts its own launchd label', async () => {
|
||||
const calls: string[][] = [];
|
||||
const launcher = createLaunchdRestartLauncher({
|
||||
env: { XPC_SERVICE_NAME: 'fun.chickliu.multi-simadmin-api' },
|
||||
runner: {
|
||||
run: async (command, args) => {
|
||||
calls.push([command, ...args]);
|
||||
return { code: 0, stdout: '', stderr: '' };
|
||||
},
|
||||
},
|
||||
uid: () => 501,
|
||||
});
|
||||
expect(launcher.supported).toBe(true);
|
||||
await launcher.restart();
|
||||
expect(calls[0]).toEqual([
|
||||
'launchctl',
|
||||
'kickstart',
|
||||
'-k',
|
||||
'gui/501/fun.chickliu.multi-simadmin-api',
|
||||
]);
|
||||
});
|
||||
|
||||
it('surfaces a launchctl failure', async () => {
|
||||
const launcher = createLaunchdRestartLauncher({
|
||||
env: { XPC_SERVICE_NAME: 'fun.chickliu.multi-simadmin-api' },
|
||||
runner: { run: async () => ({ code: 1, stdout: '', stderr: 'Could not find service' }) },
|
||||
uid: () => 501,
|
||||
});
|
||||
await expect(launcher.restart()).rejects.toMatchObject({ code: 'COMMAND', statusCode: 500 });
|
||||
});
|
||||
});
|
||||
|
||||
describe('createSystemdRestartLauncher', () => {
|
||||
it('is unsupported without a unit or with a malformed unit name', () => {
|
||||
for (const unit of [undefined, '', 'evil; rm -rf /', 'multi-simadmin']) {
|
||||
const launcher = createSystemdRestartLauncher({
|
||||
env: unit === undefined ? {} : { MULTI_SIMADMIN_SYSTEMD_UNIT: unit },
|
||||
runner: { run: async () => ({ code: 0, stdout: '', stderr: '' }) },
|
||||
});
|
||||
expect(launcher.supported).toBe(false);
|
||||
}
|
||||
});
|
||||
|
||||
it('restarts through the configured scope and unit', async () => {
|
||||
const calls: string[][] = [];
|
||||
const launcher = createSystemdRestartLauncher({
|
||||
env: { MULTI_SIMADMIN_SYSTEMD_UNIT: 'multi-simadmin-api.service' },
|
||||
runner: {
|
||||
run: async (command, args) => {
|
||||
calls.push([command, ...args]);
|
||||
return { code: 0, stdout: '', stderr: '' };
|
||||
},
|
||||
},
|
||||
});
|
||||
expect(launcher.supported).toBe(true);
|
||||
await launcher.restart();
|
||||
expect(calls[0]).toEqual(['systemctl', 'user', 'restart', 'multi-simadmin-api.service']);
|
||||
});
|
||||
|
||||
it('supports the system scope and surfaces systemctl failures', async () => {
|
||||
const calls: string[][] = [];
|
||||
const launcher = createSystemdRestartLauncher({
|
||||
env: {
|
||||
MULTI_SIMADMIN_SYSTEMD_UNIT: 'multi-simadmin-api.service',
|
||||
MULTI_SIMADMIN_SYSTEMD_SCOPE: 'system',
|
||||
},
|
||||
runner: {
|
||||
run: async (command, args) => {
|
||||
calls.push([command, ...args]);
|
||||
return { code: 1, stdout: '', stderr: 'not logged in' };
|
||||
},
|
||||
},
|
||||
});
|
||||
await launcher.restart().catch(() => {});
|
||||
expect(calls[0]).toEqual(['systemctl', 'system', 'restart', 'multi-simadmin-api.service']);
|
||||
await expect(launcher.restart()).rejects.toMatchObject({ code: 'COMMAND', statusCode: 500 });
|
||||
});
|
||||
});
|
||||
|
||||
describe('createPlatformRestartLauncher', () => {
|
||||
const runner = { run: async () => ({ code: 0, stdout: '', stderr: '' }) };
|
||||
|
||||
it('prefers systemd on Linux when a unit is configured', () => {
|
||||
const launcher = createPlatformRestartLauncher({
|
||||
env: { MULTI_SIMADMIN_HOST_PLATFORM: 'linux', MULTI_SIMADMIN_SYSTEMD_UNIT: 'api.service' },
|
||||
runner,
|
||||
});
|
||||
expect(launcher.supported).toBe(true);
|
||||
});
|
||||
|
||||
it('keeps the launchd launcher everywhere else', () => {
|
||||
const linux = createPlatformRestartLauncher({
|
||||
env: { MULTI_SIMADMIN_HOST_PLATFORM: 'linux' },
|
||||
runner,
|
||||
});
|
||||
const darwin = createPlatformRestartLauncher({
|
||||
env: { MULTI_SIMADMIN_HOST_PLATFORM: 'darwin', XPC_SERVICE_NAME: 'x' },
|
||||
runner,
|
||||
uid: () => 501,
|
||||
});
|
||||
expect(linux.supported).toBe(false);
|
||||
expect(darwin.supported).toBe(true);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,781 @@
|
||||
import { spawn } from 'node:child_process';
|
||||
import type Database from 'better-sqlite3';
|
||||
|
||||
/**
|
||||
* Online update for the console itself, modelled on the Hub update card: check a remote
|
||||
* branch, fetch and verify the candidate, install it, then hand the process back to the
|
||||
* supervisor. Each step is a separate, operator-visible phase, so a half-finished update stays
|
||||
* readable after a crash instead of silently half-applied.
|
||||
*/
|
||||
export const UPDATE_PHASES = [
|
||||
'idle',
|
||||
'checking',
|
||||
'up_to_date',
|
||||
'update_available',
|
||||
'downloading',
|
||||
'ready',
|
||||
'installing',
|
||||
'install_queued',
|
||||
'restarting',
|
||||
'failed',
|
||||
'rolled_back',
|
||||
] as const;
|
||||
|
||||
export type UpdatePhase = (typeof UPDATE_PHASES)[number];
|
||||
|
||||
/** Phases where work is in flight: the UI polls these fast and refuses a second command. */
|
||||
export const BUSY_UPDATE_PHASES: ReadonlySet<UpdatePhase> = new Set<UpdatePhase>([
|
||||
'checking',
|
||||
'downloading',
|
||||
'installing',
|
||||
'install_queued',
|
||||
'restarting',
|
||||
]);
|
||||
|
||||
export interface UpdateRelease {
|
||||
readonly status: 'latest' | 'update_available' | 'unavailable';
|
||||
readonly remote: string;
|
||||
readonly branch: string;
|
||||
readonly currentCommit: string;
|
||||
readonly latestCommit: string;
|
||||
readonly currentVersion: string;
|
||||
readonly latestVersion: string;
|
||||
readonly ahead: number;
|
||||
readonly changedFiles: number;
|
||||
readonly changedLines: number;
|
||||
readonly summary: string;
|
||||
}
|
||||
|
||||
export interface UpdateStatus {
|
||||
readonly phase: UpdatePhase;
|
||||
readonly message: string;
|
||||
readonly error: string;
|
||||
readonly progressPercent: number;
|
||||
readonly checkedAt: string;
|
||||
readonly deploymentMode: 'git' | 'manual';
|
||||
readonly installSupported: boolean;
|
||||
readonly restartSupported: boolean;
|
||||
readonly release: UpdateRelease | null;
|
||||
}
|
||||
|
||||
export interface UpdateCommandResult {
|
||||
readonly code: number;
|
||||
readonly stdout: string;
|
||||
readonly stderr: string;
|
||||
}
|
||||
|
||||
export interface UpdateCommandRunner {
|
||||
run(
|
||||
command: string,
|
||||
args: readonly string[],
|
||||
options?: { readonly timeoutMs?: number },
|
||||
): Promise<UpdateCommandResult>;
|
||||
}
|
||||
|
||||
export interface RestartLauncher {
|
||||
readonly supported: boolean;
|
||||
restart(): Promise<void>;
|
||||
}
|
||||
|
||||
export class UpdateError extends Error {
|
||||
constructor(
|
||||
readonly code: 'BUSY' | 'NOT_SUPPORTED' | 'STATE' | 'COMMAND',
|
||||
message: string,
|
||||
readonly statusCode = 409,
|
||||
) {
|
||||
super(message);
|
||||
this.name = 'UpdateError';
|
||||
}
|
||||
}
|
||||
|
||||
const STATE_KEY = 'console.update';
|
||||
const DEFAULT_TIMEOUT_MS = 120_000;
|
||||
const INSTALL_TIMEOUT_MS = 15 * 60_000;
|
||||
const MAX_OUTPUT = 4 * 1024 * 1024;
|
||||
const COMMIT = /^[0-9a-f]{7,40}$/u;
|
||||
|
||||
type PendingAction = 'install' | 'restart' | '';
|
||||
|
||||
interface PersistedState {
|
||||
readonly phase: UpdatePhase;
|
||||
readonly message: string;
|
||||
readonly error: string;
|
||||
readonly progressPercent: number;
|
||||
readonly checkedAt: string;
|
||||
readonly release: UpdateRelease | null;
|
||||
/** Where the update is aimed; lets a restarted process confirm its own success. */
|
||||
readonly pendingCommit: string;
|
||||
readonly pendingAction: PendingAction;
|
||||
}
|
||||
|
||||
function text(value: unknown, maximum = 512): string {
|
||||
return typeof value === 'string' ? value.slice(0, maximum) : '';
|
||||
}
|
||||
|
||||
function count(value: unknown): number {
|
||||
return typeof value === 'number' && Number.isSafeInteger(value) && value >= 0 ? value : 0;
|
||||
}
|
||||
|
||||
function percent(value: unknown): number {
|
||||
return typeof value === 'number' && Number.isFinite(value)
|
||||
? Math.min(100, Math.max(0, Math.round(value)))
|
||||
: 0;
|
||||
}
|
||||
|
||||
function phaseOf(value: unknown): UpdatePhase {
|
||||
return typeof value === 'string' && (UPDATE_PHASES as readonly string[]).includes(value)
|
||||
? (value as UpdatePhase)
|
||||
: 'idle';
|
||||
}
|
||||
|
||||
function readRelease(value: unknown): UpdateRelease | null {
|
||||
if (typeof value !== 'object' || value === null || Array.isArray(value)) return null;
|
||||
const source = value as Record<string, unknown>;
|
||||
const status = source.status;
|
||||
return {
|
||||
status:
|
||||
status === 'latest' || status === 'update_available' || status === 'unavailable'
|
||||
? status
|
||||
: 'unavailable',
|
||||
remote: text(source.remote, 256),
|
||||
branch: text(source.branch, 128),
|
||||
currentCommit: text(source.currentCommit, 40),
|
||||
latestCommit: text(source.latestCommit, 40),
|
||||
currentVersion: text(source.currentVersion, 64),
|
||||
latestVersion: text(source.latestVersion, 64),
|
||||
ahead: count(source.ahead),
|
||||
changedFiles: count(source.changedFiles),
|
||||
changedLines: count(source.changedLines),
|
||||
summary: text(source.summary, 512),
|
||||
};
|
||||
}
|
||||
|
||||
/** `2 files changed, 34 insertions(+), 6 deletions(-)` from `git diff --shortstat`. */
|
||||
export function parseShortStat(value: string): { files: number; lines: number } {
|
||||
const files = Number(/(\d+) files? changed/u.exec(value)?.[1] ?? 0);
|
||||
const insertions = Number(/(\d+) insertions?\(\+\)/u.exec(value)?.[1] ?? 0);
|
||||
const deletions = Number(/(\d+) deletions?\(-\)/u.exec(value)?.[1] ?? 0);
|
||||
return {
|
||||
files: Number.isFinite(files) ? files : 0,
|
||||
lines:
|
||||
(Number.isFinite(insertions) ? insertions : 0) + (Number.isFinite(deletions) ? deletions : 0),
|
||||
};
|
||||
}
|
||||
|
||||
/** Reads the version field out of a package.json blob fetched from a commit. */
|
||||
export function parseManifestVersion(value: string): string {
|
||||
try {
|
||||
const parsed: unknown = JSON.parse(value);
|
||||
if (parsed && typeof parsed === 'object' && !Array.isArray(parsed)) {
|
||||
const version = (parsed as Record<string, unknown>).version;
|
||||
if (typeof version === 'string' && version.length <= 64) return version;
|
||||
}
|
||||
} catch {
|
||||
// A malformed manifest simply means there is no version to show.
|
||||
}
|
||||
return '';
|
||||
}
|
||||
|
||||
export function createCommandRunner(options: {
|
||||
readonly cwd: string;
|
||||
readonly timeoutMs?: number;
|
||||
}): UpdateCommandRunner {
|
||||
return {
|
||||
run(command, args, runOptions) {
|
||||
return new Promise((resolve) => {
|
||||
const child = spawn(command, [...args], { cwd: options.cwd, shell: false });
|
||||
let stdout = '';
|
||||
let stderr = '';
|
||||
const timer = setTimeout(
|
||||
() => child.kill('SIGKILL'),
|
||||
runOptions?.timeoutMs ?? options.timeoutMs ?? DEFAULT_TIMEOUT_MS,
|
||||
);
|
||||
child.stdout.on('data', (chunk: Buffer) => {
|
||||
if (stdout.length < MAX_OUTPUT) stdout += chunk.toString('utf8');
|
||||
});
|
||||
child.stderr.on('data', (chunk: Buffer) => {
|
||||
if (stderr.length < MAX_OUTPUT) stderr += chunk.toString('utf8');
|
||||
});
|
||||
child.once('error', (error: Error) => {
|
||||
clearTimeout(timer);
|
||||
resolve({ code: -1, stdout, stderr: `${stderr}${error.message}` });
|
||||
});
|
||||
child.once('close', (code) => {
|
||||
clearTimeout(timer);
|
||||
resolve({ code: code ?? -1, stdout, stderr });
|
||||
});
|
||||
});
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* macOS LaunchAgents learn their own label through XPC_SERVICE_NAME, which is the only
|
||||
* dependable hint that this process is supervised and will be started again after it exits.
|
||||
*/
|
||||
export function createLaunchdRestartLauncher(input: {
|
||||
readonly env: Readonly<Record<string, string | undefined>>;
|
||||
readonly runner: UpdateCommandRunner;
|
||||
readonly uid?: () => number | undefined;
|
||||
}): RestartLauncher {
|
||||
const label = text(input.env.XPC_SERVICE_NAME, 256);
|
||||
const uid =
|
||||
input.uid?.() ?? (typeof process.getuid === 'function' ? process.getuid() : undefined);
|
||||
const supported = label.length > 0 && typeof uid === 'number';
|
||||
return {
|
||||
supported,
|
||||
async restart() {
|
||||
if (!supported) throw new UpdateError('NOT_SUPPORTED', '当前进程未受服务管理器托管。', 501);
|
||||
const result = await input.runner.run('launchctl', [
|
||||
'kickstart',
|
||||
'-k',
|
||||
`gui/${uid}/${label}`,
|
||||
]);
|
||||
if (result.code !== 0)
|
||||
throw new UpdateError(
|
||||
'COMMAND',
|
||||
text(result.stderr, 200) || 'launchctl kickstart 失败。',
|
||||
500,
|
||||
);
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Linux systemd counterpart to the launchd launcher. The unit and scope are
|
||||
* supplied explicitly through the environment so a supervised deployment can
|
||||
* hand restarts to its service manager; unset variables keep it unsupported.
|
||||
*/
|
||||
export function createSystemdRestartLauncher(input: {
|
||||
readonly env: Readonly<Record<string, string | undefined>>;
|
||||
readonly runner: UpdateCommandRunner;
|
||||
}): RestartLauncher {
|
||||
const unit = input.env.MULTI_SIMADMIN_SYSTEMD_UNIT ?? '';
|
||||
const scope = input.env.MULTI_SIMADMIN_SYSTEMD_SCOPE === 'system' ? 'system' : 'user';
|
||||
const validUnit = /^[A-Za-z0-9@_.\-]+\.service$/.test(unit);
|
||||
const supported = validUnit;
|
||||
return {
|
||||
supported,
|
||||
async restart() {
|
||||
if (!supported)
|
||||
throw new UpdateError(
|
||||
'NOT_SUPPORTED',
|
||||
'未配置 systemd 服务单元(MULTI_SIMADMIN_SYSTEMD_UNIT)。',
|
||||
501,
|
||||
);
|
||||
const result = await input.runner.run('systemctl', [scope, 'restart', unit]);
|
||||
if (result.code !== 0)
|
||||
throw new UpdateError(
|
||||
'COMMAND',
|
||||
text(result.stderr, 200) || 'systemctl restart 失败。',
|
||||
500,
|
||||
);
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
/** Picks the restart launcher matching the host service manager. */
|
||||
export function createPlatformRestartLauncher(input: {
|
||||
readonly env: Readonly<Record<string, string | undefined>>;
|
||||
readonly runner: UpdateCommandRunner;
|
||||
readonly uid?: () => number | undefined;
|
||||
}): RestartLauncher {
|
||||
const platform = input.env.MULTI_SIMADMIN_HOST_PLATFORM ?? process.platform;
|
||||
return platform === 'linux' && input.env.MULTI_SIMADMIN_SYSTEMD_UNIT !== undefined
|
||||
? createSystemdRestartLauncher(input)
|
||||
: createLaunchdRestartLauncher(input);
|
||||
}
|
||||
|
||||
export interface ConsoleUpdateServiceOptions {
|
||||
readonly db: Database.Database;
|
||||
readonly version: string;
|
||||
readonly remote?: string;
|
||||
readonly runner?: UpdateCommandRunner;
|
||||
readonly launcher?: RestartLauncher;
|
||||
readonly now?: () => Date;
|
||||
/**
|
||||
* Safety net run before the checkout moves: a new version may migrate the database in a way
|
||||
* that cannot be undone by rolling the code back, so the config is snapshotted first.
|
||||
*/
|
||||
readonly preInstallBackup?: (targetCommit: string) => Promise<string>;
|
||||
/** Test hook: the real hand-off waits a beat so the HTTP response can flush first. */
|
||||
readonly restartDelayMs?: number;
|
||||
}
|
||||
|
||||
export class ConsoleUpdateService {
|
||||
readonly #db: Database.Database;
|
||||
readonly #version: string;
|
||||
readonly #remote: string;
|
||||
readonly #runner: UpdateCommandRunner;
|
||||
readonly #launcher: RestartLauncher;
|
||||
readonly #now: () => Date;
|
||||
readonly #preInstallBackup: ((targetCommit: string) => Promise<string>) | undefined;
|
||||
readonly #restartDelayMs: number;
|
||||
#state: PersistedState;
|
||||
#inFlight: Promise<UpdateStatus> | undefined;
|
||||
#mode: 'unknown' | 'git' | 'manual' = 'unknown';
|
||||
#reconciled = false;
|
||||
|
||||
constructor(options: ConsoleUpdateServiceOptions) {
|
||||
this.#db = options.db;
|
||||
this.#version = options.version;
|
||||
this.#remote = options.remote ?? 'origin';
|
||||
this.#runner =
|
||||
options.runner ?? createCommandRunner({ cwd: process.cwd(), timeoutMs: DEFAULT_TIMEOUT_MS });
|
||||
this.#launcher = options.launcher ?? { supported: false, async restart() {} };
|
||||
this.#now = options.now ?? (() => new Date());
|
||||
this.#preInstallBackup = options.preInstallBackup;
|
||||
this.#restartDelayMs = options.restartDelayMs ?? 250;
|
||||
this.#state = this.#read();
|
||||
}
|
||||
|
||||
/** Resolves the deployment mode, because a plain status poll must not shell out to git. */
|
||||
async status(): Promise<UpdateStatus> {
|
||||
if (this.#mode === 'unknown') await this.#localHead();
|
||||
await this.#reconcile();
|
||||
return this.#view();
|
||||
}
|
||||
|
||||
async check(): Promise<UpdateStatus> {
|
||||
return this.#exclusive(async () => {
|
||||
const head = await this.#localHead();
|
||||
if (!head) return this.#view();
|
||||
const branch = (await this.#text(['rev-parse', '--abbrev-ref', 'HEAD'])).trim();
|
||||
const remoteUrl = (
|
||||
await this.#text(['config', '--get', `remote.${this.#remote}.url`])
|
||||
).trim();
|
||||
if (!branch || branch === 'HEAD')
|
||||
throw new UpdateError('STATE', '当前处于游离提交,请先切回分支再检查更新。', 409);
|
||||
if (!remoteUrl)
|
||||
throw new UpdateError('STATE', `未配置远端 ${this.#remote},无法检查更新。`, 409);
|
||||
|
||||
const listing = await this.#git(['ls-remote', remoteUrl, `refs/heads/${branch}`]);
|
||||
if (listing.code !== 0)
|
||||
throw new UpdateError('COMMAND', '无法连接更新源,请检查网络后重试。', 502);
|
||||
const latest = text(listing.stdout).trim().split(/\s+/u)[0];
|
||||
if (!latest || !COMMIT.test(latest))
|
||||
throw new UpdateError('COMMAND', `远端分支 ${branch} 不存在或不可读。`, 502);
|
||||
|
||||
// A different SHA is not enough: a checkout that already carries the remote commit is
|
||||
// ahead of the update source, not behind it. Only an ancestry test tells the two apart.
|
||||
const behind = latest !== head && !(await this.#contains(latest));
|
||||
const ahead = behind
|
||||
? 0
|
||||
: latest === head
|
||||
? 0
|
||||
: Number(await this.#text(['rev-list', '--count', `${latest}..HEAD`]));
|
||||
const upToDate = !behind;
|
||||
const release: UpdateRelease = {
|
||||
status: upToDate ? 'latest' : 'update_available',
|
||||
remote: remoteUrl,
|
||||
branch,
|
||||
currentCommit: head,
|
||||
latestCommit: latest,
|
||||
currentVersion: this.#version,
|
||||
latestVersion: '',
|
||||
ahead: behind ? 0 : Number.isFinite(ahead) ? ahead : 0,
|
||||
changedFiles: 0,
|
||||
changedLines: 0,
|
||||
summary: behind
|
||||
? '下载更新包后可以看到提交数量与改动规模。'
|
||||
: upToDate && latest !== head
|
||||
? `本地已包含 ${remoteUrl}/${branch} 的全部提交。`
|
||||
: '',
|
||||
};
|
||||
this.#save({
|
||||
phase: upToDate ? 'up_to_date' : 'update_available',
|
||||
message: upToDate
|
||||
? latest === head
|
||||
? `当前已是最新版本(${head.slice(0, 7)})。`
|
||||
: `本地版本已领先更新源 ${Number.isFinite(ahead) ? ahead : 0} 个提交,无需更新。`
|
||||
: `发现新版本 ${latest.slice(0, 7)},可以下载更新包。`,
|
||||
error: '',
|
||||
progressPercent: 100,
|
||||
release,
|
||||
pendingCommit: '',
|
||||
pendingAction: '',
|
||||
});
|
||||
return this.#view();
|
||||
});
|
||||
}
|
||||
|
||||
async download(): Promise<UpdateStatus> {
|
||||
return this.#exclusive(async () => {
|
||||
const release = this.#requireCandidate();
|
||||
this.#save({
|
||||
phase: 'downloading',
|
||||
message: '正在下载并校验更新包…',
|
||||
error: '',
|
||||
progressPercent: 35,
|
||||
release,
|
||||
pendingCommit: release.latestCommit,
|
||||
pendingAction: '',
|
||||
});
|
||||
|
||||
const fetched = await this.#git([
|
||||
'fetch',
|
||||
'--quiet',
|
||||
'--no-tags',
|
||||
release.remote,
|
||||
release.branch,
|
||||
]);
|
||||
if (fetched.code !== 0) {
|
||||
this.#fail('下载更新包失败,请检查网络后重试。');
|
||||
throw new UpdateError('COMMAND', '下载更新包失败,请检查网络后重试。', 502);
|
||||
}
|
||||
const candidate = (await this.#text(['rev-parse', 'FETCH_HEAD'])).trim();
|
||||
if (candidate !== release.latestCommit) {
|
||||
this.#fail('远端分支已变化,请重新检查更新。');
|
||||
throw new UpdateError('STATE', '远端分支已变化,请重新检查更新。', 409);
|
||||
}
|
||||
const ancestor = await this.#git(['merge-base', '--is-ancestor', 'HEAD', 'FETCH_HEAD']);
|
||||
if (ancestor.code !== 0) {
|
||||
this.#fail('本地分支已偏离远端,无法快进合并,请先处理本地提交。');
|
||||
return this.#view();
|
||||
}
|
||||
|
||||
const ahead = Number(await this.#text(['rev-list', '--count', 'HEAD..FETCH_HEAD']));
|
||||
const stat = parseShortStat(await this.#text(['diff', '--shortstat', 'HEAD', 'FETCH_HEAD']));
|
||||
const manifest = parseManifestVersion(await this.#text(['show', 'FETCH_HEAD:package.json']));
|
||||
const commits = Number.isFinite(ahead) ? ahead : 0;
|
||||
const updated: UpdateRelease = {
|
||||
...release,
|
||||
latestVersion: manifest,
|
||||
ahead: commits,
|
||||
changedFiles: stat.files,
|
||||
changedLines: stat.lines,
|
||||
summary: `领先 ${commits} 个提交,${stat.files} 个文件、${stat.lines} 行改动。`,
|
||||
};
|
||||
this.#save({
|
||||
phase: 'ready',
|
||||
message: `更新包已校验:${updated.summary}`,
|
||||
error: '',
|
||||
progressPercent: 100,
|
||||
release: updated,
|
||||
pendingCommit: updated.latestCommit,
|
||||
pendingAction: '',
|
||||
});
|
||||
return this.#view();
|
||||
});
|
||||
}
|
||||
|
||||
async install(): Promise<UpdateStatus> {
|
||||
return this.#exclusive(async () => {
|
||||
const release = this.#requireCandidate();
|
||||
if (this.#state.phase !== 'ready' && this.#state.phase !== 'rolled_back')
|
||||
throw new UpdateError('STATE', '请先下载并校验更新包。', 409);
|
||||
// The rollback below resets the working tree, so it is only ever allowed to start from a
|
||||
// clean one; otherwise an operator's uncommitted work would be the first casualty.
|
||||
if ((await this.#text(['status', '--porcelain'])).trim())
|
||||
throw new UpdateError('STATE', '工作区存在未提交改动,为避免丢失内容已中止安装。', 409);
|
||||
const previousCommit = await this.#localHead();
|
||||
if (!previousCommit) return this.#view();
|
||||
|
||||
if (this.#preInstallBackup) {
|
||||
this.#save({
|
||||
phase: 'installing',
|
||||
message: '正在创建更新前备份…',
|
||||
error: '',
|
||||
progressPercent: 5,
|
||||
release,
|
||||
pendingCommit: '',
|
||||
pendingAction: '',
|
||||
});
|
||||
try {
|
||||
await this.#preInstallBackup(release.latestCommit);
|
||||
} catch {
|
||||
this.#fail('更新前备份失败,已中止安装,代码保持原版本。');
|
||||
throw new UpdateError('COMMAND', '更新前备份失败,已中止安装。', 500);
|
||||
}
|
||||
}
|
||||
|
||||
this.#save({
|
||||
phase: 'installing',
|
||||
message: '正在安装更新…',
|
||||
error: '',
|
||||
progressPercent: 20,
|
||||
release,
|
||||
pendingCommit: release.latestCommit,
|
||||
pendingAction: 'install',
|
||||
});
|
||||
const merge = await this.#git(['merge', '--ff-only', release.latestCommit]);
|
||||
if (merge.code !== 0) {
|
||||
this.#fail('快进合并失败,代码保持原版本。');
|
||||
throw new UpdateError('COMMAND', '快进合并失败,代码保持原版本。', 500);
|
||||
}
|
||||
this.#progress(55, '依赖安装中…');
|
||||
const dependencies = await this.#runner.run(
|
||||
'corepack',
|
||||
['pnpm', 'install', '--frozen-lockfile'],
|
||||
{ timeoutMs: INSTALL_TIMEOUT_MS },
|
||||
);
|
||||
if (dependencies.code !== 0) {
|
||||
await this.#rollback(previousCommit, '依赖安装失败,已回滚到原版本。');
|
||||
throw new UpdateError('COMMAND', '依赖安装失败,已回滚到原版本。', 500);
|
||||
}
|
||||
this.#progress(80, '前端构建中…');
|
||||
const build = await this.#runner.run(
|
||||
'corepack',
|
||||
['pnpm', '--filter', '@multi-simadmin/web', 'build'],
|
||||
{ timeoutMs: INSTALL_TIMEOUT_MS },
|
||||
);
|
||||
if (build.code !== 0) {
|
||||
await this.#rollback(previousCommit, '前端构建失败,已回滚到原版本。');
|
||||
throw new UpdateError('COMMAND', '前端构建失败,已回滚到原版本。', 500);
|
||||
}
|
||||
|
||||
this.#save({
|
||||
phase: 'install_queued',
|
||||
message: '安装完成,重启后生效。',
|
||||
error: '',
|
||||
progressPercent: 100,
|
||||
release,
|
||||
pendingCommit: release.latestCommit,
|
||||
pendingAction: 'install',
|
||||
});
|
||||
return this.#view();
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* Hands the process back to the supervisor. The reply flushes first: killing the API in the
|
||||
* middle of a request leaves the browser with a connection error instead of a result.
|
||||
*/
|
||||
async restart(): Promise<UpdateStatus> {
|
||||
if (!this.#launcher.supported)
|
||||
throw new UpdateError('NOT_SUPPORTED', '当前部署方式不支持在线重启,请手动重启服务。', 501);
|
||||
if (this.#inFlight) throw new UpdateError('BUSY', '更新任务正在进行中。', 409);
|
||||
const installing = this.#state.phase === 'install_queued';
|
||||
this.#save({
|
||||
phase: 'restarting',
|
||||
message: '服务正在重启,页面连接将短暂中断。',
|
||||
error: '',
|
||||
progressPercent: 100,
|
||||
release: this.#state.release,
|
||||
pendingCommit: installing ? this.#state.pendingCommit : '',
|
||||
pendingAction: installing ? 'install' : 'restart',
|
||||
});
|
||||
setTimeout(() => {
|
||||
void this.#launcher.restart().catch(() => {
|
||||
this.#save({
|
||||
phase: 'failed',
|
||||
message: '重启请求未能交给服务管理器。',
|
||||
error: '重启请求未能交给服务管理器,请手动重启服务。',
|
||||
progressPercent: 100,
|
||||
release: this.#state.release,
|
||||
pendingCommit: '',
|
||||
pendingAction: '',
|
||||
});
|
||||
});
|
||||
}, this.#restartDelayMs);
|
||||
return this.#view();
|
||||
}
|
||||
|
||||
async #exclusive(action: () => Promise<UpdateStatus>): Promise<UpdateStatus> {
|
||||
if (this.#inFlight) throw new UpdateError('BUSY', '更新任务正在进行中,请稍候。', 409);
|
||||
const pending = (async () => {
|
||||
try {
|
||||
return await action();
|
||||
} finally {
|
||||
this.#inFlight = undefined;
|
||||
}
|
||||
})();
|
||||
this.#inFlight = pending;
|
||||
return pending;
|
||||
}
|
||||
|
||||
#requireCandidate(): UpdateRelease {
|
||||
const release = this.#state.release;
|
||||
if (!release || release.status !== 'update_available' || !COMMIT.test(release.latestCommit))
|
||||
throw new UpdateError('STATE', '请先检查更新。', 409);
|
||||
return release;
|
||||
}
|
||||
|
||||
async #localHead(): Promise<string> {
|
||||
const result = await this.#git(['rev-parse', 'HEAD']);
|
||||
const head = result.stdout.trim();
|
||||
if (result.code !== 0 || !COMMIT.test(head)) {
|
||||
this.#mode = 'manual';
|
||||
this.#save({
|
||||
phase: 'idle',
|
||||
message: '当前部署不是 Git 工作副本,请按安装文档手动更新后重启。',
|
||||
error: '',
|
||||
progressPercent: 0,
|
||||
release: null,
|
||||
pendingCommit: '',
|
||||
pendingAction: '',
|
||||
});
|
||||
return '';
|
||||
}
|
||||
this.#mode = 'git';
|
||||
return head;
|
||||
}
|
||||
|
||||
async #git(args: readonly string[]): Promise<UpdateCommandResult> {
|
||||
return this.#runner.run('git', args);
|
||||
}
|
||||
|
||||
/** True when HEAD already carries `commit`. A missing object reads as "not contained". */
|
||||
async #contains(commit: string): Promise<boolean> {
|
||||
if (!COMMIT.test(commit)) return false;
|
||||
const probe = await this.#git(['merge-base', '--is-ancestor', commit, 'HEAD']);
|
||||
return probe.code === 0;
|
||||
}
|
||||
|
||||
async #text(args: readonly string[]): Promise<string> {
|
||||
const result = await this.#git(args);
|
||||
return result.code === 0 ? text(result.stdout) : '';
|
||||
}
|
||||
|
||||
#progress(progressPercent: number, message: string): void {
|
||||
this.#save({
|
||||
phase: 'installing',
|
||||
message,
|
||||
error: '',
|
||||
progressPercent,
|
||||
release: this.#state.release,
|
||||
pendingCommit: this.#state.pendingCommit,
|
||||
pendingAction: 'install',
|
||||
});
|
||||
}
|
||||
|
||||
async #rollback(previousCommit: string, message: string): Promise<void> {
|
||||
await this.#git(['reset', '--hard', previousCommit]);
|
||||
this.#save({
|
||||
phase: 'rolled_back',
|
||||
message,
|
||||
error: message,
|
||||
progressPercent: 100,
|
||||
release: this.#state.release,
|
||||
pendingCommit: '',
|
||||
pendingAction: '',
|
||||
});
|
||||
}
|
||||
|
||||
#fail(message: string): void {
|
||||
this.#save({
|
||||
phase: 'failed',
|
||||
message,
|
||||
error: message,
|
||||
progressPercent: this.#state.progressPercent,
|
||||
release: this.#state.release,
|
||||
pendingCommit: '',
|
||||
pendingAction: '',
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* A restarted process has no memory of the install. When the recorded target commit is now
|
||||
* HEAD, the update landed, and that is the confirmation the UI has been waiting for.
|
||||
*/
|
||||
async #reconcile(): Promise<void> {
|
||||
if (this.#reconciled) return;
|
||||
this.#reconciled = true;
|
||||
const state = this.#state;
|
||||
if (!state.pendingAction || !BUSY_UPDATE_PHASES.has(state.phase)) return;
|
||||
const head = await this.#localHead();
|
||||
if (!head) return;
|
||||
if (state.pendingAction === 'install' && head !== state.pendingCommit) {
|
||||
this.#save({
|
||||
...state,
|
||||
phase: 'failed',
|
||||
message: '重启后未看到目标版本,安装可能未完成。',
|
||||
error: '重启后未看到目标版本,请查看服务日志。',
|
||||
pendingCommit: '',
|
||||
pendingAction: '',
|
||||
});
|
||||
return;
|
||||
}
|
||||
this.#save({
|
||||
...state,
|
||||
phase: 'up_to_date',
|
||||
message:
|
||||
state.pendingAction === 'install'
|
||||
? `已更新到 ${head.slice(0, 7)} 并完成重启。`
|
||||
: '服务已重启完成。',
|
||||
error: '',
|
||||
progressPercent: 100,
|
||||
release: state.release ? { ...state.release, status: 'latest', currentCommit: head } : null,
|
||||
pendingCommit: '',
|
||||
pendingAction: '',
|
||||
});
|
||||
}
|
||||
|
||||
#view(): UpdateStatus {
|
||||
const git = this.#mode === 'git';
|
||||
return {
|
||||
phase: this.#state.phase,
|
||||
message: this.#state.message,
|
||||
error: this.#state.error,
|
||||
progressPercent: this.#state.progressPercent,
|
||||
checkedAt: this.#state.checkedAt,
|
||||
deploymentMode: this.#mode === 'unknown' ? 'git' : this.#mode,
|
||||
installSupported: git,
|
||||
restartSupported: this.#launcher.supported,
|
||||
release: this.#state.release,
|
||||
};
|
||||
}
|
||||
|
||||
#emptyState(message: string): PersistedState {
|
||||
return {
|
||||
phase: 'idle',
|
||||
message,
|
||||
error: '',
|
||||
progressPercent: 0,
|
||||
checkedAt: '',
|
||||
release: null,
|
||||
pendingCommit: '',
|
||||
pendingAction: '',
|
||||
};
|
||||
}
|
||||
|
||||
#read(): PersistedState {
|
||||
let row: { value_json?: string | null } | undefined;
|
||||
try {
|
||||
row = this.#db.prepare('SELECT value_json FROM app_settings WHERE key = ?').get(STATE_KEY) as
|
||||
| { value_json?: string | null }
|
||||
| undefined;
|
||||
} catch {
|
||||
row = undefined;
|
||||
}
|
||||
if (!row?.value_json) return this.#emptyState('尚未检查更新。');
|
||||
try {
|
||||
const parsed: unknown = JSON.parse(row.value_json);
|
||||
const source =
|
||||
parsed && typeof parsed === 'object' && !Array.isArray(parsed)
|
||||
? (parsed as Record<string, unknown>)
|
||||
: {};
|
||||
const pendingAction = source.pendingAction;
|
||||
return {
|
||||
phase: phaseOf(source.phase),
|
||||
message: text(source.message, 256),
|
||||
error: text(source.error, 256),
|
||||
progressPercent: percent(source.progressPercent),
|
||||
checkedAt: text(source.checkedAt, 64),
|
||||
release: readRelease(source.release),
|
||||
pendingCommit: text(source.pendingCommit, 40),
|
||||
pendingAction:
|
||||
pendingAction === 'install' || pendingAction === 'restart' ? pendingAction : '',
|
||||
};
|
||||
} catch {
|
||||
return this.#emptyState('尚未检查更新。');
|
||||
}
|
||||
}
|
||||
|
||||
#save(input: Omit<PersistedState, 'checkedAt'>): void {
|
||||
const now = this.#now().toISOString();
|
||||
this.#state = { ...input, checkedAt: now };
|
||||
try {
|
||||
this.#db
|
||||
.prepare(
|
||||
`INSERT INTO app_settings (key,value_json,created_at,updated_at)
|
||||
VALUES (?,?,?,?)
|
||||
ON CONFLICT(key) DO UPDATE SET value_json = excluded.value_json,
|
||||
updated_at = excluded.updated_at`,
|
||||
)
|
||||
.run(STATE_KEY, JSON.stringify(this.#state), now, now);
|
||||
} catch {
|
||||
// A read-only database must not break the update flow; the in-memory state still leads.
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,189 @@
|
||||
import Database from 'better-sqlite3';
|
||||
import { afterEach, describe, expect, it } from 'vitest';
|
||||
|
||||
import { migrateDatabase } from '../../infrastructure/database/migrations.js';
|
||||
import { ConnectionLogService } from './connection-log-service.js';
|
||||
import { LogCenterService } from './log-center-service.js';
|
||||
|
||||
const databases: Database.Database[] = [];
|
||||
|
||||
function fixture() {
|
||||
const db = new Database(':memory:');
|
||||
db.pragma('foreign_keys = ON');
|
||||
migrateDatabase(db);
|
||||
databases.push(db);
|
||||
db.prepare(
|
||||
`INSERT INTO instances (id,name,base_url,enabled,config_revision,created_at,updated_at)
|
||||
VALUES ('node-a','Node A','http://node-a.local',1,3,'2026-09-01T00:00:00.000Z','2026-09-01T00:00:00.000Z')`,
|
||||
).run();
|
||||
db.prepare(
|
||||
`INSERT INTO instance_tags (instance_id,tag,created_at) VALUES ('node-a','lab','2026-09-01T00:00:00.000Z')`,
|
||||
).run();
|
||||
const connections = new ConnectionLogService({ db });
|
||||
return { db, connections, logs: new LogCenterService({ db, connections }) };
|
||||
}
|
||||
|
||||
afterEach(() => {
|
||||
for (const db of databases.splice(0)) db.close();
|
||||
});
|
||||
|
||||
function seed(db: Database.Database): void {
|
||||
db.prepare(
|
||||
`INSERT INTO event_journal (public_id,envelope_json)
|
||||
VALUES ('evt-1',@envelope)`,
|
||||
).run({
|
||||
envelope: JSON.stringify({
|
||||
id: 'evt-1',
|
||||
kind: 'instance.updated',
|
||||
instanceId: 'node-a',
|
||||
occurredAt: '2026-09-04T02:00:00.000Z',
|
||||
}),
|
||||
});
|
||||
db.prepare(
|
||||
`INSERT INTO audit_events
|
||||
(id,instance_id,actor,operation_id,risk_level,request_id,parameters_summary_json,result_code,duration_ms,created_at)
|
||||
VALUES ('aud-1','node-a','console','restart-service','R1','req-1','{}','failed',42,'2026-09-04T03:00:00.000Z')`,
|
||||
).run();
|
||||
db.prepare(
|
||||
`INSERT INTO scheduled_tasks
|
||||
(id,name,operation_type,enabled,version,cron_expression,timezone,target_selector_json,
|
||||
misfire_policy,overlap_policy,retry_policy_json,created_by,updated_by,created_at,updated_at)
|
||||
VALUES ('task-1','夜间重启','restart-service',1,1,'0 3 * * *','Asia/Shanghai','{}',
|
||||
'skip','skip','{"maximumAttempts":1}','console','console','2026-09-01T00:00:00.000Z','2026-09-01T00:00:00.000Z')`,
|
||||
).run();
|
||||
db.prepare(
|
||||
`INSERT INTO scheduled_runs
|
||||
(id,scheduled_task_id,schedule_version,task_snapshot_json,due_at,claimed_at,started_at,finished_at,
|
||||
target_snapshot_json,outcome,reason,job_ids_json,trigger_source,attempt)
|
||||
VALUES ('run-1','task-1',1,'{"name":"夜间重启"}','2026-09-04T04:00:00.000Z','2026-09-04T04:00:00.000Z',
|
||||
'2026-09-04T04:00:00.000Z','2026-09-04T04:00:05.000Z','[]','failed','UPSTREAM_TIMEOUT','[]','scheduled',1)`,
|
||||
).run();
|
||||
db.prepare(
|
||||
`INSERT INTO notification_deliveries
|
||||
(id,rule_id,channel_id,instance_id,event_type,status,detail,created_at,sent_at)
|
||||
VALUES ('del-1',NULL,NULL,'node-a','sms','failed','smtp refused','2026-09-04T05:00:00.000Z',NULL)`,
|
||||
).run();
|
||||
}
|
||||
|
||||
describe('LogCenterService', () => {
|
||||
it('merges every persisted activity source into one timeline', () => {
|
||||
const { db, logs } = fixture();
|
||||
seed(db);
|
||||
const page = logs.listRuntimeLogs();
|
||||
expect(page.page.total).toBe(4);
|
||||
expect(page.items.map((item) => item.source)).toEqual([
|
||||
'delivery',
|
||||
'schedule',
|
||||
'audit',
|
||||
'event',
|
||||
]);
|
||||
expect(page.counts).toEqual({ event: 1, audit: 1, schedule: 1, delivery: 1 });
|
||||
expect(page.items[1]).toMatchObject({ level: 'error', code: 'failed' });
|
||||
});
|
||||
|
||||
it('filters the timeline by level, source and search text', () => {
|
||||
const { db, logs } = fixture();
|
||||
seed(db);
|
||||
expect(logs.listRuntimeLogs({ level: 'error' }).items.map((item) => item.source)).toEqual([
|
||||
'delivery',
|
||||
'schedule',
|
||||
'audit',
|
||||
]);
|
||||
expect(logs.listRuntimeLogs({ source: 'event' }).page.total).toBe(1);
|
||||
expect(logs.listRuntimeLogs({ search: 'smtp' }).page.total).toBe(1);
|
||||
expect(logs.listRuntimeLogs({ from: '2026-09-04T04:30:00.000Z' }).page.total).toBe(1);
|
||||
expect(logs.listRuntimeLogs({ instanceId: 'node-a' }).page.total).toBe(3);
|
||||
});
|
||||
|
||||
it('rejects malformed pagination and unknown enum values', () => {
|
||||
const { logs } = fixture();
|
||||
expect(() => logs.listRuntimeLogs({ page: 0 })).toThrow(/positive integer/u);
|
||||
expect(() => logs.listRuntimeLogs({ pageSize: 500 })).toThrow(/between 1 and 200/u);
|
||||
expect(() => logs.listRuntimeLogs({ source: 'nope' as never })).toThrow(/source is invalid/u);
|
||||
expect(() => logs.listRuntimeLogs({ from: 'yesterday' })).toThrow(/from is invalid/u);
|
||||
});
|
||||
|
||||
it('reports connection snapshots as device health for the live probe path', () => {
|
||||
const { db, connections, logs } = fixture();
|
||||
db.prepare(
|
||||
`INSERT INTO status_snapshots
|
||||
(id,instance_id,category,state,payload_json,observed_at,expires_at,created_at)
|
||||
VALUES ('snap-1','node-a','connection','fresh','{"reachable":true,"authenticated":false}',
|
||||
'2026-09-04T06:00:00.000Z','2026-09-04T06:00:30.000Z','2026-09-04T06:00:00.000Z')`,
|
||||
).run();
|
||||
db.prepare(
|
||||
`INSERT INTO capabilities
|
||||
(instance_id,operation_id,state,observed_at,created_at,updated_at)
|
||||
VALUES ('node-a','restart-service','supported','2026-09-04T06:00:00.000Z',
|
||||
'2026-09-04T06:00:00.000Z','2026-09-04T06:00:00.000Z'),
|
||||
('node-a','reboot-system','unsupported','2026-09-04T06:00:00.000Z',
|
||||
'2026-09-04T06:00:00.000Z','2026-09-04T06:00:00.000Z')`,
|
||||
).run();
|
||||
connections.record({
|
||||
instanceId: 'node-a',
|
||||
outcome: 'stale',
|
||||
state: 'stale',
|
||||
errorCode: 'AUTH_REQUIRED',
|
||||
httpStatus: 401,
|
||||
durationMs: 15,
|
||||
observedAt: '2026-09-04T06:00:00.000Z',
|
||||
});
|
||||
const [device] = logs.diagnostics();
|
||||
expect(device).toMatchObject({
|
||||
instanceId: 'node-a',
|
||||
name: 'Node A',
|
||||
enabled: true,
|
||||
revision: 3,
|
||||
tags: ['lab'],
|
||||
capabilities: { total: 2, supported: 1, unsupported: 1 },
|
||||
});
|
||||
expect(device?.health).toMatchObject({
|
||||
category: 'connection',
|
||||
status: 'reachable',
|
||||
authenticated: false,
|
||||
errorCode: 'AUTH_REQUIRED',
|
||||
});
|
||||
expect(device?.connections).toMatchObject({ total: 1, availabilityPercent: 0, failed: 0 });
|
||||
});
|
||||
|
||||
it('prefers a full health envelope when both snapshot categories exist', () => {
|
||||
const { db, logs } = fixture();
|
||||
for (const [category, payload] of [
|
||||
['connection', '{"reachable":true,"authenticated":true}'],
|
||||
[
|
||||
'health',
|
||||
'{"schemaVersion":1,"data":{"status":"ok","version":"1.9.6","platform":"linux"},' +
|
||||
'"errorCode":null,"httpStatus":200,"fetchedAt":"2026-09-04T06:00:00.000Z","durationMs":21,' +
|
||||
'"freshness":"fresh","supported":true,"dataFetchedAt":null}',
|
||||
],
|
||||
] as const) {
|
||||
db.prepare(
|
||||
`INSERT INTO status_snapshots
|
||||
(id,instance_id,category,state,payload_json,observed_at,expires_at,created_at)
|
||||
VALUES (?, 'node-a', ?, 'fresh', ?, '2026-09-04T06:00:00.000Z', NULL, '2026-09-04T06:00:00.000Z')`,
|
||||
).run(`snap-${category}`, category, payload);
|
||||
}
|
||||
const [device] = logs.diagnostics();
|
||||
expect(device?.health).toMatchObject({
|
||||
category: 'health',
|
||||
version: '1.9.6',
|
||||
platform: 'linux',
|
||||
status: 'ok',
|
||||
durationMs: 21,
|
||||
httpStatus: 200,
|
||||
});
|
||||
expect(device?.snapshots.map((snapshot) => snapshot.category)).toEqual([
|
||||
'connection',
|
||||
'health',
|
||||
]);
|
||||
});
|
||||
|
||||
it('surfaces the most recent failed operations per device', () => {
|
||||
const { db, logs } = fixture();
|
||||
seed(db);
|
||||
const [device] = logs.diagnostics();
|
||||
expect(device?.recentFailures).toEqual([
|
||||
{ occurredAt: '2026-09-04T03:00:00.000Z', code: 'failed', message: 'restart-service' },
|
||||
]);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,445 @@
|
||||
import type Database from 'better-sqlite3';
|
||||
|
||||
import type { ConnectionLogService, ConnectionSummary } from './connection-log-service.js';
|
||||
|
||||
export type LogSource = 'event' | 'audit' | 'schedule' | 'delivery';
|
||||
export type LogLevel = 'info' | 'warning' | 'error';
|
||||
|
||||
const SOURCES: readonly LogSource[] = ['event', 'audit', 'schedule', 'delivery'];
|
||||
const LEVELS: readonly LogLevel[] = ['info', 'warning', 'error'];
|
||||
|
||||
export interface RuntimeLogEntry {
|
||||
readonly id: string;
|
||||
readonly source: LogSource;
|
||||
readonly level: LogLevel;
|
||||
readonly occurredAt: string;
|
||||
readonly instanceId: string | null;
|
||||
readonly message: string;
|
||||
readonly code: string | null;
|
||||
readonly actor: string | null;
|
||||
readonly durationMs: number | null;
|
||||
}
|
||||
|
||||
export interface RuntimeLogQuery {
|
||||
readonly page?: number;
|
||||
readonly pageSize?: number;
|
||||
readonly source?: LogSource | undefined;
|
||||
readonly level?: LogLevel | undefined;
|
||||
readonly instanceId?: string | undefined;
|
||||
readonly search?: string | undefined;
|
||||
readonly from?: string | undefined;
|
||||
readonly to?: string | undefined;
|
||||
}
|
||||
|
||||
export interface RuntimeLogPage {
|
||||
readonly items: readonly RuntimeLogEntry[];
|
||||
readonly page: { readonly page: number; readonly pageSize: number; readonly total: number };
|
||||
readonly counts: Readonly<Record<LogSource, number>>;
|
||||
}
|
||||
|
||||
export interface InstanceDiagnostics {
|
||||
readonly instanceId: string;
|
||||
readonly name: string;
|
||||
readonly origin: string;
|
||||
readonly enabled: boolean;
|
||||
readonly revision: number;
|
||||
readonly tags: readonly string[];
|
||||
readonly health: {
|
||||
readonly category: 'health' | 'connection';
|
||||
readonly state: string;
|
||||
readonly observedAt: string;
|
||||
readonly errorCode: string | null;
|
||||
readonly httpStatus: number | null;
|
||||
readonly durationMs: number | null;
|
||||
readonly version: string | null;
|
||||
readonly platform: string | null;
|
||||
readonly status: string | null;
|
||||
readonly authenticated: boolean | null;
|
||||
} | null;
|
||||
readonly snapshots: readonly {
|
||||
readonly category: string;
|
||||
readonly state: string;
|
||||
readonly observedAt: string;
|
||||
}[];
|
||||
readonly capabilities: {
|
||||
readonly total: number;
|
||||
readonly supported: number;
|
||||
readonly unsupported: number;
|
||||
readonly authRequired: number;
|
||||
readonly degraded: number;
|
||||
readonly unknown: number;
|
||||
};
|
||||
readonly connections: ConnectionSummary | null;
|
||||
readonly recentFailures: readonly {
|
||||
readonly occurredAt: string;
|
||||
readonly code: string | null;
|
||||
readonly message: string;
|
||||
}[];
|
||||
}
|
||||
|
||||
export type LogCenterErrorCode = 'VALIDATION_FAILED';
|
||||
|
||||
export class LogCenterError extends Error {
|
||||
constructor(
|
||||
readonly code: LogCenterErrorCode,
|
||||
message: string,
|
||||
) {
|
||||
super(message);
|
||||
this.name = 'LogCenterError';
|
||||
}
|
||||
}
|
||||
|
||||
export interface LogCenterOptions {
|
||||
readonly db: Database.Database;
|
||||
readonly connections: ConnectionLogService;
|
||||
}
|
||||
|
||||
const MAX_PAGE_SIZE = 200;
|
||||
|
||||
function invalid(message: string): never {
|
||||
throw new LogCenterError('VALIDATION_FAILED', message);
|
||||
}
|
||||
|
||||
function instant(value: string): string | undefined {
|
||||
const parsed = Date.parse(value);
|
||||
return Number.isFinite(parsed) ? new Date(parsed).toISOString() : undefined;
|
||||
}
|
||||
|
||||
/**
|
||||
* One UNION over the tables that already persist control-plane activity. Nothing new is
|
||||
* written here, so the timeline stays consistent with the pages that own each source.
|
||||
*/
|
||||
const TIMELINE_SQL = `
|
||||
SELECT id, source, level, occurred_at, instance_id, message, code, actor, duration_ms FROM (
|
||||
SELECT e.public_id || ':' || e.sequence AS id,
|
||||
'event' AS source,
|
||||
'info' AS level,
|
||||
json_extract(e.envelope_json,'$.occurredAt') AS occurred_at,
|
||||
json_extract(e.envelope_json,'$.instanceId') AS instance_id,
|
||||
'事件 ' || json_extract(e.envelope_json,'$.kind') || ' ' || json_extract(e.envelope_json,'$.id')
|
||||
AS message,
|
||||
json_extract(e.envelope_json,'$.kind') AS code,
|
||||
NULL AS actor,
|
||||
NULL AS duration_ms
|
||||
FROM event_journal e
|
||||
UNION ALL
|
||||
SELECT a.id,
|
||||
'audit',
|
||||
CASE WHEN a.result_code = 'success' THEN 'info' ELSE 'error' END,
|
||||
a.created_at,
|
||||
a.instance_id,
|
||||
'操作 ' || a.operation_id || ' 结果 ' || a.result_code,
|
||||
a.result_code,
|
||||
a.actor,
|
||||
a.duration_ms
|
||||
FROM audit_events a
|
||||
UNION ALL
|
||||
SELECT r.id,
|
||||
'schedule',
|
||||
CASE
|
||||
WHEN r.outcome IN ('failed','needs-attention') THEN 'error'
|
||||
WHEN r.outcome IN ('partially-succeeded','skipped','no-targets') THEN 'warning'
|
||||
ELSE 'info'
|
||||
END,
|
||||
COALESCE(r.finished_at, r.due_at),
|
||||
NULL,
|
||||
'定时任务 ' || COALESCE(json_extract(r.task_snapshot_json,'$.name'), r.scheduled_task_id)
|
||||
|| ' 结果 ' || COALESCE(r.outcome, 'running')
|
||||
|| COALESCE(' 原因 ' || r.reason, ''),
|
||||
COALESCE(r.outcome, 'running'),
|
||||
NULL,
|
||||
CAST((julianday(COALESCE(r.finished_at, r.claimed_at)) - julianday(r.started_at))
|
||||
* 86400000 AS INTEGER)
|
||||
FROM scheduled_runs r
|
||||
UNION ALL
|
||||
SELECT d.id,
|
||||
'delivery',
|
||||
CASE
|
||||
WHEN d.status = 'failed' THEN 'error'
|
||||
WHEN d.status IN ('unmatched','no_available_channel','quiet_hours','rate_limited')
|
||||
THEN 'warning'
|
||||
ELSE 'info'
|
||||
END,
|
||||
d.created_at,
|
||||
d.instance_id,
|
||||
'通知 ' || d.event_type || ' 状态 ' || d.status
|
||||
|| COALESCE(' 详情 ' || d.detail, ''),
|
||||
d.status,
|
||||
NULL,
|
||||
NULL
|
||||
FROM notification_deliveries d
|
||||
)`;
|
||||
|
||||
export class LogCenterService {
|
||||
readonly #db: Database.Database;
|
||||
readonly #connections: ConnectionLogService;
|
||||
|
||||
constructor(options: LogCenterOptions) {
|
||||
this.#db = options.db;
|
||||
this.#connections = options.connections;
|
||||
}
|
||||
|
||||
listRuntimeLogs(query: RuntimeLogQuery = {}): RuntimeLogPage {
|
||||
const page = query.page ?? 1;
|
||||
const pageSize = query.pageSize ?? 50;
|
||||
if (!Number.isSafeInteger(page) || page < 1) invalid('page must be a positive integer');
|
||||
if (!Number.isSafeInteger(pageSize) || pageSize < 1 || pageSize > MAX_PAGE_SIZE)
|
||||
invalid(`pageSize must be between 1 and ${MAX_PAGE_SIZE}`);
|
||||
if (query.source !== undefined && !SOURCES.includes(query.source)) invalid('source is invalid');
|
||||
if (query.level !== undefined && !LEVELS.includes(query.level)) invalid('level is invalid');
|
||||
if (query.search !== undefined && query.search.length > 200) invalid('search is too long');
|
||||
if (query.instanceId !== undefined && query.instanceId.trim() === '')
|
||||
invalid('instanceId is invalid');
|
||||
const bounds: Record<string, string> = {};
|
||||
for (const key of ['from', 'to'] as const) {
|
||||
const value = query[key];
|
||||
if (value === undefined) continue;
|
||||
const normalized = instant(value);
|
||||
if (!normalized) invalid(`${key} is invalid`);
|
||||
bounds[key] = normalized;
|
||||
}
|
||||
|
||||
const where: string[] = [];
|
||||
const parameters: Record<string, unknown> = {};
|
||||
if (query.source) {
|
||||
where.push('source = @source');
|
||||
parameters.source = query.source;
|
||||
}
|
||||
if (query.level) {
|
||||
where.push('level = @level');
|
||||
parameters.level = query.level;
|
||||
}
|
||||
if (query.instanceId) {
|
||||
where.push('instance_id = @instanceId');
|
||||
parameters.instanceId = query.instanceId;
|
||||
}
|
||||
if (bounds.from) {
|
||||
where.push('occurred_at >= @from');
|
||||
parameters.from = bounds.from;
|
||||
}
|
||||
if (bounds.to) {
|
||||
where.push('occurred_at <= @to');
|
||||
parameters.to = bounds.to;
|
||||
}
|
||||
const needle = query.search?.trim().toLowerCase() ?? '';
|
||||
if (needle !== '') {
|
||||
where.push(
|
||||
`(lower(message) LIKE @search OR lower(COALESCE(code,'')) LIKE @search
|
||||
OR lower(COALESCE(actor,'')) LIKE @search OR lower(COALESCE(instance_id,'')) LIKE @search)`,
|
||||
);
|
||||
parameters.search = `%${needle}%`;
|
||||
}
|
||||
const clause = where.length > 0 ? ` WHERE ${where.join(' AND ')}` : '';
|
||||
|
||||
const total = (
|
||||
this.#db
|
||||
.prepare(`SELECT COUNT(*) AS count FROM (${TIMELINE_SQL})${clause}`)
|
||||
.get(parameters) as { count: number }
|
||||
).count;
|
||||
const rows = this.#db
|
||||
.prepare(
|
||||
`SELECT id, source, level, occurred_at, instance_id, message, code, actor, duration_ms
|
||||
FROM (${TIMELINE_SQL})${clause}
|
||||
ORDER BY occurred_at DESC LIMIT @limit OFFSET @offset`,
|
||||
)
|
||||
.all({ ...parameters, limit: pageSize, offset: (page - 1) * pageSize }) as Array<
|
||||
Record<string, unknown>
|
||||
>;
|
||||
|
||||
const counts = Object.fromEntries(SOURCES.map((source) => [source, 0])) as Record<
|
||||
LogSource,
|
||||
number
|
||||
>;
|
||||
const grouped = this.#db
|
||||
.prepare(`SELECT source, COUNT(*) AS count FROM (${TIMELINE_SQL}) GROUP BY source`)
|
||||
.all() as Array<{ source: string; count: number }>;
|
||||
for (const row of grouped) {
|
||||
if (SOURCES.includes(row.source as LogSource)) counts[row.source as LogSource] = row.count;
|
||||
}
|
||||
|
||||
return Object.freeze({
|
||||
items: Object.freeze(rows.map((row) => this.#entry(row))),
|
||||
page: Object.freeze({ page, pageSize, total }),
|
||||
counts: Object.freeze(counts),
|
||||
});
|
||||
}
|
||||
|
||||
/** Per-instance diagnostics for the whole fleet, newest failure surfaced first. */
|
||||
diagnostics(): readonly InstanceDiagnostics[] {
|
||||
const instances = this.#db
|
||||
.prepare(
|
||||
`SELECT id,name,base_url,enabled,config_revision FROM instances ORDER BY name COLLATE NOCASE`,
|
||||
)
|
||||
.all() as Array<Record<string, unknown>>;
|
||||
const tagRows = this.#db.prepare('SELECT instance_id, tag FROM instance_tags').all() as Array<{
|
||||
instance_id: string;
|
||||
tag: string;
|
||||
}>;
|
||||
const tagsByInstance = new Map<string, string[]>();
|
||||
for (const row of tagRows) {
|
||||
const bucket = tagsByInstance.get(row.instance_id) ?? [];
|
||||
bucket.push(row.tag);
|
||||
tagsByInstance.set(row.instance_id, bucket);
|
||||
}
|
||||
const summaries = new Map(
|
||||
this.#connections.summarize().map((summary) => [summary.instanceId, summary]),
|
||||
);
|
||||
|
||||
return Object.freeze(
|
||||
instances.map((instance) =>
|
||||
Object.freeze(this.#diagnostics(instance, tagsByInstance, summaries)),
|
||||
),
|
||||
);
|
||||
}
|
||||
|
||||
#diagnostics(
|
||||
instance: Record<string, unknown>,
|
||||
tagsByInstance: ReadonlyMap<string, readonly string[]>,
|
||||
summaries: ReadonlyMap<string, ConnectionSummary>,
|
||||
): InstanceDiagnostics {
|
||||
const instanceId = String(instance.id);
|
||||
// The control plane persists 'health' envelopes only when the snapshot service runs; the
|
||||
// live reachability path writes 'connection' rows, so diagnostics reads whichever exists.
|
||||
const snapshot = this.#db
|
||||
.prepare(
|
||||
`SELECT category, state, payload_json, observed_at FROM status_snapshots
|
||||
WHERE instance_id = ? AND category IN ('health','connection')
|
||||
ORDER BY CASE WHEN category = 'health' THEN 0 ELSE 1 END LIMIT 1`,
|
||||
)
|
||||
.get(instanceId) as
|
||||
| { category: string; state: string; payload_json: string; observed_at: string }
|
||||
| undefined;
|
||||
const envelope = parseRecord(snapshot?.payload_json);
|
||||
const data = parseRecord(envelope?.data) ?? envelope;
|
||||
const capability = this.#db
|
||||
.prepare(
|
||||
`SELECT
|
||||
COUNT(*) AS total,
|
||||
SUM(CASE WHEN state = 'supported' THEN 1 ELSE 0 END) AS supported,
|
||||
SUM(CASE WHEN state = 'unsupported' THEN 1 ELSE 0 END) AS unsupported,
|
||||
SUM(CASE WHEN state = 'auth-required' THEN 1 ELSE 0 END) AS auth_required,
|
||||
SUM(CASE WHEN state = 'degraded' THEN 1 ELSE 0 END) AS degraded,
|
||||
SUM(CASE WHEN state = 'unknown' THEN 1 ELSE 0 END) AS unknown_state
|
||||
FROM capabilities WHERE instance_id = ?`,
|
||||
)
|
||||
.get(instanceId) as Record<string, number | null> | undefined;
|
||||
const snapshots = this.#db
|
||||
.prepare(
|
||||
`SELECT category, state, observed_at FROM status_snapshots
|
||||
WHERE instance_id = ? ORDER BY category`,
|
||||
)
|
||||
.all(instanceId) as Array<{ category: string; state: string; observed_at: string }>;
|
||||
const failures = this.#db
|
||||
.prepare(
|
||||
`SELECT id, operation_id, result_code, created_at FROM audit_events
|
||||
WHERE instance_id = ? AND result_code <> 'success'
|
||||
ORDER BY created_at DESC LIMIT 5`,
|
||||
)
|
||||
.all(instanceId) as Array<{
|
||||
id: string;
|
||||
operation_id: string;
|
||||
result_code: string;
|
||||
created_at: string;
|
||||
}>;
|
||||
|
||||
return {
|
||||
instanceId,
|
||||
name: String(instance.name ?? instanceId),
|
||||
origin: String(instance.base_url ?? ''),
|
||||
enabled: Number(instance.enabled ?? 0) === 1,
|
||||
revision: Number(instance.config_revision ?? 1),
|
||||
tags: Object.freeze([...(tagsByInstance.get(instanceId) ?? [])]),
|
||||
health: snapshot ? Object.freeze(this.#health(snapshot, envelope, data)) : null,
|
||||
snapshots: Object.freeze(
|
||||
snapshots.map((row) =>
|
||||
Object.freeze({ category: row.category, state: row.state, observedAt: row.observed_at }),
|
||||
),
|
||||
),
|
||||
capabilities: Object.freeze({
|
||||
total: Number(capability?.total ?? 0),
|
||||
supported: Number(capability?.supported ?? 0),
|
||||
unsupported: Number(capability?.unsupported ?? 0),
|
||||
authRequired: Number(capability?.auth_required ?? 0),
|
||||
degraded: Number(capability?.degraded ?? 0),
|
||||
unknown: Number(capability?.unknown_state ?? 0),
|
||||
}),
|
||||
connections: summaries.get(instanceId) ?? null,
|
||||
recentFailures: Object.freeze(
|
||||
failures.map((row) =>
|
||||
Object.freeze({
|
||||
occurredAt: row.created_at,
|
||||
code: row.result_code,
|
||||
message: row.operation_id,
|
||||
}),
|
||||
),
|
||||
),
|
||||
};
|
||||
}
|
||||
|
||||
#entry(row: Record<string, unknown>): RuntimeLogEntry {
|
||||
return Object.freeze({
|
||||
id: String(row.id),
|
||||
source: row.source as LogSource,
|
||||
level: row.level as LogLevel,
|
||||
occurredAt: String(row.occurred_at ?? ''),
|
||||
instanceId:
|
||||
row.instance_id === null || row.instance_id === undefined ? null : String(row.instance_id),
|
||||
message: String(row.message ?? ''),
|
||||
code: row.code === null || row.code === undefined ? null : String(row.code),
|
||||
actor: row.actor === null || row.actor === undefined ? null : String(row.actor),
|
||||
durationMs:
|
||||
row.duration_ms === null || row.duration_ms === undefined ? null : Number(row.duration_ms),
|
||||
});
|
||||
}
|
||||
|
||||
#health(
|
||||
snapshot: { category: string; state: string; observed_at: string },
|
||||
envelope: Record<string, unknown> | undefined,
|
||||
data: Record<string, unknown> | undefined,
|
||||
): NonNullable<InstanceDiagnostics['health']> {
|
||||
const category = snapshot.category === 'connection' ? 'connection' : 'health';
|
||||
if (category === 'connection') {
|
||||
const authenticated =
|
||||
typeof envelope?.authenticated === 'boolean' ? envelope.authenticated : null;
|
||||
const reachable = typeof envelope?.reachable === 'boolean' ? envelope.reachable : null;
|
||||
return Object.freeze({
|
||||
category,
|
||||
state: snapshot.state,
|
||||
observedAt: snapshot.observed_at,
|
||||
errorCode: authenticated === false ? 'AUTH_REQUIRED' : null,
|
||||
httpStatus: null,
|
||||
durationMs: null,
|
||||
version: null,
|
||||
platform: null,
|
||||
status: reachable === null ? null : reachable ? 'reachable' : 'unreachable',
|
||||
authenticated,
|
||||
});
|
||||
}
|
||||
return Object.freeze({
|
||||
category,
|
||||
state: snapshot.state,
|
||||
observedAt: snapshot.observed_at,
|
||||
errorCode: typeof envelope?.errorCode === 'string' ? envelope.errorCode : null,
|
||||
httpStatus: typeof envelope?.httpStatus === 'number' ? envelope.httpStatus : null,
|
||||
durationMs: typeof envelope?.durationMs === 'number' ? envelope.durationMs : null,
|
||||
version: typeof data?.version === 'string' ? data.version : null,
|
||||
platform: typeof data?.platform === 'string' ? data.platform : null,
|
||||
status: typeof data?.status === 'string' ? data.status : null,
|
||||
authenticated: null,
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
/** Accepts either a JSON string or an already-decoded object so both snapshot shapes work. */
|
||||
function parseRecord(value: unknown): Record<string, unknown> | undefined {
|
||||
if (value !== null && typeof value === 'object' && !Array.isArray(value))
|
||||
return value as Record<string, unknown>;
|
||||
if (typeof value !== 'string' || value === '') return undefined;
|
||||
try {
|
||||
const parsed: unknown = JSON.parse(value);
|
||||
if (parsed === null || typeof parsed !== 'object' || Array.isArray(parsed)) return undefined;
|
||||
return parsed as Record<string, unknown>;
|
||||
} catch {
|
||||
return undefined;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,243 @@
|
||||
import Database from 'better-sqlite3';
|
||||
import { mkdtemp, readFile, rm } from 'node:fs/promises';
|
||||
import { join } from 'node:path';
|
||||
import { tmpdir } from 'node:os';
|
||||
import { afterEach, describe, expect, it } from 'vitest';
|
||||
|
||||
import { migrateDatabase } from '../../infrastructure/database/migrations.js';
|
||||
import { SystemMaintenanceService } from './system-maintenance-service.js';
|
||||
|
||||
const roots: string[] = [];
|
||||
afterEach(async () => {
|
||||
await Promise.all(roots.splice(0).map((root) => rm(root, { recursive: true, force: true })));
|
||||
});
|
||||
|
||||
function database(): Database.Database {
|
||||
const db = new Database(':memory:');
|
||||
db.pragma('foreign_keys = ON');
|
||||
migrateDatabase(db);
|
||||
return db;
|
||||
}
|
||||
|
||||
function insertAudit(db: Database.Database, id: string, createdAt: string): void {
|
||||
db.prepare(
|
||||
`INSERT INTO audit_events
|
||||
(id,actor,operation_id,risk_level,request_id,parameters_summary_json,result_code,duration_ms,created_at)
|
||||
VALUES (?,?,?,?,?,?,?,?,?)`,
|
||||
).run(id, 'operator', 'probe', 'R0', `${id}-request`, '{}', 'success', 1, createdAt);
|
||||
}
|
||||
|
||||
function insertJob(db: Database.Database, id: string, createdAt: string): void {
|
||||
db.prepare(
|
||||
`INSERT INTO jobs
|
||||
(id,operation_id,risk_level,status,requested_by,request_id,parameters_digest,created_at,updated_at)
|
||||
VALUES (?,?,?,?,?,?,?,?,?)`,
|
||||
).run(
|
||||
id,
|
||||
'probe',
|
||||
'R0',
|
||||
'succeeded',
|
||||
'operator',
|
||||
`${id}-request`,
|
||||
'0'.repeat(64),
|
||||
createdAt,
|
||||
createdAt,
|
||||
);
|
||||
}
|
||||
|
||||
function insertEvent(db: Database.Database, publicId: string, envelope: unknown): void {
|
||||
db.prepare('INSERT INTO event_journal (public_id,envelope_json) VALUES (?,?)').run(
|
||||
publicId,
|
||||
JSON.stringify(envelope),
|
||||
);
|
||||
}
|
||||
|
||||
describe('SystemMaintenanceService', () => {
|
||||
it('reports runtime storage, component counts, and Hub-compatible default retention', async () => {
|
||||
const db = database();
|
||||
insertAudit(db, 'audit-1', '2026-01-01T00:00:00.000Z');
|
||||
insertJob(db, 'job-1', '2026-01-01T00:00:00.000Z');
|
||||
insertEvent(db, 'event-1', { kind: 'test' });
|
||||
const service = new SystemMaintenanceService(db, { version: '1.9.9' });
|
||||
|
||||
const overview = await service.overview();
|
||||
|
||||
expect(overview.runtime).toMatchObject({
|
||||
version: '1.9.9',
|
||||
platform: process.platform,
|
||||
arch: process.arch,
|
||||
});
|
||||
expect(overview.runtime.uptimeSeconds).toBeGreaterThanOrEqual(0);
|
||||
expect(overview.storage.databaseBytes).toBeGreaterThan(0);
|
||||
expect(
|
||||
Object.fromEntries(overview.storage.components.map((item) => [item.key, item.count])),
|
||||
).toEqual({
|
||||
instances: 0,
|
||||
statusSnapshots: 0,
|
||||
jobs: 1,
|
||||
auditEvents: 1,
|
||||
scheduledRuns: 0,
|
||||
notificationQueue: 0,
|
||||
smsMessages: 0,
|
||||
eventJournal: 1,
|
||||
connectionLogs: 0,
|
||||
});
|
||||
expect(overview.retention.auditEvents).toEqual({
|
||||
enabled: true,
|
||||
days: 180,
|
||||
maximumCount: 50_000,
|
||||
});
|
||||
expect(overview.retention.jobs).toEqual({
|
||||
enabled: true,
|
||||
days: 90,
|
||||
maximumCount: 5_000,
|
||||
});
|
||||
});
|
||||
|
||||
it('persists validated retention policies and cleans selected components', async () => {
|
||||
const db = database();
|
||||
const task = {
|
||||
name: 'Night restart',
|
||||
operationType: 'restart-service' as const,
|
||||
cronExpression: '0 2 * * *',
|
||||
timezone: 'Asia/Shanghai' as const,
|
||||
targetSelector: { mode: 'fixed' as const, instanceIds: ['instance-a'] },
|
||||
misfirePolicy: 'skip' as const,
|
||||
overlapPolicy: 'skip' as const,
|
||||
retryPolicy: { maxRetries: 0, intervalSeconds: 60 },
|
||||
enabled: false,
|
||||
};
|
||||
db.prepare(
|
||||
`INSERT INTO scheduled_tasks
|
||||
(id,name,operation_type,enabled,version,cron_expression,timezone,target_selector_json,
|
||||
misfire_policy,overlap_policy,retry_policy_json,created_by,updated_by,created_at,updated_at)
|
||||
VALUES ('task-1',?,?,?,?,?,?,?,?,?,?,?,?,?,?)`,
|
||||
).run(
|
||||
task.name,
|
||||
task.operationType,
|
||||
0,
|
||||
1,
|
||||
task.cronExpression,
|
||||
task.timezone,
|
||||
JSON.stringify(task.targetSelector),
|
||||
task.misfirePolicy,
|
||||
task.overlapPolicy,
|
||||
JSON.stringify(task.retryPolicy),
|
||||
'operator',
|
||||
'operator',
|
||||
'2026-01-01T00:00:00.000Z',
|
||||
'2026-01-01T00:00:00.000Z',
|
||||
);
|
||||
db.prepare(
|
||||
`INSERT INTO scheduled_runs
|
||||
(id,scheduled_task_id,schedule_version,task_snapshot_json,due_at,claimed_at,
|
||||
target_snapshot_json,job_ids_json,trigger_source,attempt)
|
||||
VALUES ('run-old','task-1',1,?,'2026-01-01T02:00:00.000Z','2026-01-01T02:00:00.000Z','[]','[]','scheduled',1)`,
|
||||
).run(JSON.stringify(task));
|
||||
insertAudit(db, 'audit-old', '2026-01-01T00:00:00.000Z');
|
||||
insertAudit(db, 'audit-new', '2026-09-01T00:00:00.000Z');
|
||||
insertJob(db, 'job-old', '2026-01-01T00:00:00.000Z');
|
||||
insertEvent(db, 'event-old', { kind: 'old' });
|
||||
const service = new SystemMaintenanceService(db, {
|
||||
version: '1.9.9',
|
||||
now: () => new Date('2026-09-03T00:00:00.000Z'),
|
||||
});
|
||||
|
||||
await service.updateRetention({
|
||||
auditEvents: { enabled: true, days: 30, maximumCount: 1 },
|
||||
jobs: { enabled: true, days: 1, maximumCount: 100 },
|
||||
eventJournal: { enabled: true, days: 1, maximumCount: 100 },
|
||||
});
|
||||
|
||||
const result = await service.cleanup(['auditEvents', 'jobs', 'eventJournal', 'scheduledRuns']);
|
||||
expect(result).toEqual({
|
||||
auditEvents: 1,
|
||||
jobs: 1,
|
||||
eventJournal: 1,
|
||||
scheduledRuns: 1,
|
||||
});
|
||||
const counts = (table: string) =>
|
||||
(db.prepare(`SELECT COUNT(*) AS count FROM ${table}`).get() as { count: number }).count;
|
||||
expect(counts('audit_events')).toBe(1);
|
||||
expect(counts('jobs')).toBe(0);
|
||||
expect(counts('event_journal')).toBe(0);
|
||||
expect(counts('scheduled_runs')).toBe(0);
|
||||
expect((await service.overview()).retention.auditEvents).toEqual({
|
||||
enabled: true,
|
||||
days: 30,
|
||||
maximumCount: 1,
|
||||
});
|
||||
});
|
||||
|
||||
it('rejects unknown components and invalid retention values without touching settings', async () => {
|
||||
const db = database();
|
||||
const service = new SystemMaintenanceService(db, { version: '1.9.9' });
|
||||
await expect(service.cleanup(['unknown' as never])).rejects.toThrow('Unknown data component');
|
||||
await expect(
|
||||
service.updateRetention({ auditEvents: { enabled: true, days: 0, maximumCount: 1 } }),
|
||||
).rejects.toThrow('Retention days must be between');
|
||||
expect(await service.getRetention()).toEqual(await service.defaultRetention());
|
||||
});
|
||||
|
||||
it('optimizes SQLite and creates validated, listed backups with retention pruning', async () => {
|
||||
const db = database();
|
||||
const backupRoot = await mkdtemp(join(tmpdir(), 'multi-simadmin-backup-'));
|
||||
roots.push(backupRoot);
|
||||
const service = new SystemMaintenanceService(db, {
|
||||
version: '1.9.9',
|
||||
backupDirectory: backupRoot,
|
||||
maximumBackups: 2,
|
||||
});
|
||||
|
||||
expect(await service.optimize()).toEqual({
|
||||
checkpointed: true,
|
||||
vacuumed: true,
|
||||
analyzed: true,
|
||||
});
|
||||
const first = await service.createBackup();
|
||||
const second = await service.createBackup();
|
||||
const third = await service.createBackup();
|
||||
const backups = await service.listBackups();
|
||||
|
||||
expect(backups).toHaveLength(2);
|
||||
expect(backups.map((item) => item.filename)).toEqual([third.filename, second.filename]);
|
||||
expect(first.sha256).toMatch(/^[a-f0-9]{64}$/u);
|
||||
// Retention is enforced on disk, not just in the listing.
|
||||
await expect(readFile(first.path)).rejects.toThrow();
|
||||
expect(await readFile(second.path)).toBeTruthy();
|
||||
expect(backups[0]).toEqual(
|
||||
expect.objectContaining({
|
||||
filename: third.filename,
|
||||
sha256: third.sha256,
|
||||
sizeBytes: expect.any(Number),
|
||||
createdAt: expect.any(String),
|
||||
}),
|
||||
);
|
||||
});
|
||||
|
||||
it('serves and deletes single backups and refuses names outside the directory', async () => {
|
||||
const db = database();
|
||||
const backupRoot = await mkdtemp(join(tmpdir(), 'multi-simadmin-backup-'));
|
||||
roots.push(backupRoot);
|
||||
const service = new SystemMaintenanceService(db, {
|
||||
version: '1.9.9',
|
||||
backupDirectory: backupRoot,
|
||||
});
|
||||
|
||||
const backup = await service.createBackup();
|
||||
expect(await service.backupFile(backup.filename)).toEqual({
|
||||
filename: backup.filename,
|
||||
path: backup.path,
|
||||
sizeBytes: backup.sizeBytes,
|
||||
createdAt: expect.any(String),
|
||||
});
|
||||
expect(await service.backupFile('multi-simadmin-1970-01-01T00-00-00-000.db')).toBeUndefined();
|
||||
|
||||
await expect(service.backupFile('multi-simadmin-../secret.db')).rejects.toThrow();
|
||||
await expect(service.backupFile('multi-simadmin-notes.txt')).rejects.toThrow();
|
||||
await expect(service.deleteBackup('multi-simadmin-%00.db')).rejects.toThrow();
|
||||
|
||||
expect(await service.deleteBackup(backup.filename)).toEqual({ filename: backup.filename });
|
||||
expect(await service.listBackups()).toEqual([]);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,387 @@
|
||||
import { createHash } from 'node:crypto';
|
||||
import { chmod, mkdir, readdir, readFile, rm, stat } from 'node:fs/promises';
|
||||
import { basename, join } from 'node:path';
|
||||
|
||||
import type { SqliteDatabase } from '../../infrastructure/database/database.js';
|
||||
|
||||
export type DataComponentKey =
|
||||
| 'instances'
|
||||
| 'statusSnapshots'
|
||||
| 'jobs'
|
||||
| 'auditEvents'
|
||||
| 'scheduledRuns'
|
||||
| 'notificationQueue'
|
||||
| 'smsMessages'
|
||||
| 'eventJournal'
|
||||
| 'connectionLogs';
|
||||
|
||||
export interface RetentionPolicy {
|
||||
readonly enabled: boolean;
|
||||
readonly days: number;
|
||||
readonly maximumCount: number;
|
||||
}
|
||||
|
||||
export interface MaintenanceOverview {
|
||||
readonly runtime: {
|
||||
readonly version: string;
|
||||
readonly platform: NodeJS.Platform;
|
||||
readonly arch: string;
|
||||
readonly uptimeSeconds: number;
|
||||
};
|
||||
readonly storage: {
|
||||
readonly databaseBytes: number;
|
||||
/** SQLite file path; empty for in-memory databases. */
|
||||
readonly databasePath: string;
|
||||
/** Write-ahead log file size; 0 when the database is not in WAL mode. */
|
||||
readonly walBytes: number;
|
||||
/** Total size of the retained backup files. */
|
||||
readonly backupBytes: number;
|
||||
/** Free pages that "整理空间" can hand back to the filesystem. */
|
||||
readonly reclaimableBytes: number;
|
||||
readonly components: readonly {
|
||||
readonly key: DataComponentKey;
|
||||
readonly count: number;
|
||||
readonly bytes: number;
|
||||
}[];
|
||||
};
|
||||
readonly retention: Readonly<Record<DataComponentKey, RetentionPolicy>>;
|
||||
}
|
||||
|
||||
export interface MaintenanceBackup {
|
||||
readonly filename: string;
|
||||
readonly path: string;
|
||||
readonly sizeBytes: number;
|
||||
readonly sha256: string;
|
||||
readonly createdAt: string;
|
||||
}
|
||||
|
||||
/** Download metadata without the full-file hash so large backups stay cheap to serve. */
|
||||
export interface MaintenanceBackupFile {
|
||||
readonly filename: string;
|
||||
readonly path: string;
|
||||
readonly sizeBytes: number;
|
||||
readonly createdAt: string;
|
||||
}
|
||||
|
||||
interface ServiceOptions {
|
||||
readonly version: string;
|
||||
readonly now?: () => Date;
|
||||
readonly backupDirectory?: string;
|
||||
readonly maximumBackups?: number;
|
||||
}
|
||||
|
||||
const COMPONENT_TABLES: Readonly<Record<DataComponentKey, string>> = Object.freeze({
|
||||
instances: 'instances',
|
||||
statusSnapshots: 'status_snapshots',
|
||||
jobs: 'jobs',
|
||||
auditEvents: 'audit_events',
|
||||
scheduledRuns: 'scheduled_runs',
|
||||
notificationQueue: 'notification_queue',
|
||||
smsMessages: 'sms_messages',
|
||||
eventJournal: 'event_journal',
|
||||
connectionLogs: 'connection_logs',
|
||||
});
|
||||
|
||||
const RETENTION_SETTING_KEY = 'system.retention';
|
||||
const DEFAULT_RETENTION: Readonly<Record<DataComponentKey, RetentionPolicy>> = Object.freeze({
|
||||
instances: Object.freeze({ enabled: false, days: 365, maximumCount: 0 }),
|
||||
statusSnapshots: Object.freeze({ enabled: false, days: 30, maximumCount: 0 }),
|
||||
jobs: Object.freeze({ enabled: true, days: 90, maximumCount: 5_000 }),
|
||||
auditEvents: Object.freeze({ enabled: true, days: 180, maximumCount: 50_000 }),
|
||||
scheduledRuns: Object.freeze({ enabled: true, days: 180, maximumCount: 5_000 }),
|
||||
notificationQueue: Object.freeze({ enabled: true, days: 180, maximumCount: 10_000 }),
|
||||
smsMessages: Object.freeze({ enabled: false, days: 365, maximumCount: 0 }),
|
||||
eventJournal: Object.freeze({ enabled: true, days: 30, maximumCount: 10_000 }),
|
||||
// The probe journal already self-caps, so retention here is opt-in by day window.
|
||||
connectionLogs: Object.freeze({ enabled: true, days: 30, maximumCount: 20_000 }),
|
||||
});
|
||||
|
||||
function cloneRetention(
|
||||
value: Readonly<Record<DataComponentKey, RetentionPolicy>>,
|
||||
): Record<DataComponentKey, RetentionPolicy> {
|
||||
return Object.fromEntries(
|
||||
Object.entries(value).map(([key, policy]) => [key, { ...policy }]),
|
||||
) as Record<DataComponentKey, RetentionPolicy>;
|
||||
}
|
||||
|
||||
function validateRetention(value: unknown): Record<DataComponentKey, RetentionPolicy> {
|
||||
const source =
|
||||
value !== null && typeof value === 'object' && !Array.isArray(value)
|
||||
? (value as Record<string, unknown>)
|
||||
: {};
|
||||
const merged = cloneRetention(DEFAULT_RETENTION);
|
||||
for (const key of Object.keys(DEFAULT_RETENTION) as DataComponentKey[]) {
|
||||
const current = source[key];
|
||||
if (current === undefined) continue;
|
||||
const policy = current as Partial<RetentionPolicy>;
|
||||
if (
|
||||
typeof policy.enabled !== 'boolean' ||
|
||||
!Number.isSafeInteger(policy.days) ||
|
||||
(policy.days ?? 0) < 1 ||
|
||||
(policy.days ?? 0) > 3_650 ||
|
||||
!Number.isSafeInteger(policy.maximumCount) ||
|
||||
(policy.maximumCount ?? 0) < 0
|
||||
)
|
||||
throw new RangeError('Retention days must be between 1 and 3650 and limits must be valid');
|
||||
merged[key] = {
|
||||
enabled: policy.enabled,
|
||||
days: policy.days!,
|
||||
maximumCount: policy.maximumCount!,
|
||||
};
|
||||
}
|
||||
if (Object.keys(source).some((key) => !(key in DEFAULT_RETENTION)))
|
||||
throw new TypeError('Unknown data component');
|
||||
return merged;
|
||||
}
|
||||
|
||||
function tableExists(db: SqliteDatabase, table: string): boolean {
|
||||
return Boolean(
|
||||
db.prepare('SELECT 1 FROM sqlite_master WHERE type = ? AND name = ?').get('table', table),
|
||||
);
|
||||
}
|
||||
|
||||
function countTable(db: SqliteDatabase, table: string): number {
|
||||
if (!tableExists(db, table)) return 0;
|
||||
return Number(
|
||||
(db.prepare(`SELECT COUNT(*) AS count FROM "${table}"`).get() as { count?: number } | undefined)
|
||||
?.count ?? 0,
|
||||
);
|
||||
}
|
||||
|
||||
/** Per-table page usage from the dbstat virtual table; 0 when SQLite was built without it. */
|
||||
function tableBytes(db: SqliteDatabase, table: string): number {
|
||||
if (!tableExists(db, table)) return 0;
|
||||
try {
|
||||
const row = db
|
||||
.prepare('SELECT COALESCE(SUM(pgsize), 0) AS bytes FROM dbstat WHERE name = ?')
|
||||
.get(table) as { bytes?: number } | undefined;
|
||||
return Number(row?.bytes ?? 0);
|
||||
} catch {
|
||||
return 0;
|
||||
}
|
||||
}
|
||||
|
||||
export class SystemMaintenanceService {
|
||||
readonly #db: SqliteDatabase;
|
||||
readonly #version: string;
|
||||
readonly #now: () => Date;
|
||||
readonly #backupDirectory: string;
|
||||
readonly #maximumBackups: number;
|
||||
|
||||
constructor(db: SqliteDatabase, options: ServiceOptions) {
|
||||
this.#db = db;
|
||||
this.#version = options.version;
|
||||
this.#now = options.now ?? (() => new Date());
|
||||
this.#backupDirectory = options.backupDirectory ?? './data/backups';
|
||||
this.#maximumBackups = Math.max(1, options.maximumBackups ?? 10);
|
||||
}
|
||||
|
||||
async overview(): Promise<MaintenanceOverview> {
|
||||
const pageCount = Number(this.#db.pragma('page_count', { simple: true }));
|
||||
const pageSize = Number(this.#db.pragma('page_size', { simple: true }));
|
||||
const databaseFile = String(
|
||||
(this.#db.prepare('PRAGMA database_list').get() as { file?: string } | undefined)?.file ?? '',
|
||||
);
|
||||
const walBytes = databaseFile
|
||||
? await stat(`${databaseFile}-wal`)
|
||||
.then((details) => (details.isFile() ? details.size : 0))
|
||||
.catch(() => 0)
|
||||
: 0;
|
||||
const freelist = Number(this.#db.pragma('freelist_count', { simple: true }));
|
||||
return {
|
||||
runtime: {
|
||||
version: this.#version,
|
||||
platform: process.platform,
|
||||
arch: process.arch,
|
||||
uptimeSeconds: Math.floor(process.uptime()),
|
||||
},
|
||||
storage: {
|
||||
databaseBytes: pageCount * pageSize,
|
||||
databasePath: databaseFile,
|
||||
walBytes,
|
||||
backupBytes: await this.#backupBytes(),
|
||||
reclaimableBytes: freelist * pageSize,
|
||||
components: (Object.keys(COMPONENT_TABLES) as DataComponentKey[]).map((key) => ({
|
||||
key,
|
||||
count: countTable(this.#db, COMPONENT_TABLES[key]),
|
||||
bytes: tableBytes(this.#db, COMPONENT_TABLES[key]),
|
||||
})),
|
||||
},
|
||||
retention: await this.getRetention(),
|
||||
};
|
||||
}
|
||||
|
||||
async #backupBytes(): Promise<number> {
|
||||
const names = await readdir(this.#backupDirectory).catch(() => [] as string[]);
|
||||
let total = 0;
|
||||
for (const name of names.filter((entry) => entry.startsWith('multi-simadmin-'))) {
|
||||
const details = await stat(join(this.#backupDirectory, name)).catch(() => undefined);
|
||||
if (details?.isFile()) total += details.size;
|
||||
}
|
||||
return total;
|
||||
}
|
||||
|
||||
defaultRetention(): Readonly<Record<DataComponentKey, RetentionPolicy>> {
|
||||
return cloneRetention(DEFAULT_RETENTION);
|
||||
}
|
||||
|
||||
async getRetention(): Promise<Readonly<Record<DataComponentKey, RetentionPolicy>>> {
|
||||
const row = this.#db
|
||||
.prepare('SELECT value_json FROM app_settings WHERE key = ?')
|
||||
.get(RETENTION_SETTING_KEY) as { value_json?: string } | undefined;
|
||||
if (!row?.value_json) return this.defaultRetention();
|
||||
return validateRetention(JSON.parse(row.value_json));
|
||||
}
|
||||
|
||||
async updateRetention(
|
||||
value: unknown,
|
||||
): Promise<Readonly<Record<DataComponentKey, RetentionPolicy>>> {
|
||||
const policy = validateRetention(value);
|
||||
const now = this.#now().toISOString();
|
||||
this.#db
|
||||
.prepare(
|
||||
`INSERT INTO app_settings (key,value_json,created_at,updated_at)
|
||||
VALUES (?,?,?,?)
|
||||
ON CONFLICT(key) DO UPDATE SET value_json = excluded.value_json,
|
||||
updated_at = excluded.updated_at`,
|
||||
)
|
||||
.run(RETENTION_SETTING_KEY, JSON.stringify(policy), now, now);
|
||||
return policy;
|
||||
}
|
||||
|
||||
async cleanup(
|
||||
components: readonly DataComponentKey[],
|
||||
): Promise<Partial<Record<DataComponentKey, number>>> {
|
||||
const known = new Set(Object.keys(COMPONENT_TABLES));
|
||||
if (components.some((component) => !known.has(component)))
|
||||
throw new Error('Unknown data component');
|
||||
const retention = await this.getRetention();
|
||||
const result: Partial<Record<DataComponentKey, number>> = {};
|
||||
|
||||
for (const component of components) {
|
||||
const policy = retention[component];
|
||||
const table = COMPONENT_TABLES[component];
|
||||
if (!policy.enabled || !tableExists(this.#db, table)) continue;
|
||||
const cutoff = new Date(this.#now().getTime() - policy.days * 86_400_000).toISOString();
|
||||
if (component === 'auditEvents')
|
||||
result[component] = this.#delete(`DELETE FROM ${table} WHERE created_at < ?`, cutoff);
|
||||
else if (component === 'jobs')
|
||||
result[component] = this.#delete(
|
||||
`DELETE FROM ${table} WHERE created_at < ? AND status NOT IN ('running','pending')`,
|
||||
cutoff,
|
||||
);
|
||||
else if (component === 'scheduledRuns')
|
||||
result[component] = this.#delete(`DELETE FROM ${table} WHERE due_at < ?`, cutoff);
|
||||
else if (component === 'eventJournal')
|
||||
result[component] = this.#delete(`DELETE FROM ${table}`);
|
||||
else if (component === 'connectionLogs')
|
||||
result[component] = this.#delete(`DELETE FROM ${table} WHERE observed_at < ?`, cutoff);
|
||||
else if (policy.maximumCount > 0)
|
||||
result[component] = this.#delete(
|
||||
`DELETE FROM ${table} WHERE rowid IN (
|
||||
SELECT rowid FROM ${table} LIMIT ?
|
||||
)`,
|
||||
Math.max(0, countTable(this.#db, table) - policy.maximumCount),
|
||||
);
|
||||
}
|
||||
return result;
|
||||
}
|
||||
|
||||
async optimize(): Promise<{ checkpointed: boolean; vacuumed: boolean; analyzed: boolean }> {
|
||||
this.#db.pragma('wal_checkpoint(TRUNCATE)');
|
||||
this.#db.exec('VACUUM');
|
||||
this.#db.exec('ANALYZE');
|
||||
return { checkpointed: true, vacuumed: true, analyzed: true };
|
||||
}
|
||||
|
||||
async createBackup(): Promise<MaintenanceBackup> {
|
||||
await mkdir(this.#backupDirectory, { recursive: true });
|
||||
const filename = `multi-simadmin-${this.#now().toISOString().replaceAll(/[:.]/gu, '-')}.db`;
|
||||
const path = join(this.#backupDirectory, filename);
|
||||
await this.#db.backup(path);
|
||||
await chmod(path, 0o600);
|
||||
const details = await stat(path);
|
||||
await this.#pruneBackups();
|
||||
return {
|
||||
filename,
|
||||
path,
|
||||
sizeBytes: details.size,
|
||||
sha256: await readFile(path).then((content) =>
|
||||
createHash('sha256').update(content).digest('hex'),
|
||||
),
|
||||
createdAt: this.#now().toISOString(),
|
||||
};
|
||||
}
|
||||
|
||||
async listBackups(): Promise<readonly MaintenanceBackup[]> {
|
||||
await mkdir(this.#backupDirectory, { recursive: true });
|
||||
const names = (await readdir(this.#backupDirectory))
|
||||
.filter((name) => name.startsWith('multi-simadmin-') && name.endsWith('.db'))
|
||||
.sort()
|
||||
.reverse();
|
||||
const backups: MaintenanceBackup[] = [];
|
||||
for (const filename of names.slice(0, this.#maximumBackups)) {
|
||||
const path = join(this.#backupDirectory, filename);
|
||||
const details = await stat(path);
|
||||
if (!details.isFile()) continue;
|
||||
backups.push({
|
||||
filename,
|
||||
path,
|
||||
sizeBytes: details.size,
|
||||
sha256: createHash('sha256')
|
||||
.update(await readFile(path))
|
||||
.digest('hex'),
|
||||
createdAt: details.mtime.toISOString(),
|
||||
});
|
||||
}
|
||||
return backups;
|
||||
}
|
||||
|
||||
/** Resolves a listed backup name to a path that stays inside the backup directory. */
|
||||
#backupPath(filename: string): string {
|
||||
if (
|
||||
!/^multi-simadmin-[A-Za-z0-9._-]{1,120}\.db$/u.test(filename) ||
|
||||
basename(filename) !== filename
|
||||
)
|
||||
throw new TypeError('Backup filename is invalid');
|
||||
return join(this.#backupDirectory, filename);
|
||||
}
|
||||
|
||||
async backupFile(filename: string): Promise<MaintenanceBackupFile | undefined> {
|
||||
const path = this.#backupPath(filename);
|
||||
const details = await stat(path).catch((error: unknown) => {
|
||||
if ((error as NodeJS.ErrnoException).code === 'ENOENT') return undefined;
|
||||
throw error;
|
||||
});
|
||||
if (!details?.isFile()) return undefined;
|
||||
return {
|
||||
filename,
|
||||
path,
|
||||
sizeBytes: details.size,
|
||||
createdAt: details.mtime.toISOString(),
|
||||
};
|
||||
}
|
||||
|
||||
async deleteBackup(filename: string): Promise<{ filename: string }> {
|
||||
await rm(this.#backupPath(filename), { force: true });
|
||||
return { filename };
|
||||
}
|
||||
|
||||
/** Names sort newest-first because they embed an ISO timestamp. */
|
||||
async #pruneBackups(): Promise<number> {
|
||||
const names = (await readdir(this.#backupDirectory))
|
||||
.filter((name) => name.startsWith('multi-simadmin-') && name.endsWith('.db'))
|
||||
.sort()
|
||||
.reverse();
|
||||
let removed = 0;
|
||||
for (const filename of names.slice(this.#maximumBackups)) {
|
||||
await rm(join(this.#backupDirectory, filename), { force: true });
|
||||
removed += 1;
|
||||
}
|
||||
return removed;
|
||||
}
|
||||
|
||||
#delete(statement: string, ...parameters: readonly unknown[]): number {
|
||||
return Number(this.#db.prepare(statement).run(...parameters).changes);
|
||||
}
|
||||
}
|
||||
@@ -75,7 +75,11 @@ async function rawSocketRequest(origin: string, request: string): Promise<string
|
||||
}
|
||||
|
||||
async function startGateway(distDir: string, upstreamPort: number): Promise<CanaryGateway> {
|
||||
const gateway = createCanaryGateway({ distDir, port: 0, upstreamPort });
|
||||
const gateway = createCanaryGateway({
|
||||
distDir,
|
||||
port: 0,
|
||||
upstreamPort,
|
||||
});
|
||||
gateways.push(gateway);
|
||||
await gateway.start();
|
||||
return gateway;
|
||||
@@ -334,6 +338,31 @@ describe('same-origin canary gateway', () => {
|
||||
await reader?.cancel();
|
||||
});
|
||||
|
||||
it('flushes SSE response headers before the first upstream event', async () => {
|
||||
const upstream = createServer((_request, response) => {
|
||||
response.writeHead(200, {
|
||||
'content-type': 'text/event-stream',
|
||||
'cache-control': 'no-cache',
|
||||
});
|
||||
response.flushHeaders();
|
||||
});
|
||||
servers.push(upstream);
|
||||
const gateway = await startGateway(await fixtureDist(), await listen(upstream));
|
||||
|
||||
const controller = new AbortController();
|
||||
const timeout = setTimeout(() => controller.abort(), 500);
|
||||
try {
|
||||
const response = await fetch(`${gateway.origin}/api/v1/events`, {
|
||||
signal: controller.signal,
|
||||
});
|
||||
expect(response.status).toBe(200);
|
||||
expect(response.headers.get('content-type')).toBe('text/event-stream');
|
||||
await response.body?.cancel();
|
||||
} finally {
|
||||
clearTimeout(timeout);
|
||||
}
|
||||
});
|
||||
|
||||
it('has reversible, idempotent start/stop lifecycle', async () => {
|
||||
const gateway = createCanaryGateway({ distDir: await fixtureDist(), port: 0, upstreamPort: 1 });
|
||||
gateways.push(gateway);
|
||||
|
||||
@@ -107,6 +107,9 @@ function trustedUpstreamHeaders(
|
||||
delete sanitized[GATEWAY_CLIENT_IP_HEADER];
|
||||
if (gatewayToken !== undefined) sanitized[GATEWAY_AUTH_HEADER] = gatewayToken;
|
||||
if (clientIp !== undefined) sanitized[GATEWAY_CLIENT_IP_HEADER] = clientIp;
|
||||
// The browser connection's protocol is the gateway's to declare; the API uses
|
||||
// it to decide whether session cookies may carry the Secure flag.
|
||||
sanitized['x-forwarded-proto'] = (headers['x-forwarded-proto'] as string | undefined) ?? 'http';
|
||||
return sanitized;
|
||||
}
|
||||
|
||||
@@ -177,6 +180,7 @@ async function proxyRequest(
|
||||
upstreamResponse.statusMessage,
|
||||
withoutHopByHop(upstreamResponse.headers),
|
||||
);
|
||||
response.flushHeaders();
|
||||
pipeline(upstreamResponse, response)
|
||||
.then(resolvePromise)
|
||||
.catch(() => {
|
||||
|
||||
@@ -91,6 +91,31 @@ async function expectPortFree(port: number): Promise<void> {
|
||||
await closeServer(server);
|
||||
}
|
||||
|
||||
async function isPortFree(port: number): Promise<boolean> {
|
||||
try {
|
||||
const server = await listenOn(port);
|
||||
await closeServer(server);
|
||||
return true;
|
||||
} catch (error) {
|
||||
if ((error as NodeJS.ErrnoException).code === 'EADDRINUSE') return false;
|
||||
throw error;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Reserves a port the OS reports as free. The canary gateway accepts a configured
|
||||
* port, so its test does not need to fight a live deployment for 8789.
|
||||
*/
|
||||
async function reserveFreePort(): Promise<number> {
|
||||
const server = await listenOn(0);
|
||||
const address = server.address();
|
||||
if (address === null || typeof address === 'string')
|
||||
throw new Error('Could not reserve a free port');
|
||||
const port = address.port;
|
||||
await closeServer(server);
|
||||
return port;
|
||||
}
|
||||
|
||||
async function listenerPids(port: number): Promise<readonly string[]> {
|
||||
try {
|
||||
const { stdout } = await execFileAsync('lsof', ['-nP', `-iTCP:${port}`, '-sTCP:LISTEN', '-t']);
|
||||
@@ -106,7 +131,9 @@ afterEach(async () => {
|
||||
await Promise.all(cleanup.splice(0).map((fn) => fn()));
|
||||
});
|
||||
|
||||
describe.sequential('executable package runtimes', () => {
|
||||
// The runtime lifecycle under test is macOS/POSIX-only: process-group SIGTERM,
|
||||
// lsof port inspection, and corepack as a direct spawn target.
|
||||
describe.sequential('executable package runtimes', { skip: process.platform === 'win32' }, () => {
|
||||
it('makes the exact production gateway executable attempt 8788 only with acknowledgement', async () => {
|
||||
await expectPortFree(8788);
|
||||
const fixture = await listenOn(8788);
|
||||
@@ -134,7 +161,10 @@ describe.sequential('executable package runtimes', () => {
|
||||
expect(acknowledged.child.exitCode).not.toBe(0);
|
||||
}, 20_000);
|
||||
|
||||
it('starts the exact production command on 8790 and shuts down on SIGTERM', async () => {
|
||||
// The production executable refuses any port other than 8790 on purpose, so this
|
||||
// check only runs when a live deployment is not already holding that port.
|
||||
it('starts the exact production command on 8790 and shuts down on SIGTERM', async (context) => {
|
||||
if (!(await isPortFree(8790))) return context.skip();
|
||||
await expectPortFree(8790);
|
||||
const legacyListeners = await listenerPids(8788);
|
||||
const directory = await mkdtemp(join(tmpdir(), 'multi-simadmin-production-cli-'));
|
||||
@@ -166,8 +196,9 @@ describe.sequential('executable package runtimes', () => {
|
||||
expect(command.output()).not.toContain(gatewayToken);
|
||||
}, 20_000);
|
||||
|
||||
it('starts the exact canary command on 8789 with synthetic dist and shuts down on SIGTERM', async () => {
|
||||
const port = 8789;
|
||||
it('starts the exact canary command with synthetic dist and shuts down on SIGTERM', async () => {
|
||||
const port = await reserveFreePort();
|
||||
const upstreamPort = await reserveFreePort();
|
||||
await expectPortFree(port);
|
||||
const legacyListeners = await listenerPids(8788);
|
||||
const directory = await mkdtemp(join(tmpdir(), 'multi-simadmin-canary-cli-'));
|
||||
@@ -177,7 +208,7 @@ describe.sequential('executable package runtimes', () => {
|
||||
const command = startPackageCommand('canary', {
|
||||
CANARY_DIST_DIR: directory,
|
||||
CANARY_PORT: String(port),
|
||||
CANARY_UPSTREAM_PORT: '8790',
|
||||
CANARY_UPSTREAM_PORT: String(upstreamPort),
|
||||
MULTI_SIMADMIN_GATEWAY_TOKEN: gatewayToken,
|
||||
});
|
||||
await waitForOutput(command, `Canary gateway listening at http://127.0.0.1:${port}`);
|
||||
|
||||
@@ -9,7 +9,12 @@ it('registers the read-only operations catalog without touching upstream', async
|
||||
let upstreamCalls = 0;
|
||||
const app = buildControlPlaneApp({
|
||||
db,
|
||||
store: { set: async () => '', get: async () => undefined, delete: async () => false },
|
||||
store: {
|
||||
provider: 'macos-keychain',
|
||||
set: async () => '',
|
||||
get: async () => undefined,
|
||||
delete: async () => false,
|
||||
},
|
||||
upstream: {
|
||||
get: async () => {
|
||||
upstreamCalls += 1;
|
||||
@@ -27,6 +32,10 @@ it('registers the read-only operations catalog without touching upstream', async
|
||||
upstreamCalls += 1;
|
||||
throw new Error('unexpected');
|
||||
},
|
||||
postBasebandRestart: async () => {
|
||||
upstreamCalls += 1;
|
||||
throw new Error('unexpected');
|
||||
},
|
||||
postSystemReboot: async () => {
|
||||
upstreamCalls += 1;
|
||||
throw new Error('unexpected');
|
||||
|
||||
+1258
-11
File diff suppressed because it is too large
Load Diff
+524
-16
@@ -12,6 +12,13 @@ import {
|
||||
ConnectionProbe,
|
||||
type ConnectionTransport,
|
||||
} from './application/connections/connection-probe.js';
|
||||
import { ConnectionSettingsService } from './application/connections/connection-settings-service.js';
|
||||
import {
|
||||
FLEET_STATE_CODES,
|
||||
FLEET_STATE_EVENTS,
|
||||
FLEET_STATE_NAMES,
|
||||
FleetHeartbeatCoordinator,
|
||||
} from './application/connections/fleet-heartbeat.js';
|
||||
import { InstanceCredentialResolver } from './application/connections/instance-credential-resolver.js';
|
||||
import { InstanceLoginService } from './application/connections/instance-login-service.js';
|
||||
import {
|
||||
@@ -20,11 +27,17 @@ import {
|
||||
type UpstreamSessionClientOptions,
|
||||
} from './application/connections/upstream-session-client.js';
|
||||
import { InstanceService } from './application/instances/instance-service.js';
|
||||
import { DeleteInstanceOperation } from './application/operations/delete-instance-operation.js';
|
||||
import { DeviceIdentityService } from './application/identity/device-identity-service.js';
|
||||
import { DeviceDiscoveryService } from './application/instances/device-discovery-service.js';
|
||||
import {
|
||||
type BindingRelease,
|
||||
DeleteInstanceOperation,
|
||||
} from './application/operations/delete-instance-operation.js';
|
||||
import type { SecretStore } from './infrastructure/secrets/secret-store.js';
|
||||
import { registerInstanceRoutes } from './interface/http/instance-routes.js';
|
||||
import { registerEventRoutes } from './interface/http/event-routes.js';
|
||||
import { JobQueryService } from './application/jobs/job-query-service.js';
|
||||
import { JobReconcileService } from './application/jobs/job-reconcile-service.js';
|
||||
import { registerJobRoutes } from './interface/http/job-routes.js';
|
||||
import { AuditQueryService } from './application/audit/audit-query-service.js';
|
||||
import { registerAuditRoutes } from './interface/http/audit-routes.js';
|
||||
@@ -32,12 +45,51 @@ import { InstanceResourceService } from './application/resources/instance-resour
|
||||
import { ConsoleAuthService } from './application/auth/console-auth-service.js';
|
||||
import { registerConsoleAuth } from './interface/http/console-auth-routes.js';
|
||||
import { InstanceMessageService } from './application/messages/instance-message-service.js';
|
||||
import { HubMessageService } from './application/messages/hub-message-service.js';
|
||||
import { SmsOutboxService } from './application/messages/sms-outbox-service.js';
|
||||
import { ScheduledTaskRepository } from './application/automation/scheduled-task-repository.js';
|
||||
import { ScheduledTaskService } from './application/automation/scheduled-task-service.js';
|
||||
import { registerAutomationRoutes } from './interface/http/automation-routes.js';
|
||||
import { registerFleetRoutes } from './interface/http/fleet-routes.js';
|
||||
import { SystemMaintenanceService } from './application/system/system-maintenance-service.js';
|
||||
import { ComponentBackupService } from './application/system/component-backup-service.js';
|
||||
import { registerSystemRoutes } from './interface/http/system-routes.js';
|
||||
import {
|
||||
ConsoleUpdateService,
|
||||
createCommandRunner,
|
||||
type RestartLauncher,
|
||||
type UpdateCommandRunner,
|
||||
} from './application/system/console-update-service.js';
|
||||
import { registerUpdateRoutes } from './interface/http/update-routes.js';
|
||||
import { InstanceNotificationService } from './application/notifications/instance-notification-service.js';
|
||||
import { CentralNotificationService } from './application/notifications/central-notification-service.js';
|
||||
import { registerCentralNotificationRoutes } from './interface/http/central-notification-routes.js';
|
||||
import { DeviceOrganizationService } from './application/organization/device-organization-service.js';
|
||||
import { registerOrganizationRoutes } from './interface/http/organization-routes.js';
|
||||
import { ConnectionLogService } from './application/system/connection-log-service.js';
|
||||
import { LogCenterService } from './application/system/log-center-service.js';
|
||||
import { registerLogCenterRoutes } from './interface/http/log-center-routes.js';
|
||||
import { InstanceModuleService } from './application/instances/instance-module-service.js';
|
||||
import { registerInstanceModuleRoutes } from './interface/http/instance-module-routes.js';
|
||||
import { DeviceActionService } from './application/instances/device-action-service.js';
|
||||
import { registerDeviceActionRoutes } from './interface/http/device-action-routes.js';
|
||||
import { registerIdentityRoutes } from './interface/http/identity-routes.js';
|
||||
import { MetricsService } from './application/observability/metrics-service.js';
|
||||
import { registerMetricsRoutes } from './interface/http/metrics-routes.js';
|
||||
import { registerDiscoveryRoutes } from './interface/http/discovery-routes.js';
|
||||
import { ScheduledOperationDispatcher } from './application/automation/scheduled-operation-dispatcher.js';
|
||||
import { SchedulerCoordinator } from './application/automation/scheduler-coordinator.js';
|
||||
|
||||
export interface SafeControlPlaneUpstream extends ConnectionTransport {
|
||||
request: UpstreamSessionClientOptions['request'];
|
||||
postNetworkRegisterAuto(origin: string): Promise<{ readonly status: number }>;
|
||||
postServiceRestart(origin: string): Promise<{ readonly status: number }>;
|
||||
postSystemReboot(origin: string, delaySeconds: number): Promise<{ readonly status: number }>;
|
||||
postNetworkRegisterAuto(origin: string, cookie?: string): Promise<{ readonly status: number }>;
|
||||
postServiceRestart(origin: string, cookie?: string): Promise<{ readonly status: number }>;
|
||||
postBasebandRestart(origin: string, cookie?: string): Promise<{ readonly status: number }>;
|
||||
postSystemReboot(
|
||||
origin: string,
|
||||
delaySeconds: number,
|
||||
cookie?: string,
|
||||
): Promise<{ readonly status: number }>;
|
||||
}
|
||||
export interface ControlPlaneOptions {
|
||||
readonly db: Database.Database;
|
||||
@@ -45,63 +97,323 @@ export interface ControlPlaneOptions {
|
||||
readonly upstream: SafeControlPlaneUpstream;
|
||||
readonly now?: () => Date;
|
||||
readonly authenticateEventStream?: (request: FastifyRequest) => boolean;
|
||||
readonly runtimeVersion?: string;
|
||||
readonly backupDirectory?: string;
|
||||
readonly smsSyncIntervalMs?: number;
|
||||
readonly queueDrainIntervalMs?: number;
|
||||
readonly logPruneIntervalMs?: number;
|
||||
readonly autoBackupIntervalMs?: number;
|
||||
/** Keep device online state fresh on its own; off in tests, on for the production gateway. */
|
||||
readonly heartbeatEnabled?: boolean;
|
||||
readonly app?: Omit<BuildAppOptions, 'registerRoutes'>;
|
||||
/** Online update of the console itself; defaults to a git checkout in the current directory. */
|
||||
readonly update?: {
|
||||
readonly remote?: string;
|
||||
readonly root?: string;
|
||||
readonly runner?: UpdateCommandRunner;
|
||||
readonly launcher?: RestartLauncher;
|
||||
/** Overrides the safety snapshot taken before the checkout moves. */
|
||||
readonly preInstallBackup?: (targetCommit: string) => Promise<string>;
|
||||
};
|
||||
}
|
||||
|
||||
const DEFAULT_SMS_SYNC_INTERVAL_MS = 300_000;
|
||||
const DEFAULT_QUEUE_DRAIN_INTERVAL_MS = 15_000;
|
||||
const DEFAULT_LOG_PRUNE_INTERVAL_MS = 900_000;
|
||||
const DEFAULT_AUTO_BACKUP_INTERVAL_MS = 300_000;
|
||||
|
||||
interface HubDeviceSummary {
|
||||
readonly id: string;
|
||||
readonly name: string;
|
||||
readonly state: 'ready' | 'unavailable' | 'unknown';
|
||||
readonly tags: readonly string[];
|
||||
readonly groupId: string | null;
|
||||
}
|
||||
|
||||
// Device reachability comes from the last ConnectionProbe result, never from a live probe:
|
||||
// the notification overview must stay cheap enough to poll.
|
||||
function createHubDeviceReader(
|
||||
db: Database.Database,
|
||||
instances: InstanceService,
|
||||
): () => Promise<readonly HubDeviceSummary[]> {
|
||||
const readConnections = db.prepare(
|
||||
`SELECT instance_id, state FROM status_snapshots
|
||||
WHERE category = 'connection' AND expires_at > ?`,
|
||||
);
|
||||
return async () => {
|
||||
const now = new Date().toISOString();
|
||||
const reachable = new Set<string>(
|
||||
(readConnections.all(now) as readonly { instance_id: string; state: string }[])
|
||||
.filter((row) => row.state === 'fresh' || row.state === 'stale')
|
||||
.map((row) => row.instance_id),
|
||||
);
|
||||
const devices: HubDeviceSummary[] = [];
|
||||
let page = 1;
|
||||
while (devices.length < 200) {
|
||||
const current = await instances.list({ page, pageSize: 100 });
|
||||
for (const instance of current.items.slice(0, 200 - devices.length)) {
|
||||
devices.push({
|
||||
id: instance.id,
|
||||
name: instance.name,
|
||||
state: reachable.has(instance.id) ? 'ready' : 'unknown',
|
||||
tags: [...instance.tags],
|
||||
groupId: instance.groupId,
|
||||
});
|
||||
}
|
||||
if (current.items.length < 100) break;
|
||||
page += 1;
|
||||
}
|
||||
return devices;
|
||||
};
|
||||
}
|
||||
|
||||
// Retention windows for operator-facing history. Preparations are short-lived
|
||||
// protocol state and cycle faster; audit records are the longest-lived.
|
||||
const RETENTION_HISTORY_MS = 30 * 24 * 60 * 60 * 1000;
|
||||
const RETENTION_AUDIT_MS = 90 * 24 * 60 * 60 * 1000;
|
||||
const RETENTION_PREPARATION_MS = 7 * 24 * 60 * 60 * 1000;
|
||||
|
||||
export interface ControlPlaneApp extends FastifyInstance {
|
||||
retryPendingSecretCleanup(): Promise<unknown>;
|
||||
}
|
||||
export function buildControlPlaneApp(options: ControlPlaneOptions): ControlPlaneApp {
|
||||
const eventJournal = new EventJournal(options.db);
|
||||
const jobs = new JobQueryService(options.db);
|
||||
const jobReconcile = new JobReconcileService({
|
||||
db: options.db,
|
||||
...(options.now ? { now: options.now } : {}),
|
||||
});
|
||||
// Nothing can be mid-flight before the first request is served, so a startup sweep closes every
|
||||
// job row a previous process left behind.
|
||||
jobReconcile.reconcile(0);
|
||||
const audit = new AuditQueryService(options.db);
|
||||
const scheduledTasks = new ScheduledTaskRepository(options.db);
|
||||
scheduledTasks.reconcileInterruptedRuns((options.now?.() ?? new Date()).toISOString());
|
||||
const automation = new ScheduledTaskService({
|
||||
repository: scheduledTasks,
|
||||
store: options.store,
|
||||
...(options.now ? { now: options.now } : {}),
|
||||
});
|
||||
const instances = options.now
|
||||
? new InstanceService({ db: options.db, store: options.store, now: options.now })
|
||||
: new InstanceService({ db: options.db, store: options.store });
|
||||
const connectionLogs = new ConnectionLogService({ db: options.db });
|
||||
const connectionSettings = options.now
|
||||
? new ConnectionSettingsService({ db: options.db, now: options.now })
|
||||
: new ConnectionSettingsService({ db: options.db });
|
||||
const connections = options.now
|
||||
? new ConnectionProbe({
|
||||
db: options.db,
|
||||
instances,
|
||||
transport: options.upstream,
|
||||
now: options.now,
|
||||
connectionLogs,
|
||||
snapshotTtlMs: () => connectionSettings.snapshotTtlMs,
|
||||
})
|
||||
: new ConnectionProbe({ db: options.db, instances, transport: options.upstream });
|
||||
: new ConnectionProbe({
|
||||
db: options.db,
|
||||
instances,
|
||||
transport: options.upstream,
|
||||
connectionLogs,
|
||||
snapshotTtlMs: () => connectionSettings.snapshotTtlMs,
|
||||
});
|
||||
const sessions = new InstanceSessionStore();
|
||||
const discovery = options.now
|
||||
? new DeviceDiscoveryService({ transport: options.upstream, instances, now: options.now })
|
||||
: new DeviceDiscoveryService({ transport: options.upstream, instances });
|
||||
const client = new UpstreamSessionClient({ sessions, request: options.upstream.request });
|
||||
const resolver = new InstanceCredentialResolver({ db: options.db, store: options.store });
|
||||
const login = options.now
|
||||
? new InstanceLoginService({ db: options.db, client, resolver, now: options.now })
|
||||
: new InstanceLoginService({ db: options.db, client, resolver });
|
||||
const pendingLogins = new Map<string, Promise<void>>();
|
||||
const ensureSession = async (
|
||||
instanceId: string,
|
||||
origin: string,
|
||||
force = false,
|
||||
): Promise<void> => {
|
||||
const existing = sessions.sessionFor(instanceId);
|
||||
if (!force && existing?.origin === origin) return;
|
||||
if (existing) sessions.clear(instanceId);
|
||||
const current = pendingLogins.get(instanceId);
|
||||
if (current) return current;
|
||||
const pending = login
|
||||
.login(instanceId)
|
||||
.then((result) => {
|
||||
if (!result.authenticated) throw new Error('INSTANCE_AUTHENTICATION_FAILED');
|
||||
})
|
||||
.finally(() => pendingLogins.delete(instanceId));
|
||||
pendingLogins.set(instanceId, pending);
|
||||
return pending;
|
||||
};
|
||||
const resources = new InstanceResourceService({
|
||||
instances,
|
||||
sessions,
|
||||
request: options.upstream.request,
|
||||
ensureSession,
|
||||
db: options.db,
|
||||
});
|
||||
const messages = new InstanceMessageService({
|
||||
instances,
|
||||
sessions,
|
||||
request: options.upstream.request,
|
||||
ensureSession,
|
||||
});
|
||||
const resolver = new InstanceCredentialResolver({ db: options.db, store: options.store });
|
||||
const login = options.now
|
||||
? new InstanceLoginService({ db: options.db, client, resolver, now: options.now })
|
||||
: new InstanceLoginService({ db: options.db, client, resolver });
|
||||
const notifications = new InstanceNotificationService({
|
||||
instances,
|
||||
sessions,
|
||||
request: options.upstream.request,
|
||||
ensureSession,
|
||||
});
|
||||
const centralNotifications = new CentralNotificationService(options.db, {
|
||||
store: options.store,
|
||||
...(options.now ? { now: options.now } : {}),
|
||||
});
|
||||
// The heartbeat owns the reachability loop, so it also owns telling the rule engine when a
|
||||
// node went offline, came back, or lost its device session.
|
||||
const heartbeat = new FleetHeartbeatCoordinator({
|
||||
instances,
|
||||
probe: connections,
|
||||
settings: connectionSettings,
|
||||
...(options.now ? { now: options.now } : {}),
|
||||
onTransition: async (transitions) => {
|
||||
for (const transition of transitions) {
|
||||
const instance = await instances.get(transition.instanceId);
|
||||
const label = instance?.name ?? transition.instanceId;
|
||||
await centralNotifications.enqueueEvent('system', {
|
||||
instanceId: transition.instanceId,
|
||||
instanceTags: instance?.tags ?? [],
|
||||
instanceGroupIds: instance?.groupId ? [instance.groupId] : [],
|
||||
fields: {
|
||||
title: `${label} ${FLEET_STATE_EVENTS[transition.to]}`,
|
||||
status: FLEET_STATE_CODES[transition.to],
|
||||
content: `节点 ${label} 从「${FLEET_STATE_NAMES[transition.from]}」变为「${
|
||||
FLEET_STATE_NAMES[transition.to]
|
||||
}」(${FLEET_STATE_EVENTS[transition.to]})。`,
|
||||
},
|
||||
});
|
||||
}
|
||||
},
|
||||
});
|
||||
const hubMessages = new HubMessageService(options.db, {
|
||||
instances,
|
||||
messages,
|
||||
...(options.now ? { now: options.now } : {}),
|
||||
onIncomingMessage: async (instanceId, message) => {
|
||||
const instance = await instances.get(instanceId);
|
||||
await centralNotifications.enqueueEvent('sms', {
|
||||
instanceId,
|
||||
instanceTags: instance?.tags ?? [],
|
||||
instanceGroupIds: instance?.groupId ? [instance.groupId] : [],
|
||||
fields: {
|
||||
sender: message.phoneNumber,
|
||||
content: message.content,
|
||||
title: `来自 ${message.phoneNumber} 的新短信`,
|
||||
status: message.status,
|
||||
},
|
||||
});
|
||||
},
|
||||
});
|
||||
// Offline sends wait here instead of failing, exactly like the Hub hands a message to a device
|
||||
// that has just come back on the LAN.
|
||||
const smsOutbox = new SmsOutboxService(options.db, {
|
||||
send: (instanceId, input) => messages.send(instanceId, input),
|
||||
offlineInstances: () => {
|
||||
const offline = new Set<string>();
|
||||
for (const [instanceId, state] of connections.reachability())
|
||||
if (!state.reachable) offline.add(instanceId);
|
||||
return offline;
|
||||
},
|
||||
...(options.now ? { now: options.now } : {}),
|
||||
});
|
||||
smsOutbox.reconcileInterrupted();
|
||||
centralNotifications.reconcileInterrupted();
|
||||
// Removal follows the hub rule the operator already knows: an online device is told to release
|
||||
// its own binding first and a refusal keeps the record, while a device that cannot be reached
|
||||
// at all is forgotten locally. The closure is late-bound because the device action service is
|
||||
// built further down, next to the session store it needs.
|
||||
const releaseBinding = async (instanceId: string, requestId: string): Promise<BindingRelease> => {
|
||||
try {
|
||||
await connections.test(instanceId);
|
||||
} catch {
|
||||
return { status: 'skipped' };
|
||||
}
|
||||
const result = await deviceActions.execute(
|
||||
instanceId,
|
||||
'hub.unbind',
|
||||
{},
|
||||
{ actor: 'loopback-control-plane', requestId, confirm: true },
|
||||
);
|
||||
// A device too old to expose the endpoint never had a binding to release.
|
||||
if (result.status === 404 || result.status === 405) return { status: 'released' };
|
||||
return result.ok ? { status: 'released' } : { status: 'failed' };
|
||||
};
|
||||
const deletion = options.now
|
||||
? new DeleteInstanceOperation({ db: options.db, instances, now: options.now })
|
||||
: new DeleteInstanceOperation({ db: options.db, instances });
|
||||
? new DeleteInstanceOperation({
|
||||
db: options.db,
|
||||
instances,
|
||||
releaseBinding,
|
||||
now: options.now,
|
||||
emit: (envelope) => eventJournal.append(envelope),
|
||||
})
|
||||
: new DeleteInstanceOperation({
|
||||
db: options.db,
|
||||
instances,
|
||||
releaseBinding,
|
||||
emit: (envelope) => eventJournal.append(envelope),
|
||||
});
|
||||
deletion.reconcileInterruptedJobs();
|
||||
const listHubDevices = createHubDeviceReader(options.db, instances);
|
||||
const resolveOperationCookie = async (
|
||||
instanceId: string,
|
||||
origin: string,
|
||||
): Promise<string | undefined> => {
|
||||
// Attach only an origin-bound in-memory session. Never invent cookies, never log them.
|
||||
const existing = sessions.sessionFor(instanceId);
|
||||
if (
|
||||
existing &&
|
||||
existing.origin === origin &&
|
||||
/^simadmin_session=[^;\s,]+$/u.test(existing.cookie)
|
||||
)
|
||||
return existing.cookie;
|
||||
// Password-protected instances need a session for reboot/restart. Best-effort Keychain login
|
||||
// keeps the control plane from silently dispatching unauthenticated upstream mutations.
|
||||
try {
|
||||
const result = await login.login(instanceId);
|
||||
if (!result.authenticated) return undefined;
|
||||
} catch {
|
||||
return undefined;
|
||||
}
|
||||
const refreshed = sessions.sessionFor(instanceId);
|
||||
return refreshed &&
|
||||
refreshed.origin === origin &&
|
||||
/^simadmin_session=[^;\s,]+$/u.test(refreshed.cookie)
|
||||
? refreshed.cookie
|
||||
: undefined;
|
||||
};
|
||||
const secureExecution = new SecureOperationExecution({
|
||||
db: options.db,
|
||||
registry: secureOperationRegistry,
|
||||
emit: (envelope) => eventJournal.append(envelope),
|
||||
transport: {
|
||||
request: async ({ origin, path, body, contentType }) => {
|
||||
request: async ({ origin, path, body, contentType, instanceId }) => {
|
||||
const cookie = await resolveOperationCookie(instanceId, origin);
|
||||
if (path === '/api/network/register-auto') {
|
||||
const response = await options.upstream.postNetworkRegisterAuto(origin);
|
||||
const response = await options.upstream.postNetworkRegisterAuto(origin, cookie);
|
||||
return { status: response.status };
|
||||
}
|
||||
if (path === '/api/service/restart') {
|
||||
const response = await options.upstream.postServiceRestart(origin);
|
||||
const response = await options.upstream.postServiceRestart(origin, cookie);
|
||||
return { status: response.status };
|
||||
}
|
||||
if (path === '/api/baseband/restart') {
|
||||
const response = await options.upstream.postBasebandRestart(origin, cookie);
|
||||
return { status: response.status };
|
||||
}
|
||||
if (path === '/api/system/reboot') {
|
||||
if (contentType !== 'application/json' || body !== JSON.stringify({ delay_seconds: 3 }))
|
||||
throw new Error('UPSTREAM_REQUEST_INVALID');
|
||||
const response = await options.upstream.postSystemReboot(origin, 3);
|
||||
const response = await options.upstream.postSystemReboot(origin, 3, cookie);
|
||||
return { status: response.status };
|
||||
}
|
||||
throw new Error('UPSTREAM_REQUEST_INVALID');
|
||||
@@ -110,10 +422,96 @@ export function buildControlPlaneApp(options: ControlPlaneOptions): ControlPlane
|
||||
...(options.now ? { now: options.now } : {}),
|
||||
});
|
||||
secureExecution.reconcileInterruptedJobs();
|
||||
const maintenance = new SystemMaintenanceService(options.db, {
|
||||
version: options.runtimeVersion ?? '0.1.0',
|
||||
...(options.now ? { now: options.now } : {}),
|
||||
...(options.backupDirectory ? { backupDirectory: options.backupDirectory } : {}),
|
||||
});
|
||||
const componentBackups = new ComponentBackupService(options.db, {
|
||||
version: options.runtimeVersion ?? '0.1.0',
|
||||
backupDirectory: options.backupDirectory ?? './data/backups',
|
||||
...(options.now ? { now: options.now } : {}),
|
||||
});
|
||||
// The console updates the same way the Hub does: a git remote is consulted, the candidate is
|
||||
// fetched and verified, and only then does the supervisor get asked for a restart.
|
||||
const updateRunner =
|
||||
options.update?.runner ??
|
||||
createCommandRunner({ cwd: options.update?.root ?? process.cwd(), timeoutMs: 120_000 });
|
||||
const updates = new ConsoleUpdateService({
|
||||
db: options.db,
|
||||
version: options.runtimeVersion ?? '0.1.0',
|
||||
runner: updateRunner,
|
||||
launcher: options.update?.launcher ?? { supported: false, async restart() {} },
|
||||
...(options.update?.remote ? { remote: options.update.remote } : {}),
|
||||
...(options.now ? { now: options.now } : {}),
|
||||
preInstallBackup:
|
||||
options.update?.preInstallBackup ??
|
||||
(async (targetCommit: string) => {
|
||||
const created = await componentBackups.create(
|
||||
['devices', 'notifications', 'automation', 'settings'],
|
||||
`更新前自动备份:${targetCommit.slice(0, 7)}`,
|
||||
true,
|
||||
);
|
||||
return created.filename;
|
||||
}),
|
||||
});
|
||||
const scheduledDispatcher = new ScheduledOperationDispatcher({
|
||||
db: options.db,
|
||||
operations: secureExecution,
|
||||
messages,
|
||||
maintenance,
|
||||
store: options.store,
|
||||
repository: scheduledTasks,
|
||||
...(options.now ? { now: options.now } : {}),
|
||||
});
|
||||
const scheduler = new SchedulerCoordinator({
|
||||
db: options.db,
|
||||
repository: scheduledTasks,
|
||||
dispatch: (task, targets, _run, context) =>
|
||||
scheduledDispatcher.dispatch(task, targets, context),
|
||||
...(options.now ? { now: options.now } : {}),
|
||||
});
|
||||
const auth = new ConsoleAuthService({
|
||||
db: options.db,
|
||||
...(options.now ? { now: options.now } : {}),
|
||||
});
|
||||
const organization = new DeviceOrganizationService({
|
||||
db: options.db,
|
||||
...(options.now ? { now: options.now } : {}),
|
||||
});
|
||||
const logCenter = new LogCenterService({ db: options.db, connections: connectionLogs });
|
||||
// The guard learns identities from the device reads the console already makes, and in return it
|
||||
// gets to stop control traffic when a node stops matching its own record.
|
||||
const identities = new DeviceIdentityService({
|
||||
db: options.db,
|
||||
...(options.now ? { now: options.now } : {}),
|
||||
authorizationMode: () => connectionSettings.get().authorizationMode,
|
||||
});
|
||||
const instanceModules = new InstanceModuleService({
|
||||
instances,
|
||||
sessions,
|
||||
request: options.upstream.request,
|
||||
ensureSession,
|
||||
onIdentity: (instanceId, evidence, origin) => {
|
||||
identities.observe(instanceId, { ...evidence, origin });
|
||||
},
|
||||
});
|
||||
const deviceActions = new DeviceActionService({
|
||||
instances,
|
||||
sessions,
|
||||
request: options.upstream.request,
|
||||
db: options.db,
|
||||
ensureSession,
|
||||
identities,
|
||||
});
|
||||
// Scraped by the operator's monitoring rather than by the console, and read-only by
|
||||
// construction: it aggregates rows the control plane already writes instead of probing devices.
|
||||
const metrics = new MetricsService({
|
||||
db: options.db,
|
||||
version: options.runtimeVersion ?? '0.1.0',
|
||||
sources: { connections: connectionLogs, identities },
|
||||
...(options.now ? { now: options.now } : {}),
|
||||
});
|
||||
const app = buildApp({
|
||||
...options.app,
|
||||
registerRoutes: (app) => {
|
||||
@@ -126,20 +524,130 @@ export function buildControlPlaneApp(options: ControlPlaneOptions): ControlPlane
|
||||
resources,
|
||||
messages,
|
||||
registerDeletionPreparationRoute: false,
|
||||
onEvent: (envelope) => eventJournal.append(envelope),
|
||||
});
|
||||
registerFleetRoutes(app, {
|
||||
instances,
|
||||
resources,
|
||||
messages: hubMessages,
|
||||
notifications,
|
||||
connections,
|
||||
outbox: smsOutbox,
|
||||
identities,
|
||||
});
|
||||
registerCentralNotificationRoutes(app, {
|
||||
notifications: centralNotifications,
|
||||
devices: listHubDevices,
|
||||
});
|
||||
registerOperationRoutes(app, operationCatalogRegistry, secureExecution, deletion);
|
||||
registerJobRoutes(app, { jobs });
|
||||
registerJobRoutes(app, { jobs, reconcile: jobReconcile });
|
||||
registerAuditRoutes(app, { audit });
|
||||
registerAutomationRoutes(app, {
|
||||
service: automation,
|
||||
repository: scheduledTasks,
|
||||
runNow: (taskId, actor, requestId) => scheduler.runNow(taskId, actor, requestId),
|
||||
});
|
||||
registerEventRoutes(app, {
|
||||
journal: eventJournal,
|
||||
...(options.authenticateEventStream
|
||||
? { authenticate: options.authenticateEventStream }
|
||||
: {}),
|
||||
});
|
||||
registerSystemRoutes(app, {
|
||||
maintenance,
|
||||
componentBackups,
|
||||
connectionSettings,
|
||||
heartbeat,
|
||||
});
|
||||
registerUpdateRoutes(app, { updates });
|
||||
registerOrganizationRoutes(app, { organization });
|
||||
registerLogCenterRoutes(app, { logs: logCenter, connections: connectionLogs });
|
||||
registerInstanceModuleRoutes(app, { modules: instanceModules });
|
||||
registerDeviceActionRoutes(app, { actions: deviceActions, identities });
|
||||
registerIdentityRoutes(app, { identities, instances, modules: instanceModules });
|
||||
registerMetricsRoutes(app, { metrics });
|
||||
registerDiscoveryRoutes(app, { discovery });
|
||||
},
|
||||
});
|
||||
Object.assign(app, {
|
||||
retryPendingSecretCleanup: () => instances.retryPendingSecretCleanup(),
|
||||
});
|
||||
// Central Hub loops: pull device SMS into the archive and drain the delivery queue.
|
||||
const smsSyncIntervalMs = options.smsSyncIntervalMs ?? DEFAULT_SMS_SYNC_INTERVAL_MS;
|
||||
const queueDrainIntervalMs = options.queueDrainIntervalMs ?? DEFAULT_QUEUE_DRAIN_INTERVAL_MS;
|
||||
const logPruneIntervalMs = options.logPruneIntervalMs ?? DEFAULT_LOG_PRUNE_INTERVAL_MS;
|
||||
const autoBackupIntervalMs = options.autoBackupIntervalMs ?? DEFAULT_AUTO_BACKUP_INTERVAL_MS;
|
||||
const messageSyncInterval =
|
||||
smsSyncIntervalMs > 0
|
||||
? setInterval(() => {
|
||||
void hubMessages.syncAll().catch(() => undefined);
|
||||
}, smsSyncIntervalMs)
|
||||
: undefined;
|
||||
const queueDrainInterval =
|
||||
queueDrainIntervalMs > 0
|
||||
? setInterval(() => {
|
||||
void centralNotifications.processQueue().catch(() => undefined);
|
||||
// Queued SMS ride the same drain tick so a returning device is picked up quickly.
|
||||
void smsOutbox.flush().catch(() => undefined);
|
||||
}, queueDrainIntervalMs)
|
||||
: undefined;
|
||||
const logPruneInterval =
|
||||
logPruneIntervalMs > 0
|
||||
? setInterval(() => {
|
||||
void Promise.resolve()
|
||||
.then(() => centralNotifications.pruneLogs())
|
||||
.catch(() => undefined);
|
||||
// Delivered and abandoned outbox rows otherwise accumulate forever.
|
||||
void Promise.resolve()
|
||||
.then(() => smsOutbox.prune(RETENTION_HISTORY_MS))
|
||||
.catch(() => undefined);
|
||||
void Promise.resolve()
|
||||
.then(() => eventJournal.prune())
|
||||
.catch(() => undefined);
|
||||
// Connection probes write a log row per beat per node; without a sweep
|
||||
// that alone is thousands of rows a day.
|
||||
void Promise.resolve()
|
||||
.then(() =>
|
||||
connectionLogs.prune({
|
||||
before: new Date(Date.now() - RETENTION_HISTORY_MS).toISOString(),
|
||||
}),
|
||||
)
|
||||
.catch(() => undefined);
|
||||
void Promise.resolve()
|
||||
.then(() => audit.pruneBefore(new Date(Date.now() - RETENTION_AUDIT_MS).toISOString()))
|
||||
.catch(() => undefined);
|
||||
void Promise.resolve()
|
||||
.then(() => secureExecution.pruneTerminalPreparations(RETENTION_PREPARATION_MS))
|
||||
.catch(() => undefined);
|
||||
void Promise.resolve()
|
||||
.then(() => centralNotifications.pruneFinishedQueue(RETENTION_HISTORY_MS))
|
||||
.catch(() => undefined);
|
||||
}, logPruneIntervalMs)
|
||||
: undefined;
|
||||
// Scheduled component backups: the tick is cheap and only writes when a period is overdue.
|
||||
const autoBackupInterval =
|
||||
autoBackupIntervalMs > 0
|
||||
? setInterval(() => {
|
||||
void Promise.resolve()
|
||||
.then(() => componentBackups.runDueAutoBackups())
|
||||
.catch(() => undefined);
|
||||
}, autoBackupIntervalMs)
|
||||
: undefined;
|
||||
messageSyncInterval?.unref?.();
|
||||
queueDrainInterval?.unref?.();
|
||||
logPruneInterval?.unref?.();
|
||||
autoBackupInterval?.unref?.();
|
||||
app.addHook('onClose', async () => {
|
||||
scheduler.stop();
|
||||
if (messageSyncInterval) clearInterval(messageSyncInterval);
|
||||
if (queueDrainInterval) clearInterval(queueDrainInterval);
|
||||
if (logPruneInterval) clearInterval(logPruneInterval);
|
||||
if (autoBackupInterval) clearInterval(autoBackupInterval);
|
||||
});
|
||||
scheduler.start();
|
||||
if (options.heartbeatEnabled) heartbeat.start();
|
||||
app.addHook('onClose', async () => {
|
||||
heartbeat.stop();
|
||||
});
|
||||
return app as ControlPlaneApp;
|
||||
}
|
||||
|
||||
@@ -23,7 +23,7 @@ const STATE_FILE = 'cutover-state.json';
|
||||
const GATEWAY_ARGV = [
|
||||
process.execPath,
|
||||
'--import',
|
||||
resolve(process.cwd(), 'node_modules/.pnpm/tsx@4.22.4/node_modules/tsx/dist/loader.mjs'),
|
||||
'tsx',
|
||||
resolve(process.cwd(), 'apps/api/src/production-gateway-cli.ts'),
|
||||
] as const;
|
||||
const DIGEST = /^[a-f0-9]{64}$/u;
|
||||
@@ -289,11 +289,14 @@ async function durableState(plan: CutoverPlan, state: CutoverState): Promise<voi
|
||||
await handle.close();
|
||||
}
|
||||
await rename(temporary, target);
|
||||
const directory = await open(plan.stateDir, constants.O_RDONLY);
|
||||
try {
|
||||
await directory.sync();
|
||||
} finally {
|
||||
await directory.close();
|
||||
if (process.platform !== 'win32') {
|
||||
// directory fsync only exists on POSIX
|
||||
const directory = await open(plan.stateDir, constants.O_RDONLY);
|
||||
try {
|
||||
await directory.sync();
|
||||
} finally {
|
||||
await directory.close();
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -311,7 +314,8 @@ function preparedState(plan: CutoverPlan): CutoverState {
|
||||
async function readState(plan: CutoverPlan): Promise<CutoverState> {
|
||||
const path = join(plan.stateDir, STATE_FILE);
|
||||
const info = await lstat(path);
|
||||
if (!info.isFile() || (info.mode & 0o777) !== 0o600)
|
||||
// Windows keeps no POSIX mode bits (chmod is a no-op); ACLs govern access there.
|
||||
if (!info.isFile() || (process.platform !== 'win32' && (info.mode & 0o777) !== 0o600))
|
||||
throw new Error('Cutover state must be a mode-0600 regular file');
|
||||
const input = record(JSON.parse(await readFile(path, 'utf8')) as unknown, 'Cutover state');
|
||||
const base = ['version', 'phase', 'legacyPid', 'legacyCommand', 'legacyIdentity', 'legacyStart'];
|
||||
|
||||
@@ -66,4 +66,6 @@ export type {
|
||||
InstanceServiceOptions,
|
||||
} from './application/instances/instance-service.js';
|
||||
export { MIGRATIONS, migrateDatabase } from './infrastructure/database/migrations.js';
|
||||
export { ScheduledTaskRepository } from './application/automation/scheduled-task-repository.js';
|
||||
export { ScheduledTaskService } from './application/automation/scheduled-task-service.js';
|
||||
export * as databaseSchema from './infrastructure/database/schema.js';
|
||||
|
||||
@@ -48,10 +48,13 @@ describe('verified backup, restore, and rollback foundation', () => {
|
||||
]);
|
||||
copy.close();
|
||||
expect(snapshot.sha256).toMatch(/^[a-f0-9]{64}$/);
|
||||
expect((await stat(backupPath)).mode & 0o777).toBe(0o600);
|
||||
if (process.platform !== 'win32') expect((await stat(backupPath)).mode & 0o777).toBe(0o600);
|
||||
});
|
||||
|
||||
it('verifies a separate candidate and digest-binds activation', async () => {
|
||||
// Activation renames a WAL file that a live connection still holds open;
|
||||
// POSIX permits that, Windows file locking does not.
|
||||
const itPosix = process.platform === 'win32' ? it.skip : it;
|
||||
itPosix('verifies a separate candidate and digest-binds activation', async () => {
|
||||
const { livePath, backupPath, database } = await fixture();
|
||||
const snapshot = await backupDatabase(database, backupPath);
|
||||
database.exec('UPDATE app_settings SET value_json = \'"after"\'');
|
||||
@@ -118,7 +121,7 @@ describe('verified backup, restore, and rollback foundation', () => {
|
||||
expect(await readFile(livePath)).toEqual(before);
|
||||
});
|
||||
|
||||
it('rejects in-place staged-byte mutation even when metadata is preserved', async () => {
|
||||
itPosix('rejects in-place staged-byte mutation even when metadata is preserved', async () => {
|
||||
const { livePath, backupPath, database } = await fixture();
|
||||
await backupDatabase(database, backupPath);
|
||||
database.close();
|
||||
|
||||
@@ -159,7 +159,8 @@ async function digestFileHandle(handle: FileHandle): Promise<string> {
|
||||
}
|
||||
|
||||
async function syncFile(path: string): Promise<void> {
|
||||
const handle = await open(path, 'r');
|
||||
// Windows FlushFileBuffers requires a writable handle; POSIX allows fsync on 'r'.
|
||||
const handle = await open(path, process.platform === 'win32' ? 'r+' : 'r');
|
||||
try {
|
||||
await handle.sync();
|
||||
} finally {
|
||||
@@ -168,6 +169,7 @@ async function syncFile(path: string): Promise<void> {
|
||||
}
|
||||
|
||||
async function syncDirectory(path: string): Promise<void> {
|
||||
if (process.platform === 'win32') return; // directory fsync only exists on POSIX
|
||||
const handle = await open(path, 'r');
|
||||
try {
|
||||
await handle.sync();
|
||||
@@ -264,7 +266,9 @@ export async function activateRestoreCandidate(
|
||||
await assertValidDatabase(stagedPath);
|
||||
if ((await digestFileHandle(stagedHandle)) !== stagedDigest)
|
||||
throw new Error('Staged restore candidate changed during validation');
|
||||
await stagedHandle.sync();
|
||||
// The staged handle is intentionally read-only after chmod(0400); Windows
|
||||
// FlushFileBuffers requires a writable handle, so skip the durability sync there.
|
||||
if (process.platform !== 'win32') await stagedHandle.sync();
|
||||
await activateStagedDatabase(
|
||||
databasePath,
|
||||
stagedPath,
|
||||
|
||||
@@ -62,19 +62,31 @@ describe('database migrations', () => {
|
||||
'app_settings',
|
||||
'audit_events',
|
||||
'capabilities',
|
||||
'connection_logs',
|
||||
'console_auth_config',
|
||||
'console_auth_sessions',
|
||||
'device_groups',
|
||||
'device_identities',
|
||||
'event_journal',
|
||||
'instance_tags',
|
||||
'instances',
|
||||
'job_attempts',
|
||||
'job_items',
|
||||
'jobs',
|
||||
'notification_channels',
|
||||
'notification_deliveries',
|
||||
'notification_queue',
|
||||
'notification_rules',
|
||||
'operation_preparations',
|
||||
'scheduled_runs',
|
||||
'scheduled_tasks',
|
||||
'schema_migrations',
|
||||
'secret_cleanup_tasks',
|
||||
'secret_references',
|
||||
'sms_messages',
|
||||
'sms_outbox',
|
||||
'status_snapshots',
|
||||
'tag_registry',
|
||||
]);
|
||||
|
||||
expect(database.pragma('foreign_keys', { simple: true })).toBe(1);
|
||||
@@ -350,7 +362,8 @@ describe('database backup and restore', () => {
|
||||
migrateDatabase(database);
|
||||
const backupPath = join(directory, 'backup.sqlite');
|
||||
await backupDatabase(database, backupPath);
|
||||
expect((await lstat(backupPath)).mode & 0o777).toBe(0o600);
|
||||
// Windows keeps no POSIX mode bits; the 0600 guarantee is POSIX-only.
|
||||
if (process.platform !== 'win32') expect((await lstat(backupPath)).mode & 0o777).toBe(0o600);
|
||||
database.close();
|
||||
});
|
||||
|
||||
|
||||
@@ -0,0 +1,103 @@
|
||||
import Database from 'better-sqlite3';
|
||||
import { describe, expect, it } from 'vitest';
|
||||
|
||||
import { MIGRATIONS, migrateDatabase } from './migrations.js';
|
||||
|
||||
function seeded(): Database.Database {
|
||||
const db = new Database(':memory:');
|
||||
db.pragma('foreign_keys=ON');
|
||||
migrateDatabase(db, MIGRATIONS.slice(0, 10));
|
||||
db.prepare(
|
||||
`INSERT INTO instances (id,name,base_url,auth_mode,enabled,config_revision,created_at,updated_at)
|
||||
VALUES ('i-1','Node One','http://a','password',1,1,'2026-01-01T00:00:00.000Z','2026-01-01T00:00:00.000Z')`,
|
||||
).run();
|
||||
db.prepare(
|
||||
`INSERT INTO instance_tags (instance_id,tag,created_at) VALUES ('i-1','office','2026-01-01T00:00:00.000Z')`,
|
||||
).run();
|
||||
db.prepare(
|
||||
`INSERT INTO scheduled_tasks (id,name,operation_type,enabled,version,cron_expression,timezone,
|
||||
target_selector_json,sms_secret_reference,sms_recipient_count,effective_start_at,effective_end_at,
|
||||
misfire_policy,overlap_policy,retry_policy_json,next_due_at,last_evaluated_at,created_by,updated_by,
|
||||
created_at,updated_at,deleted_at)
|
||||
VALUES ('t-1','Nightly','reboot-system',1,3,'0 3 * * *','Asia/Shanghai','{"mode":"fixed","instanceIds":["i-1"]}',
|
||||
NULL,NULL,NULL,NULL,'skip','skip','{"maxRetries":2,"intervalSeconds":60}',
|
||||
'2026-09-04T03:00:00.000Z',NULL,'sys','sys','2026-01-01T00:00:00.000Z','2026-01-01T00:00:00.000Z',NULL)`,
|
||||
).run();
|
||||
db.prepare(
|
||||
`INSERT INTO scheduled_runs (id,scheduled_task_id,schedule_version,task_snapshot_json,due_at,claimed_at,
|
||||
started_at,finished_at,target_snapshot_json,outcome,reason,job_ids_json,trigger_source,attempt)
|
||||
VALUES ('r-1','t-1',3,'{}','2026-09-03T03:00:00.000Z','2026-09-03T03:00:00.000Z',
|
||||
'2026-09-03T03:00:01.000Z','2026-09-03T03:00:02.000Z','["i-1"]','succeeded',NULL,'[]','scheduled',1)`,
|
||||
).run();
|
||||
return db;
|
||||
}
|
||||
|
||||
describe('migration upgrade path', () => {
|
||||
it('preserves automation rows and registers existing tags when upgrading from migration 10', () => {
|
||||
const db = seeded();
|
||||
expect(() => migrateDatabase(db)).not.toThrow();
|
||||
const task = db.prepare('SELECT * FROM scheduled_tasks WHERE id=?').get('t-1') as Record<
|
||||
string,
|
||||
unknown
|
||||
>;
|
||||
expect(task).toMatchObject({ name: 'Nightly', operation_type: 'reboot-system', version: 3 });
|
||||
expect(task.trigger_json).toBeNull();
|
||||
const run = db.prepare('SELECT * FROM scheduled_runs WHERE id=?').get('r-1') as Record<
|
||||
string,
|
||||
unknown
|
||||
>;
|
||||
expect(run).toMatchObject({ scheduled_task_id: 't-1', outcome: 'succeeded' });
|
||||
expect(db.prepare('SELECT tag FROM tag_registry ORDER BY tag').all()).toEqual([
|
||||
{ tag: 'office' },
|
||||
]);
|
||||
expect(db.prepare('PRAGMA foreign_key_check').all()).toEqual([]);
|
||||
db.prepare(
|
||||
`INSERT INTO scheduled_tasks (id,name,operation_type,enabled,version,cron_expression,timezone,
|
||||
target_selector_json,sms_secret_reference,sms_recipient_count,effective_start_at,effective_end_at,
|
||||
misfire_policy,overlap_policy,retry_policy_json,next_due_at,last_evaluated_at,created_by,updated_by,
|
||||
created_at,updated_at,deleted_at,trigger_json)
|
||||
VALUES ('t-2','Baseband','restart-baseband',1,1,'0 4 * * *','Asia/Shanghai','{"mode":"all"}',
|
||||
NULL,NULL,NULL,NULL,'skip','skip','{"maxRetries":0,"intervalSeconds":60}',NULL,NULL,'sys','sys',
|
||||
'2026-01-01T00:00:00.000Z','2026-01-01T00:00:00.000Z',NULL,'{"kind":"interval","value":6,"unit":"hours"}')`,
|
||||
).run();
|
||||
expect(
|
||||
(
|
||||
db.prepare('SELECT trigger_json FROM scheduled_tasks WHERE id=?').get('t-2') as {
|
||||
trigger_json: string;
|
||||
}
|
||||
).trigger_json,
|
||||
).toContain('interval');
|
||||
const names = (
|
||||
db
|
||||
.prepare(
|
||||
"SELECT name FROM sqlite_master WHERE type='index' AND name LIKE 'idx_scheduled%' ORDER BY name",
|
||||
)
|
||||
.all() as Array<{
|
||||
name: string;
|
||||
}>
|
||||
).map((row) => row.name);
|
||||
expect(names).toEqual([
|
||||
'idx_scheduled_runs_outcome_finished',
|
||||
'idx_scheduled_runs_task_due',
|
||||
'idx_scheduled_tasks_enabled_next_due',
|
||||
]);
|
||||
db.close();
|
||||
});
|
||||
|
||||
it('applies cleanly to an empty database', () => {
|
||||
const db = new Database(':memory:');
|
||||
db.pragma('foreign_keys=ON');
|
||||
migrateDatabase(db);
|
||||
expect(db.prepare('PRAGMA foreign_key_check').all()).toEqual([]);
|
||||
const applied = db.prepare('SELECT id,name FROM schema_migrations ORDER BY id').all() as Array<{
|
||||
id: number;
|
||||
name: string;
|
||||
}>;
|
||||
// The ledger must match the declared migration list exactly, in order, so a new migration
|
||||
// can never be added without being applied here.
|
||||
expect(applied).toEqual(
|
||||
MIGRATIONS.map((migration) => ({ id: migration.id, name: migration.name })),
|
||||
);
|
||||
db.close();
|
||||
});
|
||||
});
|
||||
@@ -304,6 +304,382 @@ export const MIGRATIONS: readonly Migration[] = [
|
||||
'CREATE INDEX idx_console_auth_sessions_expires_at ON console_auth_sessions(expires_at)',
|
||||
],
|
||||
},
|
||||
{
|
||||
id: 8,
|
||||
name: 'scheduled-automation',
|
||||
statements: [
|
||||
`CREATE TABLE scheduled_tasks (
|
||||
id TEXT PRIMARY KEY,
|
||||
name TEXT NOT NULL,
|
||||
operation_type TEXT NOT NULL CHECK (operation_type IN ('restart-service','reboot-system','send-sms')),
|
||||
enabled INTEGER NOT NULL DEFAULT 1 CHECK (enabled IN (0, 1)),
|
||||
version INTEGER NOT NULL DEFAULT 1 CHECK (version > 0),
|
||||
cron_expression TEXT NOT NULL,
|
||||
timezone TEXT NOT NULL CHECK (timezone = 'Asia/Shanghai'),
|
||||
target_selector_json TEXT NOT NULL,
|
||||
sms_secret_reference TEXT,
|
||||
sms_recipient_count INTEGER CHECK (sms_recipient_count IS NULL OR sms_recipient_count > 0),
|
||||
effective_start_at TEXT,
|
||||
effective_end_at TEXT,
|
||||
misfire_policy TEXT NOT NULL CHECK (misfire_policy IN ('skip','catch-up-once')),
|
||||
overlap_policy TEXT NOT NULL CHECK (overlap_policy IN ('skip','queue-once')),
|
||||
retry_policy_json TEXT NOT NULL,
|
||||
next_due_at TEXT,
|
||||
last_evaluated_at TEXT,
|
||||
created_by TEXT NOT NULL,
|
||||
updated_by TEXT NOT NULL,
|
||||
created_at TEXT NOT NULL,
|
||||
updated_at TEXT NOT NULL,
|
||||
deleted_at TEXT,
|
||||
CHECK ((operation_type = 'send-sms') = (sms_secret_reference IS NOT NULL)),
|
||||
CHECK ((sms_secret_reference IS NULL) = (sms_recipient_count IS NULL))
|
||||
)`,
|
||||
`CREATE TABLE scheduled_runs (
|
||||
id TEXT PRIMARY KEY,
|
||||
scheduled_task_id TEXT NOT NULL REFERENCES scheduled_tasks(id) ON DELETE RESTRICT,
|
||||
schedule_version INTEGER NOT NULL CHECK (schedule_version > 0),
|
||||
task_snapshot_json TEXT NOT NULL,
|
||||
due_at TEXT NOT NULL,
|
||||
claimed_at TEXT NOT NULL,
|
||||
started_at TEXT,
|
||||
finished_at TEXT,
|
||||
target_snapshot_json TEXT NOT NULL,
|
||||
outcome TEXT CHECK (outcome IS NULL OR outcome IN ('succeeded','partially-succeeded','failed','skipped','no-targets','needs-attention')),
|
||||
reason TEXT,
|
||||
job_ids_json TEXT NOT NULL DEFAULT '[]',
|
||||
trigger_source TEXT NOT NULL CHECK (trigger_source IN ('scheduled','manual')),
|
||||
attempt INTEGER NOT NULL DEFAULT 1 CHECK (attempt > 0),
|
||||
UNIQUE (scheduled_task_id, schedule_version, due_at, trigger_source)
|
||||
)`,
|
||||
'CREATE INDEX idx_scheduled_tasks_enabled_next_due ON scheduled_tasks(enabled, next_due_at) WHERE deleted_at IS NULL',
|
||||
'CREATE INDEX idx_scheduled_runs_task_due ON scheduled_runs(scheduled_task_id, due_at DESC)',
|
||||
'CREATE INDEX idx_scheduled_runs_outcome_finished ON scheduled_runs(outcome, finished_at DESC)',
|
||||
],
|
||||
},
|
||||
{
|
||||
id: 9,
|
||||
name: 'central-notifications',
|
||||
statements: [
|
||||
`CREATE TABLE notification_channels (
|
||||
id TEXT PRIMARY KEY,
|
||||
name TEXT NOT NULL,
|
||||
type TEXT NOT NULL CHECK (type IN (
|
||||
'webhook','bark','pushplus','wecom_app','wecom_robot','dingtalk_robot','dingtalk_app',
|
||||
'feishu_robot','telegram','email','serverchan'
|
||||
)),
|
||||
enabled INTEGER NOT NULL DEFAULT 1 CHECK (enabled IN (0, 1)),
|
||||
config_json TEXT NOT NULL,
|
||||
secret_reference TEXT,
|
||||
created_at TEXT NOT NULL,
|
||||
updated_at TEXT NOT NULL
|
||||
)`,
|
||||
`CREATE TABLE notification_rules (
|
||||
id TEXT PRIMARY KEY,
|
||||
name TEXT NOT NULL,
|
||||
event_type TEXT NOT NULL CHECK (event_type IN (
|
||||
'sms','ddns','version','system','device','automation'
|
||||
)),
|
||||
enabled INTEGER NOT NULL DEFAULT 1 CHECK (enabled IN (0, 1)),
|
||||
condition_json TEXT NOT NULL,
|
||||
scope_json TEXT NOT NULL,
|
||||
channel_ids_json TEXT NOT NULL,
|
||||
templates_json TEXT NOT NULL,
|
||||
created_at TEXT NOT NULL,
|
||||
updated_at TEXT NOT NULL
|
||||
)`,
|
||||
`CREATE TABLE notification_deliveries (
|
||||
id TEXT PRIMARY KEY,
|
||||
rule_id TEXT REFERENCES notification_rules(id) ON DELETE SET NULL,
|
||||
channel_id TEXT NOT NULL REFERENCES notification_channels(id) ON DELETE CASCADE,
|
||||
instance_id TEXT,
|
||||
event_type TEXT NOT NULL,
|
||||
status TEXT NOT NULL CHECK (status IN (
|
||||
'success','failed','pending','sending','retrying','unmatched','no_available_channel','quiet_hours'
|
||||
)),
|
||||
detail TEXT,
|
||||
created_at TEXT NOT NULL,
|
||||
sent_at TEXT
|
||||
)`,
|
||||
'CREATE INDEX idx_notification_deliveries_created_at ON notification_deliveries(created_at DESC)',
|
||||
'CREATE INDEX idx_notification_deliveries_status_created_at ON notification_deliveries(status, created_at DESC)',
|
||||
],
|
||||
},
|
||||
{
|
||||
id: 10,
|
||||
name: 'hub-message-and-notification-queue-persistence',
|
||||
statements: [
|
||||
`CREATE TABLE sms_messages (
|
||||
id TEXT PRIMARY KEY,
|
||||
instance_id TEXT NOT NULL,
|
||||
upstream_id TEXT NOT NULL,
|
||||
direction TEXT NOT NULL CHECK (direction IN ('incoming','outgoing','received','sent','unknown')),
|
||||
phone_number TEXT NOT NULL,
|
||||
content TEXT NOT NULL,
|
||||
timestamp TEXT NOT NULL,
|
||||
status TEXT NOT NULL,
|
||||
transport TEXT NOT NULL,
|
||||
synced_at TEXT NOT NULL,
|
||||
UNIQUE (instance_id, upstream_id)
|
||||
)`,
|
||||
'CREATE INDEX idx_sms_messages_instance_timestamp ON sms_messages(instance_id, timestamp DESC)',
|
||||
'CREATE INDEX idx_sms_messages_timestamp ON sms_messages(timestamp DESC)',
|
||||
'CREATE INDEX idx_sms_messages_phone_timestamp ON sms_messages(phone_number, timestamp DESC)',
|
||||
`CREATE TABLE notification_queue (
|
||||
id TEXT PRIMARY KEY,
|
||||
instance_id TEXT,
|
||||
event_type TEXT NOT NULL,
|
||||
rule_id TEXT REFERENCES notification_rules(id) ON DELETE SET NULL,
|
||||
channel_id TEXT REFERENCES notification_channels(id) ON DELETE CASCADE,
|
||||
status TEXT NOT NULL DEFAULT 'pending' CHECK (status IN (
|
||||
'pending','sending','succeeded','failed','cancelled'
|
||||
)),
|
||||
attempts INTEGER NOT NULL DEFAULT 0 CHECK (attempts >= 0),
|
||||
max_attempts INTEGER NOT NULL DEFAULT 3 CHECK (max_attempts > 0),
|
||||
payload_json TEXT NOT NULL,
|
||||
last_error TEXT,
|
||||
available_at TEXT NOT NULL,
|
||||
delivered_at TEXT,
|
||||
created_at TEXT NOT NULL,
|
||||
updated_at TEXT NOT NULL
|
||||
)`,
|
||||
'CREATE INDEX idx_notification_queue_status_available_at ON notification_queue(status, available_at)',
|
||||
'CREATE INDEX idx_notification_queue_created_at ON notification_queue(created_at DESC)',
|
||||
],
|
||||
},
|
||||
{
|
||||
id: 11,
|
||||
name: 'hub-groups-tags-and-automation-triggers',
|
||||
statements: [
|
||||
`CREATE TABLE device_groups (
|
||||
id TEXT PRIMARY KEY,
|
||||
name TEXT NOT NULL UNIQUE,
|
||||
description TEXT NOT NULL DEFAULT '',
|
||||
created_at TEXT NOT NULL,
|
||||
updated_at TEXT NOT NULL
|
||||
)`,
|
||||
'ALTER TABLE instances ADD COLUMN group_id TEXT REFERENCES device_groups(id) ON DELETE SET NULL',
|
||||
'CREATE INDEX idx_instances_group_id ON instances(group_id)',
|
||||
`CREATE TABLE tag_registry (
|
||||
tag TEXT PRIMARY KEY,
|
||||
color TEXT NOT NULL DEFAULT '',
|
||||
created_at TEXT NOT NULL,
|
||||
updated_at TEXT NOT NULL
|
||||
)`,
|
||||
`INSERT INTO tag_registry (tag, created_at, updated_at)
|
||||
SELECT tag, MIN(created_at), MAX(created_at) FROM instance_tags GROUP BY tag`,
|
||||
// scheduled_tasks carries an operation_type CHECK that has to widen, so both the
|
||||
// task table and its only child are rebuilt; renaming the child first keeps the
|
||||
// RESTRICT edge from firing while the parent is swapped out.
|
||||
'ALTER TABLE scheduled_runs RENAME TO scheduled_runs_old',
|
||||
'ALTER TABLE scheduled_tasks RENAME TO scheduled_tasks_old',
|
||||
`CREATE TABLE scheduled_tasks (
|
||||
id TEXT PRIMARY KEY,
|
||||
name TEXT NOT NULL,
|
||||
operation_type TEXT NOT NULL CHECK (operation_type IN (
|
||||
'restart-service','reboot-system','send-sms','restart-baseband','backup-data'
|
||||
)),
|
||||
enabled INTEGER NOT NULL DEFAULT 1 CHECK (enabled IN (0, 1)),
|
||||
version INTEGER NOT NULL DEFAULT 1 CHECK (version > 0),
|
||||
cron_expression TEXT NOT NULL,
|
||||
timezone TEXT NOT NULL CHECK (timezone = 'Asia/Shanghai'),
|
||||
target_selector_json TEXT NOT NULL,
|
||||
sms_secret_reference TEXT,
|
||||
sms_recipient_count INTEGER CHECK (sms_recipient_count IS NULL OR sms_recipient_count > 0),
|
||||
effective_start_at TEXT,
|
||||
effective_end_at TEXT,
|
||||
misfire_policy TEXT NOT NULL CHECK (misfire_policy IN ('skip','catch-up-once')),
|
||||
overlap_policy TEXT NOT NULL CHECK (overlap_policy IN ('skip','queue-once')),
|
||||
retry_policy_json TEXT NOT NULL,
|
||||
next_due_at TEXT,
|
||||
last_evaluated_at TEXT,
|
||||
created_by TEXT NOT NULL,
|
||||
updated_by TEXT NOT NULL,
|
||||
created_at TEXT NOT NULL,
|
||||
updated_at TEXT NOT NULL,
|
||||
deleted_at TEXT,
|
||||
trigger_json TEXT,
|
||||
CHECK ((operation_type = 'send-sms') = (sms_secret_reference IS NOT NULL)),
|
||||
CHECK ((sms_secret_reference IS NULL) = (sms_recipient_count IS NULL))
|
||||
)`,
|
||||
`CREATE TABLE scheduled_runs (
|
||||
id TEXT PRIMARY KEY,
|
||||
scheduled_task_id TEXT NOT NULL REFERENCES scheduled_tasks(id) ON DELETE RESTRICT,
|
||||
schedule_version INTEGER NOT NULL CHECK (schedule_version > 0),
|
||||
task_snapshot_json TEXT NOT NULL,
|
||||
due_at TEXT NOT NULL,
|
||||
claimed_at TEXT NOT NULL,
|
||||
started_at TEXT,
|
||||
finished_at TEXT,
|
||||
target_snapshot_json TEXT NOT NULL,
|
||||
outcome TEXT CHECK (outcome IS NULL OR outcome IN ('succeeded','partially-succeeded','failed','skipped','no-targets','needs-attention')),
|
||||
reason TEXT,
|
||||
job_ids_json TEXT NOT NULL DEFAULT '[]',
|
||||
trigger_source TEXT NOT NULL CHECK (trigger_source IN ('scheduled','manual')),
|
||||
attempt INTEGER NOT NULL DEFAULT 1 CHECK (attempt > 0),
|
||||
UNIQUE (scheduled_task_id, schedule_version, due_at, trigger_source)
|
||||
)`,
|
||||
`INSERT INTO scheduled_tasks (id,name,operation_type,enabled,version,cron_expression,timezone,
|
||||
target_selector_json,sms_secret_reference,sms_recipient_count,effective_start_at,
|
||||
effective_end_at,misfire_policy,overlap_policy,retry_policy_json,next_due_at,
|
||||
last_evaluated_at,created_by,updated_by,created_at,updated_at,deleted_at,trigger_json)
|
||||
SELECT id,name,operation_type,enabled,version,cron_expression,timezone,target_selector_json,
|
||||
sms_secret_reference,sms_recipient_count,effective_start_at,effective_end_at,misfire_policy,
|
||||
overlap_policy,retry_policy_json,next_due_at,last_evaluated_at,created_by,updated_by,
|
||||
created_at,updated_at,deleted_at,NULL FROM scheduled_tasks_old`,
|
||||
`INSERT INTO scheduled_runs (id,scheduled_task_id,schedule_version,task_snapshot_json,due_at,
|
||||
claimed_at,started_at,finished_at,target_snapshot_json,outcome,reason,job_ids_json,
|
||||
trigger_source,attempt)
|
||||
SELECT id,scheduled_task_id,schedule_version,task_snapshot_json,due_at,claimed_at,started_at,
|
||||
finished_at,target_snapshot_json,outcome,reason,job_ids_json,trigger_source,attempt
|
||||
FROM scheduled_runs_old`,
|
||||
'DROP TABLE scheduled_runs_old',
|
||||
'DROP TABLE scheduled_tasks_old',
|
||||
'CREATE INDEX idx_scheduled_tasks_enabled_next_due ON scheduled_tasks(enabled, next_due_at) WHERE deleted_at IS NULL',
|
||||
'CREATE INDEX idx_scheduled_runs_task_due ON scheduled_runs(scheduled_task_id, due_at DESC)',
|
||||
'CREATE INDEX idx_scheduled_runs_outcome_finished ON scheduled_runs(outcome, finished_at DESC)',
|
||||
],
|
||||
},
|
||||
{
|
||||
id: 12,
|
||||
name: 'schedule-action-delay',
|
||||
statements: [
|
||||
`ALTER TABLE scheduled_tasks ADD COLUMN delay_seconds INTEGER
|
||||
CHECK (delay_seconds IS NULL OR (delay_seconds >= 0 AND delay_seconds <= 3600))`,
|
||||
],
|
||||
},
|
||||
{
|
||||
id: 13,
|
||||
name: 'notification-rule-rate-limit-and-quiet-hours',
|
||||
statements: [
|
||||
`ALTER TABLE notification_rules ADD COLUMN rate_limit_json TEXT NOT NULL DEFAULT
|
||||
'{"enabled":false,"maxMessages":20,"windowSeconds":60}'`,
|
||||
`ALTER TABLE notification_rules ADD COLUMN quiet_hours_json TEXT NOT NULL DEFAULT '[]'`,
|
||||
// Suppressions (quiet hours, rate limit) are attributed to a rule rather than a channel,
|
||||
// so channel_id becomes optional. The status CHECK also has to widen, which means the
|
||||
// table is rebuilt; nothing references notification_deliveries, so the swap is contained.
|
||||
`CREATE TABLE notification_deliveries_new (
|
||||
id TEXT PRIMARY KEY,
|
||||
rule_id TEXT REFERENCES notification_rules(id) ON DELETE SET NULL,
|
||||
channel_id TEXT REFERENCES notification_channels(id) ON DELETE CASCADE,
|
||||
instance_id TEXT,
|
||||
event_type TEXT NOT NULL,
|
||||
status TEXT NOT NULL CHECK (status IN (
|
||||
'success','failed','pending','sending','retrying','unmatched','no_available_channel',
|
||||
'quiet_hours','rate_limited'
|
||||
)),
|
||||
detail TEXT,
|
||||
created_at TEXT NOT NULL,
|
||||
sent_at TEXT
|
||||
)`,
|
||||
`INSERT INTO notification_deliveries_new
|
||||
(id,rule_id,channel_id,instance_id,event_type,status,detail,created_at,sent_at)
|
||||
SELECT id,rule_id,channel_id,instance_id,event_type,status,detail,created_at,sent_at
|
||||
FROM notification_deliveries`,
|
||||
'DROP TABLE notification_deliveries',
|
||||
'ALTER TABLE notification_deliveries_new RENAME TO notification_deliveries',
|
||||
'CREATE INDEX idx_notification_deliveries_created_at ON notification_deliveries(created_at DESC)',
|
||||
'CREATE INDEX idx_notification_deliveries_status_created_at ON notification_deliveries(status, created_at DESC)',
|
||||
],
|
||||
},
|
||||
{
|
||||
id: 14,
|
||||
name: 'connection-log-history',
|
||||
statements: [
|
||||
// The Hub keeps a rolling connection history per device; the snapshot table only holds
|
||||
// the latest probe, so outcomes need their own bounded journal.
|
||||
`CREATE TABLE connection_logs (
|
||||
id TEXT PRIMARY KEY,
|
||||
instance_id TEXT NOT NULL REFERENCES instances(id) ON DELETE CASCADE,
|
||||
outcome TEXT NOT NULL CHECK (outcome IN ('success','stale','failed','unsupported')),
|
||||
state TEXT NOT NULL CHECK (state IN ('fresh','stale','expired','unknown')),
|
||||
error_code TEXT,
|
||||
http_status INTEGER,
|
||||
duration_ms INTEGER NOT NULL CHECK (duration_ms >= 0),
|
||||
observed_at TEXT NOT NULL
|
||||
)`,
|
||||
'CREATE INDEX idx_connection_logs_observed_at ON connection_logs(observed_at DESC)',
|
||||
'CREATE INDEX idx_connection_logs_instance_observed_at ON connection_logs(instance_id, observed_at DESC)',
|
||||
'CREATE INDEX idx_connection_logs_outcome_observed_at ON connection_logs(outcome, observed_at DESC)',
|
||||
],
|
||||
},
|
||||
{
|
||||
id: 15,
|
||||
name: 'sms-offline-outbox',
|
||||
statements: [
|
||||
// The Hub keeps sending while a device is offline and delivers when it returns; the
|
||||
// central archive alone cannot hold a write, so outbound SMS need their own queue.
|
||||
`CREATE TABLE sms_outbox (
|
||||
id TEXT PRIMARY KEY,
|
||||
instance_id TEXT NOT NULL REFERENCES instances(id) ON DELETE CASCADE,
|
||||
phone_number TEXT NOT NULL,
|
||||
content TEXT NOT NULL,
|
||||
status TEXT NOT NULL DEFAULT 'queued' CHECK (status IN (
|
||||
'queued','sending','sent','failed','cancelled'
|
||||
)),
|
||||
attempts INTEGER NOT NULL DEFAULT 0 CHECK (attempts >= 0),
|
||||
max_attempts INTEGER NOT NULL DEFAULT 24 CHECK (max_attempts > 0),
|
||||
last_error TEXT,
|
||||
available_at TEXT NOT NULL,
|
||||
sent_at TEXT,
|
||||
created_at TEXT NOT NULL,
|
||||
updated_at TEXT NOT NULL
|
||||
)`,
|
||||
'CREATE INDEX idx_sms_outbox_status_available_at ON sms_outbox(status, available_at)',
|
||||
'CREATE INDEX idx_sms_outbox_instance_status ON sms_outbox(instance_id, status)',
|
||||
'CREATE INDEX idx_sms_outbox_created_at ON sms_outbox(created_at DESC)',
|
||||
],
|
||||
},
|
||||
{
|
||||
id: 16,
|
||||
name: 'notification-channel-secret-fields',
|
||||
statements: [
|
||||
// Hub channels keep several credentials (DingTalk needs an access token *and* a signing
|
||||
// key), so the secret store holds a JSON map and the row records which keys it covers.
|
||||
"ALTER TABLE notification_channels ADD COLUMN secret_fields TEXT NOT NULL DEFAULT ''",
|
||||
],
|
||||
},
|
||||
{
|
||||
id: 17,
|
||||
name: 'device-identity-guard',
|
||||
statements: [
|
||||
// The Hub refuses to control a device whose identity drifted: the same IMEI claimed by
|
||||
// two records, or hardware swapped behind an address that is still registered. Both are
|
||||
// recorded here and only an operator can clear them, because the machine cannot tell
|
||||
// which of two claimants is the real device.
|
||||
`CREATE TABLE device_identities (
|
||||
instance_id TEXT PRIMARY KEY REFERENCES instances(id) ON DELETE CASCADE,
|
||||
status TEXT NOT NULL DEFAULT 'confirmed' CHECK (status IN ('confirmed','pending')),
|
||||
reasons TEXT NOT NULL DEFAULT '[]',
|
||||
imei TEXT NOT NULL DEFAULT '',
|
||||
manufacturer TEXT NOT NULL DEFAULT '',
|
||||
model TEXT NOT NULL DEFAULT '',
|
||||
revision TEXT NOT NULL DEFAULT '',
|
||||
agent TEXT NOT NULL DEFAULT '',
|
||||
origin TEXT NOT NULL DEFAULT '',
|
||||
fingerprint TEXT NOT NULL DEFAULT '',
|
||||
confirmed_fingerprint TEXT NOT NULL DEFAULT '',
|
||||
confirmed_values_json TEXT NOT NULL DEFAULT '{}',
|
||||
confirmed_peers_json TEXT NOT NULL DEFAULT '[]',
|
||||
detail_json TEXT NOT NULL DEFAULT '{}',
|
||||
observed_at TEXT NOT NULL,
|
||||
confirmed_at TEXT NOT NULL,
|
||||
created_at TEXT NOT NULL,
|
||||
updated_at TEXT NOT NULL
|
||||
)`,
|
||||
'CREATE INDEX idx_device_identities_status ON device_identities(status, instance_id)',
|
||||
'CREATE INDEX idx_device_identities_imei ON device_identities(imei, instance_id)',
|
||||
],
|
||||
},
|
||||
{
|
||||
id: 18,
|
||||
name: 'console-auth-scrypt-params',
|
||||
statements: [
|
||||
// Legacy hashes were derived with N=16384; new derivations use stronger
|
||||
// parameters. The column records the derivation per row; NULL means legacy
|
||||
// and is transparently upgraded after the next successful login.
|
||||
'ALTER TABLE console_auth_config ADD COLUMN password_kdf TEXT',
|
||||
],
|
||||
},
|
||||
];
|
||||
|
||||
const createMigrationsTable = `CREATE TABLE schema_migrations (
|
||||
|
||||
@@ -255,3 +255,67 @@ export const secretCleanupTasks = sqliteTable(
|
||||
},
|
||||
(table) => [index('idx_secret_cleanup_tasks_queued_at').on(table.queuedAt)],
|
||||
);
|
||||
|
||||
export const scheduledTasks = sqliteTable(
|
||||
'scheduled_tasks',
|
||||
{
|
||||
id: text('id').primaryKey(),
|
||||
name: text('name').notNull(),
|
||||
operationType: text('operation_type').notNull(),
|
||||
enabled: integer('enabled', { mode: 'boolean' }).notNull().default(true),
|
||||
version: integer('version').notNull().default(1),
|
||||
cronExpression: text('cron_expression').notNull(),
|
||||
timezone: text('timezone').notNull(),
|
||||
targetSelectorJson: text('target_selector_json').notNull(),
|
||||
smsSecretReference: text('sms_secret_reference'),
|
||||
smsRecipientCount: integer('sms_recipient_count'),
|
||||
effectiveStartAt: text('effective_start_at'),
|
||||
effectiveEndAt: text('effective_end_at'),
|
||||
misfirePolicy: text('misfire_policy').notNull(),
|
||||
overlapPolicy: text('overlap_policy').notNull(),
|
||||
retryPolicyJson: text('retry_policy_json').notNull(),
|
||||
nextDueAt: text('next_due_at'),
|
||||
lastEvaluatedAt: text('last_evaluated_at'),
|
||||
createdBy: text('created_by').notNull(),
|
||||
updatedBy: text('updated_by').notNull(),
|
||||
createdAt: text('created_at').notNull(),
|
||||
updatedAt: text('updated_at').notNull(),
|
||||
deletedAt: text('deleted_at'),
|
||||
},
|
||||
(table) => [
|
||||
check('scheduled_tasks_enabled_check', sql`${table.enabled} in (0, 1)`),
|
||||
check('scheduled_tasks_version_check', sql`${table.version} > 0`),
|
||||
index('idx_scheduled_tasks_enabled_next_due').on(table.enabled, table.nextDueAt),
|
||||
],
|
||||
);
|
||||
|
||||
export const scheduledRuns = sqliteTable(
|
||||
'scheduled_runs',
|
||||
{
|
||||
id: text('id').primaryKey(),
|
||||
scheduledTaskId: text('scheduled_task_id')
|
||||
.notNull()
|
||||
.references(() => scheduledTasks.id, { onDelete: 'restrict' }),
|
||||
scheduleVersion: integer('schedule_version').notNull(),
|
||||
taskSnapshotJson: text('task_snapshot_json').notNull(),
|
||||
dueAt: text('due_at').notNull(),
|
||||
claimedAt: text('claimed_at').notNull(),
|
||||
startedAt: text('started_at'),
|
||||
finishedAt: text('finished_at'),
|
||||
targetSnapshotJson: text('target_snapshot_json').notNull(),
|
||||
outcome: text('outcome'),
|
||||
reason: text('reason'),
|
||||
jobIdsJson: text('job_ids_json').notNull().default('[]'),
|
||||
triggerSource: text('trigger_source').notNull(),
|
||||
attempt: integer('attempt').notNull().default(1),
|
||||
},
|
||||
(table) => [
|
||||
unique('scheduled_runs_occurrence_unique').on(
|
||||
table.scheduledTaskId,
|
||||
table.scheduleVersion,
|
||||
table.dueAt,
|
||||
table.triggerSource,
|
||||
),
|
||||
index('idx_scheduled_runs_task_due').on(table.scheduledTaskId, desc(table.dueAt)),
|
||||
],
|
||||
);
|
||||
|
||||
@@ -0,0 +1,174 @@
|
||||
import { randomUUID } from 'node:crypto';
|
||||
import { chmod, mkdir, open, readFile, rename, rm, stat } from 'node:fs/promises';
|
||||
import { dirname, isAbsolute } from 'node:path';
|
||||
import {
|
||||
accountFor,
|
||||
parseSecretAccount,
|
||||
validateKey,
|
||||
SecretReferenceError,
|
||||
} from './secret-reference.js';
|
||||
import type { SecretKey, SecretStore } from './secret-store.js';
|
||||
|
||||
const PROVIDER = 'secret-file';
|
||||
const SERVICE = 'multi-simadmin';
|
||||
const MAX_SECRET_BYTES = 16_384;
|
||||
const MAX_STORE_BYTES = 4 * 1024 * 1024;
|
||||
|
||||
export type FileSecretStoreErrorCode =
|
||||
| 'INVALID_KEY'
|
||||
| 'INVALID_SECRET'
|
||||
| 'INVALID_REFERENCE'
|
||||
| 'INVALID_STORE_PATH'
|
||||
| 'SIZE_LIMIT'
|
||||
| 'PAYLOAD_INVALID';
|
||||
|
||||
export class FileSecretStoreError extends Error {
|
||||
constructor(
|
||||
readonly code: FileSecretStoreErrorCode,
|
||||
message: string,
|
||||
) {
|
||||
super(message);
|
||||
this.name = 'FileSecretStoreError';
|
||||
}
|
||||
}
|
||||
|
||||
function invalid(code: FileSecretStoreErrorCode, message: string): never {
|
||||
throw new FileSecretStoreError(code, message);
|
||||
}
|
||||
|
||||
/**
|
||||
* POSIX filesystem-backed secret store for hosts without a system keyring
|
||||
* (e.g. Linux servers). Secrets live in a single 0600 JSON map under the data
|
||||
* root, replaced atomically via temp file + rename, with writes serialized
|
||||
* in-process.
|
||||
*/
|
||||
export class FileSecretStore implements SecretStore {
|
||||
readonly provider = PROVIDER;
|
||||
|
||||
private queue: Promise<unknown> = Promise.resolve();
|
||||
|
||||
constructor(readonly filePath: string) {
|
||||
if (typeof filePath !== 'string' || filePath.length === 0 || !isAbsolute(filePath))
|
||||
invalid('INVALID_STORE_PATH', 'Secret file path must be absolute');
|
||||
}
|
||||
|
||||
async set(key: SecretKey, value: string): Promise<string> {
|
||||
if (
|
||||
typeof value !== 'string' ||
|
||||
value.length === 0 ||
|
||||
/[\0\r\n]/.test(value) ||
|
||||
Buffer.byteLength(value, 'utf8') > MAX_SECRET_BYTES
|
||||
) {
|
||||
invalid('INVALID_SECRET', 'Secret is not valid for file storage');
|
||||
}
|
||||
validateKey(key);
|
||||
const account = accountFor(key);
|
||||
await this.enqueue(() => this.withStore((store) => ({ ...store, [account]: value })));
|
||||
return `secret-file://${SERVICE}/${account}`;
|
||||
}
|
||||
|
||||
async get(reference: string): Promise<string | undefined> {
|
||||
const { account } = this.parse(reference);
|
||||
const store = await this.readStore();
|
||||
return store[account];
|
||||
}
|
||||
|
||||
async delete(reference: string): Promise<boolean> {
|
||||
const { account } = this.parse(reference);
|
||||
let removed = false;
|
||||
await this.enqueue(() =>
|
||||
this.withStore((store) => {
|
||||
if (!(account in store)) return store;
|
||||
removed = true;
|
||||
const next = { ...store };
|
||||
delete next[account];
|
||||
return next;
|
||||
}),
|
||||
);
|
||||
return removed;
|
||||
}
|
||||
|
||||
private parse(reference: string): { account: string } {
|
||||
if (typeof reference !== 'string' || reference.length > 512)
|
||||
invalid('INVALID_REFERENCE', 'Secret file reference is invalid');
|
||||
const match = /^secret-file:\/\/multi-simadmin\/([A-Za-z0-9_-]+)$/.exec(reference);
|
||||
if (!match?.[1]) invalid('INVALID_REFERENCE', 'Secret file reference is invalid');
|
||||
try {
|
||||
parseSecretAccount(match[1], PROVIDER);
|
||||
} catch (error) {
|
||||
if (error instanceof SecretReferenceError)
|
||||
invalid('INVALID_REFERENCE', 'Secret file reference is invalid');
|
||||
throw error;
|
||||
}
|
||||
return { account: match[1] };
|
||||
}
|
||||
|
||||
private enqueue<T>(task: () => Promise<T>): Promise<T> {
|
||||
const run = this.queue.then(task, task);
|
||||
this.queue = run.catch(() => {});
|
||||
return run;
|
||||
}
|
||||
|
||||
private async readStore(): Promise<Record<string, string>> {
|
||||
let raw: string;
|
||||
try {
|
||||
raw = await readFile(this.filePath, 'utf8');
|
||||
} catch (error) {
|
||||
if ((error as NodeJS.ErrnoException).code === 'ENOENT') return {};
|
||||
throw error;
|
||||
}
|
||||
if (Buffer.byteLength(raw, 'utf8') > MAX_STORE_BYTES)
|
||||
invalid('SIZE_LIMIT', 'Secret file exceeds the size limit');
|
||||
let parsed: unknown;
|
||||
try {
|
||||
parsed = JSON.parse(raw);
|
||||
} catch {
|
||||
invalid('PAYLOAD_INVALID', 'Secret file payload is invalid');
|
||||
}
|
||||
if (!isPlainStringMap(parsed)) invalid('PAYLOAD_INVALID', 'Secret file payload is invalid');
|
||||
return parsed;
|
||||
}
|
||||
|
||||
/** Reads the current map, applies `mutate`, and durably replaces the file. */
|
||||
private async withStore(
|
||||
mutate: (store: Record<string, string>) => Record<string, string>,
|
||||
): Promise<void> {
|
||||
const current = await this.readStore();
|
||||
const next = mutate(current);
|
||||
const directory = dirname(this.filePath);
|
||||
await mkdir(directory, { recursive: true, mode: 0o700 });
|
||||
if (process.platform !== 'win32') {
|
||||
const info = await stat(directory);
|
||||
if (info.isDirectory() && (info.mode & 0o077) !== 0) await chmod(directory, 0o700);
|
||||
}
|
||||
const temporary = `${this.filePath}.${randomUUID()}.tmp`;
|
||||
const handle = await open(temporary, 'wx', 0o600);
|
||||
try {
|
||||
await handle.writeFile(`${JSON.stringify(next, null, 2)}\n`, 'utf8');
|
||||
// Windows FlushFileBuffers is skipped along with the directory fsync below;
|
||||
// durability anchoring is POSIX-only by design.
|
||||
if (process.platform !== 'win32') await handle.sync();
|
||||
} finally {
|
||||
await handle.close();
|
||||
}
|
||||
try {
|
||||
await rename(temporary, this.filePath);
|
||||
if (process.platform !== 'win32') {
|
||||
const directoryHandle = await open(directory, 'r');
|
||||
try {
|
||||
await directoryHandle.sync();
|
||||
} finally {
|
||||
await directoryHandle.close();
|
||||
}
|
||||
}
|
||||
} catch (error) {
|
||||
await rm(temporary, { force: true });
|
||||
throw error;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
function isPlainStringMap(value: unknown): value is Record<string, string> {
|
||||
if (typeof value !== 'object' || value === null || Array.isArray(value)) return false;
|
||||
return Object.values(value).every((entry) => typeof entry === 'string');
|
||||
}
|
||||
@@ -151,7 +151,23 @@ describe('MacOSKeychainSecretStore', () => {
|
||||
}
|
||||
});
|
||||
|
||||
it.each(['line\nbreak', 'line\rbreak', 'nul\0break', 'x'.repeat(4097)])(
|
||||
it('stores a maximum-sized scheduled SMS payload', async () => {
|
||||
const runner = new FakeRunner();
|
||||
const store = new MacOSKeychainSecretStore(runner);
|
||||
const payload = JSON.stringify({
|
||||
recipients: Array.from(
|
||||
{ length: 50 },
|
||||
(_, index) => `1380013${String(index).padStart(4, '0')}`,
|
||||
),
|
||||
content: '字'.repeat(2_000),
|
||||
});
|
||||
|
||||
await expect(
|
||||
store.set({ instanceId: 'task-1', purpose: 'scheduled-sms', slot: 'rotation-1' }, payload),
|
||||
).resolves.toMatch(/^keychain:/);
|
||||
});
|
||||
|
||||
it.each(['line\nbreak', 'line\rbreak', 'nul\0break', 'x'.repeat(16_385)])(
|
||||
'rejects non-line-safe or oversized secret input before invoking the runner',
|
||||
async (value) => {
|
||||
const runner = new FakeRunner();
|
||||
|
||||
@@ -1,11 +1,11 @@
|
||||
import { spawn } from 'node:child_process';
|
||||
import { accountFor, parseSecretAccount, SecretReferenceError } from './secret-reference.js';
|
||||
import type { SecretKey, SecretStore } from './secret-store.js';
|
||||
|
||||
const SECURITY_PATH = '/usr/bin/security';
|
||||
const SERVICE = 'multi-simadmin';
|
||||
const TIMEOUT_MS = 5_000;
|
||||
const MAX_STDOUT_BYTES = 65_536;
|
||||
const safeComponent = /^[A-Za-z0-9_.-]+$/;
|
||||
|
||||
export type SecretStoreErrorCode =
|
||||
| 'INVALID_KEY'
|
||||
@@ -105,43 +105,6 @@ function invalid(code: 'INVALID_KEY' | 'INVALID_REFERENCE', message: string): ne
|
||||
throw new SecretStoreError(code, message);
|
||||
}
|
||||
|
||||
function validateKey(key: SecretKey): void {
|
||||
if (
|
||||
typeof key.instanceId !== 'string' ||
|
||||
key.instanceId.length === 0 ||
|
||||
key.instanceId.length > 128 ||
|
||||
!safeComponent.test(key.instanceId)
|
||||
) {
|
||||
invalid('INVALID_KEY', 'Secret key instance id is invalid');
|
||||
}
|
||||
if (
|
||||
typeof key.purpose !== 'string' ||
|
||||
key.purpose.length === 0 ||
|
||||
key.purpose.length > 64 ||
|
||||
!safeComponent.test(key.purpose)
|
||||
) {
|
||||
invalid('INVALID_KEY', 'Secret key purpose is invalid');
|
||||
}
|
||||
if (
|
||||
key.slot !== undefined &&
|
||||
(typeof key.slot !== 'string' ||
|
||||
key.slot.length === 0 ||
|
||||
key.slot.length > 64 ||
|
||||
!safeComponent.test(key.slot))
|
||||
) {
|
||||
invalid('INVALID_KEY', 'Secret key slot is invalid');
|
||||
}
|
||||
}
|
||||
|
||||
function accountFor(key: SecretKey): string {
|
||||
validateKey(key);
|
||||
const components =
|
||||
key.slot === undefined
|
||||
? [key.instanceId, key.purpose]
|
||||
: [key.instanceId, key.purpose, key.slot];
|
||||
return Buffer.from(JSON.stringify(components), 'utf8').toString('base64url');
|
||||
}
|
||||
|
||||
function referenceFor(account: string): string {
|
||||
return `keychain://${SERVICE}/${account}`;
|
||||
}
|
||||
@@ -152,28 +115,26 @@ export function parseKeychainReference(reference: string): ParsedKeychainReferen
|
||||
const match = /^keychain:\/\/multi-simadmin\/([A-Za-z0-9_-]+)$/.exec(reference);
|
||||
if (!match?.[1]) invalid('INVALID_REFERENCE', 'Keychain reference is invalid');
|
||||
const account = match[1];
|
||||
let decoded: unknown;
|
||||
let parsed;
|
||||
try {
|
||||
decoded = JSON.parse(Buffer.from(account, 'base64url').toString('utf8'));
|
||||
} catch {
|
||||
invalid('INVALID_REFERENCE', 'Keychain reference is invalid');
|
||||
parsed = parseSecretAccount(account, 'macos-keychain');
|
||||
} catch (error) {
|
||||
if (error instanceof SecretReferenceError)
|
||||
invalid('INVALID_REFERENCE', 'Keychain reference is invalid');
|
||||
throw error;
|
||||
}
|
||||
if (!Array.isArray(decoded) || (decoded.length !== 2 && decoded.length !== 3))
|
||||
invalid('INVALID_REFERENCE', 'Keychain reference is invalid');
|
||||
const [instanceId, purpose, slot] = decoded;
|
||||
const key = { instanceId, purpose, ...(slot === undefined ? {} : { slot }) } as SecretKey;
|
||||
validateKey(key);
|
||||
if (accountFor(key) !== account) invalid('INVALID_REFERENCE', 'Keychain reference is invalid');
|
||||
return {
|
||||
service: SERVICE,
|
||||
account,
|
||||
instanceId: key.instanceId,
|
||||
purpose: key.purpose,
|
||||
...(key.slot === undefined ? {} : { slot: key.slot }),
|
||||
instanceId: parsed.instanceId,
|
||||
purpose: parsed.purpose,
|
||||
...(parsed.slot === undefined ? {} : { slot: parsed.slot }),
|
||||
};
|
||||
}
|
||||
|
||||
export class MacOSKeychainSecretStore implements SecretStore {
|
||||
readonly provider = 'macos-keychain';
|
||||
|
||||
constructor(private readonly runner: CommandRunner = new SpawnCommandRunner()) {}
|
||||
|
||||
async set(key: SecretKey, value: string): Promise<string> {
|
||||
@@ -181,7 +142,7 @@ export class MacOSKeychainSecretStore implements SecretStore {
|
||||
typeof value !== 'string' ||
|
||||
value.length === 0 ||
|
||||
/[\0\r\n]/.test(value) ||
|
||||
Buffer.byteLength(value, 'utf8') > 4096
|
||||
Buffer.byteLength(value, 'utf8') > 16_384
|
||||
) {
|
||||
throw new SecretStoreError('INVALID_SECRET', 'Secret is not valid for Keychain storage');
|
||||
}
|
||||
|
||||
@@ -0,0 +1,165 @@
|
||||
import { describe, expect, it } from 'vitest';
|
||||
import { mkdtemp, readFile, readdir, rm, stat, writeFile, chmod } from 'node:fs/promises';
|
||||
import { tmpdir } from 'node:os';
|
||||
import { join } from 'node:path';
|
||||
import { FileSecretStore, FileSecretStoreError } from './file-secret-store.js';
|
||||
import { SecretReferenceError } from './secret-reference.js';
|
||||
import {
|
||||
createDefaultSecretStore,
|
||||
defaultSecretFileMetadataCheck,
|
||||
resolveSecretBackend,
|
||||
SecretBackendError,
|
||||
} from './secret-backend.js';
|
||||
import { MacOSKeychainSecretStore } from './keychain-secret-store.js';
|
||||
|
||||
describe('FileSecretStore', () => {
|
||||
it('round-trips set/get/delete with stable secret-file references', async () => {
|
||||
const directory = await mkdtemp(join(tmpdir(), 'msa-file-secrets-'));
|
||||
try {
|
||||
const store = new FileSecretStore(join(directory, 'secrets.json'));
|
||||
const reference = await store.set(
|
||||
{ instanceId: 'device-1', purpose: 'instance-password', slot: 'slot-1' },
|
||||
'pa$$word',
|
||||
);
|
||||
expect(reference).toMatch(/^secret-file:\/\/multi-simadmin\/[A-Za-z0-9_-]+$/);
|
||||
expect(await store.get(reference)).toBe('pa$$word');
|
||||
expect(await store.delete(reference)).toBe(true);
|
||||
expect(await store.get(reference)).toBeUndefined();
|
||||
expect(await store.delete(reference)).toBe(false);
|
||||
} finally {
|
||||
await rm(directory, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
it('rejects empty, framing, oversize, and malformed keys/values', async () => {
|
||||
const directory = await mkdtemp(join(tmpdir(), 'msa-file-secrets-'));
|
||||
try {
|
||||
const store = new FileSecretStore(join(directory, 'secrets.json'));
|
||||
await expect(store.set({ instanceId: 'x', purpose: 'p' }, '')).rejects.toThrow(
|
||||
FileSecretStoreError,
|
||||
);
|
||||
await expect(store.set({ instanceId: 'x', purpose: 'p' }, 'a\nb')).rejects.toThrow(
|
||||
FileSecretStoreError,
|
||||
);
|
||||
await expect(
|
||||
store.set({ instanceId: 'x', purpose: 'p' }, 'a'.repeat(16_385)),
|
||||
).rejects.toThrow(FileSecretStoreError);
|
||||
await expect(store.set({ instanceId: '', purpose: 'p' }, 'v')).rejects.toThrow(
|
||||
SecretReferenceError,
|
||||
);
|
||||
await expect(store.get('https://evil.example/one')).rejects.toThrow(FileSecretStoreError);
|
||||
await expect(store.get('secret-file://other-service/abc')).rejects.toThrow(
|
||||
FileSecretStoreError,
|
||||
);
|
||||
} finally {
|
||||
await rm(directory, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
it('keeps the map private (0600) and leaves no temporary files behind', async () => {
|
||||
const directory = await mkdtemp(join(tmpdir(), 'msa-file-secrets-'));
|
||||
try {
|
||||
const path = join(directory, 'nested', 'secrets.json');
|
||||
const store = new FileSecretStore(path);
|
||||
const reference = await store.set({ instanceId: 'x', purpose: 'p', slot: 's' }, 'secret');
|
||||
if (process.platform !== 'win32') expect((await stat(path)).mode & 0o777).toBe(0o600);
|
||||
expect(await readdir(join(directory, 'nested'))).toEqual(['secrets.json']);
|
||||
const raw = await readFile(path, 'utf8');
|
||||
expect(JSON.parse(raw)).toEqual({
|
||||
[reference.slice('secret-file://multi-simadmin/'.length)]: 'secret',
|
||||
});
|
||||
expect(await store.get(reference)).toBe('secret');
|
||||
} finally {
|
||||
await rm(directory, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
it('serializes concurrent writes so every mutation lands', async () => {
|
||||
const directory = await mkdtemp(join(tmpdir(), 'msa-file-secrets-'));
|
||||
try {
|
||||
const store = new FileSecretStore(join(directory, 'secrets.json'));
|
||||
const references = await Promise.all(
|
||||
Array.from({ length: 25 }, (_, index) =>
|
||||
store.set({ instanceId: 'x', purpose: `p${index}`, slot: 's' }, `v${index}`),
|
||||
),
|
||||
);
|
||||
for (const [index, reference] of references.entries())
|
||||
expect(await store.get(reference)).toBe(`v${index}`);
|
||||
} finally {
|
||||
await rm(directory, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
it('refuses a corrupted or non-map payload instead of failing open', async () => {
|
||||
const directory = await mkdtemp(join(tmpdir(), 'msa-file-secrets-'));
|
||||
try {
|
||||
const path = join(directory, 'secrets.json');
|
||||
await writeFile(path, 'not json', 'utf8');
|
||||
const store = new FileSecretStore(path);
|
||||
await expect(store.get('secret-file://multi-simadmin/abc')).rejects.toThrow(
|
||||
FileSecretStoreError,
|
||||
);
|
||||
await writeFile(path, '{"account": 42}', 'utf8');
|
||||
await expect(store.get('secret-file://multi-simadmin/abc')).rejects.toThrow(
|
||||
FileSecretStoreError,
|
||||
);
|
||||
expect(() => new FileSecretStore('relative/secrets.json')).toThrow(FileSecretStoreError);
|
||||
} finally {
|
||||
await rm(directory, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
describe('secret backend selection', () => {
|
||||
it('defaults to the keychain on darwin and the file store elsewhere', () => {
|
||||
expect(resolveSecretBackend({}, 'darwin')).toBe('macos-keychain');
|
||||
expect(resolveSecretBackend({}, 'linux')).toBe('secret-file');
|
||||
expect(resolveSecretBackend({ MULTI_SIMADMIN_SECRET_BACKEND: 'secret-file' }, 'darwin')).toBe(
|
||||
'secret-file',
|
||||
);
|
||||
expect(() =>
|
||||
resolveSecretBackend({ MULTI_SIMADMIN_SECRET_BACKEND: 'vault' }, 'darwin'),
|
||||
).toThrow(SecretBackendError);
|
||||
});
|
||||
|
||||
it('builds the matching store and refuses keychain off macOS', () => {
|
||||
const directory = join(tmpdir(), 'msa-backend-selection');
|
||||
expect(
|
||||
createDefaultSecretStore({
|
||||
env: {},
|
||||
secretFilePath: join(directory, 's.json'),
|
||||
platform: 'linux',
|
||||
}),
|
||||
).toBeInstanceOf(FileSecretStore);
|
||||
expect(
|
||||
createDefaultSecretStore({
|
||||
env: { MULTI_SIMADMIN_SECRET_BACKEND: 'macos-keychain' },
|
||||
secretFilePath: join(directory, 's.json'),
|
||||
platform: 'darwin',
|
||||
}),
|
||||
).toBeInstanceOf(MacOSKeychainSecretStore);
|
||||
expect(() =>
|
||||
createDefaultSecretStore({
|
||||
env: { MULTI_SIMADMIN_SECRET_BACKEND: 'macos-keychain' },
|
||||
secretFilePath: join(directory, 's.json'),
|
||||
platform: 'linux',
|
||||
}),
|
||||
).toThrow(SecretBackendError);
|
||||
});
|
||||
|
||||
it('file metadata probe accepts a missing or private file and rejects a public one', async () => {
|
||||
const directory = await mkdtemp(join(tmpdir(), 'msa-file-secrets-meta-'));
|
||||
try {
|
||||
const path = join(directory, 'secrets.json');
|
||||
expect(await defaultSecretFileMetadataCheck(path)).toBe(true);
|
||||
await writeFile(path, '{}\n', { mode: 0o600 });
|
||||
expect(await defaultSecretFileMetadataCheck(path)).toBe(true);
|
||||
if (process.platform !== 'win32') {
|
||||
await chmod(path, 0o644);
|
||||
expect(await defaultSecretFileMetadataCheck(path)).toBe(false);
|
||||
}
|
||||
} finally {
|
||||
await rm(directory, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
});
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user