Files

T

Codex 570edf6bb2 feat: rebuild warm operations workbench

2026-07-30 14:49:33 +08:00

20 KiB

Raw Blame History

Warm Operations Workbench Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Deliver a warm, compact operations UI and a durable Beijing-time automation center for service restarts, system reboots, and SMS delivery across fixed or tag-selected instances.

Architecture: Add strict shared contracts, a versioned SQLite schedule store, a Cron/time service, and a lifecycle-managed coordinator that claims immutable run occurrences before dispatch. Reuse the existing instance, Job, Audit, secret-store, and secure-operation boundaries, then expose the behavior through authenticated Fastify routes and a React Automation workspace. The frontend keeps Jobs and Audit as secondary Automation tabs while Fleet becomes a compact responsive node grid with opt-in batch selection.

Tech Stack: TypeScript 5.9, React 19, Fastify 5, SQLite/better-sqlite3, Drizzle schema declarations, cron-parser, Vitest, Testing Library, Vite, animal-island-ui 1.3.0.

Global Constraints

Timezone is fixed to Asia/Shanghai; APIs reject any other timezone and UI copy displays Beijing time (UTC+8).
Cron is standard five-field syntax; seconds are unsupported and no minimum interval is enforced.
Misfire defaults to skip; catch-up runs at most once. Overlap defaults to skip; queueing is bounded to one occurrence.
Dynamic selectors resolve on every claim and support any or all; every run persists an immutable target snapshot and schedule version.
System reboot defaults to zero retries. All retry counts and intervals are bounded by contracts.
SMS recipients and content live only in the existing secret store. SQLite, logs, Audit, and HTTP reads expose references or redacted summaries only.
Deleting a schedule is soft deletion for configuration and never removes Jobs, Audit, or scheduled-run history.
Preserve existing dirty-worktree changes. Do not commit overlapping user-owned modifications; use test checkpoints instead.
UI uses cream solid surfaces, warm brown text, mint actions, thin warm borders, 14-16px panel radii, no floating card shadows, and restrained 120-200ms feedback.

Task 1: Automation Contracts

Files:

Create: packages/contracts/src/automation.ts
Create: packages/contracts/src/automation.test.ts
Modify: packages/contracts/src/index.ts

Interfaces:

Produces: ScheduledTask, ScheduledRun, CreateScheduledTaskRequest, UpdateScheduledTaskRequest, ScheduleTargetSelector, CronPreview, policy and outcome constants.
Produces: parseCreateScheduledTaskRequest(value: unknown) and parseUpdateScheduledTaskRequest(value: unknown) with exact-key, length, enum, and numeric-bound validation.
Step 1: Write the failing contract tests

expect(parseCreateScheduledTaskRequest(validRestart)).toMatchObject({
  timezone: 'Asia/Shanghai', misfirePolicy: 'skip', overlapPolicy: 'skip'
});
expect(() => parseCreateScheduledTaskRequest({ ...validRestart, timezone: 'UTC' })).toThrow();
expect(() => parseCreateScheduledTaskRequest({ ...validRestart, cronExpression: '* * * * * *' })).toThrow();
expect(() => parseCreateScheduledTaskRequest({ ...validSms, sms: { recipients: [], content: '' } })).toThrow();

Step 2: Verify RED

Run: corepack pnpm vitest run packages/contracts/src/automation.test.ts

Expected: FAIL because automation.ts and parsers do not exist.

Step 3: Implement strict contracts and parsers

Use literal unions for restart-service | reboot-system | send-sms, fixed | tags, skip | catch-up-once, skip | queue-once, and the six approved run outcomes. Bound names to 120 characters, recipients to 50, fixed IDs/tags to 200, SMS content to 2,000 characters, retries to 10, and retry interval to 86,400 seconds.

Step 4: Verify GREEN

Run: corepack pnpm vitest run packages/contracts/src/automation.test.ts packages/contracts/src/index.test.ts

Expected: PASS.

Task 2: Scheduling Persistence

Files:

Modify: apps/api/src/infrastructure/database/migrations.ts
Modify: apps/api/src/infrastructure/database/schema.ts
Modify: apps/api/src/infrastructure/database/database.test.ts
Create: apps/api/src/application/automation/scheduled-task-repository.ts
Create: apps/api/src/application/automation/scheduled-task-repository.test.ts

Interfaces:

Consumes: contracts from Task 1.
Produces: ScheduledTaskRepository.create/get/list/update/setEnabled/softDelete, claimOccurrence, startRun, and finishRun.
Produces: migration 8 tables scheduled_tasks and scheduled_runs, with unique (scheduled_task_id, schedule_version, due_at, trigger_source) claim identity.
Step 1: Write failing migration and repository tests

expect(tableNames(db)).toContain('scheduled_tasks');
expect(tableNames(db)).toContain('scheduled_runs');
expect(repository.claimOccurrence(task.id, task.version, dueAt, 'scheduled', targets)).not.toBeNull();
expect(repository.claimOccurrence(task.id, task.version, dueAt, 'scheduled', targets)).toBeNull();
expect(repository.getRun(firstRun.id)?.targetSnapshot).toEqual(['instance-a', 'instance-b']);

Step 2: Verify RED

Run: corepack pnpm vitest run apps/api/src/application/automation/scheduled-task-repository.test.ts apps/api/src/infrastructure/database/database.test.ts -t "scheduled"

Expected: FAIL because migration 8 and repository are absent.

Step 3: Add migration 8, Drizzle declarations, and repository

Store selectors and policies as validated JSON, timestamps as UTC ISO strings, the fixed timezone marker, a nullable SMS secret reference, deleted_at, optimistic version, and immutable run snapshots. Wrap claim insert and overlap inspection in one better-sqlite3 transaction.

Step 4: Verify GREEN

Run: corepack pnpm vitest run apps/api/src/application/automation/scheduled-task-repository.test.ts apps/api/src/infrastructure/database/database.test.ts -t "scheduled|business tables"

Expected: PASS.

Task 3: Cron and Target Resolution

Files:

Modify: apps/api/package.json
Modify: pnpm-lock.yaml
Create: apps/api/src/application/automation/schedule-time.ts
Create: apps/api/src/application/automation/schedule-time.test.ts
Create: apps/api/src/application/automation/target-resolver.ts
Create: apps/api/src/application/automation/target-resolver.test.ts

Interfaces:

Produces: previewCron(expression, count, from?), nextOccurrence(expression, after), and reconcileOccurrence(task, now) using Asia/Shanghai.
Produces: resolveTargets(db, selector): readonly ResolvedTarget[] returning enabled instances with revisions and capability states.
Step 1: Add failing time and selector tests

expect(previewCron('0 9 * * *', 2, new Date('2026-07-30T00:30:00Z'))).toEqual([
  '2026-07-30T01:00:00.000Z', '2026-07-31T01:00:00.000Z'
]);
expect(resolveTargets(db, { mode: 'tags', match: 'all', tags: ['lab', 'east'] }).map(x => x.id)).toEqual(['a']);
expect(resolveTargets(db, { mode: 'tags', match: 'any', tags: ['lab', 'east'] }).map(x => x.id)).toEqual(['a', 'b']);

Step 2: Verify RED

Run: corepack pnpm vitest run apps/api/src/application/automation/schedule-time.test.ts apps/api/src/application/automation/target-resolver.test.ts

Expected: FAIL because services are absent.

Step 3: Install and implement

Run: corepack pnpm --filter @multi-simadmin/api add cron-parser

Parse with explicit tz: 'Asia/Shanghai', reject seconds fields before invoking the library, cap preview count at 10, and use parameterized SQL with GROUP BY/HAVING for tag matching.

Step 4: Verify GREEN

Run: corepack pnpm vitest run apps/api/src/application/automation/schedule-time.test.ts apps/api/src/application/automation/target-resolver.test.ts

Expected: PASS, including Beijing day-boundary fixtures.

Task 4: Schedule Service and SMS Secrets

Files:

Create: apps/api/src/application/automation/scheduled-task-service.ts
Create: apps/api/src/application/automation/scheduled-task-service.test.ts
Modify: apps/api/src/infrastructure/secrets/secret-store.ts

Interfaces:

Produces: ScheduledTaskService.create/update/pause/resume/remove/duplicate/preview/runNow/list/listRuns.
Uses a single JSON SMS secret payload { recipients: string[]; content: string } under a schedule-scoped SecretKey; reads never return the payload.
Step 1: Write failing lifecycle and redaction tests

const created = await service.create(actor, smsRequest);
expect(created.sms).toEqual({ configured: true, recipientCount: 2 });
expect(JSON.stringify(repository.get(created.id))).not.toContain('hello');
expect(secretStore.getCalls).toHaveLength(0);
expect((await service.update(actor, created.id, created.version, changedTargets)).version).toBe(2);

Step 2: Verify RED

Run: corepack pnpm vitest run apps/api/src/application/automation/scheduled-task-service.test.ts

Expected: FAIL because the service is absent.

Step 3: Implement lifecycle and secret compensation

Write the secret before the task transaction, compensate by deleting it when persistence fails, rotate and cleanup old references after successful sensitive edits, require optimistic versions, and preserve runs on soft delete.

Step 4: Verify GREEN

Run: corepack pnpm vitest run apps/api/src/application/automation/scheduled-task-service.test.ts

Expected: PASS with no plaintext SMS values in serialized task or errors.

Task 5: Scheduled Dispatch and Coordinator

Files:

Create: apps/api/src/application/automation/scheduled-operation-dispatcher.ts
Create: apps/api/src/application/automation/scheduled-operation-dispatcher.test.ts
Create: apps/api/src/application/automation/scheduler-coordinator.ts
Create: apps/api/src/application/automation/scheduler-coordinator.test.ts
Modify: apps/api/src/application/messages/instance-message-service.ts
Modify: apps/api/src/application/messages/instance-message-service.test.ts
Modify: apps/api/src/application/operations/secure-operation-execution.ts
Modify: apps/api/src/application/operations/secure-operation-execution.test.ts

Interfaces:

Produces: ScheduledOperationDispatcher.dispatch(run, task, targets) and SchedulerCoordinator.start/stop/tick.
Extends message execution with a Job/Audit-recorded scheduled SMS method whose body digest is computed before dispatch and whose summaries contain only redacted fields.
Step 1: Write failing aggregate execution tests

expect(await dispatch(twoTargetsOneFailure)).toMatchObject({ outcome: 'partially-succeeded' });
expect(await dispatch(noTargets)).toMatchObject({ outcome: 'no-targets' });
expect(await dispatch(missingSmsSecret)).toMatchObject({ outcome: 'needs-attention' });
expect(auditRows(db).every(row => !JSON.stringify(row).includes(phoneOrContent))).toBe(true);

Step 2: Verify RED

Run: corepack pnpm vitest run apps/api/src/application/automation/scheduled-operation-dispatcher.test.ts apps/api/src/application/automation/scheduler-coordinator.test.ts

Expected: FAIL because dispatcher and coordinator are absent.

Step 3: Implement dispatch and coordinator lifecycle

Dispatch each target independently, create correlated Jobs, aggregate outcomes, bound retries, default reboot retries to zero, reconcile at startup, claim before dispatch, skip or queue one overlap, and expose injected clock/timer dependencies for deterministic tests. Never persist interactive confirmation tokens.

Step 4: Verify GREEN

Run: corepack pnpm vitest run apps/api/src/application/automation apps/api/src/application/messages/instance-message-service.test.ts apps/api/src/application/operations/secure-operation-execution.test.ts

Expected: PASS for duplicates, misfire, overlap, retry, partial success, and redaction.

Task 6: Automation HTTP API

Files:

Create: apps/api/src/interface/http/automation-routes.ts
Create: apps/api/src/interface/http/automation-routes.test.ts
Modify: apps/api/src/control-plane.ts
Modify: apps/api/src/control-plane.test.ts
Modify: apps/api/src/production-control-plane.ts
Modify: apps/api/src/index.ts

Interfaces:

Produces authenticated /api/v1/automation/schedules, /:id, /:id/state, /:id/run, /cron/preview, and /runs routes.
Returns RFC 9457-style sanitized problems and ETags based on schedule version.
Step 1: Write failing route tests

expect((await app.inject({ method: 'POST', url: '/api/v1/automation/cron/preview', payload: { cronExpression: '0 9 * * *' } })).statusCode).toBe(200);
expect(create.statusCode).toBe(201);
expect(updateWithoutIfMatch.statusCode).toBe(428);
expect(await getJson(create)).not.toHaveProperty('sms.content');
expect(deleteResponse.statusCode).toBe(204);
expect(runHistoryAfterDelete.items).toHaveLength(1);

Step 2: Verify RED

Run: corepack pnpm vitest run apps/api/src/interface/http/automation-routes.test.ts

Expected: FAIL with 404/unregistered routes.

Step 3: Implement routes and app lifecycle

Use exact-key body parsing, authenticated actor IDs, request IDs, If-Match optimistic updates, bounded pagination, and coordinator close hooks. Register routes with the same auth scope as Jobs and Audit.

Step 4: Verify GREEN

Run: corepack pnpm vitest run apps/api/src/interface/http/automation-routes.test.ts apps/api/src/control-plane.test.ts apps/api/src/production-control-plane.test.ts

Expected: PASS.

Task 7: Automation Web Workspace

Files:

Create: apps/web/src/automation/automation-api-data-source.ts
Create: apps/web/src/automation/automation-api-data-source.test.ts
Create: apps/web/src/automation/automation-page.tsx
Create: apps/web/src/automation/automation-page.test.tsx
Modify: apps/web/src/jobs/jobs-page.tsx
Modify: apps/web/src/audit/audit-page.tsx

Interfaces:

Produces: AutomationApiDataSource CRUD/preview/run methods and AutomationPage with schedules | runs | records tabs.
Consumes existing JobsPage, AuditPage, instance summaries, and new automation contracts.
Step 1: Write failing UI behavior tests

expect(screen.getByRole('tab', { name: 'Schedules' })).toHaveAttribute('aria-selected', 'true');
await user.click(screen.getByRole('button', { name: 'Create schedule' }));
expect(screen.getByRole('dialog', { name: 'Create schedule' })).toBeVisible();
await user.selectOptions(screen.getByLabelText('Target mode'), 'tags');
expect(screen.getByLabelText('Tag matching')).toBeVisible();
expect(screen.queryByLabelText('Timezone')).not.toBeInTheDocument();

Step 2: Verify RED

Run: corepack pnpm vitest run apps/web/src/automation/automation-page.test.tsx apps/web/src/automation/automation-api-data-source.test.ts

Expected: FAIL because Automation UI is absent.

Step 3: Implement dense list and progressive drawer

Use native semantic controls where the library API does not fit; use animal-island-ui Button, Switch, Tag, Tabs, and Drawer where accessible. Include five-run Beijing preview, high-frequency restart warning plus confirmation, fixed/tag targets, SMS fields, optional window, policies, retries, row menu actions, empty/error/loading states, and redacted summaries.

Step 4: Verify GREEN

Run: corepack pnpm vitest run apps/web/src/automation

Expected: PASS for create/edit/pause/run/delete and keyboard focus behavior.

Files:

Modify: apps/web/src/app-shell.tsx
Modify: apps/web/src/app-shell.integration.test.tsx
Modify: apps/web/src/main.tsx

Interfaces:

Routes /automation to AutomationPage; aliases /jobs to Runs and /audit to Operation records.
Primary navigation contains Nodes, Automation, and Settings only.
Step 1: Write failing navigation tests

expect(within(screen.getByRole('navigation', { name: 'Primary' })).getAllByRole('link')).toHaveLength(3);
expect(screen.queryByRole('link', { name: 'Audit' })).not.toBeInTheDocument();
history.pushState({}, '', '/jobs');
expect(await screen.findByRole('tab', { name: 'Runs' })).toHaveAttribute('aria-selected', 'true');

Step 2: Verify RED

Run: corepack pnpm vitest run apps/web/src/app-shell.integration.test.tsx

Expected: FAIL against the old primary navigation.

Step 3: Implement compact shell and aliases

Keep the header near 56px, preserve deep instance routes and settings subroutes, show connection state at the trailing edge, and use the existing icon component for navigation symbols.

Step 4: Verify GREEN

Run: corepack pnpm vitest run apps/web/src/app-shell.integration.test.tsx

Expected: PASS.

Task 9: Fleet Workbench

Files:

Modify: apps/web/src/fleet/fleet-page.tsx
Modify: apps/web/src/fleet/fleet-page.test.tsx
Modify: apps/web/src/styles.css

Interfaces:

Preserves existing Fleet data-source and operation-client props.
Adds explicit selectionMode UI state; checkboxes exist only while it is true.
Step 1: Write failing Fleet interaction tests

expect(screen.queryByRole('checkbox', { name: /select/i })).not.toBeInTheDocument();
await user.click(screen.getByRole('button', { name: 'Batch select' }));
expect(screen.getAllByRole('checkbox', { name: /select/i })).toHaveLength(visibleNodeCount + 1);
await user.click(screen.getByRole('button', { name: 'Exit batch selection' }));
expect(screen.queryByRole('checkbox', { name: /select/i })).not.toBeInTheDocument();

Step 2: Verify RED

Run: corepack pnpm vitest run apps/web/src/fleet/fleet-page.test.tsx

Expected: FAIL because selection controls are permanently visible or no batch mode exists.

Step 3: Implement the warm dense Fleet layout

Remove the persistent sidebar and decorative page title, combine counts/search/filter/sort/refresh/add/batch controls, use stable node panels with identity, endpoint/phone, CPU/memory/temperature, tags/recent SMS, restart/reboot actions, and responsive 3/2/1 columns. Define semantic tokens and reduced-motion/contrast/transparency media queries in styles.css.

Step 4: Verify GREEN

Run: corepack pnpm vitest run apps/web/src/fleet/fleet-page.test.tsx apps/web/src/app-shell.integration.test.tsx

Expected: PASS.

Task 10: Integrated Verification and Visual QA

Files:

Modify: scripts/real-browser-e2e.mjs
Test: relevant API/Web suites and production build output.

Interfaces:

Adds real-browser acceptance for Automation lifecycle, Fleet selection mode, route aliases, and responsive column/overflow checks.
Step 1: Extend browser assertions before UI fixes

Add literal checks for three primary navigation items, Automation tabs, drawer creation flow, selection-checkbox absence/presence, and viewport screenshots at 390, 768, 1024, and 1440 pixels.

Step 2: Run focused and broad verification

corepack pnpm vitest run packages/contracts/src/automation.test.ts apps/api/src/application/automation apps/api/src/interface/http/automation-routes.test.ts apps/web/src/automation apps/web/src/fleet/fleet-page.test.tsx apps/web/src/app-shell.integration.test.tsx
corepack pnpm typecheck
corepack pnpm lint
corepack pnpm format:check
corepack pnpm --filter @multi-simadmin/web build
corepack pnpm test:e2e:browser

Expected: all task-related checks PASS. Record known unrelated Windows baseline failures separately rather than suppressing them.

Step 3: Inspect visual output and correct defects

Verify solid cream panels, no card shadows, correct 3/2/1 columns, no horizontal overflow, no text overlap, visible focus, 44px touch controls, and usable reduced-motion/high-contrast variants. Re-run affected tests after each correction.

Step 4: Run final regression

Run: corepack pnpm test:unit

Expected: no new failures relative to the recorded 636-pass/24-fail Windows baseline, and every new or modified task-specific suite passes.

20 KiB Raw Blame History

Warm Operations Workbench Implementation Plan

Global Constraints

Task 1: Automation Contracts

Task 2: Scheduling Persistence

Task 3: Cron and Target Resolution

Task 4: Schedule Service and SMS Secrets

Task 5: Scheduled Dispatch and Coordinator

Task 6: Automation HTTP API

Task 7: Automation Web Workspace

Task 8: Shell and Navigation

Task 9: Fleet Workbench

Task 10: Integrated Verification and Visual QA

20 KiB

Raw Blame History