Docs · Self-healing loops
docs/SELF-HEALING-LOOPS.md in the ZoeyOS source. Last verified 2026-10-08. State current, fixed against the running house on 2026-10-08. The nightly jobs run as one chain keyed off the owner's off-hours, not wall-clock timers; the table below is that chain as the schedule file gives it.Maintenance and architecture: self-healing, self-improving
The clocks that keep the box from rotting, and the research loop that turns a thin night into a card instead of a wish.
Said plainly
The house does not wait for a sitting to notice a red unit, a drifted pack, a green upgrade, or a world change. Steward and Scout write first-person, dated notes. Nightly maintenance applies the green set with smoke tests and rollback. Pack drift is a report. The Janitor samples every five minutes. Sunday Ada closes the week only with KPI-tied actions, or the week is a pass. The owner does not pick versions.
Self-heal is the clocks. Self-improve is research, then improvement cards, then the same QC stack as any other card. Loops, not a smarter 27B.
The clock, as it runs today
Jobs are keyed off the owner's off-hours window, not the wall clock. Each fires once per window, in a chain; a job that needs more time than the window has left waits for the next one.
| +2 min | day close: archive done cards older than 36 hours whose tree is closed, then garbage-collect the board |
|---|---|
| +5 min | nightly maintenance and security (below) |
| then | outcomes and lessons; pack drift check (report only); the evening collect of sources (no model) |
| then | team research (Scout) → team eval (every role scores) → the filer (one card per agreed idea) → improvement outcomes |
| then | architect logs → world note → stack note (Scout and Steward) |
| close −30 | dream night (memory consolidation) |
| close −15 | the morning brief |
| close −10 | the House Daily, the one report, built from every job's own status line |
Watched routines on their own intervals: Janitor 5 min · hold sweep and board gate 2 min · kanban loop 20 min · keeper 2 h · the restart window 5 min · steward sweep hourly · Zoey's decide watch 10 min · the Wall tab probe 2 min · the beam-mic probe 1 min. Still on the wall clock: the research catch-up at 02:25, the monthly source scan, Sunday's ops-efficiency close.
Nightly maintenance: the land loop
- Freeze each environment. Classify every package: never, risk, or green candidate.
- Drain the green pins only, in their own prefix, no dependency forks.
- Check consistency; isolate conflicts; smoke test and canary; roll the changed pins back if red.
- Hermes doctor and a security audit (OSV) over the agent runtime, plugins and MCP servers.
- File leftover risk as mitigation cards to Ada; the ledger suppresses repeats.
- Empty-room video references (refused if the room is occupied or dark).
- The test battery: pytest plus synthetic HTTP, no sound, no camera move, no occupancy flip.
| green pin | the nightly trail lands it, with smoke and rollback |
|---|---|
| major / risk | a research card stays open; a land card runs after a specialist's go, with smoke and canary |
| files, packs, units | Bash or Steward stage; Vera's production gate applies |
| render engine upgrades | a Render window, smoke, Vera's gate, then home |
Health green is a probe, not a land. Pending majors sit in a maintenance window, not a silent apply. An environment with no declared smoke test is report-only. An environment being rebuilt takes no in-place change; an app environment moves by rebuild beside the old one, checked off-GPU, flipped by symlink, the old kept fourteen days.
Architecture: watch the stack, then file work
Scout writes. Steward files. Ada dispatches. Vera still returns a miss.
| World | Scout; sourced, assumptions labelled, an honest empty is a pass |
|---|---|
| Stack | Scout; two or three pieces a day, every piece touched within seven days |
| Spike | Steward, at most one after the stack note; never on the production engines |
| Collect | no model; the bundle the notes read |
A try is an opportunity, not a design commission, until Zoey or the owner greens it.
Keep-alive: the box stays up
| Janitor, 5 min | samples units, GPU, disk, ports. One restart per cooldown on a red non-GPU unit; still red, a card. A crash-loop signature is a card. Never restarts the engines or the voice (occupancy), never touches a parked hold, never resurrects a retired unit. |
|---|---|
| Weekly janitor | aggregates the week's samples into a production review card for Steward |
| Pack drift | diffs committed agent packs against the live ones. A commit is not a deploy; drift is a report. |
| Keeper | reopens the ear, restarts the lanes it owns; never the GPU tenants |
| Restart window | a restart is owed, not fired: the window files one card in the owner's off-hours, Zoey rules go, wait, later, park or hands, and the executor restarts in tier order with post-checks, re-asking before every unit |
Parked probes in Beast or Render are expected-down, not red. The render engine up in normal occupancy is a leak. The Janitor stays up in every occupancy.
What we are not claiming
- Not a closed loop on majors. A pending major waits for a sitting or a land card. The night will not surprise-install one.
- Not "research is thick every night". Thin sources are a pass. Nothing invents a source.
- Not the owner as approver. The software next hop is Vera (files) or a coder (environments). "Needs a human" is a body in the chair.