Agent Orchestrator

A successful Symphony run alone never closes a task. Symphony only sets erledigt when the run left verifiable evidence: a linked pull request, or a reasoned closure on the task (abschlussart plus abschlussnotiz). The closure note must come from this run: Symphony snapshots abschlussnotiz when it claims the task, and a reopened task's leftover note stays stale until the run rewrites the note itself — changing only abschlussart does not make it fresh evidence. Closed, unmerged pull requests do not count as completion evidence; open and merged pull requests remain eligible. A pull request URL found in the run output only counts after the forge lookup succeeds, the PR belongs to the task repository (no fork head), and its head branch or head commit matches the checkout the run worked in; an unverifiable, nonexistent, or unrelated PR URL is ignored (and not linked) and the task falls back to the closure-note check. A repository coding run that left neither evidence nor a result comment from the Symphony agent returns the task to offen and follows the normal retry budget. Any other run that exits cleanly without evidence (for example a coding run that only posted a result comment, or a non-coding answer) moves the task to the configured waiting status and adds a localized Symphony system comment asking for review; it is not retried automatically. The waiting status is read from task_statuses before it is written: warten when it still exists, otherwise the first configured status in the waiting category. The status configuration API therefore refuses to delete the last remaining waiting status, and equally refuses to move it to another category. Plan, quota, and usage-limit refusals (for example Upgrade your plan to continue) count as provider failures even when the Fleet harness exits successfully: the runner/provider domain gets a quota cooldown, the job tries the next compatible coding-model fallback on that runner, and otherwise the task stays offen with a backoff until the cooldown ends without spending its retry budget. Tracked PR follow-ups log the provider refusal instead of a missing result block. Every Symphony status write (claim, completion, retry, pause, regression observation) is recorded in aufgaben_status_events with actor symphony, the job ID, the source (symphony.completion.local, symphony.completion.fleet, symphony.retry, …), the reason, and the evidence. The completion write is a compare-and-set from in_bearbeitung, so a status change made during the run is not overwritten — but only a human decision may bypass the evidence gate. If the task now sits in erledigt and the audit trail credits that write to a non-human actor (agent, symphony, system, or unknown, for example the executing run closing its own task through aufgaben_move_status or direct SQL), Symphony replaces it with the evidence-gated status, records the overwritten status and actor as evidence, and keeps the attempt on the retry budget instead of booking a clean completion. See Aufgaben und Status.

Regression tasks whose description contains an exact session.*.failed production signature are not marked erledigt immediately after an implementation run. Symphony moves them to the configured waiting status (warten by default, see above) and persistently waits for the linked pull request to be merged. The production observation window starts at that merge timestamp, lasts 24 hours by default (SYMPHONY_REGRESSION_OBSERVATION_MS), and survives orchestrator restarts through the durable job record. Jobs with pending observations are protected from the 12-hour job TTL cleanup. Symphony only completes the task when the signature stays absent for the full post-merge window. A recurrence or an unmerged PR closure returns the task to offen. A human status change made during the run keeps that status and cancels the observation; the cancelled observation is persisted (with bounded retries) before the run releases the task. Because that snapshot write can still fail, reconciliation re-reads the current status owner from aufgaben_status_events on every tick: an observation whose task status is credited to a human actor is dropped instead of reconciled, and a failing audit lookup keeps the observation for the next tick rather than overwriting a possibly human-managed status. Each reconciliation status write is then a compare-and-set against exactly the status that guard observed: a status changed after the guard ran but before the forge lookup or the production-signature check finished makes the write affect no row, so the observation is kept for the next tick instead of overwriting the newer decision, and that tick's guard drops it once the audit trail credits a human.

What it does

Agent Orchestrator (agent-orchestrator) runs coding agents against Git repositories and monitors them from one place. It starts detached coding-agent jobs (codex, claude, clapilot-code), hosts interactive agent sessions backed by the native ClapilotAICore broker, and keeps a small compatibility fallback for older job paths. Clapilot-code is Clapilot's own in-process coding loop; Codex and Claude remain external harnesses. GitHub and GitLab repositories share the same repository picker and automation matrix. The runtime reviews pull requests or merge requests, implements issues on a fresh branch, replies to mentions, follows review feedback, and fixes default-branch CI failures.

Settings persistence and runtime actions

The runtime's save/retry, poll, and start/stop controls sit together at the top right of its settings card on web, iOS, and macOS. Configuration edits, including repository selections, automation toggles, prompts, and model defaults, save automatically after a short pause. Polling status does not replace an edited draft. Saving configuration does not start or stop the runtime; those actions still require their own control.

Stopping Symphony jobs backed by embedded native sessions waits for confirmed run termination before closing the session, cleaning its workspace, marking the job stopped, or releasing its retry claim. Pending cancellation or runtime errors leave the job active so cancellation can be retried safely.

Repository automation settings adapt to the available panel width: wide web panels show compact columns, while smaller panels wrap the labeled controls below each repository. Repository names remain visible without repeating integration names, descriptions, or repository URLs. Webhook addresses and copy actions remain available. The native iOS and macOS settings use adaptive repository rows as well. This presentation change reuses the existing configuration API and agent tools.

How to open / enable it

• Agent Orchestrator is a Developer-mode module: the web module and the Settings -> Agent Orchestrator surface are hidden while Developer mode is disabled (src/lib/module-store/developer-mode-modules.ts).
• With Developer mode enabled, open it from the module menu; the route is /modules/agent-orchestrator. GitHub automations and runtime settings live under Settings -> Agent Orchestrator (src/app/(app)/(einstellungen)/settings/agent-orchestrator/page.tsx).
• Manifest: bundled-modules/agent-orchestrator/module.json (slug agent-orchestrator, entry index.html, renderer react, icon bot). The React UI is src/components/modules/agent-orchestrator-module.tsx, hosted by src/app/(app)/modules/[slug]/page.tsx; bundled frontend assets remain in bundled-modules/agent-orchestrator/ but the app renders the module directly in the main React tree.
• Auth for the module API: web users authenticate with the clapilot_session cookie; legacy compatibility skills/scripts use a short-lived machine Bearer token from POST /api/auth/agent/system-token; remote runners use Hub HMAC auth with the shared CLAPILOT_HUB_SHARED_SECRET.

Key workflows

Start a detached coding job

Use jobs when the user wants a detached run such as "go implement this", "prepare a PR", or "review this repo".

• + New Agent opens a compact chat-style composer. With Local selected it creates an interactive workspace session; its footer contains attachment, repository, icon-prefixed model, optional speech-to-text, and send controls. The model menu combines available Codex, Claude, and Clapilot Code models and infers the harness from the selected model.
• When at least one online remote runner is available, the composer adds a runner selector between repository and model (web and Apple clients). Auto lets any online runner claim the work, while choosing a named runner pins it to that machine. A remote selection limits the model menu to the harnesses the eligible runners advertise (Codex, plus Clapilot Code on capable runners), disables attachments, and creates a decoupled coding job instead of an interactive session; the queued job remains pinned even if that runner later goes offline.
• The Apple client's Neuer Agent surface mirrors the same session-first contract and single-row footer. It hides the generic composer emoji and keyboard-hint controls so repository, runner, and model selection remain visible without a second selector row. On macOS and iPad the Neuer Agent composer opens inline in the right detail pane (like the web module's right pane) instead of a separate sheet or window; iPhone keeps the full-screen create view.
• Detached and remote jobs remain supported for the web New Agent composer, automations, GitHub observers, remote runners, API/tool calls, existing activity rows, and job follow-ups.
• Codex-backed detached jobs keep an internal linked orchestrator session, so the same background run can be resumed later with a follow-up prompt from the web module, the native Apple client, or agent tools. Plain CLI jobs without a linked session, including Claude CLI jobs, can also receive follow-up prompts; those run in the original job workspace and append their output to the same job log. Remote Codex jobs requeue follow-up prompts for the remote runner and reuse the remote job workspace.
• Follow-ups can be sent while a job is still running. Session-backed jobs queue the turn inside the linked session runtime; remote and plain CLI jobs store the prompt in a per-job pending queue (max 20, persisted in jobs.json) and return 202 with turn.status: "queued". The prompt appears in the transcript immediately, serialized jobs expose the pending count as queuedFollowUps, and queued entries run in order once the current run finishes with succeeded or failed (a user-initiated stop pauses the queue until the next explicit follow-up). The web composer stays enabled while running and shows a subtle "runs after the current run" indicator under the input.

Model selection

• provider=clapilot-code runs through Clapilot's in-process coding loop instead of the Codex app-server or Claude CLI. It reuses the native embedded_pi adapter internally. The web creation UI shows a model picker backed only by usable non-subscription entries in the configured Provider & Modelle catalog; session creation fails clearly when none are available instead of silently falling back.
• pi, embedded_pi, embedded-pi, and clapilot_code remain accepted input aliases for clapilot-code. New jobs, session metadata, and automation settings use the canonical clapilot-code provider value.
• The web Neuer Agent panel shows model pickers for claude and codex too, backed by GET /coding-models. The picker preselects the model a job would run with by default (runtime config override, then the provider catalog default). Options merge the configured agent_provider_configs catalog with the current CLI-native model lineup, so models the CLI accepts stay selectable even when the Clapilot catalog only lists a subset.
• The current Codex lineup includes gpt-6.1-sol, gpt-6-astra, gpt-6-sol, gpt-6-luna, gpt-5.6-sol, gpt-5.6-terra, gpt-5.6-luna, gpt-5.5, gpt-5.4, gpt-5.4-mini, and the subscription-only gpt-5.3-codex-spark. Spark is a current selectable model and is not normalized through the legacy GPT-5.3 fallback rule.
• Detached Claude jobs always resolve to a concrete model: an explicit request wins, then the Settings -> Agent Orchestrator Claude runtime model, then the configured Anthropic provider catalog default. The current built-in fallback is claude-opus-5; the resolved ref (for example claude-default/claude-opus-5) is stored on the job and shown on activity cards, matching interactive sessions.
• Local Symphony Claude coding jobs recover from an explicit Fable usage/quota refusal by trying claude-opus-5 once using the same Claude credentials, job, and checkout. When the CLI supplies a session ID, fallback resumes that session; if tools already ran and no session ID is available, automatic fallback stops to avoid replaying side effects. This policy does not change the saved repository model or apply to interactive sessions, other automation types, Codex Cloud, or Fleet runners. The effective model and transition are visible in existing job logs and model fields on web, iOS, macOS, and agent status reads. A quota refusal on Opus, or any failure after switching to Opus, moves the task to warten without scheduling another attempt. Resolve model availability and explicitly return the ticket to an active state to resume. A pending waiting-state write is checkpointed and blocks dispatch until the database write succeeds, including after a restart. Stopped jobs do not reopen their tasks or schedule retries.
• Claude CLI jobs run with --output-format stream-json --verbose --include-partial-messages, so the job log streams live assistant text and tool activity while the job runs, and the job's model field is corrected to the model the CLI actually reports in its init event. Claude tool use is stored as the same structured job-log items as Codex and Clapilot Code (command_execution, file_change, web_search, mcp_tool_call, dynamic_tool_call); tool results complete the matching item with output and failed/completed status, and partial assistant text streams to live SSE viewers as transient item.updated events while only the finished message is persisted. Older persisted [tool] … text lines keep rendering through the legacy fallback.
• The job follow-up composer (web terminal panel and Apple job detail) has a model picker limited to the running job's provider catalog, preselected to the model the job last ran with. The chosen model is sent as model on POST /jobs/{id}/follow-up; detached CLI follow-ups re-run the CLI with that model, linked-session follow-ups forward it as a per-turn model.
• The Codex app-server bridge resolves the Codex service tier for each model. Codex-auth OpenAI providers can set a per-model Codex-Tier (Global, Fast, Flex, Ultrafast, Aus) in ClapilotAICore settings; these values are stored as provider metadata under modelServiceTiers. The model picker reads Codex app-server model/list and uses the returned speed-tier metadata for the per-model tier menu. Models left on Global use CLAPILOT_AGENT_CODEX_SERVICE_TIER when it is set; otherwise the tier stays unset and Codex chooses.
• Agent Orchestrator runtime settings accept max and ultra reasoning in addition to the older effort values. Codex currently advertises max for the GPT-5.6 family and ultra for Sol and Terra; unsupported model/effort combinations remain subject to the Codex runtime's own validation.

Run interactive sessions

Use sessions when the user wants to stay in the same thread/workspace and continue iteratively:

• resumable repo conversations
• thread-bound Telegram/Slack/WhatsApp coding sessions
• live turn + terminal streaming
• fork/archive lifecycle
• Missing Codex provider threads are retired consistently during archive, resume, turn startup, failure notifications, and subscription-bridge recovery. The local session closes, its provider IDs and active bindings are removed, and diagnostics record an informational retirement instead of a session failure. Duplicate errors do not repeat retirement or affect a replacement thread. The subscription bridge retries the interrupted request once on a new session; other callers receive the original provider error. Explicit archive still cleans the managed workspace.

Interactive session adapters remain codex_app_server, acp_agent, and embedded_pi. provider=auto resolves to Codex unless an explicit adapter override is provided, provider=clapilot-code defaults to the internal adapter=embedded_pi, and an explicit provider=claude defaults to adapter=acp_agent. The acp_agent adapter remains scaffolded for local ACP-compatible sidecars/agents and is still experimental. The web + New Agent panel offers interactive execution for the production-ready Codex and Clapilot-code paths.

Coding mode in the personal chat

When Agent Orchestrator is installed (and visible through Developer mode), the personal chat surfaces gain a quiet coding mode instead of a separate screen:

• Web /chat and the floating right-side chat show the Clapilot mark in the header as a toggle. Clicking it switches to the "Clapilot Code" mark and the chat body swaps to coding mode; the choice is stored per browser (clapilot.chat.codingMode.v1) and both web surfaces flip together. The iOS and macOS apps show the same toggle (a plain icon without the circular button surface) at the trailing end of the chat top bar (macOS also gains a + button there for a new chat or coding session), stored per device.
• In coding mode the session chooser lists coding threads instead of chat sessions: interactive orchestrator sessions plus remote/cloud jobs, newest first, with the active ones marked. Jobs that are backed by a listed session and mirrored runtime chat sessions are hidden. + / "Neue Coding-Session" clears the active thread so the next message starts a new one.
• The composer drops the emoji control and adds a repository picker (GitHub/GitLab connections from POST /repos) and an execution-target picker: In Clapilot (interactive session on this instance), any online remote runner (or a specific one), Codex Cloud, or Claude Cloud. A model picker appears on the right except for Codex Cloud, which owns its model. Attachments are only offered for local sessions; remote and cloud jobs reject them. A dictation mic (browser speech-to-text on the web via src/lib/use-browser-dictation.ts; the native composer's dictation control on iOS/macOS) appends spoken text to the draft. The same dictation control is available in the module's own composers: the web + New Agent panel and every follow-up composer (session, job, remote Codex, runtime mirrors), and the native compose and reply composers.
• Sending without an active thread calls POST /sessions for local execution or POST /jobs for remote/cloud execution with the same payload the module composer uses. Follow-ups go to POST /sessions/{id}/turns or POST /jobs/{id}/follow-up. The transcript reuses the module's stream-block builders and consumes GET /sessions/{id}/stream or GET /jobs/{id}/stream while the thread is active; a thread header shows the status and an "In Coding Agents öffnen" link to the full module view (/modules/agent-orchestrator?session=… or ?job=…).
• Agent access is unchanged: the chat agent keeps using the existing agent_orchestrator_* tools for sessions and jobs; coding mode is a client-side view over the same module API and adds no new tool contract.
• Web: src/lib/chat-coding-mode.ts, src/lib/use-chat-coding-mode.ts, src/components/chat-coding-mode-panel.tsx, src/components/chat-coding-mode-toggle.tsx. Apple: clients/apple/ClapilotApple/Sources/Clapilot/Views/ChatCodingModeView.swift (controller + view) wired in ChatView.swift. Icon assets: public/clapilot-code-icon.png and the clapilot-code image set.

Monitor work in the module

The React module shows detached jobs, interactive sessions, and native runtime sessions from ClapilotAICore so admins can inspect normal Assistant chats, Clapilot-code coding sessions, and bridge runtime activity without switching to the dedicated Sessions diagnostics page.

The activity rail uses a canonical read-only thread projection over the existing job, orchestrator-session, native runtime-session, and remote-session records. The projection classifies each source as chat, coding, scheduled, system, or remote, preserves its original session/chat/job/external identifiers, and advertises capabilities such as conversation inspection, reply, diagnostics, workspace files, diffs, terminal, approvals, checkpoints, and Git actions. It does not migrate or replace stored sessions.

Normal web, group, and channel chats remain first-class entries in Sessions. Their persisted chatTranscript stays the authoritative conversation, while model runs, tool events, memory/context diagnostics, raw events, and metadata remain available in the detail inspector. Coding-only workspace capabilities are enabled only for repository/workspace-backed projections, so later Changes, Files, Terminal, Preview, and checkpoint panels cannot replace or hide ordinary chat history.

The Apple client applies the same projection on iPhone, iPad, and macOS. Selecting a native runtime chat loads its full persisted transcript instead of stopping at the list summary; supported chats can continue through the existing runtime-session turn endpoint, while fork, close, terminal, diff, checkpoint, and Git actions remain limited to session types that actually support them.

• The left rail is a dense, searchable activity navigator with counted filters for Coding Agents, Sessions, Alle, Remote Agents, and Analytics. Each row keeps category, provider, repository/session identity, title, latest output, resolved model, status, latest tool use, and update time scannable without competing with the selected-thread detail pane.
• Selected-thread headers lead with the human-readable session/channel title and retain the technical session id as secondary metadata. This keeps normal chat inspection understandable without removing diagnostic identity.
• Detached-job details use the same compact identity hierarchy as session details: task title, repository/model context, provider mark, status dot, and an optional stop action. A segmented detail control separates Conversation, Activity, Terminal, Changes, and Files; changed files and diffs never render inline in chat history. Do not add a Terminal/Prompt switch, a second "Current Terminal" title, explanatory subtitle, status badge, or always-visible job/workspace metadata row above it. The original task remains visible as a user message.
• Coding Jobs shows detached coding jobs plus repository-backed interactive and runtime sessions, so coding work stays grouped with repository context.
• Session Jobs bundles chat- and runtime-backed session work: normal Assistant sessions (runtimePath=native), native Clapilot-code coding sessions (runtimePath=embedded_pi), and native sessions whose active execution harness resolves to the Claude CLI Bridge. Hidden follow-up sessions that belong to detached coding jobs are intentionally excluded; they stay attached to their owning Coding Job detail view.
• Alle mixes session-backed work and detached coding jobs in one rail.
• Session overview requests are summary-only by default: they carry compact status/model/activity previews but omit per-session event and chat-history arrays. The web module refreshes visible overview lists every 30 seconds, skips hidden-tab refreshes, and suppresses overlapping requests. Full history is loaded only for the selected session.
• Remote Agents switches the rail to a remote-machine dropdown, then lists that machine's local Codex CLI and clapilot-code sessions from the latest runner heartbeat. Clapilot Code rows carry a harness label. Selecting a remote session opens its metadata and transcript once the runner has answered the detail request in a later heartbeat. A composer at the bottom creates a decoupled resume job pinned to that machine, so replies queue even while the runner is stale or offline and continue through the matching CLI harness when a compatible runner reconnects.
• Analytics opens the internal Agent Costs & Yield dashboard for repo-bound Orchestrator runs. The first MVP derives run/yield data from existing jobs and interactive sessions, including repo, issue/PR context when detectable, provider/model, status category, duration, and simple waste hints such as failed runs or completed runs without a detected PR. Cost and token fields are explicitly marked as unavailable until reliable provider usage data exists; this analytics view does not render raw logs.

Detail-pane behavior:

• Session-backed details preserve each data mode without changing the underlying records. Conversation is the default and shows the chronological user/assistant dialog plus compact inline tool and command rows (command_execution, file_change, mcp_tool_call, dynamic_tool_call, web_search) rendered with the same collapsible tool treatment as Activity — raw terminal output, turn markers, and other meta/system noise stay out of Conversation. Activity isolates tool calls, terminal output, model runs, and lifecycle events, and Details exposes identifiers, runtime metadata, and diagnostics. The same conversation contract applies to detached jobs and session-backed jobs. Local repository-backed coding sessions additionally expose capability-gated Terminal, Changes, and Files modes. Terminal isolates command cards and raw terminal deltas already emitted by the agent runtime; Changes shows the current branch, Git status, staged diff, and working-tree diff; Files shows a bounded Git-ignore-aware relative inventory. Normal chats never receive these workspace modes.
• Workspace inspection is lazy-loaded from GET /sessions/{id}/workspace or GET /jobs/{id}/workspace. Each local job records its starting Git commit; Changes compares the current checkout against that baseline, so committed and pushed agent edits remain visible instead of appearing clean. Before an automation checkout is removed, the Orchestrator persists an immutable final workspace snapshot and serves it to both the owning job and linked session. Historical PR jobs created before snapshot support can recover a read-only changed-file view from the GitHub pull-request files API. Ownership and trusted-root checks still match workspace file references, and remote-runner workspaces remain unavailable. The Terminal inspector is read-only: it reuses the selected session's persisted and SSE-streamed command/terminal events and does not expose the unrestricted Developer-mode admin terminal or introduce an arbitrary-shell endpoint.
• Local coding sessions expose an Approvals mode on web and Apple. Codex app-server runs use approvalPolicy=on-request, approvalsReviewer=user, and workspace-write; command and file-change server requests remain pending in the native broker until the owner explicitly approves or declines them through GET/POST /sessions/{id}/approvals. The same mode can create bounded read-only workspace checkpoints through GET/POST /sessions/{id}/checkpoints; checkpoint data is stored in Clapilot runtime state outside the repository and never mutates or rolls back the worktree. Preview, checkpoint restore, and mutating Git actions remain unavailable until their separate guarded contracts are implemented.
• The Changes inspector exposes guarded Git actions through POST /sessions/{id}/git-actions and the equivalent job endpoint. Stage/unstage accepts one validated repository-relative path, commit accepts only a bounded message and requires staged changes, and push is fixed to a non-force origin push of the current branch. Every action rechecks the owner/trusted workspace plus the branch and HEAD the user reviewed; stale requests are rejected. Web uses a two-click push confirmation, and Apple mirrors the same controls. Destructive discard/reset, force push, arbitrary remotes/refspecs, and checkpoint restore are intentionally unsupported.
• Wrapper preferences persist without changing chat semantics: web retains the activity filter and resizable detail width in local storage, while Apple retains the activity filter in AppStorage. Selecting any other session still resets the detail mode to Conversation, so a previously inspected Terminal/Changes/Approvals pane can never hide a normal chat transcript. Web exposes Cmd/Ctrl+K for activity search, Cmd/Ctrl+Shift+N for a new agent, and Escape to clear search; macOS mirrors the search and new-agent shortcuts.
• Runtime conversations render oldest-to-newest across providers. Reply composers are available only in Conversation, preventing prompts from being entered while inspecting diagnostics; detached jobs retain their existing prompt/terminal workflow.
• Interactive Codex sessions render a typed timeline instead of a flat log wall: assistant replies are unboxed markdown prose, user messages are compact trailing bubbles, and Codex tool/command lifecycle items are quiet expandable rows. Tool details expose command/cwd, duration, exit code, output, and raw payload only on demand; unmatched low-level output is collapsed behind a bounded Terminal output disclosure.
• Multi-turn sessions remain one continuous chronological conversation. Turn IDs, event counts, provider avatars, and timestamps are not persistent section headers; timestamps stay secondary, while user prompts and assistant prose establish the turn boundaries naturally.
• Detached Codex jobs run codex exec --json. The job log preserves the legacy source and message fields and additionally carries typed eventType, itemType, itemId, turnId, itemStatus, item, and usage fields. Web and Apple render agent_message, command_execution, file_change, MCP/dynamic tool, and web-search items directly instead of classifying CLI stdout heuristically.
• Detached-job detail views use the same visual language on web and Apple: task and follow-up prompts appear as user messages, assistant text renders once as prose, the newest two tool actions stay visible, and older tool calls fold behind a +N previous tool calls disclosure. Routine runtime initialization/status chatter stays out of the primary transcript. The currently open job/session follows its SSE stream, preserves structured event fields across reconnect replay, and reconnects after transient transport failures. When a job is linked to a canonical session, the linked transcript owns user/assistant prose and job logs contribute only non-conversation activity, preventing duplicate answers.
• Repository-backed web timelines show a compact changed-files card with file count and diff totals; View diff opens the existing guarded Changes inspector. The composer remains anchored below the scroll content as a rounded floating surface, with attachments and the provider-scoped model selector inside its footer.
• Bridge runtime tool calls (tool.start / tool.end rows from agent_events, e.g. Claude CLI Bridge and Codex bridge runs) are reconstructed into the same expandable tool rows: calls are paired by toolCallId, labeled with the tool name plus an args-derived summary, and expose the executed command ($ …), result preview, and full raw JSON payload on demand. Older consecutive tool calls collapse while the newest actions remain visible. run.postprocess.* chatter is hidden unless it failed. The Apple client mirrors the same behavior.
• Clapilot-code entries use the same oldest-to-newest transcript direction as Codex and Claude detail panes, auto-scroll to the newest turn, and include a footer composer for follow-up prompts into the same runtime session. Raw runtime events and metadata stay hidden behind the provider icon in the pane header.
• Claude bridge entries expose attached model/tool runs plus the persisted Claude harness session id. The module footer can send follow-up prompts back into the same Claude bridge session; if the Claude CLI process is still running, the prompt is steered into the live stream, otherwise a new runtime turn resumes the persisted Claude session.
• The read-only native session detail pane shows readable timestamps on each model run and nests per-run tool usage events inside the corresponding turn/run card. When no classic chat_nachrichten transcript row exists for a native runtime session, the pane reconstructs the visible session transcript from the persisted run history instead of showing an empty transcript block. For native runtime runs that used a Mixture-of-Agents (MoA) preset, the run view also lists the reference model outputs (per-reference provider/model and response) alongside the aggregated answer.
• Interactive sessions continue directly from the terminal pane footer (no separate lower controls composer). Detached jobs expose a follow-up composer directly in the job detail pane: linked-session jobs continue the same hidden session, while plain CLI jobs continue in the same workspace and append output to the job log. The follow-up composers reuse the floating-chat input styling, minus voice/model controls, and support the same inline image/file attachments for first and follow-up turns. The new-agent composer carries repository and inferred harness/model selection plus optional speech-to-text in one footer row.
• User-owned local coding sessions and local job follow-ups support workspace file references in the composer. Type @ (or use the @ action), search the active working directory, and select a relative path. The request carries the selected paths as fileReferences[]; Codex, Claude CLI, and Clapilot-code are instructed to inspect those files relative to their active cwd without uploading or embedding the file contents. New-agent drafts have no checkout yet, legacy ownerless sessions are not claimed implicitly, and remote runner workspaces are not server-local, so the picker is intentionally unavailable in those cases.
• The web module keeps the selected interactive or linked-job session attached to GET /sessions/{id}/stream?replay=0 even while the session is ready or waiting for another turn. The initial SSE snapshot supplies recent history once; replay=0 prevents the same historical events from being emitted a second time during reconnects. New events then update the open detail pane without overview polling.
• Selected Clapilot-code and Claude bridge entries subscribe to the SSE stream of their linked external session and throttle runtime-detail reconciliation while events are arriving. Legacy runtime rows without a linked external stream, or rows whose persisted stream is no longer available after a runtime restart, use a 15-second selected-detail fallback poll instead of rapid SSE reconnects.
• The Apple client subscribes to the same SSE endpoints (GET /jobs/{id}/stream and GET /sessions/{id}/stream) for the selected activity item. Session list calls receive the new compact summary view by default, while streamed job logs and linked-session events remain the live detail path.

Run jobs on remote machines

• Detached jobs can set executionTarget: "remote" with provider codex, clapilot-code, or auto (which resolves to codex), and optionally remoteRunnerId to wait for a specific registered machine; otherwise any online runner that advertises the matching harness capability can claim the job. The + New Agent panel on web and the Apple client exposes local execution, automatic remote claiming, and pinned remote runners through its runner dropdown; the model menu offers Codex models plus, when a capable runner is online, Clapilot Code catalog models.
• Remote execution is pull-based: scripts/agent-orchestrator-codex-remote-runner.mjs runs on the remote machine, connects outbound to Clapilot with CLAPILOT_HUB_SHARED_SECRET, claims queued work, checks the repository out from a local shared bare cache into a per-job worktree, runs the assigned harness CLI (codex or clapilot-code), and reports logs/status back to the same job stream.
• Runner capability reporting: since codex-remote-runner/0.2.6 the Node runner always advertises codex (a missing Codex CLI fails loudly at spawn time, matching earlier versions) and additionally advertises clapilot-code when the CLI resolves on PATH or at ~/.clapilot-code/bin/clapilot-code.mjs. Fresh Clapilot Code jobs are only assigned to runners advertising the clapilot-code capability, so older runners are unaffected. The remote machine's clapilot-code must be logged in against an instance (clapilot-code login, or hub-driven provisioning below); job model selection uses this instance's Clapilot Code model catalog and is passed via clapilot-code exec --model. Codex stdout is parsed as --json JSONL into typed events. Clapilot Code jobs run with --output-format stream-json when the machine's CLI is 0.3.1 or newer, so assistant prose and tool calls render as the same typed conversation rows as Codex jobs; older CLIs keep emitting plain text, which is stored as raw job-log lines.
• Runner self-update (runner 0.2.8+): heartbeat/claim responses include a runnerUpdate: { version, source } payload while a self-update-capable runner is older than the hub's bundled script and idle. The runner syntax-checks the payload, verifies the declared version, atomically replaces its own script, and exits so its supervisor (fleet connector, launchd, systemd) restarts the new version; updates are never applied while jobs are running. Pre-0.2.8 runners cannot self-update — restarting the fleet connector (which re-downloads the bundled runner at startup, and self-updates itself when the hub advertises a newer connector version) upgrades them once.
• Remote provisioning (runner 0.2.7+): Settings -> Agent Orchestrator shows an Install Codex and an Install Clapilot Code button per registered machine, plus an auto-provision toggle. Actions are queued on the hub and executed by the runner on its next poll — the hub never connects to the machine. Install Clapilot Code ships the CLI source from the hub (no Developer mode needed), mints a dedicated, revocable inference:execute instance API key named remote-runner:<runnerId> (re-running rotates and revokes the previous key), and writes ~/.clapilot-code/config.json with the validated key and the first tool-capable catalog model. Install Codex installs the CLI via npm -g (falling back to npm --prefix ~/.local and brew) and writes the hub's stored Codex auth (codex_auth_json or OPENAI_API_KEY) into the machine's Codex home. When the auto-provision toggle is on, any connecting 0.2.7+ runner without the clapilot-code capability is provisioned once automatically; failed installs are never retried automatically and stay visible in the machine row. Install payloads (API keys, CLI source) travel only in runner-authenticated heartbeat/claim responses and are dropped from hub memory once the runner reports an outcome; browser-facing runner lists carry status, error, and detail only. The shell-tools kill switch blocks both queueing and delivery.
• The runner includes a read-only CLI-session snapshot in its heartbeat. It scans the remote machine's Codex home (CLAPILOT_REMOTE_RUNNER_CODEX_HOME, CODEX_HOME, or ~/.codex) for state_*.sqlite thread metadata and Codex CLI/Desktop history files (session_index.jsonl plus sessions/**/rollout-*.jsonl). It also scans ~/.clapilot-code/sessions/*.jsonl, or CLAPILOT_REMOTE_RUNNER_CLAPILOT_CODE_SESSIONS when configured. Both harnesses are exposed per machine through GET /remote-runners/{runnerId}/codex-sessions; selecting a session requests transcript detail, which the runner includes in a later heartbeat, so detail loading is asynchronous and outbound-only.
• Replying to a scanned Codex session requires the Node runner codex-remote-runner/0.2.3 or newer and runs codex exec resume with the explicit Codex home. Replying to a scanned Clapilot Code session requires codex-remote-runner/0.2.4 or newer and runs clapilot-code exec resume. Both paths use the scanned cwd when it still exists, without cloning or creating a repository worktree. The macOS Swift runner does not claim session-resume assignments.
• macOS machines can instead run the signed Clapilot Remote Runner menu-bar app. Its popup has an Overview / Connection / Log segmented control: Overview shows live worker stats, active sessions, and recent finished sessions (succeeded/failed/cancelled) with timing; Connection holds the Clapilot URL, Hub shared secret, runner id/label, workspace path, polling interval, slot count, and the Save/Test heartbeat/Quit actions; the secret is stored in Keychain. Start/stop and heartbeat status stay in the popup header and status bar. The macOS menu-bar runner advertises only the codex capability, so it never claims Clapilot Code jobs; use the Node runner on machines that should run both harnesses.
• Settings -> Agent Orchestrator lists registered remote machines with online/stale status, slots, capabilities, version, running jobs, and last heartbeat.
• The Clapilot server does not need SSH access or an inbound port on the remote machine. Treat connected runners as trusted, because repo jobs receive a GitHub token for cloning private repositories.

GitHub and GitLab automations

Configure named GitHub and GitLab accounts in Settings -> App Verbindungen, then select repositories in Settings -> Agent Orchestrator. GitLab connections accept GitLab.com or a self-managed instance URL. The settings page and the New Agent composer show one combined repository list; each row carries its GitHub or GitLab suffix icon and retains the source connection so same-named repositories do not collide. The picker caches the last loaded union locally, and a refresh button remains available.

The repository automation matrix selects a concrete model independently for PR review, issue implementation, and mention handling on every enabled repository. The picker reuses the same Codex, Claude, and Clapilot Code model catalog as New Agent; users select only the model, and the model's provider determines the coding harness. Main-CI failure fixes and Symphony coding tasks dispatched from that repository's mapped task board use the repository's issue-implementation model. Existing rows without model metadata remain backward-compatible with the former global provider defaults until the configuration is saved, at which point enabled rows persist concrete { provider, model } selections as prReviewModel, issueObserverModel, and mentionObserverModel inside agent_orchestrator_repo_automation_config.

Published automated PR/MR reviews and Issue Observer implementation PR/MR descriptions and issue completion comments include an Agent Orchestrator attribution footer with the completing harness and model. The server adds it from execution metadata, including native-session model reports and the final fallback candidate; provider-qualified Clapilot-Code model references remain visible. Cloud runs whose service does not report a model say Cloud default (not reported) instead of claiming the requested model. The implementation result checkpoints its attribution for recovery, and retries replace only the generated footer while preserving the current description and testing instructions. Older recovered PRs without that checkpoint are not relabelled; their completion comment reports unavailable metadata. Background forge messages retain their existing English default; the footer also supports German and Italian when a run supplies its UI language. This is shared server publication for web, iOS/macOS, and agent-triggered workflows, with no separate client setting.

For each repository you can independently enable:

• automatic pull-request or merge-request reviews
• automatic issue observer runs
• automatic mention replies
• default-branch CI failure fixes from GitHub or GitLab webhooks

Webhook delivery is acknowledged only after feature scans succeed or discovered work is durably queued. Scan failures and rate-limit cooldowns retain the delivery for retry after 30 seconds, and feature scan errors also surface in the overall poll status. Every five minutes, the polling loop reconciles enabled PR reviews for webhook repositories to recover missed deliveries; this does not enable issue or mention polling for those repositories. Transport failures count toward GitHub API error telemetry. CI follow-ups must repair or rerun the actual checks, preserve coverage, and report inaccessible infrastructure as a blocker (status: "blocked" with the reason in summary) rather than turning unexecuted tests into passing results; a blocked follow-up completes as a succeeded job and is retried after the usual failure backoff. Apple E2E HTTP 530 recovery waits for the exact-head PR preview and reruns the full Apple workflow to select it.

Trigger mode is configured per repository as polling or webhook. Webhook rows show a repo-specific tokenized URL for the matching GitHub or GitLab project webhook; polling rows run from the interval loop. GitLab merge-request, issue, note, pipeline, and push hooks are normalized to the same scoped automation families as their GitHub equivalents.

Integrations and providers:

• Each repository retains its named GitHub or GitLab connection. Review scans, checkout, review submission, issue assignment, follow-up tracking, mentions, and CI inspection all use that same provider-specific connection.
• GitLab parity covers nested group paths, GitLab.com and self-managed base URLs, merge requests, issues, notes/discussions, approvals, pipeline jobs, emoji reactions, and project webhooks.
• Legacy github_token / github_pr_review_token values still exist as compatibility fallbacks, but the intended admin flow is named GitHub integrations.
• GitHub automations can select codex, claude, or local clapilot-code; the OpenClaw compatibility fallback is not used for review, implementation, or mention-reply submission.

PR review: scans non-draft open PRs, deduplicates by repo + PR number + head SHA, and derives a GitHub review decision from the generated findings:

• REQUEST_CHANGES when any finding is tagged critical or the review explicitly asks for changes
• APPROVE when the review reports no issues or only medium / minor findings
• COMMENT remains the fallback when the review output cannot be classified

Issue observer: assigns the issue to the configured GitHub identity when a coding run starts, comments that implementation has started, and closes the issue immediately after pickup so it cannot be picked up by another observer run. On success it persists the issue-to-PR link and comments the PR link back onto the issue. If another active implementation session already owns the same issue, the duplicate pickup is treated as already in progress and the issue stays closed. If implementation fails and no PR can be recovered, it reopens the issue and records the failure with the normal retry backoff. Codex-backed issue implementations default to interactive sessions so the same session can be resumed for later PR follow-up work. Issue observer settings include an Agent Goal toggle: when enabled and the implementation provider resolves to Codex or Claude, implementation and PR creation runs start with a /goal command derived from the issue task. PR creation prompts require PR bodies to include a closing issue reference, a leading ## Summary section written in plain language a non-technical reader can understand (what changed, why, and what behaves differently), plus a ## Testing Instructions section; custom issue-observer prompts still receive this mandatory PR body contract after template expansion.

Main CI failure fix: webhook-only. When enabled for a webhook-mode repository, completed failing workflow_run, check_run, or check_suite events on the repository default branch first pass an evidence gate. The failing SHA must still be the current default-branch head, and GitHub/GitLab must expose a completed failing job step or equivalent check output. Lost runners, incomplete steps on completed jobs, setup/cleanup failures, known infrastructure errors, stale heads, and evidence-free status deliveries are recorded without starting a coding run. Evidence-free check deliveries remain eligible for a later, richer workflow_run delivery.

An admitted run must reproduce the cited failure before changing files. If the failure is transient, already fixed, infrastructure-owned, or otherwise lacks a safe source-controlled repair, the run finishes with no_action and creates no PR. Repair agents receive no forge write credentials: they commit locally and return repair_ready with a proposed title/body. The orchestrator then verifies that the default branch has not advanced, the commit descends from the exact failing head, the checkout is clean, the diff is non-empty, the body starts with a plain-language ## Summary section and includes ## Testing Instructions, and the change is not only a JavaScript bundle-budget increase. Only after those checks pass does the orchestrator push the branch and create the PR/MR. A final forge read-back closes any invalid or raced publication before it can enter tracked follow-up. Valid repair PRs enter the same tracked follow-up loop as issue-observer PRs. Pull-request branch failures stay on the existing tracked-PR follow-up path.

Main-CI diagnostic sessions use a durable key derived from repository and failed head SHA. After a runtime restart, a workspace that still proves it is on the verified pre-turn commit may replay the diagnostic turn. A locally committed candidate interrupted before the module-owned publication gate remains protected and is not replayed by the write-disabled coding session. Pushed candidates and matching pull requests are reconciled where forge evidence is conclusive; unsupported repository identifiers, unexpected forge responses, and unavailable Git or forge evidence remain fail-closed.

Main-CI turns also keep a five-minute finalization reserve inside the global turn timeout. Three consecutive failed tool steps, five failed tool steps in total, or entry into that reserve asks the coding agent to stop mutations, inspect Git state, report its last validation, and emit the structured Main-CI result. The runtime persists metadata.mainCiAutofixCheckpoint before this steer and again before a hard-timeout interrupt. The checkpoint contains HEAD, status, a bounded binary diff, the last recognized test/build/lint command and result, failure counters, and explicit resume/discard instructions. Its disposition remains pending until the existing module-owned publication or cleanup path makes the outcome conclusive, so a timed-out workspace is never silently treated as valid or discarded.

Tracked PRs: PRs opened by orchestrator-managed coding runs or interactive sessions are tracked durably in the database until they merge or close.

• If an issue observer run fails after a linked PR was already opened, or a later manual/chat run opens a PR that closes the watched issue, the next issue scan recovers that PR into the same durable tracked-PR follow-up state instead of leaving it unmonitored.
• While a tracked PR remains open, the orchestrator watches for merge conflicts, failing GitHub checks, PR review comments, top-level PR review bodies, GitHub Actions CI report comments, and mention activity on the open PR branch. Failing check follow-up is not gated only by the PR thread updated_at timestamp, so edited GitHub Actions smoke/e2e reports still retrigger the tracked PR worker when the check run fails again.
• Every tracked-PR scan follow-up receives a new job-specific worktree. The source job and source session remain attribution/context references only for that path and are never used as its checkout, so a PR scan cannot replace or reset an earlier development workspace. Direct mention turns continue in their existing interactive session.
• CHANGES_REQUESTED reviews, merge conflicts, and critical comments trigger an immediate isolated follow-up run, while non-critical comments keep the guarded triage fallback. Triage can choose reply for direct questions, status requests, or clarification comments and post a GitHub PR conversation reply without changing code; it chooses follow_up for small safe code fixes (which can also post a result-specific PR conversation reply after updating the branch); ignore is reserved for noise or duplicate comments.
• A follow-up that reports blocked (for example an external Codex quota or unavailable check infrastructure) persists its finding and snoozes all follow-up trigger paths for that PR/head until retry_after, an optional absolute ISO 8601 timestamp with timezone in the result JSON. Reset times more than seven days after the failure are rejected. Without a usable future reset it waits AGENT_ORCHESTRATOR_TRACKED_PR_BLOCKED_BACKOFF_MS (default one hour) initially; repeated findings with the same persisted blocker signature and head double that delay up to seven days, independently of the escalating actionable-failure backoff (30 minutes initially, capped at four hours). Older blocked: summaries with an explicit reset date are also recognized; a date without an hour waits through that UTC date. Restart recovery and supervisor cleanup preserve the snooze. A new head or changed failed-check identity/state can retry earlier; CI report edits alone cannot, and associated CI reports do not fall through as critical feedback while checks are snoozed. External conflict blockers also remain snoozed when only the base branch advances; unverified conflict repairs use the ordinary actionable-failure backoff. Review, fallback, reconciliation and restart dispatches all honor the blocker before allocating a job or workspace; a legacy blocked: record that never stored the head it applies to stays scoped to its own trigger instead of snoozing the whole PR past new commits. Administrative Apple prerequisite signatures (a Trusted Preflight gate, or Xcode first-launch/license/xcode-select setup that needs an administrator) remain snoozed even when check IDs or report timestamps change; a merely Xcode-related infrastructure wait is an ordinary blocker. Records persisted before blocker signatures existed keep the single-step fallback instead of inheriting escalation from earlier actionable attempt counts, and restart checkpoints are admitted against the current PR head. A new PR head, a changed base SHA for Trusted Preflight, or the reset time allows another attempt; for Trusted Preflight the base SHA is the base branch's current tip resolved from the forge (one extra request per scan only while such a blocker exists), because GitHub does not reliably advance pull.base.sha when the base branch moves without a head push. A record persisted before blocker signatures existed does not carry its inherited attempt count into the first re-recorded blocker. When a changed-check follow-up completes without reporting blocked again, the older ordinary check blockers it superseded on the same head are cleared, so the old reset cannot keep rejecting review, fallback or reconciliation work; Apple prerequisite blockers and comment triage results leave existing blockers untouched. Repeated findings keep escalating across base-branch advances; only Trusted Preflight starts a new attempt series when the base SHA changes, because that blocker is resolved on the base branch. For a manual override, stop the orchestrator runtime, back up <workspace>/.agent-orchestrator/tracked-pr-state.json, remove only the affected PR’s entries from failures, and restart; do not edit the state file while the runtime is active. No required check is bypassed or marked successful.
• A blocker that describes the machine the follow-up ran on is not treated as an external wait. follow-up-blockers.mjs classifies the reported summary into a host fault (disk, memory, docker, runner_offline) or an empty kind for a genuine external wait, using the same vocabulary as the MAIN_CI_INFRASTRUCTURE_TEXT check that already triages default-branch CI. An exhausted quota is never read as a host fault, because an account allowance follows the credential rather than the machine.
• A host fault waits AGENT_ORCHESTRATOR_TRACKED_PR_HOST_BLOCKER_BACKOFF_MS (default ten minutes) and ignores any reset time the agent stated, since the machine and not an external allowance has to recover. The durable failure ledger records hostBlocker (the fault kind) and blockedRunnerId (the machine). The machine itself enters the existing per-runner cooldown (RUNNER_FAILED, 15 minutes doubling to at most four hours), so the next reservation selects a different runner, local execution, or Codex/Claude Cloud according to the repository execution priority; Cloud stays ineligible whenever the job requires Docker validation. Operators are told through the Team Chat alerts described under runner host health.
• When no target can be placed at all — for example a Docker-requiring follow-up with no Docker-capable runner online — the work is queued durably with status target_unavailable instead of being dropped until a later scan rediscovers it.
• Tracked PR status exposes recent follow-up trigger outcomes and failures for debugging (lastScanResults, recent handled triggers, recent failures).
• Active tracked-PR turns are checkpointed before agent execution. After a runtime restart the original trigger is restored once with the explicit requeued_after_restart status and dispatched with the same composite idempotency key. Review/conversation replies carry a hidden operation marker, so replay after a crash reuses an already posted GitHub comment or GitLab merge-request note instead of posting it twice.
• Detached/manual jobs that explicitly implement a GitHub issue share the same issue-level lock as the issue observer: a second job for the same repo issue is rejected while an implementation session is active or once an open PR already closes the issue.

Mention observer: watches configured issue and PR conversations for @<github-login> mentions of the GitHub identity behind the configured token, adds a lightweight reaction, and then either redirects the mention into the tracked PR session or starts a target-specific session/workspace for that PR or issue.

Runtime settings information

Symphony Runtime keeps primary metric values and runtime status visible. Secondary counts, polling and API-budget details, cache totals, and board/model/speed/reasoning explanations appear in adjacent info popovers. Field labels remain above their controls. Apple clients also expose automation count breakdowns and runtime field guidance through info buttons. Active errors, rate-limit cooldowns, and throttling notices remain visible.

Repository settings

Automation settings keep titles and field labels visible while descriptions, model guidance, Agent Goal help, and prompt hints live in adjacent info popovers. The repository header also places its four live automation counts inside the title info popover, keeping the collapsed header to one line. Remote Machines keeps its online status and refresh control visible; its description and automatic-provisioning explanation use info popovers, with the setup toggle remaining directly accessible on web and Apple clients. Web popovers open on hover, keyboard focus, or click; Apple clients support hover and tap/click. The three web automation cards align their controls and prompts without fixed-height description blocks.

Settings → Agent Orchestrator shows saved repository rows immediately, even before the GitHub/GitLab catalog is loaded or when a refresh returns only part of it. The catalog enriches existing rows without determining which rows are visible. Unconfigured repositories appear only in Add repository: open the dropdown, choose a repository, configure its board, trigger, and models, then save. Loading or refreshing the catalog does not add rows automatically. Web, iOS, and macOS use this same selection flow.

Adding a repository starts with all automations disabled. Saved rows remain configured even with every automation disabled; adding a row alone does not enable an observer. Repository discovery remains available to agents through agent_orchestrator_list_repos. Automation settings continue to use the existing admin-only config endpoint; the selection UI does not expand agent authority to change automation settings.

Symphony task-board dispatch

• Symphony's poll loop is controlled by a persisted enable/disable flag in app_settings.agent_orchestrator_symphony_enabled. POST /orchestrator/start enables and starts the loop, POST /orchestrator/stop disables it and clears queued retries. GET /status includes enabled; disabled loops do not dispatch aufgaben candidates. The UI control lives under Settings -> Agent Orchestrator, not in the user-facing module view.
• The Symphony task-board toggle only controls aufgaben coding task dispatch; GitHub automations continue to run from the per-repository automation matrix even when Symphony is disabled or no task board is selected.
• Dispatch observes the configured default task board in app_settings.agent_orchestrator_symphony_task_board_id plus any repo-specific boards selected in the Settings GitHub automation matrix. If Symphony is enabled without a default board or a repo board mapping, no task dispatch happens.
• Tasks inside repo-specific boards inherit the mapped GitHub repository automatically. Tasks inside the default board still need a concrete repo:owner/name reference in the task description so the agent can safely clone the repository.
• Repository coding tasks dispatched from a mapped board resolve the same concrete issueObserverModel used for issue implementation, including its implied Codex, Claude, or Clapilot Code harness. The job record stores the requested provider/model and the provider-reported effective model remains visible in Agent Activity. Rows without a concrete model retain the legacy global issue-observer provider fallback.
• Symphony dispatch uses local capacity first. Once the local concurrency or memory limit is reached, compatible Codex and Clapilot Code repository tasks reserve a slot on an online Fleet runner (0.3.1+), subject to the per-machine Orchestrator slots setting. Without compatible capacity, work remains queued. Claude and non-coding tasks remain local. Remote failures never bypass local admission for a fallback.
• Symphony only appends mandatory branch/commit/PR instructions for explicit repository coding tasks. General aufgaben records such as document or payroll review tasks without a repo stay in the non-repository task path and must not create GitHub PRs.
• Each Symphony attempt has its own persistent job workspace, preventing retries and parallel tasks from replacing one another's checkout.
• Symphony-created PR bodies must include both the Requested-by: Symphony task ... line and a Clapilot task: .../aufgaben/{id} link for traceability (using the configured public app URL, not internal container service URLs), plus a leading plain-language ## Summary section a non-technical reader can understand and a ## Testing Instructions section with concrete validation commands or manual checks. The coding job retains the originating task ID. After detecting the created PR, Symphony adds a system comment with its canonical URL, PR number, and open status to that task, localized from the task creator's persisted UI language (de, en, or it; private-board owner and German fallbacks). Separate PRs accumulate as separate comments; a typed UUID advisory lock and URL lookup run as separate statements in one transaction so retries and concurrent completion paths do not add the same canonical PR URL twice. Linking still runs when optional tracked-PR follow-up registration fails.
• Symphony tasks and repository automations share one resource-aware dispatch budget. A slot is reserved synchronously before asynchronous workspace preparation, preventing one poll from exceeding the configured limit. The safe default is one local run. The effective limit is also capped by total and currently free memory so coding workers cannot consume the runtime's safety reserve; GET /status exposes effectiveMaxConcurrentAgents and the capacity snapshot.
• Container bootstrap does not start Symphony by default. Set CLAPILOT_AGENT_ORCHESTRATOR_BOOTSTRAP_ENABLED=true only when a deployment should force the poll loop on during boot.

How the agent can drive it (tools)

Native ClapilotAICore tool contracts cover repository job orchestration and interactive session orchestration (services/clapilot-agent/src/sessions/index.mjs):

• agent_orchestrator_list_repos
• agent_orchestrator_start_job
• agent_orchestrator_list_jobs
• agent_orchestrator_get_job
• agent_orchestrator_follow_up_job
• agent_orchestrator_stop_job
• agent_orchestrator_start_session
• agent_orchestrator_list_sessions
• agent_orchestrator_get_session
• agent_orchestrator_send_turn
• agent_orchestrator_fork_session
• agent_orchestrator_close_session

Wiring notes:

• src/lib/agent-runtime/tool-proxy.ts forwards agent_orchestrator_* tool calls into this module API on behalf of the linked Clapilot user.
• Repository start tools forward the provider, named connection, instance URL, and clone URL returned by agent_orchestrator_list_repos; the module re-resolves the named connection and derives authenticated remote URLs server-side before using its token.
• services/clapilot-agent/src/orchestrator-sessions/index.mjs brokers interactive session lifecycle, Codex app-server integration, and event streaming into agent_events.
• Native/channel agent runs should use these tools for generic repo coding and PR-preparation work instead of legacy gateway execution paths.
• Website Canvas uses the same interactive session broker internally for iterative website edits, but keeps repo sync, preview, commit, and push ownership inside Website Canvas.

Configuration & limits

Repository checkout and cleanup:

• Server-side local automation keeps shared bare repository caches under .agent-orchestrator/repos and creates per-job/per-session Git worktrees under the existing job roots (manual-job-workspaces, github-automation-workspaces/*, and .agent-orchestrator/sessions).
• GitHub and GitLab tokens are supplied to Git fetches through temporary http.extraheader arguments instead of being persisted in remote URLs.
• Shared bare repo cache configs are kept credential-free and private: every cache preparation, workspace cleanup, and hourly GC run rewrites any remote.*.url, branch.*.remote, or other value carrying a credential URL (https://<token>@...) to the plain URL, removes url.<credential-url>.* sections, restricts the config file to mode 0600, and emits agent_orchestrator_repo_cache_credentials_scrubbed (key names only, never the secret) so operators know to rotate the exposed token. Under the per-repo cache lock, local clapilot-tracked-pr/* branches that are no longer checked out in any worktree and whose commits are all published are deleted with their config sections, so the cache config does not grow without bound.
• Agent prompts instruct coding agents to push with git push origin HEAD:<branch> using the preconfigured environment authentication and never to use push -u, clone, or remote set-url with a credential URL, because Git persists such URLs in the shared config.
• Repository-backed Clapilot-code sessions keep the initiating Clapilot user id on initial and resumed turns, so user-scoped Agent Orchestrator tools remain available inside the linked coding session. Older sessions acquire and persist that binding when they are resumed from the signed-in module UI.
• Repository shells receive the selected credential only as ephemeral process environment: GitHub uses GH_TOKEN, GitLab uses GL_TOKEN and GITLAB_HOST, and Git push uses temporary process-local Git config. Named-connection and GitLab sessions use the embedded native runtime so credentials remain scoped to one session. Tokens are never written to the checkout remote URL, job log, or external-session metadata.
• Completed local automation worktrees are removed at job completion where possible, and the instance cleanup worker removes stale PR review, issue, main-CI, tracked PR, manual-job, and Symphony workspaces while protecting active work. Before either cleanup path removes a Git worktree, it checks whether HEAD contains commits absent from all remote-tracking refs. Such workspaces are retained temporarily with recovery metadata (workspace, branch, and commit), allowing operators to recover the commit without permitting abandoned worktrees to consume storage indefinitely. Native interactive session roots are excluded because the session subsystem owns their resumable lifetime.
• The crash-recovery reaper runs at startup and hourly. AGENT_ORCHESTRATOR_WORKSPACE_TTL_HOURS controls the orphan TTL (default 24); an atomic PID lock prevents concurrent reapers. Lease state plus the in-memory queued/running inventory protects active workspaces, and terminal jobs with linked interactive sessions stay protected for follow-up turns. Paused or other non-terminal Symphony task states remain active; only configured terminal task states become eligible after the TTL.
• Every GC emits structured agent_orchestrator_workspace_gc telemetry with workspace counts and bytesBefore, bytesAfter, and bytesFreed. AGENT_ORCHESTRATOR_WORKSPACE_HARD_LIMIT_GIB optionally rejects every new local manual or GitHub-automation worktree and emits agent_orchestrator_workspace_backpressure once orchestrator storage reaches the configured limit.
• Run-local dependency trees are never copied between worktrees. Removing a worktree recursively removes its node_modules; package managers should use their host/container-level shared content store (for pnpm, the configured global pnpm store).
• Bare repo caches are retained for reuse and are pruned by age only when Git reports no linked worktrees. The default cache TTL is 336 hours.
• Remote runners keep their own sibling repos cache next to the configured workspaces directory and remove stale completed job workspaces after 24 hours by default.

GitHub API budget:

• Every GitHub REST GET from the orchestrator is a conditional request. The local githubApiFetch wrapper keeps a bounded in-memory LRU of ETag + body per token hash and URL (default 2000 entries / 64 MiB, per-entry cap 4 MiB) and sends If-None-Match. GitHub answers unchanged resources with 304 Not Modified, which is replayed from the cache and does not count against the primary rate limit. Repeated polls of open PR lists, issue lists, PR details, check runs, reviews, and comments therefore only consume quota when something actually changed.
• The wrapper records x-ratelimit-limit/remaining/reset from every response (including 304s) per token. Before the GitHub section of a polling tick runs, the loop checks the most constrained token: it keeps a reserve of 15% of the limit (at least 100 requests) for interactive actions and, if the remaining budget cannot sustain the configured poll interval until the reset given what the previous scan consumed, it stretches the GitHub scan interval (up to 15 minutes) and skips ticks in between with the reason github_budget_throttled. Aufgaben/task-board dispatch is unaffected and still runs every tick.
• After a 403/429 rate-limit response the affected token enters a cooldown until GitHub's reset time. Polling and manual ticks skip the GitHub section during the cooldown (github_rate_limit_cooldown) instead of failing each feature with the same error; webhook ticks are also gated when every resolved token is cooling down.
• Session-history PR recovery (agent_external_sessions scan) runs at most every 5 minutes on polling ticks because it is a fallback path; manual ticks still run it immediately.
• GET /status exposes githubApi with rateLimit (limit, remaining, resetAt, observedAt), cooldown (resetAt, message), throttle (active, intervalMs, nextScanAt, reason, skippedTicks), cache (entries, bytes, hits, misses), and telemetry (requests, conditionalHits, quotaConsumed, errors, lastScanConsumed, lastScanAt). Settings -> Agent Orchestrator shows the remaining budget, reset time, cached-response count, and a cooldown/throttle notice on web, iOS, and macOS.

Detached Claude auth:

• Detached Claude jobs resolve auth from the configured Anthropic provider in Clapilot's database before falling back to legacy app settings or local CLI state.
• Claude subscription auth uses the stored Claude setup-token and maps it to the Claude CLI internally; it does not rely on ANTHROPIC_OAUTH_TOKEN being injected via Docker .env.

Supervisor:

• An optional supervisor loop periodically reviews orchestrator state (for example stale runs) and can be triggered or toggled via GET|POST /supervisor/run, POST /supervisor/start, and POST /supervisor/stop.
• Its configuration is persisted in app_settings (agent_orchestrator_supervisor_enabled, ..._provider, ..._model, ..._interval_ms, ..._stale_run_ms); the interval floor is 1 minute and the stale-run floor is 15 minutes.

Optional env overrides:

• AGENT_ORCHESTRATOR_CODEX_ARGS
• CLAPILOT_AGENT_CODEX_SERVICE_TIER
• AGENT_ORCHESTRATOR_CLAUDE_ARGS
• OPENCLAW_PRIMARY_MODEL
• CLAPILOT_AGENT_ORCHESTRATOR_BOOTSTRAP_ENABLED (force Symphony on at container boot)
• SYMPHONY_MAX_CONCURRENT (configured local automation concurrency, default 1, maximum 8)
• AGENT_ORCHESTRATOR_MIN_FREE_MEMORY_MIB (memory reserve protected from new local automation runs, default 3072)
• AGENT_ORCHESTRATOR_RUN_MEMORY_BUDGET_MIB (estimated memory required per new local automation run, default 4096)
• AGENT_ORCHESTRATOR_GITHUB_CONDITIONAL_CACHE_MAX_ENTRIES (ETag cache entries, default 2000)
• AGENT_ORCHESTRATOR_GITHUB_CONDITIONAL_CACHE_MAX_MIB (ETag cache byte budget, default 64)
• AGENT_ORCHESTRATOR_GITHUB_BUDGET_RESERVE_RATIO (share of the hourly limit the poll loop never spends, default 0.15)
• AGENT_ORCHESTRATOR_GITHUB_BUDGET_MIN_RESERVE (absolute request reserve floor, default 100)
• AGENT_ORCHESTRATOR_GITHUB_MAX_SCAN_INTERVAL_MS (upper bound for the stretched GitHub scan interval, default 900000)
• AGENT_ORCHESTRATOR_GITHUB_SESSION_RECOVERY_INTERVAL_MS (minimum spacing of session-history PR recovery on polling ticks, default 300000)
• AGENT_ORCHESTRATOR_GITHUB_RATE_LIMIT_FALLBACK_COOLDOWN_MS (cooldown when GitHub sends no reset header, default 300000)
• AGENT_ORCHESTRATOR_TRACKED_PR_BLOCKED_BACKOFF_MS (fallback wait for a tracked-PR blocker that states no reset time of its own, default 3600000)
• AGENT_ORCHESTRATOR_TRACKED_PR_HOST_BLOCKER_BACKOFF_MS (wait after a tracked-PR blocker classified as a runner host fault, default 600000)
• AGENT_ORCHESTRATOR_RUNNER_MIN_FREE_DISK_BYTES (free disk a runner must report to stay eligible for new automation work, default 10737418240 / 10 GiB)
• AGENT_ORCHESTRATOR_HOST_ALERT_COOLDOWN_MS (minimum spacing between repeated host-health alerts for the same machine and fault, default 1800000)

API endpoints exposed by module

Base: /api/modules/agent-orchestrator/api — implementation: bundled-modules/agent-orchestrator/api/handler.mjs; reference: bundled-modules/agent-orchestrator/README.md.

• GET /tools
• GET /auth
• POST /auth
• DELETE /auth
• POST /repos
• GET /status
• GET /poll
• POST /config
• GET /webhook/{token}
• POST /webhook/{token}
• POST /orchestrator/start
• POST /orchestrator/stop
• GET|POST /supervisor/run, POST /supervisor/start, POST /supervisor/stop
• GET /remote-runners
• POST /remote-runners/heartbeat
• POST /remote-runners/claim
• GET /remote-runners/{runnerId}/codex-sessions
• GET /remote-runners/{runnerId}/codex-sessions/{sessionId}
• POST /remote-runners/{runnerId}/codex-sessions/{sessionId}/follow-up
• POST /remote-runners/jobs/{id}/events
• GET /coding-models (model options plus resolved default for claude, codex, and clapilot-code; the Clapilot-code list contains only usable non-subscription catalog models)
• GET /jobs
• POST /jobs (supports an optional model for all coding providers — Clapilot-code direct provider models plus concrete Claude/Codex CLI models; executionTarget: "remote" plus optional remoteRunnerId remains Codex-only, and optional codexGoalEnabled enables Codex/Claude first-turn /goal bootstrapping)
• GET /jobs/{id}
• GET /jobs/{id}/stream
• GET /jobs/{id}/workspace-files?q={query}&limit={count} (local jobs only; linked jobs resolve the linked coding session cwd)
• POST /jobs/{id}/follow-up (supports an optional model override and validated relative fileReferences[] for local follow-up runs; remote jobs reject file references. Follow-ups against a running/queued remote or plain-CLI job return 202 with turn.status: "queued" and are queued per job — max 20, dispatched in order after the current run ends with succeeded/failed; serialized jobs expose the pending count as queuedFollowUps)
• DELETE /jobs/{id}
• GET /sessions
• POST /sessions (supports canonical provider: "clapilot-code" plus model; legacy pi/embedded_pi aliases normalize to Clapilot-code and use the internal embedded_pi adapter)
• GET /sessions/{id}
• DELETE /sessions/{id}
• GET /sessions/{id}/workspace-files?q={query}&limit={count} (returns bounded, Git-ignore-aware relative file suggestions for an owned local workspace)
• POST /sessions/{id}/turns (supports an optional per-turn model override plus validated relative fileReferences[], forwarded to the native session broker; stale or invalid selected paths reject the request)
• GET /sessions/{id}/stream
• POST /sessions/{id}/fork
• POST /sessions/{id}/archive

Troubleshooting

• The module is not visible: Agent Orchestrator is hidden while Developer mode is disabled; enable Developer mode first.
• Symphony is enabled but no tasks dispatch: check that a default task board (app_settings.agent_orchestrator_symphony_task_board_id) or a repo-specific board mapping is configured, and that default-board tasks contain a concrete repo:owner/name reference.
• A second job for the same GitHub issue is rejected: the issue-level lock is intentional — one implementation session or open closing PR owns the issue at a time.
• GitHub API error 403 ... rate limit exceeded in the runtime card: the token's hourly budget is exhausted. The loop pauses GitHub scans until the reset shown in the cooldown notice; nothing needs to be restarted. If this recurs, check the githubApi.telemetry.lastScanConsumed value in GET /status: a high number per scan usually means many tracked PRs with running checks or a second consumer (Issue Reporter, another instance) sharing the same token. Raise the poll interval, split tokens per feature, or lower AGENT_ORCHESTRATOR_GITHUB_BUDGET_RESERVE_RATIO only if interactive actions do not need headroom.
• Remote job never starts: verify the runner heartbeat in Settings -> Agent Orchestrator (online/stale status) and that the runner shares the correct CLAPILOT_HUB_SHARED_SECRET; remote execution is outbound-only from the runner.

Specialized agents on remote runners

Fleet remote runner 0.3.0 also accepts pinned specialized-agent sessions via the native ClapilotAICore broker. This path uses persistent Codex app-server threads, a scoped Clapilot MCP bridge, and the runner's installed native computer/browser tools. It is separate from detached coding jobs and does not require installing the Agent Orchestrator module. Fleet connector 1.4.6 starts a specialist-only companion for Fleet-enabled Mac runner apps. See Specialized Agents for setup, approvals, session lifecycle, and limitations.

Local concurrency and Fleet overflow

Max local concurrent limits automation harnesses on the instance, with the existing free-memory reserve. After local capacity is exhausted, compatible Codex and Clapilot Code work can use remote machines: Symphony coding tasks, PR reviews, issue implementation, mentions, and tracked PR follow-ups. Admission reserves capacity synchronously before workspace preparation. Remote work does not consume a local harness slot. Git/workspace preparation and result bookkeeping still run on the hub.

Each machine has an Orchestrator slots setting in Settings → Agent Orchestrator on web, iOS, and macOS. It persists by runner ID, defaults to 1, accepts 0–8, and is separate from the runner's base capacity for manually requested work. Zero disables new automatic work. Reducing the limit lets running work finish and keeps excess queued work waiting. Other active remote jobs consume machine capacity too. The fleet's usable total depends on online runners, harness capabilities, advertised model restrictions, and occupied slots; models are never silently substituted.

Overflow needs remote runner 0.3.1+, delivered by the existing runner self-update mechanism. Its assignment protocol preserves the hub-prepared commit, branch, upstream, and review refs. Runners serialize shared repository preparation to avoid concurrent cache initialization. Older runners remain available for existing manual workflows but receive no automatic overflow. Offline or incompatible machines are skipped. A failed remote assignment uses the normal workflow failure/retry handling; it never starts an unreserved local fallback.

Main-CI repair remains local because its write-disabled harness and hub-owned validation/publication require the local candidate checkout. It still obeys the local limit. Personal specialist runs and manually started jobs/sessions keep their existing scheduling semantics.

Chat/live agents can inspect machines and, with administrator access, update limits using agent_orchestrator_fleet (list, set_slots). Migration 303_agent_orchestrator_runner_limits.sql must run before the updated handler starts.

Fleet overflow reliability

Changing per-runner Orchestrator slots requires an administrator session on the module HTTP endpoint; authenticated internal service calls retain the existing trusted-service access. Agent tools retain their administrator check. Before the first successful database load, unknown runner slot limits allow no automation; after a successful load, transient failures preserve the last known limits. Manual base capacity remains available.

Runner control payloads and listings expose base capacity plus Orchestrator slots (maximum 16 on runner 0.3.2+; 0.3.1 runners retain automation capped at 8 until their idle-time self-update; unsupported native runners retain only their reported base capacity while their saved Orchestrator slot setting remains visible). Once Orchestrator slots are configured, the hub-provided combined capacity can exceed the machine-local CLAPILOT_REMOTE_RUNNER_MAX_CONCURRENT value; that environment setting controls base/manual capacity rather than imposing a hard machine ceiling. Operators should size slots for the runner's available CPU and memory. The bundled runner does not advertise a complete model catalog: absent model restrictions mean the selected model is attempted by the configured harness, and an unsupported-model error follows normal job failure handling. Optional advertised model maps constrain routing; an explicit null clears previous restrictions. Models are never silently substituted.

Codex Cloud per Repository

Unter Einstellungen → Agent Orchestrator → Repositories lässt sich Codex Cloud für jedes GitHub-Repository einzeln aktivieren. Die Cloud-Umgebungs-ID stammt aus der eingerichteten Codex-Cloud-Umgebung (codex cloud zeigt die verfügbaren Umgebungen); der Cloud-Branch muss bereits auf GitHub liegen. Die Zuordnung muss zum Repository passen. Der Clapilot-Host benötigt die Codex CLI mit cloud exec, cloud status und cloud diff sowie ein verbundenes Codex-Konto mit Cloud-Zugang. Ein lokaler GitHub-Token ersetzt diesen Zugang nicht.

Beim Anlegen eines Coding-Agenten ist Codex Cloud im Ausführungsmenü auf Web, iOS und macOS immer sichtbar, auch ohne ausgewähltes Repository oder Online-Runner. Wähle ein GitHub-Repository mit aktivierter Cloud-Konfiguration und gib die Aufgabe ein. Fehlen Freigabe, Umgebung oder Branch, erklärt der Composer die Einrichtung unter Einstellungen → Agent Orchestrator → Repositories und sperrt das Absenden, auch per Tastatur. Ein Repository-Wechsel prüft die Freigabe erneut. Die Cloud verwendet ihr Standardmodell; lokale Modell-, Workspace-, Anhangs- und MCP-Einstellungen werden nicht übertragen. Code und Aufgabenbeschreibung werden auf OpenAI-Infrastruktur verarbeitet. Bestehende lokale Jobs, Fleet-Runner und Repository-Automatisierungen behalten ihre bisherigen Einstellungen. Die Aktivierung stellt eine zusätzliche Auswahl bereit und schaltet keine automatische Cloud-Ausführung ein.

Status, Abschlussbericht und Diff werden über denselben begrenzten, schreibgeschützten Ergebnisabruf wie bei automatischen Cloud-Aufgaben gelesen. Ein Fehler des experimentellen CLI-Statusbefehls verhindert damit nicht mehr die Anzeige bereits abgeschlossener Aufgaben.

Bei manuell gestarteten Cloud-Jobs speichert Clapilot die Cloud-Aufgaben-ID und verfolgt den Status auch nach einem Neustart und zeigt abgeschlossene Änderungen unter Änderungen. Cloud-Aufgabe öffnen führt zur weiteren Bearbeitung in Codex Cloud. Lokales Stoppen, Folgenachrichten und Git-Aktionen sind für diese Aufgaben nicht verfügbar; Diffs werden nicht automatisch angewendet oder veröffentlicht. Bei Verbindungsproblemen bleibt die Aufgabe aktiv und die Statusabfrage wird wiederholt. Eine unklare Übermittlung wird niemals automatisch erneut gestartet: vor einem manuellen Wiederholungsversuch Codex Cloud prüfen, um doppelte Aufgaben zu vermeiden.

Web, iOS und macOS unterstützen dieselbe Repository-Konfiguration und Ausführungsauswahl. Chat- und Live-Agenten verwenden agent_orchestrator_cloud zum Lesen bzw. auf ausdrücklichen Wunsch eines Administrators zum Konfigurieren, agent_orchestrator_start_job mit executionTarget: "cloud" zum Start und agent_orchestrator_get_job für Status, Link und Diff.

Cloud-Ursprung in der Aktivitätsliste

Die Web-Aktivitätsliste kennzeichnet Cloud-Jobs mit einem dezenten Codex Cloud- bzw. Claude Cloud-Hinweis neben dem Repository. Der Tab Cloud-Agents steht zwischen Remote Agents und Analytics und enthält auch Jobs, die nach einem Cloud-Fehler lokal oder remote weiterlaufen. Diese Jobs bleiben zugleich unter Alle und Coding Agents sichtbar. Die Tab-Auswahl wird gespeichert.

Bei einem Fallback ergänzt der Hinweis das Ziel, etwa → Lokal; der Tooltip zeigt den vollständigen Fehlergrund. Im Job-Detail bleiben Cloud-Ursprung, Fallback-Ziel und Grund sowie Cloud-Aufgabe öffnen sichtbar. Ein lokal weiterlaufender Job lässt sich weiterhin stoppen. Die Suche findet Cloud-Jobs über cloud, codex cloud bzw. claude cloud, die Cloud-Aufgaben-ID und die Cloud-URL.

Die Anzeige nutzt die vorhandenen Job-Metadaten, auch wenn das aktuelle Ausführungsziel bereits gewechselt hat. API- und Agent-Tool-Verträge bleiben unverändert; Agenten können diese Metadaten weiterhin über die vorhandenen Job-Abfragen lesen. iOS und macOS haben keine entsprechende Orchestrator-Aktivitätsliste.

Cloud-Umgebung auswählen

Die Repository-Einstellungen zeigen passende Codex-Umgebungen als Dropdown mit Namen. Genau ein Treffer wird im Formular automatisch ausgewählt; bei mehreren Treffern wählt der Nutzer. Gespeicherte Auswahlen bleiben während des Ladens oder bei Fehlern erhalten. Die Auswahl wird mit der Repository-Konfiguration gespeichert; die Cloud-Freigabe bleibt ausdrücklich pro Repository.

„In Codex einrichten“ öffnet die Umgebungsverwaltung. Für neue Repository-Umgebungen dort dasselbe Codex-Konto wie auf dem Clapilot-Host verwenden, das Repository verbinden und anschließend in Clapilot aktualisieren. Die CLI bietet derzeit keine Umgebungserstellung; Clapilot erstellt keine Umgebungen über eine undokumentierte Schreib-API. Die Erkennung verwendet gekapselt den schreibgeschützten Repository-Lookup des offiziellen Codex-Cloud-Pickers. Bei Änderungen dieses experimentellen Endpunkts bleibt die gespeicherte Konfiguration erhalten und ein sichtbarer Fehler erlaubt erneutes Laden.

Ausführungspriorität je Repository

Web, iOS und macOS zeigen Repository-Automationen als kompakte, aufklappbare Zeilen. Die Zusammenfassung zeigt Issues/Coding-Aufgaben, PR-Reviews, Mentions und Main-CI. Im aufgeklappten Bereich stehen Task-Board, Trigger/Webhook, Modelle, Cloud-Umgebung und die Ausführungspriorität. Änderungen werden automatisch gespeichert.

Die drei Prioritäten Remote-Runner, Cloud und In Clapilot / lokal lassen sich je Repository umordnen. Beim Start wählt der Orchestrator das erste verfügbare, zum Provider passende Ziel. Remote benötigt einen kompatiblen Online-Runner mit freiem Orchestrator-Slot; lokal gelten weiterhin Speicher- und Parallelitätsgrenzen. Cloud benötigt die separate Freigabe für den gewählten Provider: Codex verwendet Umgebung und Branch, Claude eine eigene Cloud-Kontoanmeldung und einen Branch. Clapilot Code überspringt Cloud. Main-CI-Reparaturen können lokal oder in Cloud laufen; ihr bestehender Fleet-Ausschluss bleibt erhalten. Explizit ausgewählte Ziele manuell gestarteter Jobs bleiben verbindlich.

Bestehende Repositories ohne gespeicherte executionPriority bleiben lokal zuerst mit Fleet als Ausweichziel. Diese Priorität verwenden oder eine Änderung der Reihenfolge aktiviert die vollständige Liste; die separate Cloud-Freigabe bleibt erforderlich. Pro Repository ist höchstens eine automatische Cloud-Aufgabe gleichzeitig zugelassen. Die Auswahl von Cloud garantiert kein bestimmtes Codex-Modell: Cloud verwendet sein eigenes Standardmodell.

Automatische Cloud-Aufgaben erhalten den vorbereiteten Git-Commit als Referenz. Clapilot liest anschließend Bericht und Diff, prüft den unveränderten lokalen Ausgangsstand und übernimmt den Patch. Reviews und Triage dürfen keine Dateien ändern. Bei Coding-Aufgaben veröffentlicht der Host einen PR; bei PR-Folgearbeiten aktualisiert er den vorhandenen Branch ohne Force-Push. Main-CI-Reparaturen durchlaufen weiterhin die bestehende lokale Veröffentlichungsprüfung. Cloud selbst erhält keine Berechtigung zum Pushen, Kommentieren oder Mergen; bestehende Review- und Merge-Regeln bleiben maßgeblich.

Nach unklarer Übermittlung, Neustart, Zeitüberschreitung oder fehlgeschlagener Ergebnisübernahme bleibt der Cloud-Versuch zur Prüfung gespeichert. Diese Sperre betrifft das Cloud-Ziel; unabhängige Reviews und andere Aufgaben dürfen weiterhin kompatible Remote- oder lokale Kapazität nutzen. Insbesondere läuft ein gewähltes Claude-Review lokal, auch wenn Cloud zuerst priorisiert oder ein früherer Cloud-Versuch angehalten ist.

Schlägt Cloud vor der lokalen Übernahme fehl (einschließlich vier aufeinanderfolgender Fehler beim Abruf oder eines ungültigen Berichts), sperrt Clapilot dessen Ergebnis dauerhaft gegen spätere Anwendung und versucht die verbleibenden Ziele in der gespeicherten Reihenfolge. Provider und Modell bleiben erhalten. Der unveränderte, saubere Checkout und freie Kapazität werden erneut geprüft; Main-CI bleibt von Fleet ausgeschlossen. Ist kein Ziel frei, greifen die bestehenden Workflow-Wiederholungen. Nach einem Neustart dürfen unterbrochene Aufgaben vor der Ergebnisübernahme über diese kompatiblen Ziele erneut eingeplant werden. Der Cloud-Auftrag wird nicht erneut übermittelt. Bei einem nachweislich beendeten Cloud-Versuch reicht eine fünfminütige Pause vor neuen Cloud-Aufträgen; unbekannte oder noch laufende Aufträge bleiben für Cloud gesperrt. Unterbrochene Aufträge mit bekannter ID werden nach dem Neustart schreibgeschützt weiter geprüft: Ist der Auftrag beendet und hatte die lokale Übernahme noch nicht begonnen, bleiben Bericht und Diff gespeichert, werden nicht angewendet, und Cloud-Kapazität wird wieder freigegeben. Ältere Ergebnis-Checkpoints mit unklarem Übernahmestand bleiben zur manuellen Prüfung angehalten.

Sobald die lokale Anwendung eines Cloud-Patches begonnen hat, gibt es keinen automatischen Ausführungswechsel für dieselbe Aufgabe. Prüfe dann die verlinkte Cloud-Aufgabe und gegebenenfalls bereits erstellte Branches/PRs, bevor du im Job-Detail Geprüft: Wiederholung zulassen auswählst. Diese Aktion entfernt den fehlgeschlagenen Tracking-Job; sie ist auf Web, iOS und macOS verfügbar. Aktive Cloud-Automationen können nicht durch Löschen ihres lokalen Tracking-Jobs abgebrochen werden. Die Cloud-Aufgabe muss gegebenenfalls direkt in Codex Cloud beendet werden. Ausführungswechsel und Cloud-Task-ID bleiben im bestehenden Job-Detail und in Agent-Statusabfragen sichtbar.

Coding-model fallback order

Settings → Agent Orchestrator → Coding model priority stores an ordered list of coding provider/model pairs. Web, iOS and macOS support adding, editing, removing and moving entries. Models come from the existing coding catalog, with manual IDs for newly available models. Save applies the list to subsequent runs. Example: Claude Fable 5.1 → GPT-6 Astra → Claude Opus 5 → GPT-5.6 Sol.

An explicit model starts at its own position and only falls forward. A model outside the list has no configured fallback; the final entry ends the chain. Repository automations without an explicit model use the first entry. Empty lists retain the existing runtime defaults. Fallback is triggered by provider limits or model/provider unavailability, not failed tests, ordinary coding failures or cancellation. Each CLI attempt retains the same job and checkout, records the transition, and resumes the existing provider session where possible. Cross-provider attempts receive the original task and recent progress, with instructions to verify completed work and avoid repeating publication.

Fleet keeps the assigned runner and checkout, skipping candidates its harness does not support. Codex Cloud retains its explicit model/execution contract and review hold; it never moves to another provider through this list. Execution location priority remains separate. Exhausted local chains stop automatic model attempts and Symphony places the task in its waiting state.

Chat and live agents can read/discover/update this list with agent_orchestrator_model_priority; updates require administrator authority. The database migration 310_agent_orchestrator_coding_model_priority.sql adds the workspace-global list; omitted values in older settings clients do not overwrite it.

Configured Clapilot providers (including Cursor subscriptions) can be selected directly in the coding-model priority editor. Their models are discovered on selection, independently of the global chat catalog; a manual model ID is also supported. These entries always use the Clapilot-Code harness and persist the exact provider-slug/model-id r eference. Two providers offering the same model remain separate fallback candidates. Saving a coding entry does not add it to the global model list. The same editor is available on web, iOS, and macOS.

The priority editor separates Coding harness (Claude Code, Codex, Clapilot Code) from Model provider. Cursor is a provider account under Clapilot Code; its subscription transport uses the native runtime's Cursor Agent bridge. It is not a fourth orchestrator harness. Saved priority models from all three harnesses join the shared coding catalog used by repository coding/issues/PR reviews, individual jobs, and agent tools. Web selectors refresh after saving or returning to the window; Apple settings refresh on save and the coding workspace reloads its catalog during its normal refresh. Explicit repository model pins stay selectable after removal from the priority list. Clapilot-Code selections display the Clapilot logo.

Symphony retry storm protection

Symphony admits at most four executions per task (the initial run plus three retries). Failed executions wait at least 60 seconds, then 120 and 240 seconds. Capacity waiting does not consume another attempt. Admission happens before a workspace is created. Existing failed jobs since the last successful execution seed the counter on upgrade, including older jobs whose attempt was always zero.

The separate .agent-orchestrator/symphony-retry-state.json ledger retains task claims, counters and deadlines across restarts and job/workspace cleanup. A queued/running job or an in-process preparation claim prevents duplicate runs. After restart, an active ledger claim without a queued/running job is recovered, retaining its attempt count and backoff. Operator stops, reconciliation and disabling Symphony release claims without consuming a failure attempt and cancel linked native sessions. Writes use an exclusive PID lock and atomic replacement; dead-owner locks and empty legacy locks older than 30 seconds are recovered. Circuit reads use the atomically replaced snapshot without taking a writer lock. Invalid state refuses reservations without aborting the poll. Refused retry admissions return to the poll instead of creating a repeating capacity timer.

An unavailable model opens a persistent circuit for that provider/model pair. Authentication, quota, provider connection and executable failures share a provider circuit across models with exponential backoff capped at one hour. Transient circuits expire after that cooldown and reset on a successful provider run; permanent model refusals remain blocked. Other task failures consume only that task's budget. Automation reservations and local/native and Fleet model fallback selection honor these circuits. In-flight jobs can still finish. Direct clapilot-code Symphony jobs use native orchestrator sessions with the embedded_pi adapter, instead of spawning embedded-pi as a CLI. Tasks that exhaust their budget during execution or import exhausted history are moved to warten. Admission refusals log the concrete reason and backoff deadline when applicable; a failed status update leaves the durable budget blocked. Retry timers honor the configured maximum backoff, including values below 60 seconds; the durable failure deadline still governs execution admission.

Operator recovery is deliberate: stop the orchestrator process, inspect existing jobs, sessions, unpublished work and the ledger, and resolve the provider/model problem first. To grant a task a new budget, set only its entry under tasks to {"attempts":0,"active":false,"blocked":false,"dueAt":0}. Remove only the repaired model/provider entry under domains (keys are provider:model or provider:*). If a crash left symphony-retry-state.json.lock, remove that lock only after confirming the process is stopped. Restart, then explicitly reopen the task. Reopening a task alone, deleting jobs or cleaning workspaces does not clear the safety ledger. Do not delete the whole ledger as routine cleanup.

This is shared backend behavior for web, iOS and macOS; no client controls or API request shapes change.

Claude Cloud (experimentell)

Claude Cloud ist ein separater, standardmäßig deaktivierter Adapter. Unter Einstellungen → Agent Orchestrator → Repositories lassen sich Claude Cloud und der gewünschte Branch für ein GitHub-Repository aktivieren. Web, iOS und macOS bieten dieselbe Konfiguration, eine schreibgeschützte Prüfung des Kontozugriffs und einen eigenen Eintrag im Ausführungsmenü. Die gewählte Claude-Modellauswahl bleibt erhalten. Für Automationen gilt die bestehende Ausführungspriorität; eine reine Codex-Cloud-Freigabe aktiviert Claude Cloud nicht.

Auf dem Linux-Agent-Host ist eine vollständige Anmeldung bei einem Claude-Konto mit Cloud-Zugriff nötig. Ein Claude setup-token oder Anthropic-API-Key für lokale Ausführung reicht nicht. Die CLI überträgt einen Snapshot des vorbereiteten Checkouts in die Cloud. Die Anmeldung erfolgt einmal interaktiv als derselbe Betriebssystemnutzer wie der Agent:

env -u CLAUDE_CODE_OAUTH_TOKEN -u ANTHROPIC_AUTH_TOKEN -u ANTHROPIC_API_KEY \
  CLAUDE_CONFIG_DIR=/app/workspace/.clapilotaicore/claude-cloud \
  claude auth login

Bei einer frischen CLI-Konfiguration anschließend einmal claude --safe-mode mit demselben CLAUDE_CONFIG_DIR und denselben entfernten Token-Variablen starten und die Ersteinrichtung abschließen. auth login allein überspringt weder die Darstellungswahl noch die CLI-Einführung. Eine noch offene Ersteinrichtung wird vor der Cloud-Übermittlung als eindeutiger Startfehler behandelt, damit Automationen lokal fortgesetzt werden können.

CLAUDE_CLOUD_CONFIG_DIR kann dieses persistente, separate Konfigurationsverzeichnis überschreiben. Der Adapter liest die Linux-CLI-Anmeldung aus .credentials.json; macOS-Keychain-Konten werden nicht importiert. Lokale Claude-Zugangsdaten werden weder überschrieben noch anstelle des Cloud-Kontos benutzt. Kurzlebige Zugriffstoken werden mit dem gespeicherten Refresh-Token automatisch über Anthropic erneuert und atomar mit Dateirechten 0600 gespeichert. Gleichzeitige Prüfungen teilen sich die Erneuerung. Fehlt das Refresh-Token oder lehnt Anthropic die Erneuerung ab, wird das Konto als nicht bereit gemeldet; dann ist eine erneute CLI-Anmeldung nötig. Zugangsdaten gehören nicht in Repository-Einstellungen, Logs oder Chat-Nachrichten.

Die aktuelle CLI verlangt für neue Cloud-Aufträge eine interaktive Konsole. Clapilot startet den offiziellen claude --cloud-Befehl über eine begrenzte PTY, mit --safe-mode gegen lokale Repository-Hooks und MCP-Anpassungen. Die Ordnerfreigabe ist auf den zuvor geprüften Checkout beschränkt; Anmelde- und Werkzeugberechtigungsfragen werden nicht automatisch beantwortet. Das externe Konto behält seine Cloud-Berechtigungen und Nutzungsgrenzen. Code und Aufgabenbeschreibung werden auf Anthropic-Infrastruktur verarbeitet. Ultrareview und dessen gesonderte Credits werden nicht verwendet.

Ein Auftrag gilt erst als abgeschlossen, wenn ein erfolgreicher Ergebnis-Event, das tatsächlich gemeldete Modell, die individuelle Auftragskennung, die Host-Commit-Referenz und der verifizierte initiale Git-Tree zusammenpassen. Claude Cloud erstellt aus dem Snapshot einen eigenen Seed-Commit ohne garantierten Git-Remote. Deshalb wird der Dateiinhalt über den Tree-Hash geprüft; ein identischer HEAD-Commit wird in der Cloud nicht vorausgesetzt. Der Patch bezieht sich auf den Seed-Commit und wird auf dem Host weiterhin nur gegen den unveränderten ursprünglichen Commit geprüft. Der Adapter liest dieselbe experimentelle Session-/Event-Schnittstelle wie die CLI; inkompatible Antworten werden als Fehler behandelt. Der vollständige Bericht und Git-Patch müssen in einem strukturierten Ergebnis vorliegen. Ein Link, ein ruhender Worker oder ein angefordertes Modell sind kein Erfolgsnachweis. Manuelle Jobs zeigen Bericht und Patch zur Prüfung; sie wenden Änderungen nicht automatisch an. Automationen verwenden die bestehenden lokalen Prüfungen und Veröffentlichungswege.

Fehlende Cloud-Berechtigungen oder eindeutig abgelehnte Starts erlauben den Wechsel zum nächsten kompatiblen Ziel. Claude fällt derzeit auf lokale Ausführung mit demselben Modell zurück. Unklare Starts werden nicht erneut übermittelt. Bereits eingegangene Ergebnisse werden vor einem Wechsel dauerhaft gesperrt; nach begonnener lokaler Patch-Übernahme erfolgt keine automatische Wiederholung. Session-ID und Prüfsperren bleiben über Neustarts erhalten.

Validierungsstand (14.09.2026): Ein lesender Live-Lauf auf dem Produktionshost wurde mit Claude Fable 5.1, passendem initialem Tree-Hash, unverändertem Host-Checkout und leerem Patch erfolgreich abgeschlossen. Die Session-/Event-Schnittstelle bleibt experimentell. Ein Kontocheck allein beweist keinen erfolgreichen Lauf.

Nach der Anmeldung kann der lesende Live-Test auf einem sauberen GitHub-Checkout gestartet werden:

node scripts/verify-claude-cloud.mjs /absolute/prepared/repository claude-fable-5-1

Nur die Ausgabe event: verified bestätigt den vollständigen Lauf. submitted ist lediglich die Session-Quittung. Bei Unterbrechung zuerst die ausgegebene Session prüfen; der Test wiederholt die Übermittlung nicht automatisch.

Claude Cloud requires a concrete model ID (for example claude-fable-5-1); CLI aliases such as opus use a compatible local target. The uploaded snapshot must preserve the initial Git tree exactly. Repositories whose upload filters, submodules or file metadata change that tree fail verification; no patch is applied. Terminal verification failures retain the raw report for inspection and release the Cloud slot instead of polling indefinitely. Account/network errors before the CLI starts allow fallback without marking an ambiguous submission.

PR review publication accepts structured assistant output. If Claude finishes without that block, Clapilot requests it once from the same session with tools and MCP disabled; missing session identity or a second incomplete response fails the review. Repository command output and raw diagnostics cannot substitute for an assistant review. Concrete Claude selections also accept their dated service revisions while retaining the reported model for attribution.

Fleet failure recovery

Automatic Fleet jobs record quota/model refusals and terminal harness startup failures against the runner and provider that reported them. These circuits survive hub restarts and impose at least a 15-minute cooldown for transient failures. Another machine's credentials remain eligible; successful work on another runner does not clear the failed runner's cooldown.

Repeated runner failures extend the cooldown from 15 minutes to 30 minutes, one hour, two hours, and at most four hours. Fleet listings expose cooldowns with the affected domain, deadline, reason, and failure count. Codex structured failure events remain in job logs for diagnosis.

Tracked PR failures with an unchanged trigger back off for 30 minutes, one hour, two hours, then at most one attempt every four hours. Retry counts survive restarts and supervisor cleanup. A new trigger remains eligible; success clears the corresponding failure. Status responses include attempts and retryAt.

After the existing task backoff, Symphony acquires fresh compatible capacity. If its selected provider has no compatible capacity, it tries the remaining configured coding-model priority entries. Capacity waiting does not consume a task attempt. The four-attempt task budget and cancellation fences still apply; already exhausted tasks require operator recovery. Existing workspaces remain on their original machines; this routing change does not migrate uncommitted work or replay an active assignment. Once a remote tool has run, exhaustion requires recovery of that workspace instead of automatically replaying the task on a fresh machine; compatible model fallback may still use the same checkout.

Fleet Codex repair: the existing install_codex install action upgrades an installed CLI on runner 0.3.4+, using npm and a runner-owned prefix so an outdated system CLI cannot shadow it. The administrative install-action API accepts preserveAuth: true to retain the machine’s login and withhold hub credentials. Wait for the runner script update before using this repair, then verify the reported version and a model probe before restoring automatic capacity.

Runner 0.3.5+ reports its IANA timeZone. Fleet quota failures, including manual probes, quarantine that runner/provider until the reported reset (interpreted in the worker timezone), or four hours when the reset cannot be determined. They never retry another model on the same exhausted account. Other providers and machines remain eligible. Remote modelAttempts include runner, errorClass, and retryAfter (epoch milliseconds); Fleet cooldowns exposes the persisted reason and deadline.

Runner host health

codex-remote-runner/0.3.9 and newer report hostHealth in the full heartbeat and claim payload: free/total disk of the runner workspace (df -kP), free/total memory (node:os), whether the Docker daemon answered, and the measurement timestamp. The reading is cached for 60 seconds, and the lightweight job-lease heartbeat deliberately omits it; the hub keeps the previous reading whenever a payload omits the field.

A runner whose reported free disk is below AGENT_ORCHESTRATOR_RUNNER_MIN_FREE_DISK_BYTES (default 10 GiB) is not selected for new automation work. Unknown or unreported health never excludes a runner, so older Node runners stay fully eligible. The Swift remote runner in clients/apple/ClapilotApple/RemoteRunner/ does not report host health yet; a Mac running that runner is treated as unknown health and remains eligible.

Settings -> Agent Orchestrator marks an affected machine with an amber low disk status and shows its remaining free space. The orchestrator additionally posts Team Chat alerts for a host fault that stopped a follow-up, a runner crossing the low-disk threshold, a runner recovering, and a runner going offline; the target room, the per-machine alert cooldown, and the operator runbook are documented in Operations.

PR completion and repair placement

Completed, tracked GitHub implementation PRs are made ready for review when their source job, repository, branch and observed head match. Creation instructions also prohibit drafts. Human drafts and unfinished implementation jobs are not promoted. Conflict repair takes priority over failed checks; historical CI infrastructure failures do not prevent resolving and pushing a merge.

When a follow-up reports Docker unavailable, its durable failure history makes Docker a requirement for subsequent attempts. Placement uses configured provider fallbacks and only admits a worker that recently verified both Docker daemon access and Compose. Busy or incapable workers leave the job waiting without consuming an execution attempt. Cloud placement cannot claim unverified Docker support.

Repairs are serialized per PR, including Fleet jobs restored after a hub restart. Interrupted local attempts enter durable exponential backoff before recovery; changing a base SHA or check-run ID does not reset backoff for the same repair class and head. Recovery reevaluates stale triggers and retains pending work until a compatible worker is admitted. Conflict repairs are marked handled only after the forge confirms mergeability (or the PR is closed/merged), not solely from an agent's success report.

Codex's marked review-status summary comment is ignored as automation metadata; edits to that status table do not launch coding jobs. Actual inline review findings remain actionable.

Fleet 0.3.7 sends an independent lightweight heartbeat every five seconds while a coding job is active. History inventory, SQLite queries and workspace cleanup cannot hold that lease loop; SQLite inventory is also time-bounded. The lease uses current claims and delivers cancellation responses without overlapping requests. A worker lost to the watchdog enters persistent per-machine cooldown for its coding providers, so reconnecting alone does not immediately select the same failing host again.

Fleet 0.3.8 retries ordered command-log delivery through transient hub/proxy failures before reporting completion. A recovered log-upload error no longer turns a successful command into a failed job; real nonzero command exits still fail. An explicitly released claim ends delivery, matching terminal-report handling.

Learned validation capabilities are retained on the tracked PR across successful repairs, head changes, and runtime restarts, independently of the failure ledger. Existing successful job records backfill this state, so resolving one trigger cannot make the next repair forget its Docker requirement.

Independent GitHub review identity

Before publishing an automated review, the orchestrator resolves the PR author and the authenticated GitHub account. It prefers the configured reviewer; if that account authored the PR, it selects another configured GitHub integration with repository write access. Integration names alone do not establish independent identities. Invalid credentials and inaccessible repositories are skipped; rate limits and service failures retain normal retry handling. If no independent account is available, publication fails visibly rather than recording a self-review comment as a completed review. GitLab review behavior is unchanged.

Remote runner storage admission and retention

Remote runner 0.3.10 reports available bytes on both the workspace and repository-cache filesystems. Before checkout and before starting the coding harness it probes again. The Fleet scheduler admits work only with a healthy report received no more than 60 seconds ago; freshness is measured on the hub clock from the moment the telemetry arrived, so a runner whose clock runs ahead or behind is not rejected. Unknown space, old runners without telemetry, and exhausted disks are excluded; idle self-updates still work. Storage pressure never counts as "no runner online" for the stall watchdog: a queued job whose only capable runners are online but pressured (or whose pinned runner is) stays queued and logs a waiting: ... storage pressure reason until a healthy measurement arrives; only unregistered or offline runners fail the job. Automation chooses another healthy compatible runner. An unstarted initial automation assignment can move; follow-up/resume work stays on its original host to preserve local changes. A move is a fresh placement and re-applies every constraint of the original reservation: the job's validation requirements (for example Docker), host health, the per-runner failure cooldown, provider/model support, and free Orchestrator capacity on the new machine. A Docker repair therefore never lands on a runner without Docker, and the reservation follows the job to the runner that claims it.

Set CLAPILOT_REMOTE_RUNNER_MIN_FREE_GIB on runners and AGENT_ORCHESTRATOR_REMOTE_MIN_FREE_GIB on the hub (both default to 20 GiB; invalid/nonpositive values retain that default). The higher effective requirement wins. Size the floor for concurrent builds; this is admission control, not a disk reservation or a guarantee against other processes filling the disk during a build.

Pressure emits runner_storage_pressure in runner/hub logs with host, measurement time and per-path free bytes. Runner logs additionally emit runner_storage_consumers with bounded du totals for workspaces, repository caches and Codex state plus docker system df. Diagnostics are rate-limited to five minutes. Existing log monitoring should alert on these event names.

Idle workspace cleanup uses CLAPILOT_REMOTE_RUNNER_WORKSPACE_TTL_HOURS (default 24). Commits reachable from the checkout's own HEAD but absent from every remote-tracking ref are preserved without an age limit, because they exist nowhere else; other branches in the shared repository cache do not affect this decision. Dirty or untracked agent files in an otherwise published checkout are preserved for seven days and then reclaimed. Runner-written artifacts inside the checkout (.clapilot-attachments/) never count as dirt. The runner records each workspace's expected layout in .clapilot-workspace.json when it prepares the job and marks the checkout complete only after it succeeded: repository-less jobs, session-resume jobs, and checkouts that failed before completing expire normally after the TTL, whereas a completed repository checkout whose working tree is missing or cannot be verified is kept for operator inspection. Workspaces without a record (created by older runners) cannot be told apart from pre-upgrade repository-less jobs, so an unverifiable repo directory there is kept only for the previous seven-day cap and then reclaimed. Cleanup keeps running while the runner is busy and skips only the roots in use: each executing job's own root and any workspace root a resumed session is executing in through its resumeCwd; the optional Docker prune below waits until the runner is idle. A follow-up reuses its earlier checkout only while a Git working tree still exists there; if cleanup already reclaimed it, the job takes the fresh-clone path instead of running in an empty directory. Never delete preserved workspaces to recover space without first saving the work. Repository caches retain normal Git automatic garbage collection.

Docker cleanup is opt-in with CLAPILOT_REMOTE_RUNNER_DOCKER_CLEANUP=1, for a dedicated CI Docker daemon only. While idle, cleanup prunes unused builder cache older than seven days with a 20 GB retention target and dangling images older than seven days labelled com.clapilot.runner.disposable=true. It never prunes volumes, containers, or unlabelled images. Running build layers are managed by Docker; shared Docker daemons must leave this option disabled.

Incident verification for PR #1163: inspect the affected Mac's disk and diagnostic logs; preserve unpublished work before cleanup; enable dedicated-runner retention where appropriate. Wait for fresh healthy telemetry and verify the scheduler selects a healthy host. Then rerun the PR repair/build against the current PR head, recording host, SHA, free space before/after and lint/build/Docker outcomes. A GitHub Actions rerun alone is not proof of a successful Fleet repair run.

GPT-6 Sol and Luna rollout selections

Codex CLI 0.155.1 or newer is required for GPT-6 Sol and Luna. Docker images pin 0.155.1 and verify the installed version during the build; host and Fleet CLI installations must also meet this minimum.

Codex subscriptions support explicit gpt-6-sol and gpt-6-luna selections in chat and Agent Orchestrator, including web, iOS, and macOS settings. Migration 330 adds both IDs to existing Codex OAuth catalogs without changing saved defaults or the latest alias. Coding jobs and the subscription bridge preserve the selected ID. Account access remains controlled by the Codex rollout; a catalog option alone does not confirm access. Live app-server discovery supplies supported reasoning efforts and speed tiers. No unverified model-specific limits or API-key defaults are added.

GPT-6.1 Sol selections

Select gpt-6.1-sol in Codex runtime settings, coding-model priority, per-job overrides, or the new-agent model menu on web, iOS, and macOS. The exact ID reaches CLI execution and subscription sessions. Docker uses Codex 0.159.0; host and Fleet runtimes should use the same version or newer. Saved defaults remain unchanged. See Providers and models for API/subscription differences and verified capabilities.

Ultrafast speed tier

ultrafast is a service tier, separate from reasoning effort ultra. Codex 0.159.2 currently advertises it for gpt-6-astra, but not Sol. Clapilot preserves the tier in provider discovery, metadata.modelServiceTiers, interactive thread/turn requests, and Agent Orchestrator CLI service_tier overrides. Web, iOS, and macOS expose the same setting. Per-model Codex menus follow live model/list capabilities, so Sol becomes selectable when the runtime advertises it. Saved unsupported choices remain visible as not reported; execution errors are surfaced instead of silently replacing Ultrafast with Fast.

Set it under ClapilotAICore → Providers and models → model speed, or as the Agent Orchestrator Codex speed override. The global coding override must be paired with a supported model. Existing defaults remain unchanged. Host/Fleet Codex installations should use 0.159.2 or newer; Docker pins that version.

Direct first-party OpenAI API Astra requests also accept the per-model tier and send service_tier: "ultrafast" on every Responses tool-loop request. HTTP is supported; no transport change is required. Ultrafast consumes more quota/cost, and API Ultrafast supports only global/US processing, not EU regional endpoints. Existing cost estimates remain standard-tier estimates. See OpenAI Ultrafast mode.