Background Workers

This page lists every background process that entrypoint.sh starts inside the clapilot web container, what each one does, and which of them are legacy fallbacks that native ClapilotAICore system automations replace. Read it when you need to know "what runs in the background and where".

There are two automation layers:

• Web-container workers: Node scripts started by entrypoint.sh next to the app server, plus singleton in-process workers registered by src/instrumentation.ts.
• Native system automations: preinstalled jobs owned by clapilot-agent and visible in /geplante-aufgaben (Agent Inbox Sync, User Inbox Sync, Inbox Document Processing, Document Post-Extraction Queue, Learning Curator, Memory Dreaming, Skill Curator).

When CLAPILOT_SYSTEM_AUTOMATIONS_ENABLED=true (the default), the two legacy workers — task scheduler and agent email poller — are skipped at startup and their responsibilities move to the native automations. All other workers always run alongside the native runtime.

Instance performance check (user-schedulable)

In addition to the preinstalled system automations above, users can create a Performance Check automation from /geplante-aufgaben (create dialog → Task type → Performance check). It is a normal, editable scheduled task, so the user picks when it runs (once / interval / weekdays) and the report channel (the automation result target — main session, a team-chat channel, or an approved external channel) with the standard controls.

Unlike an agent task it does not spend an LLM turn: it maps to the deterministic clapilotPerformanceCheck job payload handled in services/clapilot-agent/src/jobs/system-automations.mjs (runPerformanceCheckJob). On each run it computes:

• terminal agent-run failure rate and interactive completed-run latency p50/p90 for chat, group_chat, and channel,
• provider-attempt failure rate and the top recurring run/provider errors,
• provider timeouts, both overall and per provider/model, and the total attempt duration burned by those timeouts,
• circuit-breaker state and the number of provider.circuit_open events (openings or attempts skipped while open) in the lookback window.

It fails when the terminal run failure rate exceeds maxFailureRatePct (default 10), interactive p90 exceeds maxInteractiveP90Ms (default 90000), any provider/model exceeds maxProviderTimeoutsPerModel (positive number, default 24) full-budget timeouts, or any circuit is currently open. The lookback defaults to windowHours: 24; these settings are configurable in the performance-check payload. Successful fallback runs therefore cannot hide excessive full-budget timeout load or an open circuit.

The report adds a provider-timeout summary after chat latency, a Schutzschalter: section for circuit states, and Provider-Timeouts je Modell: for the model breakdown; empty sections are omitted. Volle Zeitbudgets means timeout attempts without a reduced circuit budget; begrenzt includes both half-open probes (metadata.circuitBudget = "probe") and attempts bounded after reaching the full-budget limit ("bounded"). Both categories contribute to timeout totals and burned time, but only full-budget timeouts count toward the per-model breach threshold. The displayed model breakdown is sorted by total timeout count and limited by topErrorLimit (default 8); totals and breaches include every model.

Circuits are shown as open until their deadline, half-open during recovery, or observed/degraded otherwise, with current full-budget timeout usage and whether their budget is bounded. Usage counts timestamps newer than the circuit's rolling window (default one hour, limit two). On older instances without agent_provider_circuit_states, the result returns circuits.available: false and an empty state list while retaining timeout metrics and event counts.

The rendered report uses the existing automation result-target path (deliverSystemAutomationRunResult → /api/agent-runtime/assistant-message), honoring notify_result_mode. Failed checks remain visible in informational mode; passing checks are suppressed there. Use Run now on the task for an on-demand report.

Worker inventory

Document post-extraction review events are drained by the native clapilot-system:document-postprocess queue job. The queue is durable per document event, collapses identical burst callbacks by document_id plus a canonical payload hash, and creates a new revision when an already-seen document's extracted content changes. A revision that arrives while the previous revision is running is retained and becomes pending after the older run finishes, so updates are not lost. The worker processes up to ten documents per pass but executes model runs sequentially. Each run uses a stable queue-row/revision idempotency key, preserves configured specialist prompts/skills and delivery targets, and counts only real execution failures against the retry budget.

Only provider-originated errors are eligible for global 429/provider-limit handling; arbitrary document or model output text cannot pause the queue. Structured reset metadata and short runtime cooldown messages such as Retry in 12s are honored before the one-hour fallback. A pause records one warning event with the enabled waiting-document count, leaves failure attempts unchanged, and reschedules the job to the provider reset time. Processed and terminal rows older than DOCUMENT_POSTPROCESS_QUEUE_RETENTION_DAYS (90 by default) are pruned in bounded batches.

Task scheduler (legacy)

Script: scripts/task-scheduler.mjs

Current responsibility:

1. Process inbox trigger files.
2. Index _inbox files into dokumente placeholder rows.

Scheduled task execution is no longer performed by this worker. Planned/recurring tasks run via native clapilot-agent jobs (agent_jobs), while Clapilot keeps synchronized DB metadata in scheduled_tasks.

When CLAPILOT_SYSTEM_AUTOMATIONS_ENABLED=true (default), this legacy worker is skipped and replaced by the preinstalled Inbox Document Processing automation.

Learning curator

Runtime owner: clapilot-agent

The preinstalled Learning Curator automation runs inside the native runtime with the model selected on /geplante-aufgaben. Safe evidence-backed facts use an opt-out policy and become active immediately. The scheduled curator checks only canonical facts that are new or whose content changed since its last decision; keep is the default, while high-confidence false, contradicted, duplicate, non-durable, runtime-noise, unsafe, or clearly worthless facts are rejected. Its payload supports model, batchSize, and minimumRejectConfidence (default 0.8). A blank model fails closed. Every result records model and decision provenance, and manual admin/user decisions are authoritative. Dreaming v2 retains a targeted deterministic safety pass for its own new assertion ids so eligible assertions can reach durable recall and the Knowledge Graph immediately, while the scheduled agentic curator supplies asynchronous quality control.

Legacy upgrades may also receive the hidden one-shot Memory v2 graph bootstrap native job from migration 201_memory_v2_graph_bootstrap.sql. It is inserted only when historical Dream v1/legacy graph data exists while no active v2 graph exists; a completed Dream v2 does not suppress the repair when it produced no graph. Its version-gated job type and payload are ignored by older agents during a rolling restart. The fixed-id job runs forced, graph-only Dreaming with Wiki proposals disabled and disables itself only after all partitions, targeted curation, and projection-outbox work succeed; skipped/partial/empty or failed work remains retryable. A migration rerun never re-enables an existing completed/disabled row.

The native Memory v2 projection worker starts with the agent and polls every 30 seconds. It claims due assertion-projection outbox rows with the ledger lease, coalesces graph work per Dream or instance, and never overlaps a still-running local batch. Provider and validation failures remain in the bounded outbox retry lifecycle instead of waiting for another Dreaming or retention run to wake them.

Skill Curator

Runtime owner: clapilot-agent

The preinstalled Skill Curator runs daily at 03:40 Europe/Berlin in the isolated clapilot-system:skill-curator session. It loads every workspace skill marked origin: agent, then selects up to 20 completed non-system runs from the previous 24 hours that contain successful tool activity. Only trimmed request/outcome summaries are sent to one schema-constrained maintenance-model completion; full transcripts are not included.

The model can propose at most six create-draft, update, merge, or archive actions. The runtime independently enforces the action cap, safe slug and body limits, agent-only ownership, pinned-skill immunity, and the invariant that no skill file is deleted. Proposed new skills remain drafts until an admin activates them in the Skill Store. Every execution, including clean skips when the maintenance model is unavailable, is audited in agent_skill_curator_runs.

Agent email poller (legacy)

Script: scripts/agent-email-poller.mjs

1. Load mailbox settings from app_settings (env overrides allowed).
2. Connect via IMAP and fetch unseen UIDs.
3. Claim UID jobs in agent_email_jobs.
4. Auto-reply allowlisted senders or delegate to the runtime.
5. Persist draft/result and update job state.

Requires runtime dependency imapflow.

When CLAPILOT_SYSTEM_AUTOMATIONS_ENABLED=true (default), this legacy worker is skipped and replaced by the preinstalled Agent Inbox Sync and User Inbox Sync automations. User Inbox Sync defaults to every 10 minutes and no output node, so it writes operational history to the run log unless an admin adds an output node. Native mailbox sync writes attachment files into _inbox, inserts placeholder dokumente rows, tracks processing in email_ingestion_jobs, forwards the configured agent_email_auto_reply_allowlist safe-case flag into the delegated agent prompt for immediate agent-mailbox auto-send decisions, and also triggers the personal /emails prepared-answer workflow automatically for newly imported user mailbox messages when the global email_auto_analysis_enabled setting is on.

Worker controls

TASK_SCHEDULER_ENABLED=true
TASK_SCHEDULER_POLL_SECONDS=30
TASK_SCHEDULER_BATCH=10
TASK_SCHEDULER_RETRY_MINUTES=5

AGENT_EMAIL_POLL_ENABLED=true
AGENT_EMAIL_POLL_SECONDS=60
AGENT_EMAIL_POLL_BATCH=10
AGENT_EMAIL_POLL_RETRY_MINUTES=10
AGENT_EMAIL_POLL_STALE_MINUTES=30

RAG_INDEXER_ENABLED=true
RAG_INDEXER_POLL_SECONDS=30
RAG_INDEXER_BATCH=4
RAG_INDEXER_RETRY_MINUTES=10

GOOGLE_SYNC_ENABLED=true
# Drive: downloads per account/cycle, auto-import window (days, "all" or 0 = on demand), weekly full rescan
GOOGLE_SYNC_DRIVE_MAX_FILES=100
GOOGLE_SYNC_DRIVE_AUTO_IMPORT_DAYS=90
GOOGLE_SYNC_DRIVE_IMPORT_SECONDS=180
GOOGLE_SYNC_DRIVE_FULL_RESCAN_DAYS=7
ICAL_SYNC_ENABLED=true
CLAPILOT_INSTANCE_CLEANUP_ENABLED=true
CLAPILOT_SOCIAL_MEDIA_PUBLISHER_ENABLED=true

CLAPILOT_SYSTEM_AUTOMATIONS_ENABLED=true

Webhook release delivery regression guard

Webhook Team Chat authorization uses the persisted active run gate and automation identity, including queue fallback and specialist event sessions without an interactive user. Only the enabled automation's configured rooms are available to its active service principal.

A completed agent run is distinct from a delivered webhook result. A failed or unacknowledged runtime delivery leaves the queue eligible for delivery retry (attempt counted, task marked error, terminal failure announced); it must not be marked processed. A pending acknowledgement (another caller owns the delivery reservation) defers the row for a short time without counting an attempt or announcing a failure; duplicate and skipped acknowledgements are successful. Every configured output is attempted on each try and a single aggregate error lists the targets that failed. Retries reuse the completed output and the same per-run/target delivery key. A delivery-phase failure (including a 401/403 from the internal delivery route) is retried on the plain 60s budget without a provider auth event and leaves the completed run's error fields untouched. A confirmed direct post suppresses the automatic duplicate. Runless system job results (for example performance checks) are keyed by job occurrence so a job retry cannot repost a target that was already delivered.

Regression checks: npx vitest run services/clapilot-agent/src/jobs/index.test.mjs services/clapilot-agent/src/jobs/system-automations.test.mjs src/lib/agent-runtime/tool-proxy.test.ts src/app/api/agent-runtime/automation-delivery-dedup.test.ts. Run the real PostgreSQL authorization cases with CLAPILOT_QUEUE_INTEGRATION_DATABASE_URL=<test database URL> npx vitest run src/lib/automation-origin-session.integration.test.ts; they use connection-local temporary tables.