Clapilot Docs
Docs

Call & Fax Agent

SIP-backed calling and fax module with native RTP/live-audio bridge for realtime phone conversations on a shared line.

What it does

Call & Fax Agent is a bundled React module for SIP-based telephone and fax workflows on one shared SIP line. Outbound calls are bridged into an OpenAI GPT-Live-1, OpenAI Realtime or Google Gemini Live audio session through a native RTP worker, so the live phone agent can hold low-latency duplex conversations, execute Clapilot tools during the call, and write an automatic post-call summary. The same line also sends outbound faxes (text or document-backed) and routes externally received inbound faxes into Dokumente.

Current scope:

  • store one global SIP/router configuration and test SIP registration via the native pjsua runtime path
  • queue outbound calls from the module UI
  • queue outbound text faxes from the same module and launch fax-send popups directly from Dokumente, Word, Excel, and Notizen for document-backed sends
  • select a target customer from the Mandanten database and prefill the phone number from customer master data
  • route externally supplied inbound fax files into Dokumente via mandanten/_inbox/... and keep a dedicated fax audit log
  • capture caller speech, agent speech, and tool outcomes from live calls so summaries are based on the real conversation flow
  • keep a registered inbound SIP listener alive on the configured local SIP port when incoming calls are enabled, and hand inbound calls into the same live-audio worker
  • restrict inbound live-tool access by caller number so only approved callers get the full internal tool surface while all other callers stay in a public-information-only mode
  • enforce a mutual-exclusion line lock so one active voice or fax activity blocks the other
  • persist worker/runtime state, call history, fax history, and fax audit events in PostgreSQL

Current non-goal: multiple provider profiles at once (one shared line/configuration).

How to open / enable it

  • Module: /modules/call-agent — slug call-agent, manifest bundled-modules/call-agent/module.json, React renderer src/components/modules/call-agent-module.tsx. The module view follows the current Clapilot styleguide (display-type module title, status badge, standard navy segmented controls, clapilot-field inputs, borderless quiet tiles and hairline outlined rows).
  • Settings: /settings/call-agent (redirects to the admin-gated /admin/call-agent page), panel src/components/call-agent-settings-panel.tsx.
  • Feature toggles in the settings decide what the line does: incoming calls, outgoing calls, fax (inbound/outbound), plus default calendar/notes access for the live agent.

Key workflows

Start an outbound live call

  1. Open the module, optionally pick a customer from the Mandanten database (prefills the phone number from customer master data and biases the live agent toward that customer context).
  2. Choose the per-call permissions (see below) and start the call. The worker dials through the configured router account, attaches the RTP audio to the native bridge process, and streams the audio into the configured live provider.
  3. During the call the live agent can execute Clapilot tool calls through the existing agent tool proxy (/api/agent-runtime/tool-proxy) with the initiating user as tool context.
  4. After the call, an automatic post-call summary is written into the persisted call log, based on captured caller speech, agent speech, and tool outcomes.

Per-call permissions:

  • Kalenderzugriff exposes calendar tools to the live phone agent for that call
  • Notizenzugriff exposes Notizen tools to the live phone agent for that call
  • Kundenzugriff exposes read access to Mandanten tools during the call
  • Dokumentzugriff exposes read access to Clapilot document tools during the call
  • when a customer is selected in the module UI, the live phone agent is biased toward that customer context and document queries default to that Mandant where possible
  • without those toggles, the live phone agent is explicitly instructed not to access those domains

History and transcripts

The module opens with a compact history on the left. Call and fax rows have a fixed 80px height, with direction, status, and timestamp. Select a call to read its instructions, error details, and complete recorded transcript in the detail pane; an explicit empty state appears when no transcript has been recorded. Details refresh with the existing five-second polling cycle.

The top-right + opens the call composer. Its fax tab retains text/document fax composition. Selecting a fax opens its details and retry/cancel actions. On narrow screens the selected detail or composer replaces the history, with a back-to-history action. iOS and macOS use this same responsive module through ModuleWebContainer; there is no separate native Call Agent screen.

Page context exposes view, historyMode, selectedCallId, and selectedFaxId independently of the active call/fax IDs. Chat and live agents retrieve the selected call's transcript_text using call_agent_list_calls and match the selected ID, or use faxes_get for a selected fax.

Receive inbound calls

When incoming calls are enabled, the worker keeps a registered inbound SIP listener alive on the configured local SIP port and hands inbound calls into the same live-audio worker (mode: incoming_live_audio). Inbound live-tool access is restricted by caller number: only numbers in approved_incoming_numbers (newline- or comma-separated in the settings) get the full internal tool surface; all other callers stay in a public-information-only mode.

Send a fax

  • Queue outbound text faxes from the module UI, or launch fax-send popups directly from Dokumente, Word, Excel, and Notizen for document-backed sends (src/components/fax-send-dialog.tsx).
  • fax_transport_mode = g711 starts a real outbound SIP fax session through the native call-agent-fax-bridge, renders text-only or cover-page input into a PDF, converts the final transmission into TIFF, and only marks the fax sent after the native fax engine reports success. disabled rejects outbound fax enqueue requests immediately.
  • Inbound fax handling currently lands in the shared document inbox/audit flow only after an external handoff to POST /internal/call-agent/faxes/receive; the native SIP line work in this codebase is currently focused on inbound voice plus outbound fax delivery.

Shared line rule

Voice and fax use the same stored telephony line:

  • one active voice call blocks fax send/receive
  • one active fax blocks starting a new voice call
  • blocked inbound fax attempts are still logged in call_agent_faxes with status blocked
  • the runtime state persists active_kind plus active_call_id / active_fax_id

This mutual-exclusion rule is enforced in the native worker and surfaced in the module UI and agent tool contracts.

How the agent can drive it

Chat/live agents get a native ClapilotAICore tool family for the module (see Agent Tool Contracts):

  • call_agent_get_status — worker/SIP/shared-line state
  • call_agent_start_call — queue an outbound call on the shared line
  • call_agent_end_call — end the active call
  • call_agent_list_calls — call history
  • call_agent_search_customers — Mandanten lookup for call targets
  • faxes_list / faxes_get / faxes_send / faxes_retry / faxes_cancel — fax queue management

Separately, the live phone agent inside a call executes Realtime function calls through /api/agent-runtime/tool-proxy, scoped by the per-call permission toggles described above.

Configuration & limits

Settings fields

The settings page persists SIP fields such as:

  • router_ip, sip_domain, published_ip, outbound_published_ip
  • sip_caller_id, sip_username, sip_password, optional sip_auth_user, sip_realm
  • sip_local_port, sip_port, sip_transport
  • realtime_provider, realtime_model, realtime_voice, system_prompt
  • approved_incoming_numbers — newline- or comma-separated approved inbound caller numbers that may use the full Call Agent toolset during incoming calls
  • fax_enabled, fax_station_id, fax_header_text, fax_transport_mode
  • feature toggles for incoming/outgoing/fax/calendar/notes and auto_answer

The settings UI renders realtime_provider and realtime_model as paired dropdowns. Model choices are derived from the configured ClapilotAICore provider rows for OpenAI Realtime or Google Gemini Live, with the current persisted model kept visible as a fallback option.

Runtime architecture

Native worker source:

  • services/clapilot-agent/src/call-agent/index.mjs
  • services/clapilot-agent/src/call-agent/live-audio-session.mjs
  • services/clapilot-agent/src/call-agent/sip-proxy-runtime.mjs plus the proxy scripts scripts/sip-udp-proxy.mjs (SIP) and scripts/udp-port-proxy.mjs (RTP)
  • services/clapilot-agent/native/call_agent_live_bridge.cpp
  • services/clapilot-agent/native/call_agent_fax_bridge.cpp
  • services/clapilot-agent/src/call-agent/render_text_fax.py

Current runtime mode:

  • worker version: pjsua-live-audio-v2
  • SIP registration uses the native pjsua stack
  • Test Connect performs a temporary registration on a free port and unregisters before returning. Its result verifies that probe only; it does not prove the persistent listener or configured port works. Probe diagnostics do not overwrite the live worker's registration, heartbeat, active call, or error.
  • active calls use a native RTP bridge process (call-agent-live-bridge) plus a server-side provider session selected by realtime_provider; supported live providers are openai and google_gemini
  • the worker keeps the SIP password out of process arguments on every active path: live and fax bridges receive it through a one-shot private inherited pipe; pjsua receives an inherited regular-file descriptor, with its private temporary configuration unlinked before launch so it can reopen /proc/self/fd/5 on Linux; the password must never be included in process arguments or diagnostic output
  • an internal SIP/RTP proxy layer inside the existing clapilot-agent container fronts the native bridges; Docker publishes the UDP/TCP port range 5060-5090 for it (override via CLAPILOT_AGENT_HOST_PORT_START / CLAPILOT_AGENT_HOST_PORT_END)
  • OpenAI sessions use G.711 u-law over the Realtime WebSocket; Gemini sessions use raw PCM over Gemini Live and are downsampled back into the RTP leg; caller-side STT is explicitly biased to German so short German answers are not mis-transcribed as English
  • status/call rows persist mediaMode: realtime_live_audio while a live bridge is active
  • fax sending/receiving runs in the same worker under a dedicated fax path that reuses the stored SIP line settings; fax transport mode can be g711 or disabled

SIP credential rotation after argv exposure

Deploy the fixed web and clapilot-agent images before rotating the credential. Then change the SIP password at the router/provider, enter the new value under Einstellungen → Call Agent, save, and restart the active Call Agent session. Verify registration, place a test call, and exercise fax if it is enabled. Finally, inspect the running pjsua, call-agent-live-bridge, and call-agent-fax-bridge command lines via /proc/<pid>/cmdline (or ps auxww). The native bridges should contain --password-fd 5; pjsua should reference /proc/self/fd/5; none may contain either the old or new password. Treat any password previously passed with --password as compromised and revoke it.

Clapilot web /api/call-agent/* clapilot-agent worker pjsua2 live/fax bridge 8 kHz PCM RTP leg SIP/RTP proxy loose route (;lr) ports 5060-5090 Router registrar PSTN Live provider OpenAI Realtime / Gemini Live WebSocket audio + tools

Inbound SIP routing notes:

  • default local SIP listener: 5060/udp
  • expected forwarded UDP range for inbound voice on one line: local SIP port plus RTP local_port + 2 through local_port + 9
  • published_ip must match the IP address that the router or upstream PBX reaches after forwarding
  • the native bridges always register and dial against the real router/registrar URI and attach the internal SIP proxy as a loose-route outbound proxy (--proxy, pjsua2 sipConfig.proxies), so digest authentication stays valid on strict registrars (for example Speedport Smart 4R firmware) that reject rewritten Request-URIs
  • the SIP proxy derives its top-Via branch deterministically from Call-ID + CSeq + inner Via branch (RFC 3261 §16.6), so UDP retransmissions of one request keep their transaction identity instead of replaying the same digest nonce as distinct transactions — strict registrar firmware punishes that with 401/403 and extended 404 lockouts
  • when the router reports a different source address than the configured published_ip (SIP received= annotation), the worker surfaces a runtime warning naming both addresses; this typically means the host got a new DHCP lease after a router swap, so update published_ip/outbound_published_ip (and ideally reserve a fixed LAN address for the host in the router UI)

Runtime toggles

  • CLAPILOT_CALL_AGENT_SIP_LOCAL_PORT — optional env override for the effective inbound SIP listener port; when set, it overrides the stored sip_local_port from the settings UI at runtime without changing the saved DB value
  • CLAPILOT_CALL_AGENT_LIVE_AUDIO_ENABLED — default enabled; set to false to fall back to prompt-file playback instead of live duplex audio
  • CLAPILOT_CALL_AGENT_TRANSCRIPTION_LANGUAGE — default de; pins caller-side speech transcription to German for phone calls unless intentionally overridden

Endpoints

Internal native endpoints:

  • GET /internal/call-agent/status
  • GET /internal/call-agent/calls
  • POST /internal/call-agent/calls/start
  • POST /internal/call-agent/calls/:id/end
  • GET /internal/call-agent/faxes
  • GET /internal/call-agent/faxes/:id
  • POST /internal/call-agent/faxes/send
  • POST /internal/call-agent/faxes/:id/retry
  • POST /internal/call-agent/faxes/:id/cancel
  • POST /internal/call-agent/faxes/receive
  • POST /internal/call-agent/test-connection

Web endpoints:

  • GET/POST /api/call-agent/config
  • GET /api/call-agent/status
  • GET /api/call-agent/calls
  • POST /api/call-agent/calls/start
  • POST /api/call-agent/calls/:id/end
  • GET /api/call-agent/customers (Mandanten picker for the module UI)
  • GET /api/call-agent/faxes
  • GET /api/call-agent/faxes/:id
  • POST /api/call-agent/faxes/send
  • POST /api/call-agent/faxes/:id/retry
  • POST /api/call-agent/faxes/:id/cancel
  • POST /api/call-agent/test-connection

Data model

Tables introduced for the module:

  • call_agent_configs
  • call_agent_runtime_state
  • call_agent_calls
  • call_agent_faxes
  • call_agent_fax_events

Migrations: db/migrations/041_call_agent.sql (base tables) plus the follow-ups 042 (customer/document access), 043 (published IPs), 054 (realtime provider), 059_call_agent_fax.sql (fax tables), 060, 061, 063 (fax status/transport refinements), 064 (default SIP local port), and 068 (inbound tool allowlist).

Troubleshooting

  • Registration suddenly answers 401/403, then 404 to everything — strict registrar firmware (Speedport Smart 4R) rejects digest mismatches and rate-limits aggressively. Two worker-side causes are already fixed: the bridges now register against the real router URI with the internal proxy as a loose-route hop (so the digest URI matches), and the proxy's Via branch is deterministic per SIP transaction (so UDP retransmissions are not treated as nonce replays). If a lockout was triggered, the router may keep answering 404 for an extended period before recovering.
  • Silent calls / no inbound audio after a router swap — the router now sees a different source address than published_ip (surfaced as a runtime warning naming both addresses, from the SIP received= annotation). Update published_ip/outbound_published_ip and reserve a fixed LAN address for the host.
  • Fax enqueue rejected immediately — fax_transport_mode is disabled, or the shared line is busy with a voice call (blocked inbound attempts are logged with status blocked).
  • Inbound callers cannot use internal tools — only numbers listed in approved_incoming_numbers get the full tool surface; everyone else is intentionally kept in public-information-only mode.

GPT-Live-1

Select gpt-live-1 in the telephone module's Realtime model field. This uses the Live session protocol with G.711 mu-law audio at 8 kHz and client delegation to ClapilotAICore. Existing call permissions, user/service-principal context and post-call summaries still apply. The shared Live Voice setting does not overwrite the telephone module's independent selection. Unsupported legacy voice names resolve to Marin for GPT-Live; Realtime voice selection is unchanged.

GPT Live tool routing

GPT Live delegates spoken requests to the native runtime through clapilot_delegate. Calendar and other app tools are available only when enabled on the saved call record. The internal proxy verifies the active call and its user or approved service principal, restricts the run to the call permissions, and checks them again when executing tools. Hang-up requests target the originating call. A delegation failure is returned to the voice agent rather than reported as a completed action.

For a connected OpenAI-Codex provider, select gpt-live-1-codex explicitly in the telephone model field. Its OAuth WebRTC gateway converts telephone G.711 audio into the same existing delegation flow. The subscription default voice is Cove. Other telephone model selections keep their existing API provider routing.