> ## Documentation Index
> Fetch the complete documentation index at: https://docs.rumik.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# realtime socket

> talk to an agent over a plain websocket carrying pcm.

`wss://silk-api.rumik.ai/v1/agent/connect?token=<access_token>`

a bidirectional json socket: you send microphone audio, the agent sends its
audio back, plus events telling you when it starts and stops speaking. no
webrtc, no SDK, no ICE — if your platform can open a websocket and move audio
buffers, it can run an agent.

## connect

mint a token with [`/v1/register-call`](/register-call), then open the socket
with it. the token is single-use.

```
wss://silk-api.rumik.ai/v1/agent/connect?token=wct_9f8c2b1e…
```

the call starts when the socket connects — that is the moment billing and
concurrency begin, not when you registered.

## audio format

both directions carry **base64-encoded raw pcm**:

|             |                                |
| ----------- | ------------------------------ |
| encoding    | signed 16-bit, little-endian   |
| channels    | 1 (mono)                       |
| sample rate | 24 000 Hz                      |
| frame size  | 20 ms (480 samples, 960 bytes) |

<Note>
  24 kHz is fixed for this version. send whatever frame size is convenient —
  20 ms is what we recommend and what we emit — but resample to 24 kHz before
  sending, and expect 24 kHz back.
</Note>

## lifecycle

<Steps>
  <Step title="connect">
    open the socket with your `wct_` token.
  </Step>

  <Step title="wait for session.created">
    the first frame is always `session.created`. do not send audio before it —
    the agent is not in the room yet.
  </Step>

  <Step title="stream both ways">
    send `input_audio_buffer.append` continuously; read `output_audio.delta` and
    play it. voice activity and turn-taking are handled for you.
  </Step>

  <Step title="close">
    send `session.close`, or just close the socket. you get `session.closed`
    with a reason, then a normal close.
  </Step>
</Steps>

## events you receive

<ResponseField name="session.created" type="object">
  the call is live. carries `session_id`, `call_id`, `sample_rate` and
  `channels`. `call_id` is the id `register-call` already gave you — what
  appears in **conversations** and your usage, and what your tools receive as
  `{call_id}`.

  ```json theme={null}
  { "type": "session.created", "session_id": "01a05b4e…", "call_id": "01a05b4e…", "sample_rate": 24000, "channels": 1 }
  ```
</ResponseField>

<ResponseField name="output_audio.delta" type="object">
  one 20 ms chunk of the agent's speech, base64 pcm. play these back-to-back.

  ```json theme={null}
  { "type": "output_audio.delta", "audio": "AAABAAIA…" }
  ```
</ResponseField>

<ResponseField name="agent_start_talking / agent_stop_talking" type="object">
  speech boundaries, with an ISO-8601 `timestamp`. useful for driving an
  animation or a "speaking" indicator.
</ResponseField>

<ResponseField name="interruption" type="object">
  the caller started speaking over the agent. a good cue to flush whatever
  audio you have buffered for playback.
</ResponseField>

<ResponseField name="transcript / transcript.delta" type="object">
  what was said, with `role` (`user` or `assistant`) and `text`. `transcript`
  is a settled turn; `transcript.delta` is the turn so far.
</ResponseField>

<ResponseField name="session.closed" type="object">
  the call is over, with a `reason` — `client_requested`, `ended`,
  `max_duration` or `error`. the socket closes immediately after.
</ResponseField>

<ResponseField name="error" type="object">
  something went wrong, with a `code` and `message`. a malformed frame is
  reported and the call continues; a fatal one is followed by close `4000`.
</ResponseField>

<Info>
  `transcript`, `transcript.delta`, `agent_start_talking`, `agent_stop_talking`
  and `interruption` are best-effort: treat them as enrichment, and never gate
  your audio pipeline on one arriving.
</Info>

## events you send

<ParamField body="input_audio_buffer.append" type="object">
  a chunk of microphone audio: `{ "type": "input_audio_buffer.append", "audio": "<base64 PCM>" }`.
  send continuously while the caller talks. one frame may not exceed one second
  of audio.
</ParamField>

<ParamField body="input_audio_buffer.commit" type="object">
  marks the end of a turn. accepted for compatibility — the agent's own voice
  activity detection decides turns, so you do not need it.
</ParamField>

<ParamField body="input_text.send" type="object">
  `{ "type": "input_text.send", "text": "…" }` — send text instead of speech.
</ParamField>

<ParamField body="session.close" type="object">
  end the call politely. you get `session.closed` back before the socket closes.
</ParamField>

## example

a complete call: register, connect, stream a microphone, play the reply.

<CodeGroup>
  ```python python theme={null}
  import asyncio, base64, json, requests, websockets

  API_KEY = "rk_live_•••••••••"
  BASE = "https://silk-api.rumik.ai"
  AGENT = "ua_fe3277d8"
  RATE, FRAME = 24000, 480          # 20 ms of 24 kHz mono

  async def main():
      # 1. mint a single-use token on the server
      reg = requests.post(f"{BASE}/v1/register-call",
                          headers={"Authorization": f"Bearer {API_KEY}"},
                          json={"agent_id": AGENT}).json()

      url = f'{BASE.replace("https", "wss")}/v1/agent/connect?token={reg["access_token"]}'
      async with websockets.connect(url) as ws:
          reply = json.loads(await ws.recv())
          assert reply["type"] == "session.created", reply
          print("call", reply["call_id"], "at", reply["sample_rate"], "Hz")

          async def send_mic():
              # replace with real capture; here: 10 s of silence, correctly framed
              for _ in range(50 * 10):
                  pcm = b"\x00\x00" * FRAME
                  await ws.send(json.dumps({
                      "type": "input_audio_buffer.append",
                      "audio": base64.b64encode(pcm).decode(),
                  }))
                  await asyncio.sleep(0.02)
              await ws.send(json.dumps({"type": "session.close"}))

          async def receive():
              speech = bytearray()
              async for raw in ws:
                  event = json.loads(raw)
                  if event["type"] == "output_audio.delta":
                      speech.extend(base64.b64decode(event["audio"]))
                  elif event["type"] in ("transcript", "transcript.delta"):
                      print(f'{event["role"]}: {event["text"]}')
                  elif event["type"] == "session.closed":
                      print("closed:", event["reason"])
                      break
                  elif event["type"] == "error":
                      print("error:", event["code"], event["message"])
              return bytes(speech)      # 24 kHz mono PCM — write a WAV header to save

          _, audio = await asyncio.gather(send_mic(), receive())
          print(f"{len(audio) / (RATE * 2):.1f}s of agent audio")

  asyncio.run(main())
  ```

  ```javascript browser theme={null}
  // The page never sees the API key — your server mints the token.
  const { access_token } = await fetch("/api/agent-token", { method: "POST" }).then((r) => r.json());
  const ws = new WebSocket(`wss://silk-api.rumik.ai/v1/agent/connect?token=${access_token}`);

  const RATE = 24000;
  const play = new AudioContext({ sampleRate: RATE });
  let playhead = 0;

  ws.onmessage = async (event) => {
    const msg = JSON.parse(event.data);

    if (msg.type === "session.created") {
      startMic(ws);                                   // see below
    } else if (msg.type === "output_audio.delta") {
      // schedule each chunk back-to-back so playback is gapless
      const bytes = Uint8Array.from(atob(msg.audio), (c) => c.charCodeAt(0));
      const pcm = new Int16Array(bytes.buffer);
      const buffer = play.createBuffer(1, pcm.length, RATE);
      const channel = buffer.getChannelData(0);
      for (let i = 0; i < pcm.length; i++) channel[i] = pcm[i] / 32768;

      const src = play.createBufferSource();
      src.buffer = buffer;
      src.connect(play.destination);
      const at = Math.max(play.currentTime + 0.05, playhead);
      src.start(at);
      playhead = at + buffer.duration;
    } else if (msg.type === "transcript") {
      console.log(`${msg.role}: ${msg.text}`);
    } else if (msg.type === "session.closed") {
      console.log("closed:", msg.reason);
    }
  };

  async function startMic(ws) {
    const stream = await navigator.mediaDevices.getUserMedia({ audio: { channelCount: 1 } });
    const ctx = new AudioContext({ sampleRate: RATE });
    const source = ctx.createMediaStreamSource(stream);

    // A worklet keeps capture off the main thread.
    const code = `class Tap extends AudioWorkletProcessor {
      process(inputs) { const c = inputs[0]?.[0]; if (c) this.port.postMessage(new Float32Array(c)); return true; }
    } registerProcessor('tap', Tap);`;
    await ctx.audioWorklet.addModule(URL.createObjectURL(new Blob([code], { type: "application/javascript" })));

    const tap = new AudioWorkletNode(ctx, "tap");
    tap.port.onmessage = (e) => {
      if (ws.readyState !== WebSocket.OPEN) return;
      const pcm = new Int16Array(e.data.length);
      for (let i = 0; i < e.data.length; i++) {
        const s = Math.max(-1, Math.min(1, e.data[i]));
        pcm[i] = s < 0 ? s * 0x8000 : s * 0x7fff;
      }
      const bytes = new Uint8Array(pcm.buffer);
      let bin = "";
      for (const b of bytes) bin += String.fromCharCode(b);
      ws.send(JSON.stringify({ type: "input_audio_buffer.append", audio: btoa(bin) }));
    };
    source.connect(tap).connect(ctx.destination);
  }
  ```
</CodeGroup>

## connection errors

a failure before the call starts is still delivered on the socket: we accept the
connection, send one `error` frame so you know why, and then close.

| close code | meaning                                                                                                                                                     |
| ---------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `4401`     | the token was unknown, already spent, or expired — mint a new one                                                                                           |
| `4000`     | the call could not start. the `error` frame's `code` says why: `insufficient_balance`, `concurrency_limit_exceeded`, `agent_start_failed`, `not_configured` |
| `1000`     | normal end, after `session.closed`                                                                                                                          |

```json theme={null}
{ "type": "error", "code": "concurrency_limit_exceeded", "message": "Silk is already processing 4 requests for this account. …" }
```

<Tip>
  a `4000` close with `concurrency_limit_exceeded` is the socket equivalent of a
  `429`. check [`/v1/agent/limits`](/agent-limits) before connecting if you want
  to queue callers rather than turn them away.
</Tip>

## ending and billing

the call ends when you send `session.close`, when the socket drops, when the
agent hangs up, or when it hits the maximum session length. you are billed for
the seconds it actually ran, and the transcript and recording appear in
**conversations** shortly afterwards.