vision agents

a vision agent is a voice agent that can see. while you talk, it watches your camera and answers with what it sees and hears, in its voice, exactly like any agent.


voices	silk mulberry 1.5 (`mulberry`: a named `voice`, a style `description`, one `language`) or silk muga 1 (`muga`: one `language`), set up exactly as on a voice agent
calls	web calls (webrtc), with the caller’s camera on
phone	not supported — a phone call has no camera
price	₹3.99 / min (india), $0.0831 / min elsewhere

everything else — prompt, greeting, language, turn-taking, tools, variables, versions — works exactly as it does for a voice agent.

the quickest way to try one is the dashboard: vision in the sidebar → new vision agent → test live. it asks for your camera and microphone, and you talk to it while it watches.

create one

create the agent with "kind": "vision", then deploy it — like any agent, it answers api calls only once a version is live.

curl -X POST https://silk-api.rumik.ai/v1/agents \
  -H "Authorization: Bearer rk_live_•••••••••" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "shop assistant",
    "kind": "vision",
    "systemInstruction": "you are asha, a helpful shop assistant on a video call. help the caller pick clothes they show you.",
    "greeting": "hi! show me what you are looking at.",
    "ttsConfig": {
      "model": "mulberry",
      "voice": "Emma",
      "description": "a warm, friendly 30s voice, conversational pace"
    },
    "language": "english"
  }'

curl -X POST https://silk-api.rumik.ai/v1/agents/ua_fe3277d8/deploy \
  -H "Authorization: Bearer rk_live_•••••••••"

string

default:"voice"

vision for an agent that sees the caller’s camera. set it when you create the agent; it cannot be changed afterwards (a PATCH that sends it is refused).

object

a vision agent speaks with silk mulberry 1.5 or muga: model is mulberry (with the same named voice and style description a mulberry voice agent takes) or muga (see voices). leave model out and you get mulberry; any other voice is refused.

the agent already knows it is on a video call — you don’t need to tell it that in your prompt. write the prompt about the job: what it should look for, what to do with it.

call it (web call)

the browser joins the call with a livekit client and turns on its camera next to its microphone. nothing else changes from a web call.

start the call on your server

POST /v1/webcall with the vision agent’s id, exactly as for a voice agent.

join and turn on the mic and the camera

connect with the returned host and token, then enable the microphone and the camera.

talk

the agent watches your camera while you talk. turn the camera off and it knows it can’t see you; turn it back on and it sees again.

browser (livekit-client)

import { Room, VideoPresets } from "livekit-client";

const { host, token } = await fetch("/api/start-vision-call").then((r) => r.json());

const room = new Room();
room.on("trackSubscribed", (track) => {
  if (track.kind === "audio") document.body.appendChild(track.attach()); // the agent's voice
});
await room.connect(host, token);
await room.localParticipant.setMicrophoneEnabled(true);
// 540p at 15 fps is plenty for the agent.
await room.localParticipant.setCameraEnabled(true, {
  resolution: { ...VideoPresets.h540.resolution, frameRate: 15 },
});

what the agent sees

what your camera shows while you talk, and it uses that in its replies.
camera off — the agent knows it can’t see you, so it says so rather than guessing.
nothing it sees is stored, and call recordings stay audio-only.

errors

status	code	when
400	`agent_vision_voice_not_supported`	creating, updating or calling a vision agent on a voice other than silk mulberry 1.5 (`mulberry`) or muga (`muga`)
422	`agent_vision_phone_unsupported`	a phone call (outbound, or a number pointed at it) for a vision agent

documentation

sdk

integrations

cookbook

api reference

create one

call it (web call)

what the agent sees

errors

​create one

​call it (web call)

​what the agent sees

​errors

create one

call it (web call)

what the agent sees

errors