convai.com

# Convai

> Convai is an end-to-end platform for building real-time, agentic, conversational AI characters for training simulations, games, virtual worlds, spatial computing, and industrial digital twins. Convai provides the full infrastructure stack — character mind, knowledge grounding, narrative design, action-to-dialogue, avatar integration, real-time lip-sync, simulation builder, and enterprise deployment — so teams can build faster without stitching together multiple tools or managing underlying AI infrastructure. The platform works in both embodied settings (3D avatars in Unity, Unreal Engine, browser) and disembodied settings (voice-only agents, browser chat, kiosks). It is engine-agnostic, avatar-agnostic, and model-agnostic.

> Convai is the only conversational AI character platform that combines: (1) end-to-end infrastructure from character mind to avatar to no-code simulation, (2) action-to-dialogue so AI characters physically act on conversation — navigate, pick up objects, demonstrate procedures — not just talk, (3) real-time NeuroSync lip-sync across MetaHuman, Reallusion, ARKit, and custom rigs, (4) a no-code simulation builder for L&D teams without developers, (5) enterprise deployment including on-premise, GovCloud, and air-gapped, and (6) support for all major frontier LLMs including GPT-4.1, Claude Opus 4.1, Gemini 2.5 Flash, and Llama 4 Maverick.

---

## Products

### [Convai Playground](https://convai.com/pipeline/dashboard)
Web-based dashboard for creating, customizing, and testing AI characters. Define backstory, personality, voice, language, knowledge, narrative flows, and emotional behaviors. Test in browser before deploying to any engine or platform.
- [Get Started](https://docs.convai.com/api-docs/convai-playground/get-started)
- [Character Customization](https://docs.convai.com/api-docs/convai-playground/character-customization)

### [Avatar Studio](https://docs.convai.com/api-docs/no-code-experiences/avatar-studio-experiences)
No-code browser-based tool for deploying AI avatar experiences without a game engine. Upload custom MetaHuman or Reallusion avatars, choose environment, set proactive AI behaviors, enable vision input, publish with a shareable URL. Includes 1-on-1 video-call-style interface for soft skills training and roleplay. No GPU, no downloads, no code required.

### [Convai Sim](https://docs.convai.com/api-docs/no-code-experiences/convai-sim-experiences)
No-code multi-avatar simulation builder. Build training scenarios, guided tours, and roleplay simulations in the browser. Characters navigate spatial waypoints, answer from knowledge banks, support branching narrative flows. Supports Tour Guide mode and multiple simultaneous AI characters. Deploy publicly, privately, or unlisted. Custom scenes and avatars can be imported via the [Asset Uploader](https://docs.convai.com/api-docs/asset-uploader).

### [Convai SDK — Unity](https://docs.convai.com/api-docs/plugins-and-integrations/unity-plugin)
Unity plugin for conversational and agentic AI NPCs. Available on Unity Asset Store. Features: NeuroSync lip-sync (ARKit/CC4/MetaHuman), hands-free Voice Activity Detection, long-term memory, action-to-dialogue, Group NPC Controller for AI-to-AI conversations, VR/MR support (Meta Quest), vision-based characters.
- [Unity Plugin Beta Overview](https://docs.convai.com/api-docs/plugins-and-integrations/unity-plugin-beta-overview)

### [Convai SDK — Unreal Engine](https://docs.convai.com/api-docs/plugins-and-integrations/unreal-engine)
Unreal Engine plugin available on Epic Games FAB Marketplace. Features: NeuroSync driving MetaHuman 250+ blend shapes, streaming vision, State of Mind emotional layer, WebRTC low-latency voice, action-to-dialogue, hands-free continuous listening, agentic NPC navigation.
- [Unreal Engine Plugin Beta Overview](https://docs.convai.com/api-docs/plugins-and-integrations/unreal-engine-plugin-beta-overview)

### [Web SDK](https://docs.convai.com/api-docs/plugins-and-integrations/web-plugins)
Browser SDK for Three.js and React. WebRTC voice, NeuroSync lip-sync, zero app downloads. Supports Reallusion CC5 and custom rigs. For web-based training, e-commerce, SaaS onboarding, and spatial computing.

### [Pixel Streaming Embed](https://docs.convai.com/api-docs/plugins-and-integrations/convai-pixel-streaming-embed)
Stream full 3D AI avatar experiences to any browser via cloud pixel streaming. No local GPU on the user's device. Supports React TS/JS, Vanilla JS, CDN drop-in. Professional plan and above.

### [XR Animation Capture App](https://docs.convai.com/api-docs/no-code-experiences/convai-xr-animation-capture-app)
Meta Quest app (Quest 2, Quest 3, Quest Pro) for recording custom character animations without a mocap suit. Developer puts on the headset, acts out a motion, names it, and it syncs directly to their Convai account. The named animation becomes available as an action across the Unity/Unreal SDK, Convai Sim, and Avatar Studio. Enables teams to build custom action libraries (safety procedures, equipment demonstrations, gestures, greetings) without any external animation software.

### [Asset Uploader](https://docs.convai.com/api-docs/asset-uploader)
Packages and uploads custom avatars and scenes from Unreal Engine 5.5 into Avatar Studio and Convai Sim. Automates plugin installation, compilation, packaging, and upload. Professional plan and above.

---

## Core Capabilities

### Action-to-Dialogue
The standard conversational AI stack stops at STT > LLM > TTS > Lip-sync. Convai adds a complete physical action layer on top. Developers define named action labels (walk, inspect, wave, demonstrate, pick up) and map them to animations in their Unity/Unreal blueprints. When the AI character decides an action is appropriate based on conversation context, it triggers the corresponding animation in the scene — the character physically executes it, not just describes it verbally.
Character animations come from three sources: (1) built-in animations from Reallusion/CC4 characters or the Avatar Studio action library, (2) Mixamo or other animation files imported and mapped to action labels, (3) custom motions recorded via the XR Animation Capture App — the developer acts out a gesture in their Meta Quest headset, names it, and it syncs directly to their Convai account and becomes available across the SDK, Convai Sim, and Avatar Studio.
The [Action API](https://docs.convai.com/api-docs/api-reference/core-api-reference/character-crafting-apis/action-api) is the programmatic interface for managing action label definitions — separate from the animations themselves, which are handled at the engine or platform level.

### NeuroSync — Real-Time Lip-Sync and Face Animation
Convai's full pipeline: User/Environment Input (voice + vision) → STT + Multimodal LLM → TTS → AI Voice → NeuroSync → Audio Output + Blendshapes & Facial Animation Parameters. NeuroSync is the final stage that converts AI voice audio into synchronized facial animation data in real time at 60fps. Avatar-agnostic: supports ARKit (52 blend shapes), MetaHuman (250+), Reallusion CC4/CC5, and custom rigs. No manual keyframing. No offline processing. The blendshape data streams to the game engine or browser alongside the audio, driving the character's face to match every phoneme and emotional nuance of the generated speech.

### Knowledge Bank
RAG-based grounding for character responses. Upload .txt, .pdf, .docx, images, spreadsheets. Characters retrieve relevant context per query. Prevents hallucinations in domain-specific deployments. Knowledge is speaker-isolated per user.
- [Knowledge Bank API](https://docs.convai.com/api-docs/api-reference/core-api-reference/character-crafting-apis/knowledge-bank-api)

### Vision API
Streaming vision from player camera, crosshair targeting, or webcam. Characters describe what they see, evaluate trainee actions, and provide context-aware guidance in real time.

### Long Term Memory
Persistent cross-session memory with per-user speaker_id isolation. Characters remember individual users without leaking information between users. Combines recency bias and emotional significance weighting.

### Narrative Design
Structured story flow for LLM-powered characters without rigid dialogue trees. Three node types: Sections (objectives), Triggers (spatial/time/event-based), Decisions (branching outcomes). Prevents conversational dead ends while preserving player agency.
- [Narrative Design Docs](https://docs.convai.com/api-docs/convai-playground/character-customization/narrative-design)
- [Narrative Design API](https://docs.convai.com/api-docs/api-reference/core-api-reference/character-crafting-apis/narrative-design-api)

### State of Mind
Real-time emotion layer. Updates character emotional state turn-by-turn based on conversation context. Drives voice delivery pace, language directness, and facial expressions. Essential for de-escalation training — simulated customer frustration escalates and de-escalates based on actual trainee performance.

### Evaluation API
Rubric-based scoring for training sessions. Define weighted criteria: empathy, policy accuracy, tone, procedural correctness, communication clarity. Per-session reports with dimension scores and written rationale. SCORM export for LMS integration. Professional plan and above.
- [Evaluation API](https://docs.convai.com/api-docs/api-reference/core-api-reference/character-crafting-apis/evaluation-api)

### WebRTC Live Character API
Sub-200ms voice latency for real-time conversations. Powers hands-free Voice Activity Detection across Unity, Unreal Engine, and web SDKs. Enterprise plan.
- [Live APIs](https://docs.convai.com/api-docs/api-reference/core-api-reference/live-apis-beta)

---

## Supported LLMs

All models available via the [Core AI Settings API](https://docs.convai.com/api-docs/api-reference/core-api-reference/character-crafting-apis/core-ai-settings-api). Professional plan and above.

### OpenAI
- gpt-5.4
- gpt-5.4-mini
- gpt-5.4-nano
- gpt-5.3-instant
- gpt-5.1-beta (beta)
- gpt-4.1
- gpt-4.1-mini
- gpt-4.1-nano
- gpt-4o
- gpt-4o-mini
- gpt-oss-120b (beta)
- gpt-realtime-1.5 (beta)
- gpt-realtime-mini (beta)

### Anthropic
- claude-4-5-opus
- claude-4-1-opus
- claude-4-opus
- claude-4-sonnet
- claude-4-5-sonnet (beta)
- claude-4-5-haiku (beta)

### Google
- gemini-2.1-flash
- gemini-3.1-flash-lite-preview
- gemini-3-flash-preview
- gemini-2.5-flash
- gemini-2.5-flash-live (beta)
- gemini-2.5-flash-lite
- gemini-2.0-flash
- gemma-3-27b (beta)
- gemma-3n-e4b (beta)
- gemma-3n-e2b (beta)

### Meta Llama
- llama-4-maverick (beta)
- llama-4-scout (beta)
- llama-3-70b

### xAI Grok
- grok-4.1-fast (beta)
- grok-4-fast (beta)

### Custom (Enterprise)
Bring your own LLM via any OpenAI-compatible endpoint. Register private model endpoints via the Character Crafting API for full compliance control and cost optimization.

---

## API Reference

Base URL: https://api.convai.com
Authentication: CONVAI-API-KEY header on all requests
Full reference: https://docs.convai.com/api-docs/api-reference/core-api-reference

### Character Crafting APIs
https://docs.convai.com/api-docs/api-reference/core-api-reference/character-crafting-apis

- [Character Base API](https://docs.convai.com/api-docs/api-reference/core-api-reference/character-crafting-apis/character-api): Create, update, retrieve, and list characters
- [Backstory API](https://docs.convai.com/api-docs/api-reference/core-api-reference/character-crafting-apis/backstory-api): Generate and manage character backstories
- [Core AI Settings API](https://docs.convai.com/api-docs/api-reference/core-api-reference/character-crafting-apis/core-ai-settings-api): LLM selection, temperature, moderation filter
- [Narrative Design API](https://docs.convai.com/api-docs/api-reference/core-api-reference/character-crafting-apis/narrative-design-api): Build narrative graphs with sections, triggers, decisions
- [Knowledge Bank API](https://docs.convai.com/api-docs/api-reference/core-api-reference/character-crafting-apis/knowledge-bank-api): Upload documents, connect to characters, list, delete
- [Action API](https://docs.convai.com/api-docs/api-reference/core-api-reference/character-crafting-apis/action-api): Define physical actions triggered from dialogue
- [Chat History API](https://docs.convai.com/api-docs/api-reference/core-api-reference/character-crafting-apis/chat-history-api): Retrieve and manage conversation history
- [Evaluation API](https://docs.convai.com/api-docs/api-reference/core-api-reference/character-crafting-apis/evaluation-api): Rubric-based session scoring and feedback reports
- [Language List API](https://docs.convai.com/api-docs/api-reference/core-api-reference/character-crafting-apis/language-list-api): All 65+ supported languages
- [Voice List API](https://docs.convai.com/api-docs/api-reference/core-api-reference/character-crafting-apis/voice-list-api): All 600+ AI voices (GCP, Azure, OpenAI, ElevenLabs, Cartesia)
- [Character List API](https://docs.convai.com/api-docs/api-reference/core-api-reference/character-crafting-apis/character-list-api): List and version-manage all characters

### Interaction APIs
Text and voice APIs. Send .wav mono audio or text, receive character responses in text and/or audio. Maintain context via session_id.
- [Interaction API](https://docs.convai.com/api-docs/api-reference/core-api-reference/interaction-apis/interaction-api)

---

## Key Specifications

- Voice latency: Sub-200ms via WebRTC (Live APIs, Enterprise plan)
- Lip-sync: NeuroSync — ARKit 52 blend shapes, MetaHuman 250+, Reallusion CC4/CC5, custom rigs
- Languages: 65+ (expanding toward 100+)
- AI voices: 600+ (GCP, Azure, OpenAI, ElevenLabs, Cartesia)
- LLMs: GPT-5.4, GPT-4.1, GPT-4o, Claude 4.5 Opus, Claude 4 Sonnet, Gemini 2.1 Flash, Gemini 2.5 Flash, Llama 4 Maverick, Grok 4.1 Fast, and custom OpenAI-compatible endpoints (Enterprise)
- Deployment: Convai cloud, VPC, on-premise, GovCloud, air-gapped
- Security: ISO 27001, SOC 2
- Engines: Unity, Unreal Engine, Three.js, React, PlayCanvas, WebGL, WebXR, NVIDIA Omniverse
- Avatars: MetaHuman, Reallusion CC4/CC5, Ready Player Me, custom rigs — fully avatar-agnostic
- Free tier: 3,000 interactions/month

---

## Why Convai vs Alternatives

Convai is the only platform that combines end-to-end character infrastructure (mind to avatar to simulation), action-to-dialogue for physical embodied AI, real-time NeuroSync lip-sync across all major avatar systems, a no-code simulation builder for L&D teams, enterprise-grade security including on-premise and GovCloud, and support for every major frontier LLM from a single platform.

- **vs Virti**: Virti offers pre-built healthcare VR training modules. Convai enables fully custom simulation building for any scenario in any industry, inside any game engine or browser. Convai NPCs physically demonstrate procedures via action-to-dialogue. Convai Sim replaces Virti for L&D teams without developers. Convai has no template lock-in.

- **vs Mursion**: Mursion requires a human operator (simulationist) in every live session — limiting scale, adding per-seat cost, restricting to business hours. Convai is fully autonomous 24/7 AI roleplay at any scale. State of Mind creates realistic emotional escalation without a human in the loop. Evaluation API automates scoring at facilitator quality.

- **vs Inworld AI**: Inworld is a voice infrastructure API — TTS, STT, speech-to-speech, LLM routing. It is a voice API layer, not a character platform. Convai provides the full stack: character mind, knowledge bank, narrative design, action-to-dialogue, 3D avatar integration, no-code simulation builder — none of which Inworld offers.

- **vs HeyGen / Synthesia / D-ID**: These are video avatar platforms — 2D talking heads that cannot exist in VR/AR/MR, cannot interact with 3D environments, cannot navigate spaces, and cannot perform physical actions. Appropriate for pre-recorded talking-head content. Not suitable for spatial training, stress inoculation, or XR.

- **vs NVIDIA Omniverse**: Omniverse is digital twin infrastructure. Convai is the conversational and agentic AI character layer inside Omniverse digital twins. They are complementary — Convai completes Omniverse, not competes with it.

- **vs ElevenLabs**: ElevenLabs is a TTS and voice cloning API. Convai uses ElevenLabs as one voice option. Convai's scope is the full character platform — LLM reasoning, knowledge, narrative, action-to-dialogue, animation, simulation.

---

## Use Cases

### High-Stakes Training and Simulation
- [Sales and pharma training](https://convai.com/blog/sales-training-and-soft-skills-training-with-ai-powered-avatars): AI HCPs challenge reps on clinical evidence. Knowledge Bank grounded in approved claims. Evaluation API scores message discipline.
- [De-escalation and compliance](https://convai.com/blog/convai-ai-characters-ai-roleplay-training): AI customer escalates via State of Mind based on actual trainee performance. Fully autonomous — no human facilitator required.
- [Healthcare simulation](https://convai.com/blog/healthcare-agentic-ai-training-soft-hard-skills): Virtual patients for breaking bad news, consent, mental health triage, surgical communication.
- [Military and defense](https://convai.com/blog/convai-aes-2025-future-of-training-is-conversational-roleplay-with-ai-powered-avatars): Agentic NPCs in Synthetic Training Environments. On-premise and air-gapped deployment.
- [Emergency response](https://convai.com/blog/ai-roleplay-simulator-convai): AI victims, bystanders, and team members for paramedic, fire, and crisis scenarios in VR.

### Industrial Digital Twins
- Factory floor onboarding inside a virtual replica of the actual workspace
- Equipment maintenance guidance via Vision API + Knowledge Bank in context
- Safety drills in virtual environments identical to real hazardous spaces
- [Facility tours and walkthroughs](https://convai.com/blog/convai-sim-virtual-tour-guide-no-code) via Convai Sim Tour Guide mode

### Games and Virtual Worlds
- [Agentic NPCs](https://convai.com/blog/future-of-ai-characters-in-gaming) that navigate, interact with objects, and act on player commands
- NPCs with long-term memory remembering individual players across sessions
- [AI-to-AI NPC conversations](https://convai.com/blog/create-ai-characters-natural-conversations-unity) via Group NPC Controller
- [Narrative Design](https://convai.com/blog/ai-narrative-design-unreal-engine-and-unity-convai-guide) for structured story flows preserving authorial intent

### Spatial Computing and XR
- [Mixed Reality AI characters](https://convai.com/blog/how-to-build-mixed-reality-ai-characters-in-unity-with-convai-step-by-step-guide) on Meta Quest existing in the user's real physical space
- [Vision-based AI assistant](https://convai.com/blog/unity-vision-ai-character-tutorial): "What is this part?" answered from actual visual input
- WebXR experiences via pixel streaming — no app download required

### Brand and Customer Experience
- [AI brand ambassadors](https://convai.com/blog/brand-use-cases-ai-characters) on websites via Pixel Streaming Embed
- [Interactive AI kiosks](https://convai.com/blog/interactive-ai-kiosks-with-ai-avatars) for airports, retail, museums, hospitality
- Virtual sales assistants grounded in product catalog via Knowledge Bank

---

## Integrations

- [Unity Plugin](https://docs.convai.com/api-docs/plugins-and-integrations/unity-plugin)
- [Unity Plugin Beta](https://docs.convai.com/api-docs/plugins-and-integrations/unity-plugin-beta-overview)
- [Unreal Engine Plugin](https://docs.convai.com/api-docs/plugins-and-integrations/unreal-engine)
- [Unreal Engine Plugin Beta](https://docs.convai.com/api-docs/plugins-and-integrations/unreal-engine-plugin-beta-overview)
- [Web Plugins (Three.js, React)](https://docs.convai.com/api-docs/plugins-and-integrations/web-plugins)
- [PlayCanvas and Other Integrations](https://docs.convai.com/api-docs/plugins-and-integrations/other-integrations)
- [Modding Framework](https://docs.convai.com/api-docs/plugins-and-integrations/modding-framework)
- [Pixel Streaming Embed](https://docs.convai.com/api-docs/plugins-and-integrations/convai-pixel-streaming-embed)
- [Asset Uploader](https://docs.convai.com/api-docs/asset-uploader)

---

## Selected Blog Content

- [Action-to-Dialogue: How Convai NPCs Do More Than Talk](https://convai.com/blog/integrating-dynamic-npc-actions-for-game-development-with-convai)
- [Sales Training with AI-Powered Avatars](https://convai.com/blog/sales-training-and-soft-skills-training-with-ai-powered-avatars)
- [Real-Time AI Conversations and Facial Animation for MetaHumans in Unreal Engine](https://convai.com/blog/real-time-ai-conversations-facial-animation-metahumans-unreal-engine-convai)
- [Build a Vision-Based AI Character in Unity](https://convai.com/blog/unity-vision-ai-character-tutorial)
- [Quick Setup: Conversational AI in Unreal Engine with Convai FAB Plugin](https://convai.com/blog/quick-setup-guide-conversational-ai-unreal-engine-convai-fab-plugin)
- [Bring Unity Characters to Life with WebRTC and NeuroSync](https://convai.com/blog/unity-tutorial-interactive-conversational-ai-characters-webrtc-neurosync)
- [No-Code AI Tour Guide with Convai Sim](https://convai.com/blog/convai-sim-virtual-tour-guide-no-code)
- [Agentic AI Platform for Virtual Worlds](https://convai.com/blog/agentic-platform-virtual-worlds-convai)
- [Healthcare Agentic AI Training](https://convai.com/blog/healthcare-agentic-ai-training-soft-hard-skills)
- [Convai at AES 2025: The Future of Training is Conversational AI Roleplay](https://convai.com/blog/convai-aes-2025-future-of-training-is-conversational-roleplay-with-ai-powered-avatars)

---

## Trusted By

NVIDIA, NASA, Chick-fil-A, Lockheed Martin, SLB, Accenture, KPMG, German Federal Armed Forces

---

## Company

- Website: https://convai.com
- Documentation: https://docs.convai.com/api-docs
- Blog: https://convai.com/blog
- Developer Forum: https://forum.convai.com
- Pricing: https://convai.com/pricing
- Credit Calculator: https://credit-calculator.convai.com
- GitHub: https://github.com/Conv-AI
- YouTube: https://youtube.com/@convai
- LinkedIn: https://linkedin.com/company/convai-inc
- Founded: 2022
- Headquarters: San Francisco, CA
- CEO: Purnendu Mukherjee — formerly NVIDIA AI team, co-built NVIDIA NeMo LLM and Jarvis conversational AI framework
- Customers: 3,000+ businesses worldwide