How to Build a Secure, Encrypted Video Calling App With LiveKit on a VPS (2026)

A private video-calling service can run on a virtual private server without routing every meeting through a proprietary conferencing product. LiveKit is an open-source WebRTC selective forwarding unit (SFU). Your app authenticates people and issues room-specific access tokens; a LiveKit server relays real-time audio and video; a browser or mobile SDK captures the microphone, camera and screen. Unlike a simple one-to-one peer connection, an SFU is practical for group rooms because each participant sends media to the server, which forwards suitable streams to subscribers. The server is still part of the trust boundary unless you enable true end-to-end encryption.

Use three logical components. First, a Next.js, React or native application presents the join screen, device permissions and call controls. Second, a trusted backend authenticates users, checks meeting membership and signs short-lived LiveKit JWT access tokens with the room and allowed actions. Third, LiveKit handles WebSocket signaling and WebRTC transport. Redis supports the production server configuration. Caddy can terminate TLS and manage public certificates. The app should connect to a wss:// address, not an unencrypted websocket endpoint. A separate TURN hostname improves access from restrictive networks. Keep the application API secret entirely on the backend; never put it in client-side JavaScript.

For a small pilot, choose a Linux VPS with a public IPv4 address, predictable bandwidth, at least a few vCPUs and enough RAM for the operating system, Redis and LiveKit. Two vCPUs and 4 GB RAM can be a starting experiment rather than a guaranteed concurrency rating. Real capacity depends on number of publishers, subscribers, video resolution, codecs, simulcast, packet loss and whether the server must relay TURN traffic. Video calls are often network-bound: a rough egress estimate for a four-person room with each person receiving three streams at 1 Mbps is 12 Mbps, before overhead. Benchmark your actual mix before promising customer capacity. Add DNS A records for a signaling hostname such as meet.example.com and a TURN hostname such as turn.example.com.

Start with a maintained Ubuntu server, install security updates, Docker Engine and the Compose plugin using the official Docker instructions, and verify docker compose version. LiveKit's documented VM path uses an official configuration generator rather than a hand-written production compose file. On a trusted development machine run the following commands, then follow the interactive questions about domain, TURN, certificates and optional recording services:

The generator creates docker-compose.yaml, livekit.yaml, redis.conf, caddy.yaml and either a startup script or cloud-init file. Inspect generated files and secrets before transferring them. Follow the official instructions for the exact cloud provider: either provide the generated cloud-init as instance user data or run the generated startup script on the VPS. The documented startup method installs Docker and the systemd livekit-docker service under /opt/livekit.

LiveKit requires more than a conventional web server's TCP 443. The official single-VM deployment guide lists TCP 80 for certificate issuance, TCP 443 for HTTPS and TURN over TLS, TCP 7881 for ICE over TCP, UDP 3478 for TURN over UDP, and UDP 50000–60000 for WebRTC media. Open these in both the VPS firewall and your cloud provider's network security rules. Do not expose LiveKit's internal API port 7880 or Redis port 6379 to the public internet. Keep SSH restricted to trusted IP addresses when practical. Your generated configuration is the source of truth: if it uses UDP multiplexing on port 7882, firewall needs differ. Never blindly copy firewall commands that might lock you out of your own SSH session.

Both the primary signaling hostname and TURN hostname must resolve to the VPS public IP before automated TLS issuance succeeds. Caddy in the official deployment provisions certificates from a supported certificate authority. Clients normally prefer low-latency UDP media paths; when a corporate firewall or mobile network blocks them, TURN relays traffic, potentially over TLS. This improves connectivity but can increase bandwidth cost and latency. Test calls from two different networks, including a phone on mobile data and a laptop behind Wi-Fi. A call that works with two tabs on one laptop is not proof that external NAT traversal works. Check certificates and Caddy logs if signaling connects only over insecure development URLs.

Every joining participant needs an identity and an access token signed using your LiveKit API key and secret. The backend should authenticate the user, verify invitation or membership, assign a room name that cannot be guessed as an authorization bypass, and grant only the needed rights. For example, a viewer may subscribe without permission to publish. Use short token expiry, rate-limit the token endpoint and do not accept arbitrary room names without membership checks. Protect meeting creation and administrative endpoints with role-based access controls. A JWT is not a substitute for application authorization: if your backend hands tokens to anyone, the room is not private even though transport is encrypted.

Install the LiveKit client SDK and React components in a Next.js frontend. A typical page requests a token from your authenticated backend, then connects using LiveKitRoom, VideoConference or custom track components. Request camera and microphone access only when the user joins or explicitly previews devices. Provide mute, camera toggle, device selector, screen sharing, connection status and a visible leave button. Show who is present and who can publish. Handle browser permission refusal and devices that disappear mid-call. Do not log signed room tokens, user audio, sensitive room metadata or chat payloads to analytics. Use a content security policy and keep dependencies patched.

WebRTC encrypts media in transit and TLS protects signaling, but in a conventional SFU design the server is a trusted media intermediary. For a stronger privacy guarantee, LiveKit supports application-level end-to-end encryption for media tracks and data channels. Each participant must enable encryption in the SDK and receive the correct room key through a separate secure application channel. LiveKit says it does not store or distribute those E2EE keys. If the keys are known only to participants, the media server cannot read the protected content. Signaling and API messages are still TLS-protected rather than end-to-end encrypted, so server-visible metadata remains. Do not advertise the entire system as zero-knowledge without analyzing authentication, logs, metadata, backups and key handling.

For a small room, a shared symmetric key is the simplest pattern, but generating and distributing it safely is a separate engineering problem. Create the key with a cryptographically secure random source; send it only to authorized participants over authenticated HTTPS after membership checks. Avoid embedding a fixed global key in the frontend bundle or committing it to Git. Consider a per-room key, expiry, revocation, and rotation when a participant is removed. Advanced threat models may require participant-specific keys or a more sophisticated key agreement scheme. Remember that an authorized participant can still record their own screen, and malware on a participant device can capture decrypted content. E2EE does not solve endpoint compromise.

Server-side recording and transcription require access to media content. If you enable genuine participant-only E2EE, a normal server recorder cannot decrypt it without receiving the key. Likewise, a LiveKit voice or video AI agent joining an encrypted room is a participant that needs the room key to process speech or frames. That may be appropriate with informed consent, but it changes who can access the conversation. Make recording opt-in, show an unmistakable indicator, establish retention periods and encrypt stored files at rest. For confidential calls, leave recording and automated transcription disabled by default. Do not claim E2EE while silently handing decryption keys to an analytics or recording service.

Run LiveKit and its supporting services as maintained containers, pin tested image versions for production and plan updates in a maintenance window. Back up configuration and secrets securely, not user media unless retention is explicitly required. Configure a firewall, automatic security updates where appropriate, intrusion detection, and least-privilege SSH. Monitor CPU, memory, network throughput, packet loss, reconnects, room count and TURN usage. Avoid logging raw credentials or personal meeting content. Review dependencies and access logs after incidents. Set limits for room size, publishing permissions and API requests so one abusive client cannot exhaust the VPS. Establish a documented incident response and secret-rotation procedure.

Run a matrix of calls across Chrome, Firefox, Safari, Android and iOS, on Wi-Fi and mobile data. Test one-to-one calls, three- and four-person rooms, camera switching, screen sharing, backgrounding, reconnect after network loss, revoked permissions, expired tokens and a user being removed from a room. Verify that unauthorized users cannot join using a copied room URL. Confirm that encryption fails closed when keys do not match, rather than silently falling back to unencrypted content. Validate that no server-side recording is possible without the intended key access. Measure latency and packet loss during sustained calls and use a repeatable load test before estimating how many simultaneous rooms a VPS can serve.

Self-hosting avoids per-minute SaaS charges from some vendors but not compute, bandwidth, storage, domain, operations or support costs. TURN-heavy traffic can make network egress the largest expense. For a small community or internal pilot, a single VPS is easier to operate than a cluster; for high availability and many concurrent rooms, plan distributed nodes, load balancing, Redis, monitoring and failover. Budget for ongoing patching and on-call support, especially if users rely on the system for important conversations. The recommended launch order is: build the app and token service, deploy LiveKit with official generated configuration, test connectivity, add E2EE with secure key management, perform security review, then invite a small pilot group. No VPS configuration alone makes a call service absolutely secure.

Can LiveKit run without a GPU? Yes. The SFU forwards media rather than generating AI video; GPU resources are generally unnecessary for basic conferencing. Is it free? The open-source server and SDKs can be self-hosted, but infrastructure and bandwidth are not free. Can users join from a browser? Yes, with WebRTC-capable browsers and HTTPS permissions. Is a VPN required? No, though some organizations may choose one for additional access control. Does a password-protected meeting automatically have E2EE? No. Meeting access, encrypted transport and end-to-end media encryption are distinct controls. Can you build a WhatsApp-style calling product? You can implement core voice/video features, but contacts, notifications, identity, moderation, reliability and mobile app distribution require substantial additional engineering.