Capacity and scaling

This page gives measured numbers for one Liyagent instance and the order in which its limits appear as chat volume grows. It also lists the steps that raise each limit, so you can plan capacity before you need it.

How we measured#

We ran one app process with its own PostgreSQL 16 database and the embedded vector store, as the single-VM install runs them. The machine had 18 cores. The model was a stub that answers after a fixed delay, so the numbers show what the platform costs and not what a model provider costs. We tried delays of 0.3 seconds, 1.5 seconds and 5 seconds.

Each simulated person signed in with their own account, kept one chat session, and sent 10, 25 or 50 messages with about 2 seconds of think time between them. Real people usually think for much longer than 2 seconds, so each simulated person stands for several real ones. Every scenario stopped sending new messages after 4 minutes.

No request failed in any scenario. When the instance was busy, people waited longer instead.

Results with a 1.5 second model#

These runs started from an empty database.

The first 10 person run includes the warm-up of a freshly started process, which is why it is slower than the 25 person run.

Results with other model speeds#

Results on an instance that has been running for a while#

An instance keeps up to 5,000 recent conversations in its transcript list. Once that list is close to full, every chat turn costs more. We repeated the 10 message runs on a database that already held about 4,100 conversations.

What one VM serves today#

The platform adds about 0.25 seconds to each reply on a new instance and about 2.7 seconds on an instance whose transcript list is nearly full.

One app process completes at most about 10 chat turns a second on a new instance and about 1.6 a second on a long running one. If people pause about 30 seconds between messages, that is roughly 300 people chatting at the same moment on a new instance and roughly 50 on a long running one. Plan with the long running figure, because every busy instance reaches it.

Adding CPU cores to the VM does not raise these numbers. The app runs as one process, and that process uses about one core for chat work.

Which limit you reach first#

1. The transcript list. All conversations of the instance share one list, and every chat turn reads and rewrites it several times. The cost of a turn grows with the number of kept conversations, up to the 5,000 cap. This is the limit a long running instance meets first.
2. Worker threads. Each chat turn holds one of 40 worker threads for its whole length, including the time spent waiting for the model. With a slow model, 40 turns in flight is the ceiling. At a 5 second model this shows as about 40 calls open at the model and replies queueing behind them.
3. One CPU core. With a fast model, the process spends about 90 ms of CPU on each turn, which caps it near 10 turns a second.
4. Session summaries. When a chat passes 30 exchanges, the oldest turns are folded into a summary by an extra model call. That call happens before the reply, so it adds one model call to the turn that triggers it, and again about every 10 exchanges after that.
5. The per caller rate limit. Each signed in person may send 30 messages a minute by default (TICKETIQ_RATE_LIMIT_EXPENSIVE). Callers that the instance cannot tie to a person, such as Google Chat events and anonymous web chat visitors, share the limit of their network address.

The database was not the limit in any run. It stayed below 55 connections and was mostly idle while the app process did its work.

Settings you can change today#

The path to ten times the load#

Ten times means about 100 turns a second, or roughly 3,000 people chatting at the same moment. These steps come in order.

1. Store transcripts and session lists as database rows. Each turn then touches only its own conversation, so a turn costs the same on a long running instance as on a new one. This alone moves a long running instance from about 1.6 to about 10 turns a second.
2. Size worker threads and database connections together. Raise the worker thread count above 40, and replace one connection per thread with a connection pool. Put PgBouncer in session mode in front of PostgreSQL, because the app uses session advisory locks.
3. Move the summary fold after the reply, so a long chat never waits for it.
4. Run the vector store as a Qdrant server instead of inside the app.
5. Run several app replicas behind the load balancer. Today a second process against the same database refuses to start, because sign-ins in flight, rate limit counters, the live event stream and several write locks are held in one process's memory. Each of these moves to PostgreSQL first. Scheduled jobs already run on one elected process, so they need no change.
6. Give PostgreSQL its own machine or a managed service with at least 4 cores, and set max_connections for the pool size of all replicas together.

The path to a hundred times the load#

A hundred times means about 1,000 turns a second.

1. Put long agent runs on a durable queue. A web request then starts a run and receives the answer by streaming or polling, and workers take runs from the queue. Teams already works this way. Google Chat and web chat move to the same queue.
2. Share the queue fairly between organisations. Each organisation gets its own share of workers, so one busy organisation cannot hold every worker.
3. Set per organisation rate limits and model quotas with scoped rate limits, which already live in the database and apply across replicas.
4. Configure a fallback chain for each agent's model, and buy enough provider quota for the peak. At this volume the provider quota, not the platform, is usually the limit.
5. Autoscale app replicas and queue workers on queue depth and on the time a request waits for a worker thread.
6. Keep the audit chain from becoming a single line. Every audited action takes one database wide lock. Write audit rows in batches, or keep one chain per organisation.
7. Add a PostgreSQL read replica for console reads and reports, and run Qdrant as a cluster.

One instance for Teams, Google Chat and web chat#

Picture three kinds of organisation on one instance. One uses Microsoft Teams, one uses Google Chat, and the others use the web chat. Today they share the same 40 worker threads, the same transcript list and the same CPU core.

• Teams messages go to a durable inbox and are answered 4 at a time across the whole instance. Teams retries a message it did not see acknowledged, and the inbox answers it once.
• Google Chat events start a new background thread each. They are not queued, so a burst can open many model calls and database connections at once, and an event in flight when the process restarts is lost. Events arrive from Google's network, so they share the rate limit of that address.
• Web chat runs inside the request, on the shared worker threads, and is limited per visitor address.

Until the steps above are done, size such an instance with the long running figure and leave room for the busiest organisation's peak. After the queue and the per organisation shares are in place, each organisation's channel draws on its own share of workers and its own rate limits, and a burst in one channel no longer slows the others.