LLM chat streaming

Three conversations with a mock LLM. Each reply streams in token by token, and all three can stream at the same time — but only the chat you’re looking at is on screen.

Things to try:

Open a chat and watch the reply stream in. Later tokens update in place — the Suspense fallback only ever shows before the first token.
Switch to another chat mid-stream, wait a moment, and switch back: the first chat kept streaming while hidden and reveals instantly, fully caught up.
Hover a chat you haven’t opened yet before clicking it: the reply starts streaming in the background, so opening it skips the fallback entirely.

import {
  Activity,
  Suspense,
  use,
  useState,
} from 'react'
import {
  preloadObservablePromise,
  useObservablePromise,
} from 'react-rx'

import {
  CHATS,
  conversation$,
  type Chat,
} from './chat'

function ChatView({chat}: {chat: Chat}) {
  // Suspends until the first token, then streams in place — later emissions
  // never re-trigger the Suspense fallback.
  const messages = use(
    useObservablePromise(conversation$(chat)),
  )

  return (
    <div>
      {messages.map((message) =>
        message.role === 'user' ? (
          <p key="user">
            <strong>You:</strong>{' '}
            {message.content}
          </p>
        ) : (
          <blockquote key="assistant">
            {message.content}
          </blockquote>
        ),
      )}
    </div>
  )
}

export default function App() {
  const [activeId, setActiveId] = useState(
    CHATS[0].id,
  )
  const [visitedIds, setVisitedIds] = useState([
    CHATS[0].id,
  ])

  const open = (chat: Chat) => {
    setActiveId(chat.id)
    setVisitedIds((ids) =>
      ids.includes(chat.id)
        ? ids
        : [...ids, chat.id],
    )
  }

  return (
    <>
      <div role="group">
        {CHATS.map((chat) => (
          <button
            key={chat.id}
            type="button"
            // Hovering a chat you haven't opened yet starts its reply
            // streaming in the background, so opening it skips the fallback.
            onMouseEnter={() =>
              preloadObservablePromise(
                conversation$(chat),
                {ttl: 30_000},
              )
            }
            onClick={() => open(chat)}
            aria-current={
              chat.id === activeId || undefined
            }
            style={{
              fontWeight:
                chat.id === activeId ? 700 : 400,
            }}
          >
            {chat.title}
          </button>
        ))}
      </div>

      <Suspense
        fallback={
          <p aria-busy="true">
            Waiting for the first token…
          </p>
        }
      >
        {CHATS.filter((chat) =>
          visitedIds.includes(chat.id),
        ).map((chat) => (
          // Visited chats stay mounted but hidden: they keep their state and
          // reveal instantly — including every token that streamed while you
          // were looking at another chat.
          <Activity
            key={chat.id}
            mode={
              chat.id === activeId
                ? 'visible'
                : 'hidden'
            }
          >
            <ChatView chat={chat} />
          </Activity>
        ))}
      </Suspense>
    </>
  )
}

Open Sandbox

The mock vs. your code

llm.ts is the only mock in this demo: a stand-in for a real streaming LLM API that emits token deltas as an observable. In a real app it would wrap a fetch ReadableStream or an SSE connection. Everything else — chat.ts and App.tsx — is what your own code would look like.

The userland recipe is small:

scan folds tokens into the reply. The conversation stream emits the whole message list on every token, so components just render the latest value.
shareReplay({bufferSize: 1, refCount: false}) makes the stream independent of who’s watching. The reply keeps streaming while its chat is hidden or unmounted, and any subscriber — new or returning — immediately gets the latest state. The source completes when the reply ends, so nothing leaks.

Why the React side “just works”

useObservablePromise + use() suspend until the first emission and then update in place. Streaming tokens never re-trigger the fallback — that’s the semantic difference from re-fetch-per-render approaches.
<Activity mode="hidden"> keeps visited chats mounted with their state intact. Hiding a chat tears down its live subscription (like an effect), but the shared conversation stream keeps running, and on reveal the hook synchronously reads the current snapshot — no flash, no refetch, all the tokens that arrived meanwhile.
preloadObservablePromise warms the same cache the hook reads from, outside of render. Hover-to-preload is one line on the button.

See Activity and preload for a side-by-side comparison of prefetch strategies, and Suspense data fetching for the promise semantics on their own.