I nearly shipped a nasty AI integration as soon as. The function labored technically: the mannequin answered questions accurately, the UI regarded clear, and everybody nodded alongside in our inner demo. Then we put actual customers on it, and inside 48 hours a product supervisor Slack’d me asking why 60% of classes have been ending with no response acquired.
The issue wasn’t the API or the server. It was eight seconds of silence.
Customers despatched a message, noticed nothing occur, assumed the request had failed, and closed the tab. The response would arrive shortly after, to no person. We had constructed a purposeful AI function that felt fully damaged, and the repair wasn’t an architectural overhaul or a efficiency optimization. It was streaming: begin sending tokens the second the mannequin produces them, as a substitute of batching the total response and delivering it without delay.
I’ve seen this error in codebases throughout a number of corporations now. The non-streaming model is easier to write down, exams move, and no person notices in improvement as a result of builders look forward to issues, whereas actual customers normally don’t.
This tutorial builds the entire streaming implementation: a Node.js backend utilizing Server-Despatched Occasions to push tokens to the browser as they arrive, with session administration that offers the mannequin reminiscence throughout turns, and correct abort dealing with so you aren’t paying for tokens no person reads.
The whole code is at github.com/ziaongit/nodejs-openai-streaming.
No OpenAI key? I examined this whole article utilizing Groq as a substitute. It’s free, it speaks the identical OpenAI SDK format, and getting a key takes about two minutes. The repo has a
USE_GROQ=trueflag already wired in. Copy.env.instanceto.env, paste your Groq key, and you might be operating. I usedllama-3.3-70b-versatile. The output is sweet sufficient that you wouldn’t discover the distinction in a dev session.
Stipulations
- Node.js v18.11 or later
- An OpenAI API key
- Stable familiarity with Categorical and async/await
The Structure Earlier than Any Code
Earlier than we write any code, it helps to see what is definitely taking place throughout the three layers. Debugging streaming points with out this image will get painful quick.
┌──────────────────────────────────────────────────────────────────┐
│ Browser │
│ │
│ 1. POST /session ─────────────────────► receives sessionId │
│ 2. POST /chat { message, sessionId } │
│ 3. response.physique.getReader() ──► ReadableStream │
│ 4. decode chunks ──► parse SSE occasions ──► append tokens to UI │
└────────────────────────────┬─────────────────────────────────────┘
│ HTTP (textual content/event-stream)
▼
┌──────────────────────────────────────────────────────────────────┐
│ Categorical Server │
│ │
│ classes Map ── { sessionId: { messages: […] } } │
│ │
│ POST /chat │
│ ├─ load session historical past │
│ ├─ push person message │
│ ├─ set SSE headers + flushHeaders() │
│ ├─ openai.create({ stream: true, sign: controller.sign }) │
│ ├─ for await chunk → res.write(`knowledge: ${token}nn`) │
│ ├─ accumulate fullResponse → push to session │
│ └─ ship { finished: true } → res.finish() │
└────────────────────────────┬─────────────────────────────────────┘
│ HTTPS chunked switch encoding
▼
┌──────────────────────────────────────────────────────────────────┐
│ OpenAI API │
│ gpt-4o-mini · stream: true │
└──────────────────────────────────────────────────────────────────┘
The session retailer lives on the server. Each time the person sends a message, that full dialog historical past goes to OpenAI contained in the messages array, which is how the mannequin is aware of what was mentioned earlier. The API itself remembers nothing between calls, so that you carry the state.
The mechanism is easier than it sounds: open a daily HTTP response and by no means shut it. The server retains writing chunks into the socket as tokens arrive, and when the mannequin finishes you name res.finish() and the connection drops.
Challenge Setup
mkdir nodejs-openai-streaming
cd nodejs-openai-streaming
npm init -y
npm set up specific openai cors dotenv
mkdir public
Add `"kind": "module"` to `package deal.json`:
{
"title": "nodejs-openai-streaming",
"model": "1.0.0",
"kind": "module",
"scripts": {
"begin": "node server.js",
"dev": "node --watch server.js"
},
"dependencies": {
"cors": "^2.8.6",
"dotenv": "^16.6.1",
"specific": "^4.22.2",
"openai": "^4.104.0"
}
}
Create .env:
Try our hands-on, sensible information to studying Git, with best-practices, industry-accepted requirements, and included cheat sheet. Cease Googling Git instructions and truly study it!
OPENAI_API_KEY=your_api_key_here
PORT=3000
Challenge construction:
nodejs-openai-streaming/
├── server.js
├── public/
│ └── index.html
├── .env
├── .env.instance
├── .gitignore
└── package deal.json
Why SSE and Not WebSockets
Folks ask about WebSockets each time, normally as a result of they sound extra succesful and subsequently extra appropriate.
For AI chat, they normally aren’t. Responses journey a technique, server to consumer, and SSE is constructed for precisely that. It runs over plain HTTP with no protocol improve and no particular reverse proxy configuration, and it really works by means of HTTP/2 with out touching a config file. I’ve shipped SSE to manufacturing a number of instances and by no means as soon as fearful about browser help.
WebSockets make sense when the consumer must push knowledge again on the similar frequency the server pushes it: gaming, shared whiteboards, reside bidding. In an AI chat, the person varieties one message and sits there studying, so SSE suits the site visitors sample higher.
One real limitation of SSE value realizing: the browser’s built-in EventSource API solely helps GET requests. Since you’ll want to submit a message physique, you employ fetch with a ReadableStream reader as a substitute. The SSE wire protocol is similar; you might be simply studying the stream manually slightly than by means of the browser’s occasion wrapper.
Session Administration
The primary chatbot I constructed on high of this API had a bug I couldn’t determine for 2 days. Customers would ask a follow-up query and the mannequin would reply prefer it was listening to from them for the primary time, with no context or reminiscence of the sooner turns.
The API doesn’t retailer something between calls, so every request goes in chilly. The messages array is the way you repair that: pack the total dialog historical past into each request, person turns and assistant turns each, and the mannequin responds as if it remembers the entire thread. Depart it out and also you get that very same damaged expertise I shipped by chance.
const classes = new Map();
const SESSION_TTL_MS = 30 * 60 * 1000;
operate getOrCreateSession(sessionId) {
if (!classes.has(sessionId)) {
classes.set(sessionId, {
messages: [],
lastActive: Date.now(),
});
}
const session = classes.get(sessionId);
session.lastActive = Date.now();
return session;
}
setInterval(() => {
const now = Date.now();
for (const [id, session] of classes) {
if (now - session.lastActive > SESSION_TTL_MS) {
classes.delete(id);
}
}
}, 10 * 60 * 1000);
The Map is okay at improvement scale and in single-process deployments. The lastActive timestamp will get bumped on each request. The cleanup interval runs each ten minutes and removes classes which have gone quiet for half an hour.
The onerous restrict of this method is {that a} Node.js course of restart wipes it totally. If you happen to run a number of cases behind a load balancer, every occasion has its personal Map, and a person whose requests hit totally different cases will lose their dialog mid-sentence. The repair is Redis: one consumer, similar knowledge construction, TTL dealt with natively. I’m not going to construct that right here as a result of it might double the size of this text with out including something about streaming, however don’t ship this to manufacturing with out it.
The Streaming Endpoint
4 issues occur in sequence earlier than tokens begin flowing: enter validation, SSE setup, abort dealing with, and the OpenAI name with its token loop. Every step has not less than one non-obvious determination.
SSE Setup
res.setHeader('Content material-Kind', 'textual content/event-stream');
res.setHeader('Cache-Management', 'no-cache');
res.setHeader('Connection', 'keep-alive');
res.flushHeaders();
flushHeaders() is the road I see lacking in additional streaming PRs than I can depend. Skip it and Categorical holds onto the response headers till the primary res.write() name. The browser opens the request, will get nothing again, sits there, and after a number of seconds the entire thing appears to be like like a hung request. You’ll spend an hour checking your OpenAI name, your async logic, and your middleware, and the repair is one line on the high. Name flushHeaders() proper after setting the headers; nothing else modifications.
AbortController
const controller = new AbortController();
res.on('shut', () => {
if (!res.writableEnded) controller.abort();
});
One essential element: connect the shut listener to res, not req. The req object fires shut when the request physique is absolutely consumed, which occurs instantly after the POST physique arrives and would abort the stream earlier than it begins. res.on('shut') fires when the precise response connection drops, which is what you need. The !res.writableEnded guard prevents calling abort() after a clear end.
When the connection drops, controller.abort() propagates a cancellation sign to the OpenAI SDK and the for await loop exits on the following iteration. With out this, you retain pulling tokens from the API and writing them right into a closed socket, which suggests you might be spending cash and reaching nothing.
The OpenAI Streaming Name
const stream = await openai.chat.completions.create(
{
mannequin: 'gpt-4o-mini',
messages: [
{
role: 'system',
content: 'You are a helpful assistant. Be concise and clear.',
},
...session.messages,
],
stream: true,
max_tokens: 800,
},
{ sign: controller.sign }
);
Set max_tokens. This isn’t non-compulsory. I left it uncapped throughout a testing session and watched my utilization dashboard spike in actual time. Left alone, the mannequin will write a dissertation in response to a three-word query. I exploit 800 for chat, which works positive for many issues, and you’ll tune it when you see what your customers really ship.
The sign parameter threads the AbortController by means of to the SDK. If the consumer disconnects, the SDK cancels the underlying HTTP request to OpenAI, not simply the native loop.
Token Loop and Historical past
let fullResponse = '';
for await (const chunk of stream) {
if (req.socket.destroyed) break;
const token = chunk.selections[0]?.delta?.content material;
if (token) {
fullResponse += token;
res.write(`knowledge: ${JSON.stringify({ token })}nn`);
}
}
if (fullResponse) {
session.messages.push({ function: 'assistant', content material: fullResponse });
}
Every chunk comprises one token in delta.content material. You concatenate them into fullResponse as they arrive, relay every one to the consumer, and save the entire string to the session as soon as the loop ends.
The double newline in res.write() is obligatory; it’s how SSE alerts the tip of a single occasion. The browser buffers knowledge till it sees nn earlier than firing. Omit it and the consumer receives all tokens without delay when the connection closes, which is identical damaged conduct as not streaming in any respect.
Don’t save fullResponse if the stream was aborted. A truncated assistant reply within the session historical past will corrupt the dialog context, and the mannequin’s subsequent flip will likely be constructed on incomplete info.
The Full server.js
import 'dotenv/config';
import specific from 'specific';
import cors from 'cors';
import OpenAI from 'openai';
import path from 'path';
import crypto from 'crypto';
import { fileURLToPath } from 'url';
const __filename = fileURLToPath(import.meta.url);
const __dirname = path.dirname(__filename);
const app = specific();
const openai = new OpenAI({ apiKey: course of.env.OPENAI_API_KEY });
app.use(cors());
app.use(specific.json());
app.use(specific.static(path.be a part of(__dirname, 'public')));
const classes = new Map();
const SESSION_TTL_MS = 30 * 60 * 1000;
operate getOrCreateSession(sessionId) {
if (!classes.has(sessionId)) {
classes.set(sessionId, { messages: [], lastActive: Date.now() });
}
const session = classes.get(sessionId);
session.lastActive = Date.now();
return session;
}
setInterval(() => {
const now = Date.now();
for (const [id, session] of classes) {
if (now - session.lastActive > SESSION_TTL_MS) classes.delete(id);
}
}, 10 * 60 * 1000);
app.submit('/session', (req, res) => {
const sessionId = crypto.randomUUID();
getOrCreateSession(sessionId);
console.log(`[/session] created: ${sessionId}`);
res.json({ sessionId });
});
app.delete('/session/:sessionId', (req, res) => {
classes.delete(req.params.sessionId);
res.json({ okay: true });
});
app.submit('/chat', async (req, res) => {
const { message, sessionId } = req.physique;
if (!message || typeof message !== 'string') {
return res.standing(400).json({ error: 'message is required' });
}
if (!sessionId || typeof sessionId !== 'string') {
return res.standing(400).json({ error: 'sessionId is required' });
}
const session = getOrCreateSession(sessionId);
session.messages.push({ function: 'person', content material: message });
console.log(`[/chat] acquired:`, { message, sessionId });
res.setHeader('Content material-Kind', 'textual content/event-stream');
res.setHeader('Cache-Management', 'no-cache');
res.setHeader('Connection', 'keep-alive');
res.flushHeaders();
const controller = new AbortController();
res.on('shut', () => {
if (!res.writableEnded) controller.abort();
});
let fullResponse = '';
attempt {
const stream = await openai.chat.completions.create(
{
mannequin: 'gpt-4o-mini',
messages: [
{ role: 'system', content: 'You are a helpful assistant. Be concise and clear.' },
...session.messages,
],
stream: true,
max_tokens: 800,
},
{ sign: controller.sign }
);
for await (const chunk of stream) {
if (req.socket.destroyed) break;
const token = chunk.selections[0]?.delta?.content material;
if (token) {
fullResponse += token;
res.write(`knowledge: ${JSON.stringify({ token })}nn`);
}
}
if (fullResponse) {
session.messages.push({ function: 'assistant', content material: fullResponse });
console.log(`[stream complete] ${fullResponse.size} chars`);
}
res.write(`knowledge: ${JSON.stringify({ finished: true })}nn`);
res.finish();
} catch (err) {
if (err.title === 'AbortError' || err.title === 'APIUserAbortError' || err.message === 'Request was aborted.') {
res.finish();
return;
}
console.error('Stream error:', err.message);
res.write(`knowledge: ${JSON.stringify({ error: err.message })}nn`);
res.finish();
}
});
const PORT = course of.env.PORT || 3000;
app.pay attention(PORT, () => console.log(`Server operating at http://localhost:${PORT}`));
Constructing the Frontend
The consumer has two jobs: handle the session lifecycle and render the stream. The session lifecycle is straightforward (create on load, delete on “New Chat”), and the stream rendering is the place most implementations get tripped up.
html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta title="viewport" content material="width=device-width, initial-scale=1.0">
<title>AI Streaming Chattitle>
<model>
* { box-sizing: border-box; margin: 0; padding: 0; }
physique {
font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, sans-serif;
background: #f0f2f5;
top: 100vh;
show: flex;
flex-direction: column;
align-items: middle;
padding: 1.5rem 1rem;
}
.chat-wrapper { width: 100%; max-width: 700px; show: flex; flex-direction: column; top: 100%; }
h1 { font-size: 1.2rem; shade: #333; margin-bottom: 1rem; }
#messages {
flex: 1; overflow-y: auto; background: white; border: 1px stable #e0e0e0;
border-radius: 10px; padding: 1rem; show: flex; flex-direction: column; hole: 0.75rem;
}
.message {
max-width: 85%; padding: 0.65rem 0.9rem; border-radius: 10px;
line-height: 1.6; font-size: 0.93rem; white-space: pre-wrap; word-break: break-word;
}
.person { background: #0070f3; shade: white; align-self: flex-end; }
.assistant { background: #f1f1f1; shade: #111; align-self: flex-start; }
.input-row { show: flex; hole: 0.5rem; margin-top: 0.75rem; align-items: flex-end; }
textarea {
flex: 1; padding: 0.7rem; border: 1px stable #ddd; border-radius: 8px;
font-size: 0.93rem; font-family: inherit; resize: none; top: 52px; overflow-y: auto;
}
button { padding: 0.65rem 1.2rem; border: none; border-radius: 8px; font-size: 0.93rem; cursor: pointer; }
#ship { background: #0070f3; shade: white; }
#ship:disabled { background: #aaa; cursor: not-allowed; }
#clear { background: #f1f1f1; shade: #555; }
.standing { font-size: 0.78rem; shade: #aaa; margin-top: 0.3rem; min-height: 1rem; }
model>
head>
<physique>
<div class="chat-wrapper">
<h1>AI Chath1>
<div id="messages">div>
<p class="standing" id="standing">p>
<div class="input-row">
<textarea id="enter" placeholder="Ask one thing... (Ctrl+Enter to ship)">textarea>
<button id="ship">Shipbutton>
<button id="clear">New Chatbutton>
div>
div>
<script>
let sessionId = null;
async operate initSession() {
const res = await fetch('/session', { technique: 'POST' });
const knowledge = await res.json();
sessionId = knowledge.sessionId;
}
operate appendMessage(function, textual content) {
const div = doc.createElement('div');
div.className = `message ${function}`;
div.textContent = textual content;
doc.getElementById('messages').appendChild(div);
div.scrollIntoView({ conduct: 'clean' });
return div;
}
doc.getElementById('ship').addEventListener('click on', async () => {
const enter = doc.getElementById('enter');
const message = enter.worth.trim();
if (!message || !sessionId) return;
const sendBtn = doc.getElementById('ship');
const standing = doc.getElementById('standing');
enter.worth = '';
sendBtn.disabled = true;
standing.textContent = 'Producing...';
appendMessage('person', message);
const assistantBubble = appendMessage('assistant', '');
attempt {
const response = await fetch('/chat', {
technique: 'POST',
headers: { 'Content material-Kind': 'software/json' },
physique: JSON.stringify({ message, sessionId }),
});
if (!response.okay) {
const err = await response.json();
assistantBubble.textContent = `Error: ${err.error}`;
return;
}
const reader = response.physique.getReader();
const decoder = new TextDecoder();
whereas (true) {
const { worth, finished } = await reader.learn();
if (finished) break;
const strains = decoder.decode(worth, { stream: true }).break up('n');
for (const line of strains) {
if (!line.startsWith('knowledge: ')) proceed;
attempt {
const knowledge = JSON.parse(line.slice(6));
if (knowledge.finished) break;
if (knowledge.error) { assistantBubble.textContent += `n[Error: ${data.error}]`; break; }
if (knowledge.token) {
assistantBubble.textContent += knowledge.token;
assistantBubble.scrollIntoView({ conduct: 'clean' });
}
} catch { }
}
}
standing.textContent = '';
} catch (err) {
assistantBubble.textContent = `Request failed: ${err.message}`;
} lastly {
sendBtn.disabled = false;
standing.textContent = '';
enter.focus();
}
});
doc.getElementById('clear').addEventListener('click on', async () => {
if (!sessionId) return;
await fetch(`/session/${sessionId}`, { technique: 'DELETE' });
await initSession();
doc.getElementById('messages').innerHTML = '';
doc.getElementById('standing').textContent = '';
});
doc.getElementById('enter').addEventListener('keydown', (e) => {
if (e.key === 'Enter' && e.ctrlKey) doc.getElementById('ship').click on();
});
initSession();
script>
physique>
html>
The attempt/catch contained in the inside loop handles an actual edge case: SSE occasions often span two community reads. When that occurs, JSON.parse throws on the unfinished fragment, and catching it so the following learn can ship the remaining retains the stream alive. With out the catch, the entire stream crashes on what is basically regular community conduct.
Operating It
npm run dev
The terminal confirms the server began:
Server operating at http://localhost:3000
Hit http://localhost:3000 in your browser and kind one thing. The terminal ought to print:
[/session] created: 2dfb2586-4601-4781-932d-cb3dcca5d5a5
[/chat] acquired: {
message: 'Have you learnt about StackAbuse?',
sessionId: '2dfb2586-4601-4781-932d-cb3dcca5d5a5'
}
[stream complete] 304 chars
Within the browser, you will note the response construct out token by token, which suggests the streaming is working. The interface appears to be like like this with a reside response:
Ship a follow-up that references the primary reply and the mannequin will bear in mind it. The session historical past is included in each request, so context carries throughout turns. Click on “New Chat” and the server deletes the session and creates a recent one. The mannequin begins over with no reminiscence of the earlier dialog.
To examine the uncooked stream, open DevTools, go to Community, discover the /chat request, and have a look at the EventStream tab. You will notice every knowledge: occasion arrive individually, which is essentially the most helpful factor I do know of for debugging streaming conduct that appears appropriate within the UI however will not be working the way in which you assume.
Manufacturing Gaps to Handle Earlier than Launch
Earlier than you ship this, know what the implementation doesn’t deal with. I’ve seen builders take tutorial code and deploy it unchanged.
Context overflow. Each request sends the total session historical past. gpt-4o-mini has a 128,000-token context window, which is giant sufficient that the majority chat classes won’t ever hit it. However a session operating for hours will finally method the restrict. When it does, the API returns a context_length_exceeded error. The usual repair is a sliding window: maintain the system immediate, trim previous messages from the entrance of the historical past, optionally run a summarization name to compress previous context earlier than it falls off. It is a actual engineering downside and there’s no one-size reply.
Fee limiting. There may be none right here, so one consumer can exhaust your month-to-month OpenAI quota. Add express-rate-limit earlier than this goes wherever close to the web.
Authentication. The /chat endpoint accepts requests from anybody. In an actual deployment, you want session-based auth or JWT validation earlier than the route handler runs. With out it, anybody who is aware of your URL is spending your API credit.
Multi-process classes. Coated above. Redis or equal earlier than you scale horizontally.
Conclusion
I’ve shipped variations of this precise setup to manufacturing twice now, throughout totally different corporations and merchandise, with the identical core sample: session retailer on the server, SSE for transport, and AbortController to cease losing cash on disconnected shoppers. The gaps I listed aren’t hypothetical warnings; I hit most of them myself earlier than determining the fixes.
The half that journeys most builders will not be the OpenAI API itself, however the plumbing round it: getting tokens to the browser the second they arrive, retaining the connection alive with out burning credit on disconnected shoppers, and sustaining dialog context with out letting it develop unbounded. That’s the place the actual work sits. Hopefully this text saved you the afternoon I spent studying most of it the onerous manner.
The total code is at github.com/ziaongit/nodejs-openai-streaming. Clone it, break it, and make it your personal.
