Streaming Responses

Receive model responses as they're generated for better user experience.

Why Stream?

• Faster perceived response: Users see output immediately
• Better UX: Progressive display feels more responsive
• Memory efficiency: Process chunks without buffering full response
• Real-time applications: Chat interfaces, live transcription

Enabling Streaming

Set stream: true in your request:

{
  "model": "meta-llama/Llama-3.1-8B-Instruct",
  "messages": [...],
  "stream": true
}

Response Format

Streaming uses Server-Sent Events (SSE):

data: {"id":"chatcmpl-123","object":"chat.completion.chunk","created":1704067200,"model":"meta-llama/Llama-3.1-8B-Instruct","choices":[{"index":0,"delta":{"role":"assistant"},"finish_reason":null}]}

data: {"id":"chatcmpl-123","object":"chat.completion.chunk","created":1704067200,"model":"meta-llama/Llama-3.1-8B-Instruct","choices":[{"index":0,"delta":{"content":"Hello"},"finish_reason":null}]}

data: {"id":"chatcmpl-123","object":"chat.completion.chunk","created":1704067200,"model":"meta-llama/Llama-3.1-8B-Instruct","choices":[{"index":0,"delta":{"content":"!"},"finish_reason":null}]}

data: {"id":"chatcmpl-123","object":"chat.completion.chunk","created":1704067200,"model":"meta-llama/Llama-3.1-8B-Instruct","choices":[{"index":0,"delta":{},"finish_reason":"stop"}]}

data: [DONE]

Chunk Structure

Field	Description
id	Same ID for all chunks
object	chat.completion.chunk
delta.content	New content piece
delta.role	Role (first chunk only)
finish_reason	stop, length, or null

note

For vLLM-backed models, chunks may also carry a usage object with running token counts. Ignore it if you don't need it — most SDKs expose it as an optional field.

Python Implementation

Basic Streaming

from openai import OpenAI

client = OpenAI(
    api_key="sk-proj-your-api-key",
    base_url="https://api.bulutistan.ai/v1"
)

stream = client.chat.completions.create(
    model="meta-llama/Llama-3.1-8B-Instruct",
    messages=[{"role": "user", "content": "Tell me a story"}],
    stream=True
)

for chunk in stream:
    content = chunk.choices[0].delta.content
    if content:
        print(content, end="", flush=True)

Collecting Full Response

full_response = ""

for chunk in stream:
    content = chunk.choices[0].delta.content
    if content:
        full_response += content
        print(content, end="", flush=True)

print(f"\n\nFull response length: {len(full_response)}")

Async Streaming

from openai import AsyncOpenAI
import asyncio

async_client = AsyncOpenAI(
    api_key="sk-proj-your-api-key",
    base_url="https://api.bulutistan.ai/v1"
)

async def stream_response():
    stream = await async_client.chat.completions.create(
        model="meta-llama/Llama-3.1-8B-Instruct",
        messages=[{"role": "user", "content": "Hello"}],
        stream=True
    )

    async for chunk in stream:
        content = chunk.choices[0].delta.content
        if content:
            print(content, end="", flush=True)

asyncio.run(stream_response())

Node.js Implementation

Basic Streaming

import OpenAI from 'openai';

const client = new OpenAI({
    apiKey: 'sk-proj-your-api-key',
    baseURL: 'https://api.bulutistan.ai/v1'
});

const stream = await client.chat.completions.create({
    model: 'meta-llama/Llama-3.1-8B-Instruct',
    messages: [{ role: 'user', content: 'Tell me a story' }],
    stream: true
});

for await (const chunk of stream) {
    const content = chunk.choices[0]?.delta?.content;
    if (content) {
        process.stdout.write(content);
    }
}

Express.js SSE

app.post('/chat', async (req, res) => {
    res.setHeader('Content-Type', 'text/event-stream');
    res.setHeader('Cache-Control', 'no-cache');
    res.setHeader('Connection', 'keep-alive');

    const stream = await client.chat.completions.create({
        model: 'meta-llama/Llama-3.1-8B-Instruct',
        messages: req.body.messages,
        stream: true
    });

    for await (const chunk of stream) {
        const content = chunk.choices[0]?.delta?.content;
        if (content) {
            res.write(`data: ${JSON.stringify({ content })}\n\n`);
        }
    }

    res.write('data: [DONE]\n\n');
    res.end();
});

Frontend Implementation

React with fetch

function ChatComponent() {
    const [response, setResponse] = useState('');

    const handleStream = async (message) => {
        const res = await fetch('/api/chat', {
            method: 'POST',
            headers: { 'Content-Type': 'application/json' },
            body: JSON.stringify({ message })
        });

        const reader = res.body.getReader();
        const decoder = new TextDecoder();

        while (true) {
            const { done, value } = await reader.read();
            if (done) break;

            const text = decoder.decode(value);
            const lines = text.split('\n');

            for (const line of lines) {
                if (line.startsWith('data: ')) {
                    const data = line.slice(6);
                    if (data === '[DONE]') break;

                    const parsed = JSON.parse(data);
                    setResponse(prev => prev + parsed.content);
                }
            }
        }
    };

    return (
        <div>
            <button onClick={() => handleStream('Hello')}>Send</button>
            <div>{response}</div>
        </div>
    );
}

Error Handling

try:
    stream = client.chat.completions.create(
        model="meta-llama/Llama-3.1-8B-Instruct",
        messages=[{"role": "user", "content": "Hello"}],
        stream=True
    )

    for chunk in stream:
        content = chunk.choices[0].delta.content
        if content:
            print(content, end="")

except Exception as e:
    print(f"\nStream error: {e}")

Cancelling a Stream

Streams can be cancelled mid-flight by closing the connection client-side (e.g. AbortController in the browser, breaking out of the iterator in Python/Node). Browser aborts are propagated all the way through the portal's Next.js proxy to the upstream gateway, so the gateway sees a real client-disconnect rather than continuing to generate tokens that no one is reading. The platform records partial usage when this happens:

• Tokens generated up to the cancel point are billed. vLLM-backed models emit a running token count on every chunk, so the gateway has accurate numbers even when the client disconnects before the final chunk.

• The usage event is recorded with status="cancelled" and finish_reason="client_disconnect" so you can distinguish cancelled requests from completed ones in the usage breakdown.

• Other dispositions exposed on the same event:

  • status="completed" — normal end of stream
  • status="cancelled" — client disconnected mid-stream
  • status="error" — upstream model returned an error

  (A timeout status also exists on the platform, but only for background/async jobs — streaming requests never resolve to it.)

Cancellation no longer bypasses billing — there is no way to consume tokens for free by closing the connection just before the final chunk.

Best Practices

1. Always flush output: Use flush=True or equivalent
2. Handle connection drops: Implement reconnection logic
3. Set timeouts: Long streams need appropriate timeouts
4. Buffer carefully: Balance responsiveness vs. flickering
5. Show loading state: Indicate when waiting for first chunk
6. Don't rely on cancel-to-skip-billing: Cancelled streams are billed for tokens already produced.