Rate Limiting

Understand and handle API rate limits effectively.

How Rate Limiting Works

Rate limits protect the platform and ensure fair usage:

Limit Type	Scope	Reset
Requests per minute	Per API key / user	Rolling window
Tokens per minute	Per API key / user	Rolling window
Tokens per day	Per API key / user	Daily
Concurrent requests	Per tenant / user	Active connections
Cost per day	Per tenant / user	Daily
Cost per month	Per tenant / user	Monthly

Concurrent Limit Accuracy

The concurrent-request limit is enforced atomically: every admission decision (read current count → compare to cap → increment) runs as a single Redis Lua step, so parallel callers cannot squeeze past the cap by hitting the gateway in the same millisecond. A tenant with concurrent_requests=10 will admit exactly 10 in-flight requests at once; the 11th gets a 429 until one of the in-flight ones finishes. The x-ratelimit-active-concurrent header reflects the same atomic counter.

Rate Limit Headers

Rate limit headers are included in API responses, but their format differs between successful and rate-limited responses.

On Successful Responses (200)

Successful responses include OpenAI-compatible headers with Unix timestamp reset values, plus additional platform-specific headers:

# OpenAI-compatible headers
x-ratelimit-limit-requests: 100
x-ratelimit-remaining-requests: 95
x-ratelimit-limit-tokens: 100000
x-ratelimit-remaining-tokens: 85000
x-ratelimit-reset-requests: 1704067260
x-ratelimit-reset-tokens: 1704067260

# Platform-specific headers
x-ratelimit-limit-concurrent: 50
x-ratelimit-active-concurrent: 3
x-ratelimit-limit-cost: 1000.00
x-ratelimit-used-cost: 12.50
x-ratelimit-limit-tpd: 10000000
x-ratelimit-used-tpd: 150000

Header	Description
x-ratelimit-limit-*	Maximum allowed
x-ratelimit-remaining-*	Currently available
x-ratelimit-reset-*	Unix timestamp of next reset (e.g., 1704067260)
x-ratelimit-limit-requests	Requests per minute limit
x-ratelimit-remaining-requests	Remaining requests this minute
x-ratelimit-limit-concurrent	Max concurrent requests
x-ratelimit-active-concurrent	Currently active concurrent requests
x-ratelimit-limit-cost	Cost limit (daily or monthly)
x-ratelimit-used-cost	Cost consumed so far
x-ratelimit-limit-tpd	Tokens per day limit
x-ratelimit-used-tpd	Tokens consumed today

All header names are lowercase.

On Rate-Limited Responses (429)

When a rate limit is exceeded, the response includes the same Unix timestamp reset values plus a Retry-After header:

x-ratelimit-limit-requests: 100
x-ratelimit-remaining-requests: 0
x-ratelimit-reset-requests: 1704067260
x-ratelimit-limit-tokens: 100000
x-ratelimit-remaining-tokens: 0
x-ratelimit-reset-tokens: 1704067260
Retry-After: 30

Rate Limit Errors

When exceeded, you receive an OpenAI-style error envelope:

{
  "error": {
    "message": "Rate limit exceeded for requests_per_minute. Try again in 30 seconds.",
    "type": "rate_limit_error",
    "param": null,
    "code": "too_many_requests",
    "limit_type": "requests_per_minute",
    "limit": 100,
    "current": 100,
    "retry_after": 30
  }
}

HTTP Status: 429 Too Many Requests

Field	Type	Description
error.message	string	Rate limit exceeded for {limit_type}. Try again in {n} seconds.
error.type	string	Always rate_limit_error for 429
error.param	null	Always present, always null for 429
error.code	string	too_many_requests
error.limit_type	string	Which limit was exceeded (e.g. requests_per_minute, tokens_per_day, concurrent_requests, cost_per_day)
error.limit	number	The configured limit value
error.current	number	Current consumption that triggered the violation
error.retry_after	integer	Seconds to wait before retrying
Retry-After header	integer	Same value, in HTTP header form

error.message is always a plain string. Older versions of the platform sometimes nested a {message, retry_after} object inside error.message; that is no longer the case. SDKs and UIs that render error.message directly always receive a clean string.

Handling Rate Limits

Python with Retry

import time
from openai import OpenAI, RateLimitError

client = OpenAI(
    api_key="sk-proj-your-api-key",
    base_url="https://api.bulutistan.ai/v1"
)

def chat_with_retry(messages, max_retries=5):
    for attempt in range(max_retries):
        try:
            return client.chat.completions.create(
                model="meta-llama/Llama-3.1-8B-Instruct",
                messages=messages
            )
        except RateLimitError as e:
            if attempt == max_retries - 1:
                raise

            # Exponential backoff
            wait_time = min(2 ** attempt, 60)
            print(f"Rate limited. Waiting {wait_time}s...")
            time.sleep(wait_time)

Using tenacity Library

from tenacity import (
    retry,
    stop_after_attempt,
    wait_exponential,
    retry_if_exception_type
)
from openai import RateLimitError

@retry(
    retry=retry_if_exception_type(RateLimitError),
    wait=wait_exponential(multiplier=1, min=1, max=60),
    stop=stop_after_attempt(5)
)
def chat_completion(messages):
    return client.chat.completions.create(
        model="meta-llama/Llama-3.1-8B-Instruct",
        messages=messages
    )

Node.js with Retry

async function chatWithRetry(messages, maxRetries = 5) {
    for (let attempt = 0; attempt < maxRetries; attempt++) {
        try {
            return await client.chat.completions.create({
                model: 'meta-llama/Llama-3.1-8B-Instruct',
                messages
            });
        } catch (error) {
            if (error instanceof OpenAI.RateLimitError) {
                if (attempt === maxRetries - 1) throw error;

                const waitTime = Math.min(Math.pow(2, attempt) * 1000, 60000);
                console.log(`Rate limited. Waiting ${waitTime}ms...`);
                await new Promise(r => setTimeout(r, waitTime));
            } else {
                throw error;
            }
        }
    }
}

Proactive Rate Management

Monitor Remaining Limits

def check_rate_limits(response):
    headers = response.headers
    remaining = int(headers.get('x-ratelimit-remaining-requests', 0))
    reset_timestamp = int(headers.get('x-ratelimit-reset-requests', '0'))

    # Calculate seconds until reset from Unix timestamp
    wait_seconds = max(reset_timestamp - int(time.time()), 0)

    if remaining < 10:
        print(f"Low on requests. {remaining} left. Resets in {wait_seconds}s")
        if remaining < 5:
            time.sleep(max(wait_seconds, 1))

Request Throttling

import time
from collections import deque

class RateLimiter:
    def __init__(self, max_requests_per_minute=60):
        self.max_rpm = max_requests_per_minute
        self.requests = deque()

    def wait_if_needed(self):
        now = time.time()

        # Remove old requests
        while self.requests and self.requests[0] < now - 60:
            self.requests.popleft()

        if len(self.requests) >= self.max_rpm:
            sleep_time = 60 - (now - self.requests[0])
            if sleep_time > 0:
                print(f"Throttling: waiting {sleep_time:.1f}s")
                time.sleep(sleep_time)

        self.requests.append(time.time())

# Usage
limiter = RateLimiter(max_requests_per_minute=50)

for message in messages:
    limiter.wait_if_needed()
    response = client.chat.completions.create(...)

Batch Processing

For high-volume processing, use batching:

import asyncio
from openai import AsyncOpenAI

async_client = AsyncOpenAI(
    api_key="sk-proj-your-api-key",
    base_url="https://api.bulutistan.ai/v1"
)

async def process_batch(messages_list, batch_size=10, delay=1):
    results = []

    for i in range(0, len(messages_list), batch_size):
        batch = messages_list[i:i + batch_size]

        # Process batch concurrently
        tasks = [
            async_client.chat.completions.create(
                model="meta-llama/Llama-3.1-8B-Instruct",
                messages=msgs
            )
            for msgs in batch
        ]

        batch_results = await asyncio.gather(*tasks, return_exceptions=True)
        results.extend(batch_results)

        # Delay between batches
        if i + batch_size < len(messages_list):
            await asyncio.sleep(delay)

    return results

Best Practices

1. Implement exponential backoff: Start small, increase wait time
2. Monitor rate limit headers: Adjust behavior proactively
3. Use request queuing: Smooth out burst traffic
4. Cache responses: Avoid repeated identical requests
5. Use async jobs: For batch processing, use async job API
6. Distribute across keys: Use multiple API keys for higher limits

Your Rate Limits

Rate limits are configured per tenant. Check your current limits:

curl https://api.bulutistan.ai/api/v1/profile/rate-limits \
  -H "Authorization: Bearer <your-jwt-token>"

Or view in the Portal Dashboard.

Contact your administrator for limit adjustments.

Rate Limits API

Get Your Effective Rate Limits

Retrieve your current rate limits with source information:

curl https://api.bulutistan.ai/api/v1/profile/rate-limits \
  -H "Authorization: Bearer <your-jwt-token>"

note

This endpoint also accepts a management API key with the rate_limits:read scope in place of a JWT.

Response:

{
  "requests_per_minute": {
    "value": 1000,
    "source": "platform",
    "is_custom": false
  },
  "tokens_per_minute": {
    "value": 100000,
    "source": "platform",
    "is_custom": false
  },
  "tokens_per_day": {
    "value": 10000000,
    "source": "platform",
    "is_custom": false
  },
  "concurrent_requests": {
    "value": 100,
    "source": "tenant",
    "is_custom": true
  },
  "cost_per_day": {
    "value": 10000.0,
    "source": "platform",
    "is_custom": false
  },
  "cost_per_month": {
    "value": 2000.0,
    "source": "platform",
    "is_custom": false
  }
}

Understanding Limit Sources

Rate limits follow a hierarchy. The effective limit is the most restrictive:

API Key Level (most restrictive)
    └── User Level
            └── Tenant Level
                    └── Platform Level (least restrictive)

Source	Description
api_key	Limit set on specific API key
user	Your personal limit
tenant	Organization-wide limit
platform	System default

Each limit also includes an is_custom field indicating whether the limit was explicitly configured (true) or inherited from a parent level (false).

Python Example

import requests

jwt_token = "<your-jwt-token>"
base_url = "https://api.bulutistan.ai"

response = requests.get(
    f"{base_url}/api/v1/profile/rate-limits",
    headers={"Authorization": f"Bearer {jwt_token}"}
)

limits = response.json()

print("Your Rate Limits:")
for limit_name, info in limits.items():
    custom = " (custom)" if info['is_custom'] else ""
    print(f"  {limit_name}: {info['value']} (from {info['source']}){custom}")