AI API Parameters Explained: Temperature, System Prompt & Tokens

👁 123 reads

System prompt establishes who the model is and what rules it follows. Temperature decides how predictable responses are. Context window is the limit of everything the model sees at once, and max tokens is the response length limit. Below I explain each parameter in plain language and show real results: the same question at temperature 0, 0.7, and 1.2, three different system prompts, and a response cut mid-word. This is part of the AI API from scratch learning path.

System prompt: the instruction that governs the entire conversation

In an API request, you send a list of messages. The first one, with role system (in Anthropic a separate system field, in OpenAI Responses an instructions field), tells the model who to be and what to watch for. The rest are exchanges of user and assistant. The model takes system instructions more seriously than user requests, so you put rules there that can’t change: language, tone, format, what not to do.

I tested how much the system prompt changes the response to the same question - we tested it in Polish - in the Mistral Small 4 model:

"""System prompt changes model behavior, and max_tokens cuts the response (finish_reason=length)."""
import os
from dotenv import load_dotenv
from openai import OpenAI

load_dotenv()
client = OpenAI(base_url="https://openrouter.ai/api/v1", api_key=os.environ["OPENROUTER_API_KEY"])
MODEL = "mistralai/mistral-small-2603"
QUESTION = "Czy warto uczyć się programowania w 2026 roku?"

for system in (
    None,
    "Jesteś zrzędliwym programistą z 30-letnim stażem. Odpowiadasz w maksymalnie dwóch zdaniach.",
    "Odpowiadasz wyłącznie jednym słowem: TAK albo NIE.",
):
    messages = [{"role": "user", "content": QUESTION}]
    if system:
        messages.insert(0, {"role": "system", "content": system})
    r = client.chat.completions.create(model=MODEL, messages=messages, max_tokens=150, temperature=0.3)
    print(f"n### system: {system or '(none)'}")
    print(r.choices[0].message.content.strip())
    print(f"[finish_reason={r.choices[0].finish_reason}, output tokens={r.usage.completion_tokens}]")

# max_tokens too small: response cut mid-sentence
r = client.chat.completions.create(model=MODEL, messages=[{"role": "user", "content": QUESTION}], max_tokens=25)
print("n### max_tokens=25")
print(r.choices[0].message.content.strip())
print(f"[finish_reason={r.choices[0].finish_reason}]")

### system: (none)
Tak, **uczenie się programowania w 2026 roku nadal będzie bardzo wartościowe**, a nawet bardziej niż obecnie. Oto kluczowe powody, dlaczego warto zainwestować w te umiejętności:

### **1. Wysokie zapotrzebowanie na programistów**
- **AI i automatyzacja** będą dalej zastępować rutynowe zadania, ale **programiści będą potrzebni do tworzenia, utrzymania i rozwijania systemów AI, narzędzi i infrastruktury**.
- Według **World Economic Forum**, do 2025 roku **ponad połowa wszystkich nowych miejsc pracy** będzie wymagać umieję
[finish_reason=length, output tokens=150]

### system: Jesteś zrzędliwym programistą z 30-letnim stażem. Odpowiadasz w maksymalnie dwóch zdaniach.
Nawet w 2026 programowanie będzie potrzebne, ale nie ucz się tylko samego kodu - zrozum problem, który rozwiązujesz, bo inaczej stracisz pracę na rzecz AI. I nie marudź, że to trudne, bo w moich czasach mieliśmy dziury w kartach perforowanych.
[finish_reason=stop, output tokens=79]

### system: Odpowiadasz wyłącznie jednym słowem: TAK albo NIE.
TAK
[finish_reason=stop, output tokens=3]

Three observations:

• Without a system prompt the model wrote at length with bullet points, headings and bold text, and didn’t fit within the 150 token limit (finish_reason=length). It also cited a forecast “by 2025,” meaning outdated knowledge from training data.
• With the “grumpy programmer” role it fit in two sentences, changed tone, and finished the response itself (stop).
• With the one-word constraint it answered “TAK” (YES) and used 3 tokens instead of 150. This is the cheapest way to do classification.

Rules I follow when writing system prompts:

1. Concrete instead of adjectives: “maximum 3 sentences” instead of “concisely.”
2. Response format explicitly: list, JSON, one word. For strict JSON I use structured output mode, not just a request.
3. What not to do, with a reason: “don’t provide prices, they change weekly.”
4. Long, fixed system prompt at the start of the request - then the provider can keep it in cache and charge less.

More on the art of writing instructions: Prompt: what it is and how to write AI commands and System prompts in Claude.

Temperature: experiment with 0 / 0.7 / 1.2

At each token, the model chooses from a list of candidates with probabilities. Temperature “flattens” or “sharpens” this distribution. At 0 the model almost always takes the most probable token, so the response is repeatable. At higher values it reaches for less obvious words more often: responses are more varied, but also riskier.

I sent the same command nine times to Mistral Small 4: three times each at temperature 0, 0.7, and 1.2.

"""Experiment: same query at temperature 0, 0.7 and 1.2 (3 times each)."""
import os, sys
from dotenv import load_dotenv
from openai import OpenAI

load_dotenv()
client = OpenAI(base_url="https://openrouter.ai/api/v1", api_key=os.environ["OPENROUTER_API_KEY"])

# Model must support temperature. OpenAI reasoning models (GPT-6) don't accept it.
MODEL = sys.argv[1] if len(sys.argv) > 1 else "mistralai/mistral-small-2603"
QUESTION = "Wymyśl nazwę i jednozdaniowy slogan dla małej kawiarni w Gdańsku. Odpowiedz w jednej linii."

for temperature in (0, 0.7, 1.2):
    print(f"n=== temperature={temperature} ===")
    for attempt in range(1, 4):
        r = client.chat.completions.create(
            model=MODEL,
            messages=[{"role": "user", "content": QUESTION}],
            temperature=temperature,
            max_tokens=80,
            # only providers that truly support temperature
            extra_body={"provider": {"require_parameters": True}},
        )
        print(f"{attempt}. {r.choices[0].message.content.strip()}")

=== temperature=0 ===
1. **"Bursztynowa Kawa" - "Gdański smak, który rozgrzewa duszę."*
2. **"Bursztynowa Kawa" - "Gdański smak, który rozgrzewa duszę."*
3. **"Bursztynowa Kawa" - "Gdański smak, który rozgrzewa duszę."*

=== temperature=0.7 ===
1. **"Morska Kawa"** - "Gdańskie smaki w sercu miasta."
2. **"Kawa na Darszu"** - *Gdańska kawa, lokalny smak.*
3. **"Gdańska Mała Kawa" - "Gdzie każda filiżanka to kawałek Gdańska."*

=== temperature=1.2 ===
1. **"Kawa Mrugu"** - *Twój ulubiony smak spotyka gdański zachód słońca.*
2. **"Bursztynowa Kawiarnia
- Serce Gdańska w filiżance."**
3. **"Bursztyn Café"** - "Gdzie rodzinna atmosfera spotyka się z gdańskim niebem."

What you can see in the results:

• Temperature 0: three identical responses, character for character, including an unclosed formatting asterisk. This is what you want for data extraction, classification, and tests.
• Temperature 0.7: three different, sensible proposals. Good compromise for marketing copy and chat responses.
• Temperature 1.2: oddities appear: “Kawa Mrugu” (a non-existent word), line breaks mid-name, mixed quotation marks. Sometimes you get a gem, but for production it’s too much randomness.

Two traps I encountered:

1. Temperature 0 doesn’t guarantee identical responses. For me it came out the same three times, but providers don’t promise full repeatability (GPU calculations, different servers). Why chat gives different answers to the same question is explained in Why ChatGPT gives different answers.
2. Not every model accepts temperature. OpenAI reasoning models (GPT-6 Luna, Sol, Astra) don’t support it. OpenRouter by default silently ignores such a parameter, so the code works but temperature changes nothing. I verified this:

"""Trap: GPT-6 doesn't support temperature, and OpenRouter by default silently ignores this parameter."""
import os
from dotenv import load_dotenv
import openai
from openai import OpenAI

load_dotenv()
client = OpenAI(base_url="https://openrouter.ai/api/v1", api_key=os.environ["OPENROUTER_API_KEY"])

for require in (False, True):
    try:
        r = client.chat.completions.create(
            model="openai/gpt-6-luna",
            messages=[{"role": "user", "content": "Podaj jedną nazwę dla kawiarni w Gdańsku."}],
            temperature=1.2,
            max_tokens=60,
            extra_body={"provider": {"require_parameters": require}},
        )
        print(f"require_parameters={require}: 200 OK -> {r.choices[0].message.content}")
    except openai.APIStatusError as e:
        print(f"require_parameters={require}: {e.status_code} -> {e.body.get('message') if isinstance(e.body, dict) else e.message}")

require_parameters=False: 200 OK -> Mokka Motława
require_parameters=True: 404 -> No endpoints found that can handle the requested parameters. To learn more about provider routing, visit: https://openrouter.ai/docs/guides/routing/provider-selection

Without require_parameters the request went through and temperature 1.2 was ignored. With this setting, OpenRouter honestly refused (404 - no provider that will handle all parameters). In reasoning models, instead of temperature you control the reasoning_effort parameter (how much the model thinks) and verbosity (how long it writes).

The related top_p parameter limits the choice to the most probable tokens whose cumulative probability doesn’t exceed the given value. Practical rule: change either temperature or top_p, not both at once.

Context window: how much the model sees at once

The context window is the maximum number of tokens in one request: system prompt, entire conversation history, pasted documents, tool descriptions, and the model’s response. The API doesn’t remember previous requests - if you want a conversation, each time you send the entire history again and each time you pay for it.

A million tokens is several books worth of text, but in practice three things spoil this picture:

• Cost. A request with 500k input tokens to Claude Sonnet 5.5 costs $1 before the model writes a word. GPT-6 and Grok above thresholds (272k and 200k) charge higher rates for the entire request.
• Quality. The model is worse at finding information in the middle of very long text. I tested this in Models promise a million tokens of context - two of three models failed.
• Non-English takes more. The same text in Polish is 1.4 to 2.1 times more tokens than in English (measurement in AI Tokens).

When there are more documents than sensibly fit in the window, instead of pasting everything you use fragment retrieval, or RAG. I explain it in RAG: how it works and show it in code in RAG step by step.

max tokens: response length limit

max_tokens (in newer OpenAI API max_completion_tokens, in Gemini max_output_tokens) is the ceiling on the number of tokens the model can generate. The model doesn’t “plan” the response under this limit - it simply gets cut when it reaches it:

### max_tokens=25
Tak, warto uczyć się programowania w **2026 roku** - i to z kilku ważnych powod
[finish_reason=length]

How to use it:

• Always check finish_reason. stop - model finished on its own, length - cut by limit, tool_calls - wants to call a tool, content_filter - blocked.
• The limit is cost protection, not a way to get short answers. You enforce brevity in the system prompt (“maximum 3 sentences”), and set max_tokens with headroom.
• In reasoning models the limit includes thinking. When you set 200 tokens and the model uses 190 on thinking, the response will have 10 tokens or be empty. For such models I give a limit of at least several thousand.
• In Anthropic max_tokens is mandatory - a request without it will get a 400 error.

Other parameters worth knowing

My starting settings

Frequently asked questions

What is a system prompt?

It’s an instruction for the model sent at the start of a request, with role system. It establishes the rules for the entire conversation: who the model is, what language and format it responds in, what it doesn’t do. In chat applications the user doesn’t see it.

What temperature should I set?

0 for tasks with one correct answer (data, classification, code), around 0.7 for regular text, 0.9-1.1 for finding ideas. Above 1.2 responses start to fall apart, as you can see in my test.

What is a context window?

The maximum number of tokens the model will process in one request, together with the response. In October 2026 leading models have around 1 million tokens, smaller ones 128-256k.

How does max_tokens differ from context window?

The window is the limit for everything (input plus output), set by the model. max_tokens is your limit for just the response, which you set in the request and which can’t exceed the model’s maximum response.

Why does the model ignore my system prompt?

Most often instructions are contradictory or vague, and the user’s request says something different. Concrete helps (“maximum 3 sentences”), an example of the expected response, and repeating the most important rule at the end of the instruction.

Google · Your sourcesQuantslant higher in your Google - one clickAdd to preferred sources →

Sources and verification date

I ran the experiments on October 4, 2026 on Windows 11, Python 3.13.15, openai 3.24.0, through OpenRouter. Cost of all requests in this article: about $0.002.

• OpenRouter - request parameters
• OpenRouter - require_parameters
• Anthropic - models, context windows and response limits
• Google - Gemini models
• OpenAI - models
• DeepSeek - models and limits

Kinetyra