What does temperature do in ChatGPT?

It sets how adventurous each draw is. OpenAI's API takes a temperature from 0 to 2 and defaults to 1. Lower values sharpen the odds toward the likeliest tokens, so output is more focused; higher values flatten them, so rarer tokens get picked more often. OpenAI's own example: 0.8 for more random output, 0.2 for more focused and deterministic.

What's the difference between top-p and temperature?

Temperature changes the shape of the odds; top-p decides which tokens may be picked at all. Top-p 0.9 keeps the fewest likeliest tokens whose chances add up to 90% and rescales them. OpenAI's API defaults both to 1 and recommends changing one or the other, not both. On the stage, the same top-p keeps 39 words at temperature 0.5 and 807 at 2.0.

Why does ChatGPT give different answers to the same question?

Each token is a random draw from the odds. Two runs see the same odds but get different rolls, and one different word early on changes everything after it. Change the seed on the stage: the odds stay the same and the text doesn't.

Does ChatGPT always pick the most likely word?

Not at the default settings. It samples, so a less likely token wins whenever the roll lands on it. Always taking the likeliest text tends to be bland and repetitive, which is why sampling exists. On the stage the first pick, “had”, is only the second likeliest word after “alice”.

Is ChatGPT just autocomplete?

It runs the same loop: predict the next token from what came before, add it, repeat. The difference is the predictor. A network trained on an enormous amount of text and reading the whole conversation gives far better odds than counts of word pairs from one book.

Live model · ChatGPT

How ChatGPT Actually Picks the Next Word

ChatGPT writes one token at a time. For each one, its network scores every token in its vocabulary, and a softmax turns the scores into odds. Temperature reshapes those odds, top-p cuts off the unlikely tail, and a weighted random draw picks the next token. Then it repeats, with that token added.

Writing one word at a time

The model counted which word follows which in the 6,929 words of Alice's first three chapters, 1,170 different ones. It reads one word back, “alice”, and will write 12 words, one at a time, at temperature 1.0 and top-p 0.9.

StopNext word

PromptTemperatureTop-pWords to writeSeed

CH1 Kept by top-p
CH2 The word it reads
CH4 Words it wrote

How to read the stage

A small model stands in for the network: it counted which word follows which in the 6,929 words of Alice in Wonderland's first three chapters, and reads only the last word. After “alice” it gives “,” 20% and “had” 11%. Top-p 0.9 keeps 34 words and cuts 1,136. The roll, 0.250, lands on “had”, the second likeliest. Twelve rolls write “had no pictures hung upon its face brightened up by wild beasts”.

Temperature reshapes, top-p cuts

Temperature divides the network's scores by T before the softmax. That's the same as raising every chance to the power 1/T and rescaling. At 0.5 each chance is squared, so after “the”, “mouse” jumps from 8.3% to 43%; at 2.0 it's square-rooted and falls to 1.6%. Top-p then keeps the fewest likeliest words holding at least p. With 0.9, that's 1 word at temperature 0, 39 at 0.5, 140 at 1.0 and 807 at 2.0.

One set of odds at four temperatures

285 words follow “the” in the book. The model's odds start at “mouse” with 8.3% and run down through all 1,170 words it knows.

StopTemperature

Word it readsTop-p

CH1 Kept by top-p

Why temperature 0 repeats itself

Temperature 0 always takes the top word, so there's no roll. This model reads one word back, so the first time a word returns, the text loops: from “alice” it writes “, and she was not a little door” and then those eight words forever. Real models read much further back and loop less tightly, but Holtzman and colleagues found that always choosing the likeliest text makes output bland, incoherent or stuck in repetitive loops.

Temperature 0: no roll

StopNext word

PromptTemperatureTop-pWords to writeSeed

CH1 Kept by top-p
CH2 The word it reads
CH4 Words it wrote

What the model leaves out

ChatGPT's network reads the conversation so far, as far back as its context window allows, and its scores come from a trained network, not counts. It picks tokens, often pieces of words. This model reads one word and learned from one short book, so every word pair it writes is from the book and the sentences are nonsense. The sampling is the same: odds, temperature, top-p, a weighted roll. It offers only temperatures whose power is exact.

Questions

What does temperature do in ChatGPT?: It sets how adventurous each draw is. OpenAI's API takes a temperature from 0 to 2 and defaults to 1. Lower values sharpen the odds toward the likeliest tokens, so output is more focused; higher values flatten them, so rarer tokens get picked more often. OpenAI's own example: 0.8 for more random output, 0.2 for more focused and deterministic.
What's the difference between top-p and temperature?: Temperature changes the shape of the odds; top-p decides which tokens may be picked at all. Top-p 0.9 keeps the fewest likeliest tokens whose chances add up to 90% and rescales them. OpenAI's API defaults both to 1 and recommends changing one or the other, not both. On the stage, the same top-p keeps 39 words at temperature 0.5 and 807 at 2.0.
Why does ChatGPT give different answers to the same question?: Each token is a random draw from the odds. Two runs see the same odds but get different rolls, and one different word early on changes everything after it. Change the seed on the stage: the odds stay the same and the text doesn't.
Does ChatGPT always pick the most likely word?: Not at the default settings. It samples, so a less likely token wins whenever the roll lands on it. Always taking the likeliest text tends to be bland and repetitive, which is why sampling exists. On the stage the first pick, “had”, is only the second likeliest word after “alice”.
Is ChatGPT just autocomplete?: It runs the same loop: predict the next token from what came before, add it, repeat. The difference is the predictor. A network trained on an enormous amount of text and reading the whole conversation gives far better odds than counts of word pairs from one book.

Drill the mechanism

Drawing the bars is a counting job: a hash map from each word to the words after it and how often, then the eight most frequent. That's Top K Frequent Elements. A heap of size k finds them without sorting all 1,170 words, which Kth Largest Element in an Array drills.