8

Most LLMs these days have an output limit. Meaning the output cannot go further than X amount of tokens.

I've seen many "hacks" on social media saying how you can ask these LLMs to "continue where you left off" but none of them are really effective. I presume this is something to do with the models ability of retaining context and thinking under the hood (otherwise there wouldn't be any limit, no?)

I'm wondering why is this and how the limits work. Like does it get exponentially harder to run as more text is outputted? I've noticed that ChatGPT now has a continue generating button, that only works for so much times.

3
  • " "continue where you left off" but none of them are really effective. " Please share Jul 28, 2023 at 14:56
  • 1
    What I mean by that is, it's hit or miss. A lot of times when generating code, it doesn't just pick up where it left off. It either restarts from the beginning providing code or provides a completely different code sample that doesn't match with what it previously provided. Not just in code, this happens in many other areas too, for example writing an essay. It cannot pick up exactly from the last token. Jul 28, 2023 at 15:13
  • Please take a look to genai.meta.stackexchange.com/a/112/12
    – Wicket
    Jul 28, 2023 at 16:18

2 Answers 2

5

There are a few main reasons why large language models have output limitations:

  • Computing resources - Generating text is computationally expensive, especially for very large models with billions of parameters. More text generation requires more computation, which costs money. Limits help manage this cost.
  • Context Window - Maintaining context and coherence becomes more difficult as more text is generated. Limits help prevent the output from degrading or becoming repetitive/contradictory.
  • Safety - Longer outputs increase the chances of generating harmful, biased, or nonsensical text. Limits help mitigate the risk.

To expand more on "Context Window":

Context Window in large language models has to do with the information captured in the model's internal representations, especially in the output layer.

In particular, as text generation progresses, information about the existing context must be retained in the model's neural network activations. The output layer contains the model's representation of the currently generated text. As more text is produced, earlier context begins to fade from the output layer activations as new text overwrites it.

This is because there is a limit to how much context information can be captured and propagated through the deep neural networks that make up these large models. The connections have a certain fixed capacity.

3
  • Interesting, yeah this is exactly what I'm curious about. I've noticed this a lot with the OpenAI API's (especially the gpt-3.5-turbo model). The longer the chat goes, the further the AI deviates from the initial "system" prompt. Jul 28, 2023 at 20:26
  • 1
    What is "harmful text"?
    – user253751
    Jul 28, 2023 at 22:02
  • 1
    @user253751, ...text that the fine-tuning dataset was curated to avoid generating. Going beyond that would require finding the definition used by whichever entity curated that dataset, and also would arguably be shifting from a technical discussion into a political one. Jul 28, 2023 at 22:08
3

While this doesn't show to the User in a chat client, every API response includes a finish_reason.

The possible values for finish_reason are:

  • stop: API returned complete model output.
  • length: Incomplete model output due to max_tokens parameter or token limit.
  • content_filter: Omitted content due to a flag from our content filters.
  • null:API response still in progress or incomplete.

(source)

When the reason is 'length' (the answer is clearly un-finished), the user can prompt 'continue' and get the rest of the response. Sometimes this is the only way to get the complete response.

When the response has stopped, prompting 'continue' works like a new prompt but has very little information for the AI to work from, hence it might say almost anything.

2
  • I'm experienced with the OpenAI API. Though the crux of the question was why, what are the limitations behind the current architecture making it harder to generate longer and longer text. For example is the complexity like O(n^2) and why? Jul 28, 2023 at 15:08
  • 1
    @CoderGautamYT, complexity is one reason: the standard method of evaluating a GPT model is O(n^2) with respect to context length.
    – Mark
    Jul 29, 2023 at 2:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.