LLM Context Windows: Why AI Forgets Your Instructions

6 min read · Last updated: October 25, 2026

An LLM context window is all the text a model can reference when it answers, including its own reply. Your instructions can sit inside it and still get ignored, because according to Anthropic's documentation, accuracy and recall degrade as the token count grows. That effect is called context rot.

Came here from Instagram, LinkedIn, X or YouTube? Here is the full source list you came for. Everything below is built from these, nothing else.

The post that started this

• LLM Tokens and Context Windows: Why AI Forgets Your Conversation, Medium (I could only read the RSS teaser, not the full text, so nothing below is taken from its body)

The docs

• Anthropic: Context windows, opened October 25, 2026

The research

• Liu et al., Lost in the Middle: How Language Models Use Long Contexts, arXiv 2307.03172 (v1 July 6, 2023, v3 November 20, 2023)

Key takeaways

• A context window is the model's working memory, not its training data, and it includes the reply being written.
• Every turn resends all previous turns, so a long chat grows with each message.
• Most current Claude models have a 1M-token window. A bigger window does not mean better recall.
• Anthropic's docs say accuracy and recall degrade as token count grows. They call it context rot.
• A 2023 study found models do worse when the key text sits in the middle of a long input.
• Fix it with a short, restated instruction block, fresh chats, and compaction, not a bigger window.

What happened: the assistant ignored instructions that were still in the chat

A recent Medium post, "LLM Tokens and Context Windows: Why AI Forgets Your Conversation", starts from a familiar bug. Your instructions are still visible in the chat. The assistant ignored them anyway. The teaser says it follows one debugging session from the full conversation onward.

I could not open the full post, so I am not going to guess at its details. The question is a good one, and the primary sources answer it. Anthropic's documentation and a peer-reviewed paper explain why a model can "have" your instruction and still fail to follow it.

What is a context window and what is a token?

According to Anthropic's documentation, the context window refers to all the text a model can reference when generating a response, including the response itself. It is a working memory, separate from the large corpus the model was trained on.

A token is the unit the window is counted in. Models read text as tokens, not as words or characters, so a window of 1M tokens is a budget measured in those units. Everything in the request spends that budget.

What is in the request	Counts toward the window?
System prompt	Yes
Every earlier user and assistant message	Yes
Tool results, images and documents	Yes
Tool definitions	Yes
The reply being generated, including extended thinking	Yes
Cached prompt prefixes	Yes, caching changes the price, not the count

That table comes straight from Anthropic's context window page. It matters because tool output and pasted logs spend the same budget as your own words.

How big are context windows now?

As of October 25, 2026, Anthropic's docs list a 1M-token window for models including Claude Opus 5.5, Sonnet 5.5, Haiku 5.5 and Fable 5.1. A single request can generate up to 128k output tokens. Other Claude models, such as the deprecated Sonnet 4.5, have a 200k-token window.

For every 1M-token model, 1M is the default, with no beta header. Long-context requests are billed at standard pricing, except on Haiku 5.5, where prompts over 100,000 tokens cost more. Other vendors publish their own sizes, so check each model card.

The point for developers: size is no longer the bottleneck for most chats. Attention to what is inside is.

Why does the model ignore instructions that are still in the chat?

Because being in the window is not the same as being used. Anthropic's docs say that as token count grows, accuracy and recall degrade, a phenomenon known as context rot. They add that more context is not automatically better, so curating what is in context matters as much as how much space you have.

There is research behind the intuition. In "Lost in the Middle" (Nelson F. Liu and colleagues, first posted July 6, 2023), the authors tested multi-document question answering and key-value retrieval. Performance dropped when the relevant information sat in the middle of a long input rather than near the start or end.

Their conclusion, quoted from the paper: "current language models do not robustly make use of information in long input contexts". The study covers models from 2023, so treat the exact U-shape as a pattern to watch for, not a law for every current model.

Here is what that looks like in a real debugging session:

1. Message 1: you state the rules. "Use TypeScript, no new dependencies, keep changes minimal."
2. Messages 2 to 30: pasted stack traces, file dumps, failed attempts.
3. Message 31: the assistant adds a dependency.

Your rule is now buried under tens of thousands of tokens of logs. It is still technically in the window. It is also exactly the kind of text that ends up in the middle.

A second cause is simpler. Some chat apps trim history. Anthropic notes that chat interfaces such as claude.ai can manage the window on a rolling first-in, first-out basis, which means the oldest messages, usually your instructions, go first. In the raw API, previous turns are preserved completely.

Why it matters to a developer

Three costs hide in this behavior.

Cost 1: correctness. The model stops following constraints you set early. You get code that looks fine and breaks your own rules.

Cost 2: money. Per the docs, each turn's input holds all previous history plus the new message. Long chats resend the same tokens every time. If you use prompt caching, the docs say input is split across input_tokens, cache_read_input_tokens and cache_creation_input_tokens, and all three count toward the window. For the price side, see our Claude cache reads math.

Cost 3: hard failures. If input alone exceeds the window, the API returns a 400 invalid_request_error ("prompt is too long"). On Claude 4.5 models and newer, if input plus max_tokens exceeds the window, generation can stop with stop_reason: "model_context_window_exceeded". Your code should handle both.

What changes in practice?

The old advice was to wait for a bigger window. The current advice is to manage what is in it.

Old habit	Better habit
One endless chat per project	One chat per task
Instructions only in message 1	Instructions in the system prompt, restated before big tasks
Paste whole files and logs	Paste the failing function and the error line
Hope the window is big enough	Count tokens before sending
Trim history by hand	Use compaction or context editing

The docs describe tooling for this. Server-side compaction automatically summarizes earlier parts of the conversation so it can continue past the limit. It is in beta for Claude 4.6 and later models. Context editing adds tool result clearing, to drop old tool output in agent workflows, and thinking block clearing. To estimate a request before sending it, Anthropic points to its token counting API.

Some models also track their remaining budget. According to the docs, Claude Sonnet 5, Sonnet 4.6 and Haiku 4.5 have context awareness, where the API injects the remaining token count. Opus 4.7 and later, and Sonnet 5.5, do not get those tags. For them, the docs point to task budgets, which are in beta.

What to do this week

1. Move rules out of the chat and into the system prompt. Short, numbered, no more than ten lines.
2. Restate the rules before a big task. Paste them again as the last thing in your message. The end of the input is a stronger position than the middle, per the Lost in the Middle findings.
3. Start a new chat when the task changes. Carry over a short summary, not the whole transcript.
4. Stop pasting entire logs. Trim to the error and a few lines around it.
5. If you call an API, count tokens first and handle the "prompt is too long" error and the context-exceeded stop reason.
6. Test your own setup. Put a rule in message 1, add 20 messages of logs, then see whether it still holds. That tells you more than any benchmark.

I build with these models daily, and the fix that works most often is the dullest one: a fresh chat with a tight brief. Do not assume a 1M window means 1M tokens of reliable memory. Anthropic's own docs say the opposite.

FAQ

What is the difference between tokens and a context window?

A token is the unit of text a model reads and writes. The context window is the maximum number of tokens the model can handle in one request, covering input and output. Your system prompt, history, tool results and the reply all spend from that same budget.

Does a bigger context window fix forgetting?

Not by itself. Anthropic's documentation says more context is not automatically better, and that accuracy and recall degrade as token count grows. A larger window gives you room, but you still need to keep the important instructions short, recent and visible.

What is context rot?

Context rot is Anthropic's term for the drop in accuracy and recall as the number of tokens in the context grows. A model can have your instruction in its window and still follow it less reliably after tens of thousands of tokens of other material.

Why does the AI forget in ChatGPT or Claude chat apps but not in the API?

The docs say API requests preserve previous turns completely, while chat interfaces such as claude.ai can manage the window on a first-in, first-out basis. That means the oldest content, often your original rules, can be dropped first. Other apps may behave differently, so check their docs.

Want to test prompts and payloads quickly? Try the free tools on Art of Code. For another look at where AI falls short, read AI is not ready: why we're being sold an unfinished product, or browse AI jobs.

Written by Adithya Guttha, Founder of Art of Code.