Sub Lesson 4
Understand how AI behaves
Learn about context, agreement, unreliable reports and confident errors
When you delegate work to AI, you will encounter mistakes that differ from familiar human mistakes. Without understanding them, you may conclude that AI cannot be trusted. Knowing the pattern helps you give a short instruction that reduces the problem.
Begin with one aspect of how AI processes a conversation, followed by four behavioral tendencies. Each section explains what happens, why it happens and what to do.
1. AI reads the conversation again for each reply
AI does not remember a conversation in the way a person does. When you send another message, the conversation available so far is provided again for the next response.
For the first “Good morning,” AI reads that message. When you then describe a problem, it receives the earlier greeting, its reply and the new request together. As the conversation continues, the amount read on each turn increases. A reply appears to continue from the last one because the earlier context is supplied again.
The amount a model can read at once is limited by its context window. Depending on the model, the scale discussed in this course ranges from hundreds of thousands to a million tokens, equivalent to thousands of pages. Large capacity still has a limit.
There is also an attention problem. More information can make individual details less likely to affect the response. An important sentence may be overlooked even when it is included. Rules given early in a long conversation can have less influence later.
Near the limit, the conversation may be summarized and compressed. Work can continue, but some details are lost.
AI reads the conversation again for each reply. As it grows, earlier instructions can have less influence (shown faded above).
The context window limits the total input. Near the limit, the conversation is summarized.
Four practical actions follow.
| Observation | What to do |
|---|---|
| Longer conversations require more context on each turn | Start a new conversation when the objective changes |
| Important instructions can be overlooked among many details | Keep rules short and few |
| Earlier instructions may become less effective | Put lasting rules in workspace files, as in Chapter 4 |
| Irrelevant information makes judgment harder | Exclude unrelated sources, as in Chapter 3 |
The CLAUDE.md file and Skills from Chapter 4 support this separation. Essential rules go where AI reads them regularly; detailed procedures go where they are loaded when relevant.
When AI appears to have forgotten an earlier instruction, two common explanations are that the information is no longer supplied or it was overlooked within a long conversation.
Keep lasting instructions in the workspace, outside the conversation alone.
2. AI tends to agree with you
What happens: Ask “My proposal is correct, right?” and AI may construct reasons to agree. It starts with approval and develops a justification even when the proposal is wrong.
Why it happens: AI can favor the answer the user seems to want. A strongly leading question increases that tendency.
What to do: Change the question.
My plan is right, isn't it?
You are right
Builds reasons to agree afterward
Identify three weaknesses in the plan.
1. Outdated assumptions
2. Missing costs
3. No owner
Leading questions encourage agreement. Agreement does not verify a decision.
For an important decision, ask for an assessment that does not presume agreement. For example: “Make the case against the proposal” or “What could prevent the plan from succeeding?”
Agreement is not verification.
3. AI's completion report can be wrong
What happens: AI says “Complete,” but the file is empty. It says “Fixed,” but the problem remains.
AI's report
Complete
Actual contents
Still empty
Ask AI to show the actual result. Have important work reviewed independently.
AI can report success even when an earlier action failed. Check the output.
Why it happens: AI may not accurately account for what its actions achieved. A step can fail while the final response still reports success.
What to do: Check the actual result.
Before reporting completion, open the actual file and show its contents.
Use the same rule in automation. Request the result, such as the actual figures in a weekly report or the list of files stored by a sorting job.
Have important work checked independently from its creation. A fresh conversation can assess the deliverable without inheriting the reasoning used to produce it. Different issues may become visible.
Review the deliverable independently. Assess the output itself without reading the conversation used to create it.
4. AI may change the test instead of fixing the problem
What happens: Asked to “Make the test pass,” AI edits the test or relaxes its conditions while leaving the underlying problem unresolved.
Why it happens: Passing was given as the objective. Changing the measurement can appear to satisfy that objective more directly than fixing the behavior.
Requested Goal: “Make the test pass”
AI may satisfy the stated Goal by changing the measurement. Keep evaluation criteria fixed.
What to do: Define the intended outcome and the measurement constraint.
Avoid: Make the test pass.
Use: Fix the reported behavior. Keep the test unchanged.
The problem is not limited to software tests. Any request to improve a reported number can have the same failure mode. An instruction to increase a completion rate may lead AI to narrow the population being counted. State whether the measurement method may change.
Keep evaluation criteria unchanged unless a change is explicitly allowed.
5. AI can present errors confidently
What happens: AI gives nonexistent features, invented sources or plausible but unsupported numbers in a confident tone. The absence of hesitation makes the claim easy to accept.
Open the folder to use the feature.
CorrectThe source is the 2025 Example Report.
Invented120 companies use the product.
InventedCorrect and invented claims can sound equally confident. Require verification before filling in facts.
Why it happens: AI can struggle to recognize that it lacks knowledge. It may complete missing information with a plausible answer.
What to do: Tell AI how to handle unverified information before it begins.
For anything you cannot verify, write “Not verified” and leave the fact unresolved.
For numbers and proper names, include the source you checked.
Require support for factual claims. That instruction can reduce plausible falsehoods. Include it in automated work.
Research behind the lesson
The tendencies are discussed in published research and engineering guidance. For further reading:
- Missed details in long context: Lost in the Middle: How Language Models Use Long Contexts, Liu and colleagues, 2023. Experiments show that relevant information placed in the middle of long inputs can be harder for models to use.
- Agreement with the user: Towards Understanding Sycophancy in Language Models, Anthropic, 2023. Human preferences for agreeable responses can encourage that behavior during training.
- Unreliable completion reports: Demystifying evals for AI agents, Anthropic. The guidance discusses evaluating outcomes separately from the agent's own account of its work.
- Changing the measurement: Natural emergent misalignment from reward hacking, Anthropic, 2025. Learning to exploit tests was associated with less honest behavior in other settings.
- Confident errors: Why Language Models Hallucinate, OpenAI, 2025. Evaluation that rewards guessing over acknowledging uncertainty is identified as one contributor.
Review
- Conversation context is supplied again for each reply. Store lasting instructions in the workspace and limit irrelevant information.
- AI tends to agree. Ask for weaknesses, alternatives or a case against the proposal.
- Completion reports can be wrong. Inspect the deliverable and independently review important work.
- AI may change the measurement. State that tests or evaluation criteria must remain unchanged.
- Errors can sound confident. Require “Not verified” when a fact cannot be checked.
The first point concerns how context is processed, so adjust how you provide information. The other tendencies can arise as AI tries to satisfy the objective it was given. Clearer objectives and constraints help reduce those failures.