- Chain-of-thought prompting guides AI models to reason step by step, helping them solve complex tasks more accurately than simply predicting the next word.
- It’s best used for multi-step problems, like math, logic puzzles, or procedural tasks, but unnecessary for simple factual or single-step queries.
- Prompt chaining differs from chain-of-thought prompting by splitting a task into multiple separate prompts, whereas chain-of-thought stays within one prompt-response flow.
If you’ve used a GPT chatbot like ChatGPT, you’ve probably noticed the varied quality of output.
Sometimes it spits out exactly what you need. Other times you suspect that the ‘intelligence’ in AI is a bit of a farce.
You can take your ChatGPT game up a notch by improving how you prompt it. Chain-of-thought prompting encourages LLM agents to reason through a task step by step before generating a response.
Newer AI models and features are starting to build in chain-of-thought reasoning directly, so that their models automatically reason through the problem without any extra prompting.
What is chain-of-thought prompting?
Chain-of-thought prompting is a prompt engineering technique in AI that instructs models to break down complex tasks, reasoning through each step before responding.
You might also hear the term ‘chain-of-thought reasoning’. This refers to the step by step process the model will take to reason through the task at hand.
The OpenAI o1 models don’t require chain-of-thought prompting, because they already have chain-of-thought reasoning built in. But you can use chain-of-thought prompting on any LLM-powered chatbot.
How does chain-of-thought reasoning work?
Chain-of-thought reasoning entails breaking down a problem into smaller, logical steps for the AI chatbot to solve in sequence.
First, the AI identifies the key parts of the problem. Then it processes each part in sequence, considering how one step leads to the next. Each step builds on the previous one, allowing the AI to methodically move toward a logical conclusion.
Examples of chain-of-thought prompting
The famous ‘strawberry’ prompt
ChatGPT and other LLMs have well-documented weaknesses. One is their inability to correctly identify how many 'R's are in the word ‘strawberry’. (Likely the famous limitation behind the o1 models' code name: Strawberry.)
ChatGPT-4o doesn’t use chain-of-thought reasoning. Instead, it references its training data and generates a response based on how likely it is for each word to follow the previous one. Although it may sound correct most of the time, it’s only generating to mimic human language – not reasoning or conducting research.
When you ask ChatGPT-4o the famous strawberry question, it’s unable to give the correct answer:
However, you can use a chain-of-thought prompting technique to help the LLM-powered chatbot arrive at the correct answer:
The latest iteration of ChatGPT, powered by OpenAI o1-preview, is the first major LLM to use chain-of-thought reasoning without any additional prompting.
It cracks the answer on the first try, because it’s been instructed to automatically follow the same process as the second ChatGPT-4o prompt above. The only difference is that it does this process without additional prompting.
Math
If you asked an older version of ChatGPT a math question out of an elementary school textbook, it wouldn’t always get it right.
Multi-step math problems require reasoning, which wasn’t present in earlier LLMs. You could break down each step of the problem, but if you didn’t know the correct steps, an LLM couldn’t help.
ChatGPT-4o is able to reason out the answer to the question by breaking down the series of steps in the problem:
AI Agents connected to Hubspot
For a real world application, let’s take an LLM-powered AI agent that has been integrated into Hubspot. A sales team uses this AI agent to process new leads as they’re gathered across channels.
Scenario
A salesperson sends a new lead to the AI agent and asks it to register it in Hubspot and send a first touchpoint email, but to not fill it in if the lead works at a company that’s already a prospect.
LLM without chain-of-thought reasoning
The LLM-powered AI agent registers the lead and sends the email without checking if the company is already a prospect, missing the key condition.
LLM with chain-of-thought reasoning
The LLM-powered AI agent checks if the company is already a prospect before taking action. If it’s a prospect, it skips registration and emailing; if not, it registers the lead and sends the email, following the salesperson's instructions accurately.
When should I use chain of thought prompting?
Chain-of-thought prompting is best used in scenarios that require step-by-step reasoning.
Prime candidates are tasks that involve logical deductions, math problems, procedural tasks, or any situation requiring multi-step answers.
But wait: reasoning sounds great – why wouldn’t I use it all the time?
Good question. Not all questions require reasoning. For example:
- Simple factual questions, like ‘What is the capital of Canada?’
- Single step problems, like ‘What is 145 + 37?’
- Content generation tasks, like ‘Write a polite 3-sentence email that asks my colleague if they’re done with their project yet.’
Prompting changing vs chain-of-thought prompting
Though similar in name, prompt chaining and chain-of-thought prompting are different prompting strategies to improve generative AI output.
Chain-of-thought prompting
With chain-of-thought prompting, a user guides the AI to explain the reasoning behind its answer in a single response. This prompts the AI to walk through each step of the problem-solving process, but it’s accomplished in a single prompt and response.
For example, a chain-of-thought prompt can be accomplished in one message:
"An HR team needs to review 5 employee performance evaluations. Each will take 30 minutes and they need 15 minutes to prep beforehand. Senior evals will require an extra 10 minutes each. How long will it take to complete 5 senior and 25 junior evals? Break down your reasoning step by step."
Prompt chaining
With prompt chaining, the task is broken into separate steps with multiple prompts, each building on the previous result. This helps structure and guide the AI through a complex task that likely involves reasoning.
The first prompt might look like:
Prompt 1: Identify the main challenges a company might face when transitioning to remote work.
Output:
- Communication gaps
- Maintaining productivity
- Technology infrastructure
- Employee engagement
The next prompts might dive further into these concepts. For example:
Prompt 2: Please tell me how a company can find solutions to communication gaps when transitioning to remote work.
After the next round of output, the next link of the chain may be:
Prompt 3: What are the common challenges companies face when they adopt these solutions?
So while the two are similar, they take different approaches to extracting the most in-depth and relevant content from generative AI tools.
Chain-of-thought prompting on Botpress
Botpress users are already familiar with a feature that employs chain-of-thought reasoning.
The Autonomous Node debuted in July of 2024 on Botpress, a platform for building AI agents. The Autonomous Node is capable of automating multi step workflows and making decisions autonomously.
An Autonomous Node can be created and prompted with a simple line of text, like ‘Your purpose is to generate qualified leads. Create leads in Salesforce when a user indicates purchasing intent.’
The AI agent you build using this Autonomous Node will take a variety of actions to achieve its goal, independent of workflows designed by humans. It can also switch between different LLMs as needed, taking a decision to prioritize speed or power.
Build a custom autonomous agent
Botpress is the only AI agent platform that allows you to build truly autonomous agents.
The open and flexible Botpress Studio allows for endless use cases across industries, from HR to lead generation. Our pre-built integration library and extensive tutorials let users easily build AI agents from scratch.
Start building today. It’s free.
FAQs
Is chain-of-thought prompting only useful for AI models, or does it reflect human problem-solving as well?
It actually mirrors how humans solve complex problems by breaking them into steps so it’s both a useful technique for AI and a reflection of natural human reasoning.
How is chain-of-thought reasoning different from just “thinking step by step”?
They’re closely related, but chain-of-thought reasoning is more structured and intentional. It nudges the AI to explain its logic out loud, rather than just jumping to an answer.
Why do some LLMs not use chain-of-thought reasoning by default?
Earlier or smaller models weren’t trained to reason explicitly, so they tend to predict answers based on patterns rather than logical steps unless prompted otherwise.
Is chain-of-thought prompting a form of “training” the model during inference?
Not exactly. You're not changing the model's knowledge, just influencing how it thinks through a problem in the moment by guiding its behavior with better instructions.
Does chain-of-thought prompting always improve accuracy?
Not always. It works best on tasks that need reasoning, but it can actually overcomplicate things for simple questions and lead to unnecessary steps or even mistakes.
.png)
