On this article, you’ll learn the way an agent’s strategy to managing state — stateless or stateful — shapes each its implementation and the deployment structure constructed round it.
Matters we’ll cowl embrace:
- What separates stateless from stateful brokers, and the tradeoffs every design imposes on scaling.
- The best way to implement a stateless agent that relies upon totally on the consumer to produce dialog historical past.
- The best way to implement a stateful agent that manages its personal reminiscence by a database layer.
Introduction
A earlier article laid out a complete architectural roadmap for AI agent deployment, analyzing the infrastructure wanted to carry brokers into manufacturing settings.
As a follow-up, we now flip to a elementary, sensible query that needs to be answered earlier than any load balancer is configured: the place does the agent’s reminiscence reside? Brokers could deal with their state (the context gained to date and the dialog historical past) in numerous methods, and this code-level determination can considerably impression the complete deployment structure.
This text breaks down the 2 major paradigms for dealing with an agent’s state: stateless and stateful design. A simplified model of a real-world implementation, utilizing open language fashions served by the quick Groq API, will illustrate these concepts in follow.
Preliminary Setup
If that is the primary time you’re utilizing language fashions from Groq in a Python program, you’ll want to put in the required library: pip set up groq.
After that, we import it and set our Groq API key within the code beneath:
|
import os from groq import Groq
# Get an API key in https://console.groq.com/keys and set it right here os.environ[“GROQ_API_KEY”] = “PASTE_YOUR_GROQ_API_KEY_HERE”
# Initializing the consumer consumer = Groq()
# Utilizing an environment friendly mannequin from Groq: Llama 3.1 8B Immediate MODEL_ID = “llama-3.1-8b-instant” |
An vital setup determination right here is the selection of a selected mannequin. llama-3.1-8b-instant is a extremely cost-efficient mannequin that’s, on the time of writing, generously supported on Groq’s 2026 free tier: it permits as much as 14,400 requests per day. That makes it a great selection for illustrating the stateless and stateful agent paradigms beneath.
Stateless Brokers: Fireplace and Neglect
Stateless brokers deal with every request as utterly remoted and impartial. The agent reads the person immediate, invokes the LLM inference engine, and delivers the output. As soon as that execution cycle ends, every part is forgotten.
The Tradeoff
Architectures primarily based on stateless brokers might be scaled horizontally with outstanding ease. Since no person reminiscence is saved on a backend server, incoming requests might be forwarded to any accessible occasion. There’s, nonetheless, an vital limitation in multi-turn conversations: the frontend should re-send the entire dialog historical past alongside each new request. Consequently, the context window grows with a snowballing impact, rapidly driving up token utilization.
Illustrative Instance
This runnable code illustrates, by a primary situation, how a stateless agent sometimes interacts with a Groq language mannequin.
First, we outline a stateless_agent operate that emulates an agent’s interplay with our chosen mannequin. Importantly, no state or reminiscence of the dialog is stored internally. As a substitute, the earlier dialog historical past can optionally be handed in as a parameter and appended to the present immediate. The API name to the Groq mannequin takes place in consumer.chat.completions.create().
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 |
def stateless_agent(immediate: str, provided_history: checklist = None) -> str: “”“ The agent depends utterly on the consumer to offer context. It retains no info from previous interactions in native reminiscence. ““” # Initializing with a system immediate messages = [{“role”: “system”, “content”: “You are a helpful, concise assistant.”}]
# Appending no matter historical past the consumer offered if provided_history: messages.lengthen(provided_history)
# Appending the brand new immediate messages.append({“function”: “person”, “content material”: immediate})
# The LLM processes the complete chain of messages response = consumer.chat.completions.create( mannequin=MODEL_ID, messages=messages, max_tokens=100 )
return response.decisions[0].message.content material.strip() |
To know the constraints of a stateless agent, we simulate a easy user-model dialog by it:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 |
# — Testing the Stateless Agent —
print(“— Flip 1 —“) prompt_1 = “Hello, my title is Alice and I’m studying about API infrastructure.” response_1 = stateless_agent(prompt_1) print(f“Agent: {response_1}”)
print(“n— Flip 2 (With out Consumer Context) —“) # The agent fails right here as a result of it retained no reminiscence of Flip 1 prompt_2 = “What’s my title and what am I studying about?” response_2 = stateless_agent(prompt_2) print(f“Agent: {response_2}”)
print(“n— Flip 2 (With Consumer Context) —“) # The frontend MUST inject the historical past into the payload for the agent to succeed frontend_payload = [ {“role”: “user”, “content”: prompt_1}, {“role”: “assistant”, “content”: response_1} ] response_3 = stateless_agent(prompt_2, provided_history=frontend_payload) print(f“Agent: {response_3}”) |
Output:
|
—– Flip 1 —– Agent: Hiya Alice, good to meet you. Studying about API infrastructure can be a fascinating and rewarding matter. What particular features of API infrastructure would you like to discover or talk about? Are you trying for info on API administration, safety, deployment, or one thing else?
—– Flip 2 (With out Consumer Context) —– Agent: Sadly, I don‘t have any details about you, together with your title. Our dialog simply began, so I’m right here to assist you with any questions or subjects you‘d prefer to study. Please be happy to share your title and a subject you’re in studying about.
—– Flip 2 (With Consumer Context) —– Agent: Your title is Alice, and you are studying about API infrastructure. |
The implementation is easy, however and not using a consumer or frontend that sends the total dialog historical past to the agent on each flip, the agent’s LLM lacks the context it must reply sure questions correctly.
Stateful Brokers: Context-driven Continuity
Below this strategy, the agent takes on the reminiscence burden itself. The consumer, in the meantime, solely must ship the most recent person immediate along with a singular identifier, usually related to the present session. The agent then retrieves the session historical past or context from a database and appends the brand new message to it. As soon as the LLM inference has been processed, the agent updates the context within the database.
The Tradeoff
This can be a a lot neater expertise from the consumer facet. It additionally facilitates complicated and asynchronous workflows wherein brokers could must pause their execution and look ahead to instruments, app responses, or human approval. But it surely all comes with a price: scaling this answer turns into a lot tougher, beginning with the necessity for a persistent database layer within the structure. In infrastructures that scale horizontally, methods corresponding to centralized reminiscence caching with Redis might also develop into essential to keep away from “localized amnesia”, the place a session’s historical past is stranded on the one occasion that occurred to serve the sooner turns.
Illustrative Instance
We illustrate the fundamental concepts behind a stateful agent by incorporating a “persistent” database layer. For simplicity, we use a tiny SQLite database. The secret is to have the agent handle its personal dialog reminiscence as a substitute of relying on a frontend to offer it externally:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 |
import sqlite3 import json
# Initializing an in-memory SQLite database for pocket book testing conn = sqlite3.join(‘:reminiscence:’) cursor = conn.cursor() cursor.execute(”‘CREATE TABLE IF NOT EXISTS agent_memory (session_id TEXT PRIMARY KEY, historical past TEXT)’”) conn.commit()
def stateful_agent(session_id: str, new_prompt: str) -> str: “”“ The agent manages its personal state utilizing a database. The consumer solely sends the brand new immediate and their session ID. ““” # 1. Retrieving current state from the database cursor.execute(“SELECT historical past FROM agent_memory WHERE session_id=?”, (session_id,)) row = cursor.fetchone()
if row: conversation_history = json.masses(row[0]) else: # Initializing with system immediate for brand spanking new periods conversation_history = [{“role”: “system”, “content”: “You are a helpful, concise assistant.”}]
# 2. Appending the brand new person immediate conversation_history.append({“function”: “person”, “content material”: new_prompt})
# 3. Processing the LLM name utilizing the retrieved historical past response = consumer.chat.completions.create( mannequin=MODEL_ID, messages=conversation_history, max_tokens=100 ).decisions[0].message.content material.strip()
# 4. Updating the state with the assistant’s reply conversation_history.append({“function”: “assistant”, “content material”: response})
# 5. Saving the brand new state again to the database cursor.execute(”‘ INSERT INTO agent_memory (session_id, historical past) VALUES (?, ?) ON CONFLICT(session_id) DO UPDATE SET historical past=excluded.historical past ‘”, (session_id, json.dumps(conversation_history))) conn.commit()
return response |
Discover how the session identifier is used to question the related info from previous interactions within the dialog at hand.
Now let’s attempt all of it in a dialog just like the earlier one, however this time with the person asking the agent to recall the person’s personal title:
|
# — Testing the Stateful Agent —
print(“— Flip 1 —“) print(f“Agent: {stateful_agent(‘user_123’, ‘Hello, I’m Bob and I wish to scale my AI app.’)}”)
print(“n— Flip 2 —“) # Discover how the consumer NO LONGER sends the context payload. Simply the session ID. print(f“Agent: {stateful_agent(‘user_123’, ‘What was my title once more?’)}”) |
Output:
|
—– Flip 1 —– Agent: Hiya Bob, scaling an AI app can be a complicated course of. Could I ask:
1. What kind of AI expertise is your app constructed on? (e.g., machine studying, pure language processing, laptop imaginative and prescient) 2. Are you utilizing any cloud providers like AWS, Google Cloud, or Azure? 3. What are your scalability targets (e.g., enhance person rely, scale back latency, enhance response occasions)?
This info will assist me higher perceive your necessities and present extra efficient help.
—– Flip 2 —– Agent: Your title is Bob. |
This instance is, after all, a great distance from a scaled-up manufacturing structure, however it serves to make clear the important thing distinction between how stateful and stateless brokers work.
Wrapping Up: The Tradeoffs
The selection between a stateful and a stateless architectural design boils all the way down to correctly matching the infrastructure to the workflow:
- Stateless brokers are most well-liked in easy pipelines oriented to very particular duties, like textual content extraction, summarization, or single-turn classification chatbots. They hold the structure light-weight, which is normally sufficient in such use circumstances, avoiding database bottlenecks and permitting seamless horizontal scaling.
- Stateful brokers make way more sense if we intend to develop long-running assistants, coding assistants, or multi-turn bots in purposes like customer support. As a result of the agent owns the historical past, the consumer payload stays small on each flip, and the dialog might be trimmed or summarized server-side as a substitute of being resent in full because it grows.
