Agent Memory and RAG: Giving AI Agents Context and Knowledge

Here is something that surprises many beginners: a language model has no memory at all between calls. Every time your code calls it, it sees only what you send in that request. Everything that feels like memory — remembering your name, the earlier steps, your college's rules — is something the application puts into the context on purpose.

Three kinds of memory an agent needs

Working memory: managing the context window

As an agent loops, the conversation grows: tool calls, results, more calls. Long runs hit the context limit or simply get slow and expensive. Common techniques:

• Trim tool results to the fields that matter before adding them.
• Summarise old steps — replace ten finished steps with a short "so far we have…" note.
• Keep a scratchpad — a small, structured notes object (plan, findings, open questions) the agent updates, instead of rereading everything.

RAG: answering from your own documents

Suppose you want an assistant that answers questions about your college's academic rules — attendance, re-exams, grace marks. The model was never trained on that PDF, and even if it had been, rules change. Retrieval-Augmented Generation (RAG) solves this: before the model answers, you retrieve the most relevant passages and put them in the prompt.

Step 1 — Prepare the documents (once)

1. Split each document into chunks of a few hundred words, with a little overlap so sentences are not cut in half.
2. Embed each chunk: an embedding model turns text into a list of numbers (a vector) that captures its meaning. Texts with similar meaning get similar vectors.
3. Store the vectors with their text in a vector database or search index.

Step 2 — Answer a question (every time)

1. Embed the user's question.
2. Find the chunks whose vectors are closest to it (semantic search).
3. Send the question plus those chunks to the model with an instruction like: "Answer only from the context below. If the answer is not there, say you don't know."

# The core of RAG, without any framework
question = "How much attendance do I need to sit for the end-semester exam?"
chunks   = search_index(embed(question), top_k=4)          # most similar passages

context = "\n\n".join(f"[{c['source']}, p.{c['page']}]\n{c['text']}" for c in chunks)
prompt  = f"""Answer the student's question using ONLY the context.
Quote the rule and give the source in brackets. If the context does not answer it,
reply: "I couldn't find this in the academic rules."

<context>
{context}
</context>

Question: {question}"""
answer = call_model(prompt)

Agentic RAG: let the agent decide when to search

Basic RAG searches once for every question. In an agent, retrieval becomes just another tool — search_documents(query). The agent can search, notice the result is about the wrong semester, search again with a better query, and combine two documents. This handles harder questions, at the cost of more calls.

Why RAG answers go wrong (and fixes)

• Retrieval missed the right chunk. Most "the model is wrong" bugs are actually search bugs. Test retrieval separately: for 20 real questions, is the right passage in the top results?
• Chunks are too big or too small. Huge chunks bury the answer; tiny ones lose context. Split along headings when you can.
• Keyword-heavy questions. Course codes, roll numbers and exact names are better found by keyword search. Many systems combine keyword and semantic search (hybrid search).
• Old documents. If the 2023 rules and the 2026 rules are both in the index, the model may quote the wrong one. Store dates and filter on them.