​Building a Context Engineering Workflow​

hands-on

​Building a Context Engineering Workflow​

A few days back, we discussed covering a demo on context engineering.

So today, we'll build a multi-agent research assistant using context engineering principles.

This Agent will gather its context across 4 sources: Documents, Memory, Web search, and Arxiv.

Here’s our workflow:

• User submits query.
• Fetch context from docs, web, arxiv API, and memory.
• Pass the aggregated context to an agent for filtering.
• Pass the filtered context to another agent to generate a response.
• Save the final response to memory.

Tech stack:

• ​Tensorlake​ to get RAG-ready data from complex docs
• ​Zep​ for memory
• ​Firecrawl​ for web search
• ​Milvus​ for vector DB
• CrewAI for orchestration

The code is linked later in the issue.

Let's go!

First, what is context engineering (CE)?

LLMs aren’t mind readers. They can only work with what you give them.

Prompt engineering primarily focuses on “magic words” with an expectation of getting a better response.

CE involves creating dynamic systems that offer:

• The right info
• The right tools
• In the right format

This ensures the LLM can effectively complete the task.

Crew flow

We'll follow a top-down approach to understand the code.

Here's an outline of what our flow looks like:

Note that this is one of many blueprints to implement a context engineering workflow. Your pipeline will likely vary based on the use case.

Prepare data for RAG

We use Tensorlake to convert the document into RAG-ready markdown chunks for each section.

The extracted data can be directly embedded and stored in a vector DB without further processing.

Indexing and retrieval

Now that we have RAG-ready chunks along with the metadata, it's time to store them in a self-hosted Milvus vector database.

We retrieve the top-k most similar chunks to our query:

Build memory layer

Zep acts as the core memory layer of our workflow. It creates temporal knowledge graphs to organize and retrieve context for each interaction.

We use it to store and retrieve context from chat history and user data.

Firecrawl web search

We use Firecrawl web search to fetch the latest news and developments related to the user query.

Firecrawl's v2 endpoint provides 10x faster scraping, semantic crawling, and image search, turning any website into LLM-ready data.

ArXiv API search

To further support research queries, we use the arXiv API to retrieve relevant results from their data repository based on the user query.

Filter context

Now, we pass our combined context to the context evaluation agent that filters out irrelevant context.

This filtered context is then passed to the synthesizer agent that generates the final response.

Kick off the workflow

Finally, we kick off our context engineering workflow with a query.

Based on the query, we notice that the RAG tool, powered by Tensorlake, was the most relevant source for the LLM to generate a response.

We also translated this workflow into a streamlit app that:

• Provides citations with links and metadata.
• Provides insights into relevant sources.

That said, the workflow explained in the thread is one of the many blueprints. Your implementation can vary.

In the project, we used:

• Tensorlake:
  • It lets you transform any unstructured doc into AI-ready data.
  • ​GitHub Repo →​
• Zep:
  • It lets you build human-like memory for your Agents.
  • ​GitHub Repo →​
• Firecrawl:
  • It lets you power LLM apps with clean data from the web.
  • ​GitHub Repo →​
• Milvus:
  • It gives a high-performance vector DB for scalable vector search.
  • ​GitHub Repo →​

​You can find the code here →​

Thanks for reading!

IN CASE YOU MISSED IT

​Function calling & MCP for LLMs

Before MCPs became mainstream (or popular like they are right now), most AI workflows relied on traditional Function Calling.

Now, MCP (Model Context Protocol) is introducing a shift in how developers structure tool access and orchestration for Agents.

Here’s a visual that explains Function calling & MCP:

​Learn more about it with visual and code in our recent issue here →

PRIVACY-PRESERVING ML

​Train models on private data with federated learning

There’s so much data on your mobile phone right now — images, text messages, etc.

And this is just about one user—you.

But applications can have millions of users. The amount of data we can train ML models on is unfathomable.

The problem?

This data is private.

So consolidating this data into a single place to train a model.

The solution?

​Federated learning​ is a smart way to address this challenge.

The core idea is to ship models to devices, train the model on the device, and retrieve the updates:

But this isn't as simple as it sounds.

1) Since the model is trained on the client side, how to reduce its size?

2) How do we aggregate different models received from the client side?

3) [IMPORTANT] Privacy-sensitive datasets are always biased with personal likings and beliefs. For instance, in an image-related task:

• Some devices may only have pet images.
• Some devices may only have car images.
• Some people may love to travel, and may primarily have travel-related images.
• How to handle such skewness in data distribution?

​Learn how to implement federated learning systems (beginner-friendly) →