Module 9: Building a RAG Application

"Now we build the pipeline for real: chop the documents, turn meaning into numbers, search by similarity, and answer from what we find."

Level: Beginner–Intermediate · Time: ~4–5 days · Prerequisites: Module 8

Learning Objectives

By the end of this module, you will be able to:

Chunk documents and understand why overlap matters
Explain what an embedding model does
Run a semantic search over a vector store
Assemble retrieved context into a grounded prompt

Documents are too big to embed whole, so we split them into chunks. Spring AI's TokenTextSplitter cuts text into token-sized pieces — with a little overlap so ideas that straddle a boundary aren't lost.

Chunk size is a balancing act. Too large, and a single chunk mixes several topics, so a search match drags in irrelevant text; too small, and a chunk loses the surrounding context that made it meaningful. A few hundred tokens per chunk is a sensible default for most prose.

java

[object Object], ,[object Object], ,[object Object], ,[object Object], ,[object Object],(content, Map.of(,[object Object],, filename));
,[object Object], ,[object Object], ,[object Object], TokenTextSplitter.builder().build();
List<Document> chunks = splitter.split(document);

Chunks with a small overlap preserve context at the edges

Concept: Overlap (e.g., 30 tokens shared between neighbors) is like starting each new page with the last sentence of the previous one — no thought gets cut in half.

2. Creating Embeddings

Each chunk is converted into an embedding: a list of numbers that captures meaning. An embedding model isn't a chatbot — it can't reason. It's a brilliant filing system that places similar meanings near each other, so "car" lands next to "automobile."

Each embedding is just a long list of numbers — often hundreds or thousands of them — and each number captures some subtle aspect of meaning the model learned during training. You never read these numbers yourself; the vector store uses them to measure how close two pieces of text are.

An embedding model turns text into vectors; similar meanings sit close together

In Spring AI, the VectorStore handles embedding for you when you add documents:

java

[object Object],
VectorStore ,[object Object],[object Object], {
    ,[object Object], SimpleVectorStore.builder(embeddingModel).build();
}

,[object Object],
vectorStore.add(chunks);

Concept: Embedding happens once at ingestion. After that, searching is fast because you're comparing numbers, not re-reading documents.

3. Semantic Search

When a user asks a question, the VectorStore embeds the question too, then finds the chunks whose vectors are most similar — by meaning, not keywords.

java

[object Object], ,[object Object], ,[object Object], SearchRequest.builder()
        .query(question)
        .topK(,[object Object],)                 ,[object Object],
        .similarityThreshold(,[object Object],)
        .build();

List<Document> matches = vectorStore.similaritySearch(request);

Semantic search finds meaning, not just matching words

Similarity is measured with cosine similarity — "do these two arrows point the same way?" Aligned vectors score near 1.0, even if the words differ.

Cosine similarity: aligned meaning vectors score near 1.0

Explain like I'm new: A keyword search for "vehicle" misses a paragraph about "cars and trucks." Semantic search knows they mean the same thing and returns it anyway.

Try this: Search your store for a concept using words that never appear in the source text — e.g., ask about "time off" when the handbook only says "paid leave" and "vacation." Watching semantic search return the right chunk anyway is the moment embeddings click.

4. Generating the Answer

Finally, glue the top chunks into a prompt with clear instructions, and let the model answer only from that context:

java

[object Object], ,[object Object], ,[object Object], matches.stream()
        .map(Document::getText)
        .collect(Collectors.joining(,[object Object],));

,[object Object], ,[object Object], ,[object Object], ,[object Object],.formatted(context, question);

,[object Object], ,[object Object], ,[object Object], chatClient.prompt(prompt).call().content();

Two details make this prompt trustworthy: the instruction to use only the context (so the model can't wander back to its training data) and the permission to say "I don't know" (so it admits gaps instead of inventing answers). Together they are your main defense against hallucination.

Common mistake: Setting topK too high or the similarityThreshold too low. You then stuff loosely related chunks into the prompt, confusing the model and wasting tokens. Start with a few high-quality matches and tune from there.

✅ Checkpoint

Why do we add overlap between chunks?
What does an embedding model produce, and what does it not do?
What do topK and similarityThreshold control?

Answers: 1) So ideas spanning a chunk boundary aren't lost. 2) It produces embeddings (meaning-as-numbers); it doesn't reason or answer questions. 3) topK = how many chunks to return; similarityThreshold = the minimum relevance score to include.

Key Takeaway: A RAG app is built in four concrete steps: chunk documents (with overlap), embed each chunk into a vector via an embedding model, search the vector store by cosine similarity to get the most relevant chunks, and generate an answer from those chunks. Spring AI's TokenTextSplitter, VectorStore, and similaritySearch make each step a few lines of code.

Further Learning

Part of "Spring AI for Beginners." Adapted from Microsoft's open Spring AI curriculum (MIT License).

Learning Objectives

1. Chunking Documents

2. Creating Embeddings

3. Semantic Search

4. Generating the Answer

✅ Checkpoint

Further Learning