How OpenAI GPT and Embedding Models Power Grounded RAG

TL;DR

This article is based on the real implementation experience of building AI Engineering Circle - Knowledge Bot, a RAG-based LLM chatbot for asking grounded questions from approved knowledge sources.

A reliable Retrieval-Augmented Generation (RAG) application uses different models for different jobs:

Embeddings find relevant evidence.
The chat model turns that evidence into a readable answer.
Application code controls access, citations, and validation.

For the reference implementation in this article, text-embedding-3-small creates 1,536-dimension vectors for document chunks, Supabase stores and searches those vectors, and gpt-4o-mini produces the final grounded answer. Those are configuration choices, not requirements: model availability and fit should be checked against the current OpenAI model documentation.

The important design decision is not the brand of model. It is keeping retrieval, answer generation, access control, and validation as separate responsibilities.

Why an LLM needs retrieval for private knowledge

An LLM can explain, summarize, and write fluent text. It cannot automatically know a team's private meeting notes, the latest internal policy, or a document uploaded after the model was trained. It can also produce a plausible statement without supporting evidence.

RAG addresses that gap:

• Retrieval finds relevant information in approved source documents.
• Augmentation supplies the selected evidence to the answer model.
• Generation turns that evidence into an answer with source references.

Instead of asking, “What happened in Week 11?” and hoping the model knows, the application retrieves the permitted Week 11 notes first. The model then answers only from those notes.

This is the practical distinction between a chat interface and an evidence system. For the broader decision of when an app should use RAG rather than pass an entire document to a model, see LLM-only vs RAG GenAI apps.

The terms that make the architecture understandable

A RAG design becomes easier to reason about when its data units are clear.

• A document is an original source: a policy, transcript, support guide, or meeting note.
• A chunk is a small, meaningful section of that document, such as a decision, a takeaway, or a troubleshooting step.
• An embedding is a numerical representation of a chunk's meaning. Similar ideas tend to be close in vector space even when they use different words.
• Metadata is structured information attached to a chunk, such as tenant, document ID, week number, heading, visibility, topic, or author.
• A grounded answer is limited to the retrieved evidence instead of the model's unsupported assumptions.
• A citation maps an answer claim, such as [S2], back to an exact retrieved chunk.
• A hallucination is a generated claim that the retrieved material does not support.

Metadata matters as much as embeddings. If a user explicitly requests Week 11, a highly similar Week 10 chunk must not replace it. The week number is a hard filter, not a ranking preference.

The grounded RAG request flow

The application should resolve identity and permissions before it calls a model. It then uses the original question and any safe conversation context to retrieve only the evidence the user may see.

flowchart LR
  Q["User question"] --> U["Query understanding"]
  U --> R["Retrieve permitted chunks"]
  R --> S["Rank and select context"]
  S --> G["GPT generates grounded answer"]
  G --> V["Validate citations and evidence"]
  V --> A["Answer with citations"]

Query understanding does not need to be elaborate. It can identify an explicit week number, expand a known acronym such as MCP, or turn a follow-up such as “What about that?” into a standalone retrieval query. The original user request remains the source of truth; rewritten text is only a search aid.

The retrieval stage should combine three signals where appropriate:

1. Semantic search finds related concepts through vector similarity.
2. Metadata filtering enforces exact constraints such as tenant, visibility, week, region, or document type.
3. Lexical search catches exact terms, names, IDs, and acronyms that semantic similarity may miss.

The selected context should be small and deliberate. Apply a similarity threshold, prefer chunks that match the requested topic, cap chunks per document, and include diverse evidence instead of five near-duplicates from the same paragraph. More context is not automatically better context.

What embeddings do, and what GPT does

The embedding model and chat model solve different problems.

flowchart LR
  C["Document chunk"] --> E["OpenAI embedding model"]
  E --> V["1,536-number vector"]
  V --> DB["Supabase vector storage"]

  Q["Question"] --> QE["OpenAI embedding model"]
  QE --> QV["Question vector"]
  QV --> VS["Vector similarity search"]
  DB --> VS
  VS --> R["Relevant chunk text"]

During ingestion, the application cleans a document, splits it into meaningful chunks, creates an embedding for each chunk, and stores the vector beside the original text and metadata. During a question, it embeds the question and searches for nearby stored vectors. The result is still text: the retrieved chunk content, not the vectors themselves, becomes the evidence sent to GPT.

In this reference implementation:

Responsibility	Configuration	Output
Semantic retrieval	text-embedding-3-small, 1,536 dimensions	Relevant source chunks
Follow-up rewrite	gpt-4o-mini, temperature 0, maximum 120 tokens	Standalone search query
Recovery planner	gpt-4o-mini, temperature 0, JSON-only, maximum 160 tokens	Validated retrieval plan
Answer generation	gpt-4o-mini, temperature 0.1	Cited grounded answer

The chat model should not receive database credentials or query the vector store directly. It receives the user question, a system policy, and a numbered set of approved chunks. The application owns data access and source selection.

flowchart TB
  I["Server resolves identity and access"] --> D["Database returns permitted chunks"]
  D --> C["Application selects context"]
  C --> L["GPT receives selected text only"]
  L --> O["Draft answer with source IDs"]

This boundary prevents a model instruction from becoming an access-control bypass. Row-level security, tenant filters, and visibility checks must be enforced before retrieval—not described only in a prompt.

Grounding, citations, and recovery

The answer prompt should clearly state that it must use only the supplied sources, cite factual statements, and say when the evidence is insufficient. A useful output can look like this:

Week 11 covered context engineering and reusable context files. [S1] NotebookLM was discussed as a way to organise meeting notes. [S2]

[S1] and [S2] are not decorative. They are source IDs assigned by the application to the exact retrieved chunks. This makes the answer reviewable and gives the system something concrete to validate.

When normal retrieval has weak or empty results, do not ask the answer model to guess. A small recovery-planning call can return a structured retrieval plan, for example:

{
  "searchQuery": "Model Context Protocol discussion in Week 11",
  "intent": "session_detail",
  "scope": "week",
  "weekNumber": 11,
  "topicTags": ["mcp"]
}

The application validates every field in that JSON, runs one additional retrieval attempt, and falls back to a transparent “I couldn't find supporting material for that” response if evidence is still absent. The planner helps search; it never writes the final answer.

How to validate responses returned by the RAG system

Prompt instructions reduce errors, but they do not prove a response is reliable. Validation should happen at two levels: runtime checks for each answer and repeatable evaluations for the whole system.

Runtime validation before an answer is displayed

Treat the generated answer as a draft until it passes deterministic checks.

• Extract every citation and reject any ID that was not in the retrieved source set. An answer citing [S99] when only [S1] through [S4] were supplied is invalid.
• Require citations for material factual claims. A citation-free answer should either be repaired with the same approved context or replaced with an evidence-not-found response.
• Preserve a source map containing the document ID, chunk ID, heading, and version for every citation. This enables a user or reviewer to inspect the evidence.
• Check that explicit filters were honoured. A Week 11 answer cannot cite Week 10; a response for Tenant A cannot cite a Tenant B chunk; a private chunk must never enter the source map.
• Log the retrieval IDs, model configuration, validation result, latency, and token usage without recording more sensitive content than the product requires.

A citation parser answers “is this source ID valid?” It cannot by itself prove that the source supports the claim. For high-value answers, add an evidence check: compare each factual claim with its cited chunk, flag unsupported or partially supported claims, and send contradictions to human review. This can use a constrained verifier model, but the final policy decision stays in application code.

flowchart LR
  D["GPT draft answer"] --> P["Parse citations"]
  P --> F["Validate filters and source IDs"]
  F --> E["Check claim-to-evidence support"]
  E -->|"Pass"| A["Return cited answer"]
  E -->|"Fail or weak evidence"| R["Repair once or return safe fallback"]

The safe fallback is a product feature, not an error message. It protects users from confident answers when the system lacks approved evidence.

Regression evaluation: measure the behaviour over time

Maintain a small, versioned evaluation dataset rather than relying on a few impressive demos. Each record should include a question, user-access context, expected retrieval constraints, expected source chunks when an answer exists, and an expected abstention when it does not.

Include representative cases:

• answerable questions with one clear supporting chunk;
• multi-source questions requiring a synthesis across documents;
• unanswerable or stale questions that must produce a safe fallback;
• ambiguous questions that should request clarification;
• hard metadata constraints for week, tenant, visibility, region, or document type; and
• prompt-injection text inside retrieved documents that must not override application policy.

Track more than whether the prose “sounds good.” Useful measures include citation validity and coverage, claim groundedness, retrieval recall of expected chunks, answer relevance, abstention accuracy, access-control failures, latency, and token cost. Review failures by category: poor chunking, weak metadata, missed retrieval, unsupported generation, or faulty validation each demand a different fix.

Run this suite whenever documents are re-indexed or the chunking rules, embedding model, retrieval logic, prompts, answer model, or access policy changes. This makes quality a regression-tested engineering property instead of a subjective last-minute review.

Try the Knowledge Bot

Want to see this RAG pattern in practice? Try AI Engineering Circle - Knowledge Bot, a grounded LLM chatbot built to answer questions from approved knowledge sources with traceable evidence.

Key takeaway

A grounded RAG application is a controlled evidence pipeline:

approved documents + chunking + embeddings + metadata filters
→ focused retrieval + server-side access control
→ GPT synthesis + citations + response validation

The practical takeaways are:

• Treat embeddings as a retrieval layer, not a source of answers. Store the original chunk text and metadata with each vector, then give GPT the retrieved text as evidence.
• Make metadata and access control non-negotiable. Semantic similarity can rank results, but it must never override a tenant boundary, visibility rule, date, or explicitly requested filter.
• Keep context focused. Better chunking, ranking, and diversity usually improve answer quality and cost more effectively than sending a larger prompt or choosing a larger model.
• Make citations machine-checkable. The application should assign source IDs, validate them against the retrieved set, and retain enough source metadata for a reviewer to trace every material claim.
• Design for abstention. “I couldn't find supporting material” is the correct outcome when evidence is weak; an honest fallback is more useful than a fluent guess.
• Evaluate every change. Re-indexing content, changing chunking, updating prompts, switching models, or modifying permissions can change retrieval and answer quality. A versioned evaluation set turns those changes into measurable regressions.

Embeddings make the right evidence findable. GPT makes that evidence readable. The surrounding application makes the answer permitted, traceable, and honest about what the system does not know.