LLM-Only vs RAG GenAI Apps: Architecture Impacts Cost, Quality, Trust

TL;DR

In Part 1, we looked at the hardware and runtime side of GenAI applications — CPU, GPU, NPU, inference, embeddings, model serving, vector databases, and token generation.

In this Part 2, we look at a more practical application architecture question:

Should your GenAI app depend fully on the LLM, or should it use a RAG-based architecture?

The short answer:

LLM-only = simpler to build, useful for small and one-time tasks
RAG-based = better for large documents, repeated queries, source-grounded answers, cost control, and auditability
Hybrid = often the most practical real-world approach

A common mistake in GenAI app development is sending too much context to the LLM, too often. That may work during a demo. But when real users start uploading larger documents, asking repeated questions, expecting source-based answers, and using the system daily, poor architecture can quickly increase token usage, API cost, latency, hallucination risk, infrastructure load, and operational complexity.

The problem is not always the model. Many times, the problem is the way the GenAI application is engineered.

The common mistake: sending everything to the LLM

Many GenAI apps start like this:

User uploads document
→ app extracts text
→ app sends full document to LLM
→ user asks a question
→ LLM generates answer

This is simple. It is also attractive during early prototyping because there is less engineering work. No chunking, no embeddings, no vector database, no retrieval logic, no metadata strategy, no ranking, no source mapping. Just pass the content to the model and get the response.

For a small document, this can work well. For a proof of concept, this can be enough.

But this design starts creating problems when documents become large, users ask multiple questions on the same document, multiple users upload files, source citations are required, latency matters, API cost matters, access control matters, answers must be auditable, or the system must say "not found" when evidence is missing.

This is where the architecture decision becomes important.

How an LLM-only GenAI app works

An LLM-only GenAI app directly sends the source content to the model.

flowchart LR
    A[User Uploads Document or Image] --> B[Extract Text / OCR / Vision]
    B --> C[Build Prompt with Full Context]
    C --> D[LLM Inference]
    D --> E[Token-by-token Response]
    E --> F[Display Answer]

Example prompt structure:

System:
You are a contract review assistant. Answer only from the provided document.

Document:
<full extracted document text>

User:
What is the liability limit?

Output format:
Return summary, risk level, and relevant clause if available.

The important point: in an LLM-only system, the model may need to process the same source content again and again for each query. That is where token cost and latency can increase.

Why LLM-only looks attractive in the beginning

LLM-only architecture has some genuine benefits. It is not wrong by default. It is useful when applied to the right use case.

Benefit	Why it helps
Faster to build	Fewer moving parts
Lower engineering complexity	No vector DB or retrieval pipeline
Good for prototypes	Easy to validate product idea
Good for small documents	Full context can fit within model limits
Good for one-time analysis	No need to build reusable retrieval
Useful for structured extraction	Works well for small forms, invoices, short contracts, screenshots

The mistake is not using LLM-only. The mistake is using LLM-only for every use case without considering document size, repeat usage, source traceability, and cost.

Where LLM-only starts becoming expensive

LLM usage is usually charged based on tokens. Tokens are consumed in two directions: input tokens (what you send to the model) and output tokens (what the model generates).

In an LLM-only document Q&A app, every query may include the system prompt, user question, full document content, and output instructions.

Example:

System prompt: 500 tokens
User question: 50 tokens
Document content: 40,000 tokens
Expected answer: 1,000 tokens

One query may consume approximately 40,550 input tokens + 1,000 output tokens. If the user asks ten questions on the same document, the app may repeatedly send the same 40,000-token document context.

That becomes expensive. It also increases latency. The model has to process a large input every time.

This is one of the hidden engineering traps in GenAI app development: the app looks simple, but the token bill keeps growing because the same context is repeatedly pushed to the model.

How a RAG-based GenAI app works

RAG stands for Retrieval-Augmented Generation. RAG is not a model. It is an application architecture pattern.

Instead of sending the full source content to the LLM every time, the app retrieves only the relevant parts and sends those to the model.

flowchart TB
    A[User Uploads Document / Image] --> B[Extract Content]
    B --> C[Clean and Structure]
    C --> D[Chunk Content]
    D --> E[Create Embeddings]
    E --> F[Store Embeddings + Text + Metadata]
    G[User Query] --> H[Create Query Embedding]
    H --> I[Search Vector Database]
    I --> J[Retrieve Relevant Chunks]
    J --> K[Build Prompt with Retrieved Context]
    K --> L[LLM Inference]
    L --> M[Display Answer with Sources]

The key idea: embeddings are used to find relevant context. The LLM still generates the answer from the retrieved text.

Embeddings are for retrieval, not answering

This is one of the most important concepts in GenAI app architecture.

An embedding is a numerical representation of text, image, or content meaning. The vector is useful for similarity search. It helps answer: which document chunks are semantically close to this user query?

But the LLM does not usually read the vector and answer from it. The app retrieves the original chunk text associated with the vector. Then the LLM receives that text as context.

Correct mental model:

Embedding = search helper
Retrieved chunks = evidence
LLM = reasoning and response generation layer

If this is not understood clearly, teams may design poor retrieval pipelines and expect the LLM to compensate. That usually leads to weak answers.

Upload-time workload vs query-time workload

A good way to understand GenAI app architecture is to separate upload-time processing from query-time processing.

Upload-time (when user uploads source material):

extract text → clean content → split into chunks → create embeddings → store chunks, vectors, and metadata

Query-time (when user asks a question):

embed user query → search vector DB → retrieve relevant chunks → build prompt → call LLM → generate answer

This distinction matters because RAG moves some work to upload time so that query time can be more efficient.

LLM-only vs RAG: architecture comparison

Area	LLM-only	RAG-based
Initial build effort	Low	Medium to high
Engineering complexity	Simple	More components
Small document handling	Good	Good
Large document handling	Expensive and slower	Better
Repeated questions	Costly	More efficient
Token usage	Higher for large context	Lower at query time
Latency	Can increase with document size	Usually more controlled
Source traceability	Possible but harder	Easier with metadata
Multi-document search	Weak at scale	Stronger
Access control	Harder	Easier with metadata filters
Auditability	Harder	Better
Hallucination control	Prompt-dependent	Better if retrieval is strong
Production suitability	Use carefully	Usually better for knowledge apps

The table does not mean RAG is always better. It means RAG solves a different class of problems.

Token cost: the hidden engineering constraint

Token cost is not only a billing issue. It affects architecture. If the system sends too much context to the LLM, several things happen: API cost increases, response latency increases, the model may become distracted, large prompts become harder to control, repeated queries become inefficient, long-context limits may be reached, and the app becomes harder to scale.

A poor architecture may look fine with five test documents. It may start failing when real users upload contracts, policy documents, technical manuals, support tickets, product documentation, invoices, reports, logs, or codebases.

The cost problem usually appears after the prototype phase. That is why GenAI app architecture should be discussed early, not after the API bill becomes painful.

Full context does not always mean better context

A common assumption is: if I send the full document to the LLM, the answer will be better.

Not always. Large context can create problems: the relevant section may be buried, irrelevant text may distract the model, the model may overgeneralize, source citation may become weaker, latency increases, cost increases, and answer quality may become inconsistent.

Sometimes, a smaller but more relevant context produces a better result. That is the core purpose of RAG: do not send everything. Send what is relevant.

But this only works when retrieval is well designed. Bad RAG can also fail. If the retrieval pipeline fetches the wrong chunks, the LLM will answer from poor context.

Chunking is an architecture decision

Chunking is not just splitting text every 500 words. Chunking should follow the purpose of the GenAI app.

For example, in a contract review system, random text chunks may break important legal meaning. A better chunk may preserve clause number, section heading, obligation, party name, effective date, exception, liability cap, termination condition, and page number.

For a code assistant, chunking may follow repository, file path, class, function, module, dependency, and test file. For a log analysis system, chunking may follow timestamp, service, trace ID, error type, deployment version, and incident window.

Good RAG starts with good content structure. Poor chunking leads to poor retrieval. Poor retrieval leads to weak answers.

Metadata matters as much as embeddings

A vector database should not store only embeddings. It should store embeddings along with useful metadata.

{
  "document_id": "agreement-123",
  "chunk_id": "clause-8.2",
  "text": "The liability of the consultant shall be capped...",
  "embedding": [0.023, -0.118, 0.442],
  "metadata": {
    "page": 7,
    "section": "Limitation of Liability",
    "clause": "8.2",
    "document_type": "consulting_agreement",
    "uploaded_by": "user-456"
  }
}

Metadata helps with filtering, access control, source citation, document-level search, tenant isolation, auditability, relevance improvement, and response explanation.

For serious GenAI apps, metadata is not optional. It is part of the trust layer.

Prompt contracts matter

Prompting is not just writing a good instruction. In an application, prompts are part of the system contract.

System prompt defines the role and boundaries:

You are a contract review assistant.
Use only the provided context.
Do not invent missing clauses.
If the answer is not present, say "Not found in the provided document."
Always cite the source clause and page when available.

Retrieval contract defines what context should be fetched:

Retrieve chunks from the same document.
Prioritize sections related to liability, indemnity, damages, and limitation of liability.
Return top 5 chunks with page and clause metadata.

Output contract defines the response structure:

{
  "answer": "...",
  "risk_level": "low | medium | high",
  "source_references": [{ "clause": "...", "page": "..." }],
  "not_found": false
}

A GenAI app should not depend only on casual prompts. It needs structured contracts between the application, retrieval layer, model layer, and response layer.

Source traceability matters

In business applications, users do not only need an answer. They need to know where the answer came from.

A weak answer: "The liability appears to be limited."

A better answer: "The liability is capped at the fees paid in the previous three months, based on Clause 8.2 on page 7."

The second answer is more useful because it includes evidence. RAG makes this easier because retrieved chunks already carry source metadata.

When LLM-only is good enough

LLM-only can be a good option when the document is small, the task is one-time, risk is low, source citation is not critical, the number of users is limited, latency is acceptable, cost is not a major concern, or the app is still in MVP validation.

Good use cases: summarize a short document, extract fields from a small invoice, analyze a screenshot, classify a short message, generate a one-time report, create a draft from a small input, convert a small document into structured JSON.

When RAG becomes necessary

RAG becomes more useful when documents are large, users ask repeated questions, multiple documents are involved, source citations are required, answers need auditability, access control matters, API cost must be controlled, latency matters, the source of truth changes over time, the app needs tenant-level data isolation, or the system must answer "not found" reliably.

Good use cases: internal knowledge base, policy assistant, contract Q&A, compliance document search, technical documentation assistant, customer support assistant, incident knowledge base, codebase assistant, research archive assistant, product documentation chatbot.

Hybrid architecture: the practical middle path

The best architecture is often hybrid. Not every request needs RAG. Not every request needs full-context LLM processing.

flowchart TB
    A[User Uploads Source] --> B[Extract Content]
    B --> C{Use Case and Size Check}
    C -->|Small / One-time| D[LLM-only Processing]
    C -->|Large / Repeated Queries| E[RAG Pipeline]
    C -->|Structured Form| F[Extraction Pipeline]
    C -->|High-risk / Audit Need| G[RAG + Validation + Citations]
    D --> H[LLM Response]
    E --> I[Retrieve Context]
    I --> H
    F --> H
    G --> I
    H --> J[Formatted Output]

Example routing logic:

If document is under 10 pages and task is summary → LLM-only
If document is reused for Q&A → RAG
If exact fields are needed → structured extraction
If compliance/audit evidence is needed → RAG with citations
If answer confidence is low → ask for clarification or say not found

This avoids overengineering while still preparing for real usage.

Caching can help, but it does not replace architecture

Some model providers support prompt caching or context caching mechanisms. Caching can reduce cost and latency for repeated prompts. But caching does not replace good architecture.

Caching helps when the same large input is reused. But it does not solve poor retrieval, missing metadata, source citation, access control, hallucination, irrelevant context, bad chunking, or weak output contracts.

Caching is an optimization. Architecture is still the foundation.

Hardware and infrastructure considerations

The architecture choice also affects infrastructure.

LLM-only systems may increase input token volume, model context length requirement, latency, memory pressure, API cost, and dependency on large-context models.

RAG systems add more components: document processing, embedding generation, vector database, metadata storage, retrieval service, reranking layer, source citation logic, and indexing pipeline.

So RAG is not free. It reduces repeated token usage, but increases system complexity.

LLM-only cost = token and latency cost
RAG cost = engineering and infrastructure cost

The right choice depends on usage pattern.

Engineering decision framework

Before choosing LLM-only or RAG, ask:

1. How large is the source content?
2. Will users ask repeated questions?
3. Do users need source references?
4. Is this a one-time workflow or reusable knowledge system?
5. Is the app high-risk?
6. Is latency important?
7. Is cost predictable?
8. Is access control required?
9. Can the system say "not found"?
10. Are you building a demo or a product?

A simple decision table

Scenario	Better starting architecture
Short one-time summary	LLM-only
Invoice field extraction	LLM-only or structured extraction
Single small contract review	LLM-only may work
Large contract Q&A	RAG
Internal policy chatbot	RAG
Product documentation assistant	RAG
Codebase assistant	RAG with code-aware chunking
Compliance evidence assistant	RAG with citations and audit logs
Support ticket knowledge assistant	RAG
MVP validation	LLM-only first, then evolve
Multi-tenant enterprise app	RAG with metadata and access control
High-risk decision support	RAG + validation + human review

Example: poor architecture vs better architecture

Poor architecture:

Every user question sends the full document to the LLM.
No chunking. No metadata. No source references.
No retrieval logic. No cost control. No audit trail.

This may work in a demo. But it can become expensive and unreliable when usage increases.

Better architecture:

Upload once.
Extract and structure content.
Chunk based on domain meaning.
Create embeddings.
Store text, vectors, and metadata.
Retrieve relevant context per query.
Build a controlled prompt.
Generate answer with source references.
Log query, retrieval result, and model output.

This is more engineering work. But it gives better control over cost, quality, and trust.

Production-readiness concerns

A GenAI app should be treated like any other software system. It needs engineering discipline.

Important production concerns include input validation, file scanning, content extraction reliability, prompt injection handling, tenant isolation, access control, model timeout handling, retry strategy, fallback response, token usage tracking, cost monitoring, observability, audit logs, source traceability, model versioning, evaluation datasets, and response quality testing.

The model is only one part of the system. The application architecture around it determines whether the system is reliable.

Don't bankrupt your GenAI app with poor architecture

The phrase sounds slightly dramatic, but the point is real. A poor architecture can quietly burn money.

It may happen through sending the same document repeatedly, using large models for simple tasks, passing irrelevant context, failing to cache reusable context, using LLMs where rules or extraction would work, not separating upload-time and query-time workloads, skipping retrieval and metadata design, or ignoring token monitoring until cost becomes visible.

The goal is not to avoid LLM usage. The goal is to use the LLM where it adds value.

Final thought

The question is not whether your GenAI app should use an LLM. It obviously will.

The real question is: how much unnecessary context are you sending to the LLM, how often are you sending it, and can your architecture survive real usage without burning your token budget?

LLM-only architecture is useful for small, simple, and early-stage workflows. RAG architecture is useful when scale, repeated queries, source traceability, access control, and cost predictability matter. Hybrid architecture is often the practical middle path.

Good GenAI engineering is not just about writing prompts. It is about designing the right flow between source content, extraction, chunking, embeddings, retrieval, prompt contracts, model inference, output contracts, observability, and cost controls. That is where GenAI apps move from demo to dependable software.

Further reading

• Part 1: Understanding GenAI Hardware — CPU, GPU, NPU, Inference, and Model Serving