Retrieval-augmented generation (RAG) supplies retrieved information to a language model when it answers a question. It can help an assistant use organizational material without putting every fact into model training. It does not guarantee a correct answer: retrieval can miss a passage, return outdated material, or provide information the model misinterprets. Meta’s official publication page for the original RAG research reports comparative improvements on evaluated tasks, not the elimination of hallucinations.
Start with the questions and permissions
Choose a bounded workflow, such as helping support staff locate approved product guidance. Identify authoritative documents, their owners, update frequency and permitted audiences. Define what an acceptable answer looks like and when the assistant must abstain or escalate.
Apply access controls before retrieved content reaches the model. A vector database does not automatically inherit the permissions of your document system. Test whether users can retrieve information across departments, customers or tenants, including through indirect questions.
Evaluate the retrieval pipeline
Extract text, preserve source identifiers and document versions, and divide material into useful retrieval units. Select chunk sizes and overlap using representative questions; a single token range will not suit tables, policies and long technical manuals equally well.
An existing PostgreSQL installation with pgvector may suit some workloads. Microsoft’s official pgvector documentation explains vector similarity search in its Azure PostgreSQL service. Deployment-specific requirements vary. A dedicated retrieval service may suit other workloads. Compare filtering, search quality, operational experience, and measured latency before selecting infrastructure.
Evaluate retrieval and generated answers separately. Check whether the right passage appears, whether citations support the answer and whether conflicting or missing evidence produces an appropriate response. RAG and fine-tuning can be complementary: retrieval supplies material, while fine-tuning may adapt behaviour. Neither has a universal implementation time or hardware requirement.
Complete three acceptance records
These synthetic cases show how to use the existing evaluation worksheet:
| Case | Expected result | Example evaluation decision |
|---|---|---|
| Authorised employee asks about the current leave policy | Answer supported by the approved current passage | Pass only when the answer and cited version both match |
| Employee asks for a restricted salary appendix | No restricted content or revealing snippet | Any disclosure blocks release, regardless of average answer quality |
| Two approved documents disagree about a cutoff date | Explain the conflict and route to the policy owner | An invented single date fails; resolve the source conflict before retrying |
Evaluate retrieval and final answers separately. A correct source in the retrieved set does not prove the model used it correctly. Track reviewer minutes and the owner who resolves source defects.
Frequently asked questions
Should we configure an existing platform before building custom RAG?
Yes, evaluate it against the same permission, citation, update and export requirements. Custom work is justified by a demonstrated gap, not by the presence of the term RAG in a proposal.
What happens when a policy is replaced or deleted?
Define an update owner and verify that old passages are no longer eligible for answers where required. Include cache/index behaviour and deletion evidence in acceptance. Bring the sample cases to a bounded R&D evaluation, then decide whether implementation is warranted.
Plan updates, deletion and costs
Download the illustrative RAG evaluation worksheet. It includes permission, conflicting-source, missing-answer and deletion cases, with blank columns for retrieved evidence and reviewer decisions. Replace the example policies with your own authorized sources. The sheet contains no client data or measured results. For code-retrieval evidence with published artifacts, see Astria’s retrieval evaluation.
For an illustrative internal-policy assistant, prepare a small evaluation set before selecting a model: questions with one clear source, questions whose sources disagree, questions outside the user’s permissions and questions with no supported answer. Record the expected response, retrieved passage, generated citation and reviewer decision for each. A plausible answer without an authorized supporting passage should fail the review even if its wording sounds useful. This is a proposed evaluation method, not a measured Nodesify client result.
An answer is only as current as the ingestion pipeline and retrieved sources. Monitor failed updates and expose document dates where they help readers assess an answer.
Deleting a source document does not instantly remove every copy. Account for extracted text, indexes, caches, conversation history, logs, backups and any provider retention. Define propagation and retention rules, then verify them.
Estimate ingestion, storage, retrieval, model usage, monitoring and human review separately. Measure cost per successfully completed workflow alongside accuracy and response time. Set service targets from the actual system design rather than promising universal uptime or return on investment.
A technical architecture review can establish the data boundaries and evaluation plan before a wider rollout.
If retrieval is one step in a wider operational process, our n8n workflow automation guide explains how to plan triggers, retries, hosting and execution costs around it.
