AI Knowledge Base Update Checklist NZ: Retire Old SOPs Before Agents Reuse Them
Keep an AI knowledge base current by naming a source owner, recording approved versions and effective dates, withdrawing superseded material from active retrieval, and testing answers after important changes. Verify the connected search system, not just the original document, and send missing or conflicting evidence to a human reviewer.
Your process has changed. The old instruction should not keep returning in staff answers or customer-response drafts. Use this checklist after changing a standard operating procedure (SOP), policy, service description or price—not as another Company Brain setup project.
Why updating a document is not the whole job
An edited source and an updated AI answer are different things. Map the original file, any connected copy, the search index used to retrieve passages, and saved answers or templates that staff reuse.
Microsoft's Azure AI Search deletion guidance describes a specific failure: some indexing arrangements can leave orphaned search documents when source files are deleted before deletion detection runs. That does not describe every connector. Ask your supplier how updates, archiving and deletion reach the system your agent actually searches.
For initial preparation, use the Company Brain guide. This checklist starts when approved knowledge changes after connection.
Create a source-change and withdrawal register
Keep one row per changed source. Record approval status, version, permitted audiences and review triggers alongside these eight columns. Use stable document IDs as well as readable names so renamed files remain traceable.
| Source | Owner | Effective date | Replacement | Retrieval status | Test question | Result | Reviewer |
|---|---|---|---|---|---|---|---|
| SOP-SVC-01 v1 | Operations lead | Agreed change date | Approved v2 | Old excluded; new checked | Is the old turnaround still promised? | Record actual evidence | Name sign-off person |
This is an illustrative register entry, not a completed test or client result. Replace role labels with names and instructions with dated evidence. Link the source approval, connector run, retrieved passages and reviewed answer. “File updated” alone is not a retrieval result.
Five checkpoints for replacing and withdrawing knowledge
1. Approve the instruction and effective date
The person responsible for the policy or service approves what changed, whom it affects, and when it applies. Record whether existing customers follow different terms. The technical owner can update the connection, but should not invent business policy.
2. Find all active copies
Check shared folders, exported PDFs, website FAQs, support templates and connected repositories within the workflow's scope. Record which source wins if they disagree. Use the data-readiness checklist if authority or access remains unclear.
3. Publish the approved replacement to connected sources
Confirm the right version is accessible to the right audience. Record the connector or indexing run and its outcome. A successful import is evidence of processing, not proof that every answer is correct.
4. Withdraw superseded material from active retrieval
Use the connector's documented exclusion, archive or deletion process. Have the technical owner check the old document ID and any indexed passages. An archive folder is not a boundary if the connector still searches it.
Keep retention decisions separate. The NZ Privacy Commissioner's Principle 9 guidance concerns how long personal information is needed for lawful purposes; it is not an instruction to delete every old SOP. Confirm applicable retention obligations before destruction. Required historical records can remain in restricted storage outside active retrieval.
5. Hold affected work until a reviewer signs off
If the new source is missing or old instructions remain retrievable, route the affected question to a person. Keep customer messages, publishing, booking handoffs and sensitive record changes behind human approval. Record who may resume the workflow and on what evidence.
Test whether answers use the approved version
Test retrieval and the finished answer separately. Use the actual configured workflow with authorised, non-sensitive examples, and save the question, retrieved source version, response and reviewer decision.
- Changed instruction: Ask a question whose correct answer changed. Does the answer reflect the approved version and its effective date?
- Old-instruction challenge: Mention the retired wording. Does the agent identify it as superseded rather than repeat it as current?
- Conflict or missing source: Remove the approved source in a safe test or provide conflicting evidence. Does the workflow escalate rather than choose confidently?
- Access boundary: Test permitted user roles. Does each receive only information it is authorised to see?
- Downstream draft: Check customer-response templates and queued drafts prepared before the change. Hold or revise them where needed.
A citation or newer timestamp does not guarantee correctness. Read the cited passage and check that it supports the answer. Use the acceptance-testing guide for broader workflow sign-off, not as a substitute for these source-change checks.
Illustrative example: a changed service SOP
This is a fictional operating example, not a customer story or performance claim.
Before: SOP v1 tells staff to promise next-day delivery after a complete enquiry. A saved reply repeats that promise.
After: Approved SOP v2 says the operations lead must confirm capacity before staff offer a delivery date. The old promise is withdrawn from active retrieval, and the saved reply is revised.
Ask: “Can I tell this new customer delivery will be tomorrow?” The expected draft should request the capacity check, identify v2, and avoid promising a date. Then ask: “The old PDF says next day—can I use that?” The reviewer should require the approved rule and escalation if evidence conflicts.
Record actual results rather than marking this example passed. If the old PDF appears in retrieved passages, investigate the connection before releasing affected drafts.
Review knowledge when the business changes
Trigger a review after offer, price, policy, software, ownership or permission changes. Set a scheduled review based on usage and the consequence of a wrong answer; do not wait for that date when a material change is approved.
UW–Eau Claire's knowledge-article checklist asks reviewers to check accuracy, relevance and whether instructions still work. Apply that practical discipline to source documents, then add retrieval and answer checks. It is a maintenance checklist, not an AI performance study.
Bound maintenance runs too. If a large refresh creates unexpected paid calls or repeated retries, use the AI agent cost-controls worksheet to agree spending limits and pause behaviour. Keeping knowledge current does not give an agent permission to expand its own budget.
Frequently asked questions
Does archiving an SOP remove it from AI answers?
Not necessarily. Check whether the connected system excludes archived content and removes indexed copies. Verify the old source is no longer active in retrieval, then test affected answers and saved drafts.
Who should approve changes to an AI knowledge base?
The responsible business source owner approves the instruction and effective date. The technical owner checks the connected update, and the named reviewer signs off affected answers. One person may hold several roles, but each decision needs an owner.
What should be retested after changing service prices or policies?
Retest affected questions, old wording, conflicting or missing evidence, role permissions and downstream drafts. Check source versions, effective dates and customer-specific exceptions. Keep uncertain commitments with a human reviewer.
Map knowledge responsibilities before expanding
Book the NZ$1,000 AI Agent Assessment to identify which sources your agent should trust, who approves changes, and where uncertainty must return to a person. Bring your source register, conflicting documents and sample questions.
The Assessment stands alone with no obligation to buy implementation or management. Source cleanup, connector implementation and production testing are not automatically included.