What can go wrong with unmanaged personal AI use
The risk is not that ChatGPT exists. The risk is that the business cannot see what context entered the tool, whether the output was checked, or whether a customer, staff member, supplier, or system was affected.
Customer, staff, or client information entered into the wrong place
Staff may paste customer emails, contracts, complaint notes, employee information, financial details, pricing context, or confidential project material into a tool the business has not approved. The AI Data Access Audit for NZ Businesses gives a practical pre-check.
No shared record of prompts, outputs, or decisions
Personal accounts usually sit outside business records. Managers may not know which prompt was used, what answer came back, what was copied, or whether the output influenced a decision.
AI-generated content sent without review
The safest rule is: AI can prepare, people approve consequences. Customer replies, proposal language, pricing statements, HR wording, public claims, website copy, privacy wording, and complaints need human review. The AI Approval Gates for Business Automation page explains those review points.
Source-of-truth confusion
AI can sound confident when the source material is incomplete. If a staff member asks ChatGPT to summarise a customer situation from pasted fragments, the output may miss CRM notes, job history, contractual details, or recent emails.
Tool sprawl and shadow AI
Personal accounts are one type of shadow AI. The same issue can appear through browser extensions, meeting assistants, CRM add-ons, document tools, and spreadsheet automations.