Let me tell you how a shadow AI incident actually starts. Not the dramatic version. The real one. It starts with someone doing something completely reasonable on a Tuesday afternoon.
Note: the following is a composite scenario drawn from publicly reported incident patterns, not a single documented case. The details are illustrative of how shadow AI incidents typically unfold.
Alex is a senior associate at a mid-size consulting firm. She's preparing a proposal for a new client and needs to summarize a complex services agreement from an existing engagement — just to pull out the scope language and key terms for reference.
She opens ChatGPT — her personal account, the free tier — pastes in the agreement, and asks it to summarize the key provisions. It takes 30 seconds. She gets what she needs. She closes the tab.
Alex didn't think of this as a data incident. She thought of it as using a tool to work faster. She'd done it dozens of times before. Her colleagues do the same thing. Nobody told her not to.
Here's what she didn't know:
- The services agreement she pasted contained her client's internal pricing structure, project scope, and a named contact at a competitor the client was benchmarking against.
- Her free-tier ChatGPT account's default data settings, at the time she used it, allowed the provider to use conversation content for model improvement — meaning her client's confidential information potentially became training data. (Data practices change over time and vary by account tier, so it's worth checking current terms directly rather than assuming.)
- Her firm's services agreement with the client almost certainly included a confidentiality clause covering exactly this kind of disclosure to third parties. The AI platform is a third party.
- Her firm had no AI policy and no approved alternative. Alex wasn't breaking a rule. There was no rule to break. That's the whole problem.
How It Escalates — Three Ways This Gets Discovered
The client's legal counsel is doing a vendor review. They ask about AI tool usage. The firm says it has no formal AI policy and can't confirm what tools staff use. The client's counsel recognizes that this means confidential data has likely been processed by unauthorized AI tools. The conversation becomes contractual. It may become litigious.
This scenario is happening more frequently as enterprise procurement teams add AI governance requirements to their vendor questionnaires. The question "do you have an AI acceptable use policy?" is becoming routine.
A regulatory examination or audit surfaces the question of how client data is handled. The firm can document its data security practices for traditional systems. It cannot document its AI practices because there are none to document. The examination becomes more intensive.
For financial services firms under FINRA or SEC oversight, healthcare practices under HIPAA, and law firms under bar jurisdiction requirements, "we don't have a formal policy yet" is not a neutral answer. It's a finding.
This is the lowest-probability but highest-consequence route, and the one that generated the most public attention with the Samsung case in 2023. Internal code, client strategy, competitive intelligence — entered into a tool, potentially surfacing in ways the firm didn't anticipate and can't control after the fact.
The exact mechanism by which inputs to large language models can surface in outputs to other users is technically complex and much debated — the risk is real but not simple to quantify. What is clear is that once sensitive data enters a third-party model, you've lost control of it in a meaningful sense: you can't retrieve it, you can't confirm what happened to it, and you can't verify it doesn't persist somewhere.
Ask Jordan
One of our senior consultants told me she uses ChatGPT to help draft client deliverables. She says she never puts in client data — just uses it for structure and writing. Is that actually safe, or do I need to be worried?
That's actually the right question to be asking. For structure and writing with no client data, the risk is significantly lower — but there are two things worth knowing. First, "no client data" is harder to enforce in practice than people expect: once someone is in the habit of using a tool, the line between structure and content blurs quickly under deadline pressure. Second, if she's on a free personal account, the tool's data practices still apply to those prompts. The more important question is: what account tier is she on, and does your firm know the data practices of that specific plan?
Alex's scenario is preventable — not through a ban, but through a policy that gives people an approved alternative and clear rules before the Tuesday afternoon moment happens. Start with the 5-question audit to find out where your firm actually stands, then use the policy framework and one-page template to close the gap.