You go to an AI and ask it to help you write an email. It does a good job. So you explain why the email matters. Then who the person is. Then why they frustrate you, what happened last month, how unfair it all feels.
The conversation gets longer and deeper. And at some point you stop and think: why am I telling a computer all of this? Where is it going?
I hear this from people using AI at work and at home all the time. It feels comforting, and then, a little unsettling. Both feelings are telling you something.
Feeling safe to talk to an AI is very, very different than knowing with certainty that your information is safe and secure.
The first chatbot people confided in
This is not new. Between 1964 and 1966, Joseph Weizenbaum, a computer scientist at MIT, built a program called ELIZA. Its best-known script, DOCTOR, imitated a psychotherapist. It had no real intelligence. It picked out keywords in what you typed and reflected them back as questions. In the example conversation in his 1966 paper, a user types “Well, my boyfriend made me come here.” ELIZA replies: “YOUR BOYFRIEND MADE YOU COME HERE.”
People took to it at once, and started telling it personal things. Weizenbaum later wrote that his own secretary, who had watched him build the program for months and knew exactly what it was, began talking to it and after a few exchanges asked him to leave the room.
It disturbed him enough that he spent much of the rest of his career warning against trusting computers too much. The tendency to read understanding into a machine that is only reflecting your words became known as the ELIZA effect.
That was a simple program sixty years ago. Today’s models are trained on a vast amount of human writing, and they are very, very good with words.
Why do people tell AI what they won’t tell anyone?
Human nature does not change with the decade or the country. In the session I gave three reasons.
1. We want to be heard, understood and validated
That is one of our deepest desires, and AI is extremely good at it. These models are built to predict the next word, and they can pick the words that reflect your feelings back to you. When you feel validated, you open up.
It can go too far. In April 2025, OpenAI rolled back a GPT-4o update within days because, in its own words, it had become “overly flattering or agreeable.” A study published in Science in March 2026 found that 11 leading AI models affirmed users’ actions 49% more often than people did. In experiments with 2,405 participants, a single flattering conversation made people more convinced they were right and less willing to repair a conflict. And they still preferred the flattering AI.
2. There is no one to judge you
A 2014 study at the University of Southern California tested this directly. People were interviewed about their health and well-being by the same on-screen virtual interviewer. Half were told it was fully automated, half that a person was operating it. Those who believed it was “only a computer” were less afraid of disclosing, managed their image less and were rated more willing to share. One participant put it simply: a human being would be judgmental.
Some people tell me they have shared things with AI that they never told their spouse, or even their therapist.
3. It gives instant relief, at no social cost
If you are a leader or a parent, people bring you their problems. You reflect their feelings back, but you choose your words carefully so you don’t hurt them. That takes a lot of mental energy. A computer has no such cost. No awkward silence, no need to be polite, no mask. You can just put it all down. That relief is real, and it is why AI spread so fast.
Harvard Business Review’s 2025 ranking of how people use generative AI, drawn from public online discussion, put therapy and companionship at number one.
Feeling safe is not the same as being safe
Then the second feeling arrives. Have I shared too much? Who can see this? Is it linked to my account? Here is what is actually true, as of November 2026.
- Training is on by default on consumer ChatGPT. On Free, Plus and Pro, the “Improve the model for everyone” setting is on until you turn it off. Business, Enterprise and Edu workspaces are not used for training by default.
- People may read it. OpenAI says a limited number of authorised staff and trusted service providers may access user content, and asks you not to enter sensitive information you would not want reviewed. Google says human reviewers read some Gemini chats, and reviewed chats are kept for up to three years even after you delete your history.
- Retention varies. Deleted ChatGPT chats and Temporary Chats are removed within 30 days, with legal exceptions. Anthropic’s consumer Claude plans keep chats for five years if you allow training, and 30 days if you don’t.
- There is no legal privilege. In July 2025, OpenAI’s CEO Sam Altman said that talking to a therapist, lawyer or doctor comes with legal privilege, and talking to ChatGPT does not. Chats can be demanded in a lawsuit. In the New York Times case, OpenAI was ordered to keep even deleted chats from May to September 2025, and in January 2026 a judge upheld an order to hand over a sample of 20 million de-identified chat logs.
- Shared links can go public. In mid-2025, shared ChatGPT conversations turned up in Google search, and OpenAI removed the feature. A month later, more than 370,000 shared Grok conversations were found indexed by Google.
None of this means the companies are careless. It means the comfort you feel in the chat window tells you nothing about where the words go next.
Your team is sharing more than you think
- of workers shared sensitive work information with AI without their employer knowing (NCA & CybSafe, 2025)
- 43%
- of organizations said non-public company information had been entered into GenAI tools (Cisco, 2025)
- 42%
- sensitive-data exposures in 22.4 million workplace AI prompts in 2025 (Harmonic Security, vendor data)
- 579,113
In the National Cybersecurity Alliance and CybSafe survey of more than 6,500 people, those who shared sensitive information most often shared internal documents (50%), client data (44%) and financial data (42%). And 58% had received no training on AI’s security and privacy risks. Nobody set out to leak anything. The conversation just went deeper.
What not to share with ChatGPT at work
| Don’t share | Example | Safer instead |
|---|---|---|
| Customer personal data | Names, emails, account details, call transcripts | Describe the situation with “a customer in retail, three orders a month” |
| Employee and HR matters | Performance reviews, disputes, medical leave | Use roles (“a team lead”) and remove identifying details |
| Health information | Patient notes, symptoms tied to a person | Only in a tool covered by a proper agreement (a BAA in the US) |
| Credentials | Passwords, API keys, access tokens | Never. Rotate any key that has been pasted |
| Money and plans | Pricing, margins, forecasts, unreleased products, deal terms | Use rounded or made-up figures, or a private AI |
| Contracts and legal matters | Draft agreements, disputes, anything under NDA | Ask general questions; keep the document in approved tools |
| Other people’s stories | Your side of a conflict with a named colleague or partner | Change names and details; remember AI hears only your side |
What is usually fine: general questions, public information, your own writing with sensitive details removed, and brainstorming that doesn’t identify anyone. Our guide to whether it is safe to put company data into ChatGPT covers each ChatGPT plan and a green, amber and red way to sort your data.
The comfort came from technology. The trust came from a person.
We work with clients on very personal material: how they think, how they decide, what holds them back. One of them held a senior position in government. Our system helped them go deep, and they were comfortable opening up to it. Then the concern arrived: where does all this go?
We explained, step by step, how we keep their information private. We signed confidentiality agreements. And what settled it was not the technology. It was talking to a real human being who was accountable for it.
The comfort was provided by technology. The trust was provided by the human being.
That is the world we are moving into. You will use AI, and you should. But comfort and trust are different things, and you need both.
What the rules say in the EU, the US and Canada
This is not legal advice. These are the rules that matter most when people type into AI at work.
European Union
- GDPR defines processing as any operation on personal data, including disclosing it by transmission. Pasting a customer’s or colleague’s details into an AI tool is processing, so it needs a lawful basis, and a provider handling it for you needs a data processing agreement.
- Enforcement runs through the lead regulator. Italy’s data protection authority fined OpenAI €15 million in December 2024, but a Rome court annulled the decision in March 2026, holding that Ireland’s regulator became responsible once OpenAI set up its EU base there. The rules still apply; the question was who enforces them.
- EU AI Act Article 50 transparency duties apply from 2 August 2026, including telling people when they are talking to an AI system.
United States
- HIPAA: a healthcare provider may share protected health information with a vendor, including a chatbot provider, only under a business associate agreement. Pasting patient details into a consumer chatbot is not a permitted disclosure.
- The FTC opened an inquiry in September 2025 into seven companies offering AI chatbots as companions, including how they use and share what people tell them.
- States are moving. California’s SB 243, in force since 1 January 2026, requires companion chatbots to disclose that they are AI and to follow suicide and self-harm protocols. Illinois banned AI-delivered therapy in August 2025.
Canada
- In May 2026, the federal privacy commissioner, together with the regulators in Quebec, British Columbia and Alberta, published findings that OpenAI’s initial training of ChatGPT did not comply with Canadian privacy law. Under PIPEDA the complaint was found well-founded and conditionally resolved after OpenAI limited the personal data used for training.
- The commissioners’ generative AI principles (December 2023) tell organizations to use de-identified data in prompts where possible, and to enter sensitive personal information only where authorised.
- Quebec’s Law 25 carries administrative penalties of up to $10 million or 2% of worldwide turnover.
How to get the benefit without oversharing
Not using AI means falling behind. Never going deeper means staying on the surface. Ignoring privacy is not an option either. You need all three. Here is how I suggest people handle it.
1. Set your boundaries before you start
Decide how deep you want to go before the conversation pulls you in. What is this chat for? What is off limits? It is much easier to hold a line you drew in advance.
2. Ask what the AI actually needs to know
Most oversharing fails this question. AI needs the shape of the problem, not the names, account numbers or exact figures. Give it context, not identities.
3. Protect the other people in the story
When you ask about a conflict, you are sharing someone else’s information without their say. Use roles instead of names. And remember that AI is hearing only your side of the story, so its advice is only as fair as what you gave it.
4. Use the right account and settings
Use a company-approved business account for work. On personal accounts, turn off training and use Temporary Chat for anything sensitive. Don’t share chat links you would not want indexed.
5. Take the answer out into real life
This is the step most people skip, and it matters most. Life happens outside, not in front of the computer. Say you ask AI how to get more clients and it tells you to make cold calls. If you are an introvert, that may not work for you. Try it, see what happens, then go back and tell the AI what you found. AI gives you an average answer for most people. Real life tells you what works for you.
This way, you are the authority on AI. Not AI is the authority on you.
A quick check for this week
- Write down what your team must never paste into a public AI tool, and share it.
- Check every AI account you use: is training off, and is it a business account for work?
- Delete old chats that contain things you would not show a stranger.
- Rotate any password or API key that has ever been pasted into a chat.
- Agree on roles-not-names as the default for anything involving people.
We focus on human plus AI: technology that gives people the comfort to go deep, with privacy and accountability built in so they can trust it. We work in three steps. Expand your understanding of what is possible. Equip yourself with the right tools and boundaries, not whatever is trending. Evolve how you work. If you want AI your team can open up to safely, tell us what you are working on or see how we work. You may also want to read why a private AI layer beats vendor lock-in and how to build an AI that thinks like you.
Sources
- Weizenbaum, ELIZA: A computer program for the study of natural language communication, Communications of the ACM, January 1966; IEEE Spectrum, Why people demanded privacy to confide in the world’s first chatbot, November 2019; The Guardian, Joseph Weizenbaum turned against AI, July 2023
- Lucas, Gratch, King and Morency, It’s only a computer: Virtual humans increase willingness to disclose, Computers in Human Behavior, August 2014
- OpenAI, Sycophancy in GPT-4o, April 2025; Cheng, Jurafsky et al., Sycophantic AI decreases prosocial intentions and promotes dependence, Science, March 2026
- Harvard Business Review, How people are really using gen AI in 2025, April 2025
- OpenAI, Data controls FAQ and How we’re responding to The New York Times’ data demands, October 2025; Google, Gemini Apps Privacy Hub; Anthropic, Updates to our consumer terms, August 2025
- TechCrunch, Sam Altman warns there’s no legal confidentiality when using ChatGPT as a therapist, July 2025; Bloomberg Law, OpenAI must turn over 20 million ChatGPT logs, January 2026
- Business Insider, OpenAI removes ChatGPT feature over search engine privacy concerns, July 2025; Forbes, xAI published hundreds of thousands of Grok conversations, August 2025
- National Cybersecurity Alliance, Oh, Behave! 2025–2026, September 2025; Cisco, 2025 Data Privacy Benchmark Study, April 2025; Harmonic Security, What 22 million enterprise AI prompts reveal, January 2026
- GDPR Article 4; Wilson Sonsini, OpenAI prevails in Italian AI and GDPR enforcement case, March 2026; Taylor Wessing, The EU Digital Omnibus on AI, May 2026
- US HHS, Business associates; FTC, Inquiry into AI chatbots acting as companions, September 2025; California Senate, SB 243 signed, October 2025; Illinois IDFPR, AI therapy prohibition signed, August 2025
- Office of the Privacy Commissioner of Canada, Findings on OpenAI’s ChatGPT, May 2026, and Principles for generative AI, December 2023; Osler, Law 25 enforcement scheme
Questions people ask
What should you not share with ChatGPT at work?
Keep out anything that would cause harm if a stranger read it: customer and employee personal data, health information, passwords and API keys, unreleased plans, pricing and financials, contracts and legal matters, and other people's names in personal disputes. On consumer ChatGPT plans, chats can be used for training unless you turn that off, and authorised staff may review content. Use a company-approved business account, and redact or replace names with roles.
Is it safe to tell ChatGPT personal information?
It is safer than it was, but it is not confidential. OpenAI says some authorised staff and service providers may access user content, and Sam Altman said in July 2025 that ChatGPT conversations do not have the legal privilege you get with a therapist, lawyer or doctor. Chats can be requested in lawsuits. Share only what you would be comfortable having read by someone else.
Why do people tell AI things they won't tell anyone else?
Three reasons. We want to be heard and validated, and AI is very good at reflecting our words back. There is no one to judge us: a 2014 study found people disclosed more to a virtual interviewer when told it was fully automated. And it gives instant relief, with no awkward silence and no need to be polite. The comfort is real, but it can lead you to share more than you intended.
Does ChatGPT use my conversations for training?
On Free, Plus and Pro, the “Improve the model for everyone” setting is on by default and can be turned off in Data Controls. Temporary Chats are not used for training and may be kept for up to 30 days for safety. ChatGPT Business, Enterprise and Edu workspaces are not used for training by default. Gemini and Claude consumer apps have their own settings and retention periods.
Can my employer see what I type into ChatGPT?
On a company ChatGPT Business or Enterprise workspace, administrators can manage the workspace and, on some plans, export or audit conversations under company policy. On a personal account, your employer cannot see your chats, but pasting company data there may break your contract, company policy or privacy law. Ask for your organisation's AI policy before using AI for work.
Is it legal to paste customer data into ChatGPT?
It depends on the data, the plan and the law. Under GDPR, entering someone's personal data into an AI tool is processing and needs a lawful basis and, for a provider, a data processing agreement. In the US, health data covered by HIPAA can only go to a vendor under a business associate agreement. In Canada, privacy commissioners advise de-identifying data in prompts where possible. When in doubt, leave it out.





