This happened a couple of years or so ago.

"These numbers are wrong," the manager wrote in the public company Slack channel, tagging me about an entire multiple-tab dashboard containing several reports that I had created.

Thirty minutes or so later, while trying to understand what it was that either them or me did wrong, they finally revealed it: they had fed the entire dashboard to ChatGPT and, alas, the bot's response was that all the numbers were incorrect.

It turns out that the manager did not know about the relevant and necessary criteria to produce it. They didn't bother to ask. They didn't even bother to read the - really short - documentation on it!

So I pointed them out to that information. And there they went, back to the oracle, repeating the question — now with the relevant information attached — and their god confirmed: the data was correct, after all.

"Oh I see," they said.

"But you don't," I thought.


Former Pingu Child Actor (BANISHED)'s avatar

lol this is gonna burst so fucking hard

r/analytics
• IOh
We just found out our AI has been making up
analytics data for 3 months and I'm gonna
throw up.
Support
So we've been using an Al agent since November to
answer leadership questions about metrics. It seemed
amazing at first fast answers, detailed explanations,
everyone loved it.
I just found out it's been hallucinating numbers this
entire time.
Our VP of sales made territory decisions based on
data that didn't exist. Our CFO showed the board a
deck with fake insights. The Al was just inventing
plausible sounding percentages.
I only caught it by accident when someone asked me
to double check something. I started digging, and
holy shit, it's bad.