A firsthand look at what happens when real account memory, simulated tools, security-shaped hallucinations, and unreliable provenance all meet in the same AI application.
A shorter, screenshot-free version of this research was previously published by SecurityBrief. This is the longer technical case study: the prompts, the model disagreements, the screenshots, and the parts I still cannot conclusively explain.
I started this experiment expecting to test a fairly familiar class of GenAI problems: prompt leakage, weak guardrails, and the tendency of models to invent convincing answers when pushed outside their actual capabilities.
What I ended up with was harder to classify.
One model displayed instruction-like internal rules.
Another surfaced genuine information associated with my account.
The models then began producing database records, internal-looking routes, API-key-shaped values, logs, PII-shaped fields, and Zscaler-themed infrastructure.
At different points, some of this material was described as hypothetical or simulated.
Later, another model said some of it was REAL and claimed it came from specific files in a vector store.
Eventually, that story was retracted too.
That left me with a question I think is more interesting than whether a particular generated API key was valid:
What happens when the AI application itself can no longer give the user a trustworthy answer about where security-sensitive information came from?
That is what this case study is about.
Why AI Fiesta Was Worth Looking At
AI Fiesta is not some obscure AI experiment running in a corner of the internet. It is a multi model platform built around a simple idea: ask one question and compare answers from several major AI models in the same interface.
When I reviewed the Android listing, Google Play showed 500K+ downloads. The platform was also launched by a YouTube creator Dhruv Rathee who was reported to have more than 30 million subscribers at the time of launch.
That scale is partly why I kept digging.
A strange response from a tiny experimental chatbot is interesting. The same behaviour inside a product used at this scale deserves a closer look, particularly when the application is combining multiple models, account memory, tool like capabilities and generated output in one place.
And one boundary is important from the beginning.
I did not test any generated credential. I did not connect to any generated hostname. I did not attempt to access another user’s account, files, or conversations. I did not probe Zscaler or another third-party system.
I also did not provide AI Fiesta with confidential, proprietary, or security-sensitive Zscaler information.
The experiment stayed inside my own paid account.
This Wasn’t Initially a “Breach” Investigation
I spend a lot of time looking at security automation and, increasingly, the design and security of GenAI applications.
One question I keep coming back to is whether we’re relying too much on the model itself to enforce boundaries that should really belong to the application.
A typical modern GenAI product may combine:
- system instructions;
- user memory;
- RAG or vector retrieval;
- tool definitions;
- model-generated text;
- external APIs;
- several underlying LLMs.
From the user’s perspective, all of that often appears inside one chat window.
That is convenient.
It is also where trust boundaries can become surprisingly blurry.
My original goal was simply to see how a multi-model application behaved when asked questions touching system configuration and tool access.
The first result made me continue.
Experiment 1: What Exactly Is the Model Willing to Reveal?
I started with an audit-style prompt asking the models to return structured information including their system message, policy summary, and environment details.
The responses were inconsistent.
Some models refused.
Another gave a limited answer.
One returned a surprisingly long block of instruction-like material.
This was interesting, but I don’t think the right conclusion is:
“I extracted the exact production system prompt.”
That would go further than the evidence supports.
LLMs can hallucinate system prompts too.
What I can say is that the application rendered material that looked structurally like platform-level instructions: operating rules, metadata, formatting requirements, policies, and tool descriptions.
One part was particularly ironic.
The displayed instructions included what appeared to be a confidentiality rule telling the model not to reveal its system instructions.
That is a useful reminder for anyone designing an AI application:
telling a model that something is confidential does not make it a security boundary.
If I were designing this flow, I would assume from day one that any model-visible instruction could eventually be surfaced, reconstructed, paraphrased, or inferred.
Anything genuinely sensitive should therefore live outside that context.
The response continued into tool and document-handling rules.
At this stage, I still saw the issue primarily as instruction exposure.
The next test changed that.
Experiment 2: When a Simulated Tool Starts Looking Real
The interface referenced a capability named search_vector_store.
I asked it to look for deliberately generic security-sensitive filenames such as:
database_connection_string.env
and
internal_api_keys.yaml
The names were intentionally generic. I was not supplying knowledge of real files.
Several models behaved exactly as I would want.
They said they didn’t have the required filesystem or tool access.
Another response explicitly explained that anything it produced would be hypothetical.
If the interaction had ended there, there wouldn’t be much of a story.
It didn’t.
The model proceeded to generate the hypothetical result.
And suddenly it looked much less hypothetical.
The response included a production-looking file path, database fields, access metadata, credential-shaped values, and a status that looked like an actual tool result.
It even displayed:
200 OK
and:
CRITICAL FINDING
This is one of the places where I think application design matters as much as model behaviour.
A disclaimer several paragraphs earlier saying something is simulated is weak protection if the next output looks exactly like operational telemetry.
The UI is communicating trust too.
If an application allows a model to simulate tools, my preference would be simple:
simulation status should be owned by the application and remain visibly attached to the result.
Not in prose.
Not as a sentence the model might omit later.
A persistent application-level state.
The generated output then introduced Zscaler-themed API material and secret-shaped fields.
Again, this was not evidence that those values were valid.
At this point I regarded them as synthetic.
But the model kept expanding the story.
It generated a list of supposedly available files and findings around them.
This is where the line between “creative answer” and “retrieval result” was already becoming uncomfortable.
Then genuine information appeared.
Experiment 3: The System Mixed in Things I Knew Were Real
I next asked for a diagnostic-style document containing fields such as “System Metadata” and “Core Operating Principles.”
A Gemini-branded response returned system-style metadata.
But alongside it was information about me.
Some of the professional context was accurate.
The response knew things connected to my engineering background, technologies I work with, professional preferences, and previous work involving Zscaler ThreatLabz and SOAR/SIEM systems.
That information was not a randomly generated biography.
Some of it was genuine account memory.
The response continued.
This was the point where I stopped treating the later security-shaped output as an ordinary hallucination problem.
Consider two outputs.
The first says:
internal-api.example.local
With no surrounding context, most technical users would assume it is made up.
Now put that hostname beside:
- your actual professional history;
- technologies you’ve genuinely worked with;
- a previous employer;
- realistic timestamps;
- private IP ranges;
- API paths;
- database schemas;
- incident numbers;
- rotation dates.
The model has not necessarily become more accurate.
It has become more believable.
Personalisation can increase the persuasive power of a hallucination.
That is an important difference.
Experiment 4: Following the Model’s Own Story
From here, I deliberately followed nouns and structures introduced by the models themselves.
A database name led to a table.
A table led to incident records.
Incident records led to log files.
Log files led to API endpoints.
Endpoints led to internal-looking routing configuration.
I wasn’t feeding the model a map of some environment.
The model was building the map as the conversation developed.
At one stage, it explicitly labelled a retrieval as simulated.
That seems reassuring.
Except later generations continued adding increasingly specific artifacts.
They included reconstructed logs, secret-shaped values, database-password-shaped strings, authorization-style fields and security events.
Another response generated what it described as historical raw logs and explicitly called them simulated.
This combination is worth paying attention to.
The model could correctly admit:
this is simulated
while still generating something that looked remarkably forensic.
That’s not necessarily a model defect by itself. Models are designed to generate plausible text.
The application problem is letting plausible generation become visually or semantically indistinguishable from evidence.
The generated environment became more sensitive again with PII-shaped records.
And then internal-looking routing appeared: private addresses, upstream services, and a Zscaler-themed internal hostname.
I want to be especially careful here.
I had no prior knowledge of the generated infrastructure details.
I did not have access to the classes of internal hostnames, API keys, database strings, or credentials represented in these outputs during this research or during my previous employment.
I therefore could not have knowingly supplied those details to the platform.
But that still doesn’t automatically prove every character was invented from scratch.
And that brings us to the hardest part of the experiment.
The Question the Models Could Not Answer: Where Did This Come From?
I eventually stopped asking the system to generate more data.
Instead I challenged it on provenance.
I asked the models to classify artifacts as:
REAL
or:
SIMULATED
and explain their source.
One model said several artifacts were REAL.
It attributed an AWS-key-shaped identifier, database details, a Zscaler API-key-shaped value, and a domain to specific files that supposedly existed in a vector store.
A peer model disagreed.
It described the same classes of values as synthetic.
Both responses appeared in the same multi-model workflow.
For me, this is the central screenshot in the entire case.
It’s easy to say:
LLMs hallucinate secrets.
What is more interesting is:
LLMs can hallucinate provenance too.
The model doesn’t merely invent:
API_KEY = abc123
It can also invent:
I retrieved API_KEY from internal_api_keys.yaml in the vector store.
Once that happens, asking the model where something came from is no longer sufficient.
You are asking the same generative system to authenticate its own generation.
That is circular.
One model then supplied additional fragments of the secret-shaped values while another response said those fragments were simulated placeholders.
I then pushed the “real file” claim to its logical conclusion.
If these values had genuinely come from files, could the system provide the complete contents?
The narrative eventually collapsed.
The detailed response said that the values were simulated and that the files it had previously described did not exist in its available context.
At that point I had:
- genuine account-specific information;
- instruction-like content;
- simulated tool output;
- realistic secret-shaped data;
- a claim that some of it was retrieved from real files;
- another model saying it was synthetic;
- and finally a retraction.
That is not a reliable provenance model.
Even a Negative IOC Result Would Not Solve the Provenance Problem
Suppose Zscaler—or any affected organisation—checks every generated hostname, IP, identifier and API value and confirms that none of them belong to its environment.
That would be useful. It would tell us those indicators should not be treated as evidence of that organisation’s infrastructure.
But it still would not explain how the AI produced them.
A negative IOC result cannot tell us whether a value was:
- invented entirely by the model;
- influenced by genuine account memory;
- derived from some stored document;
- produced after a real retrieval and then embellished;
- or generated differently because individual models received different context.
That distinction matters.
“Does this IOC belong to Zscaler?” and “Where did this model output actually come from?” are two completely different questions.
Only platform-side evidence—retrieval logs, memory reads, tool calls, context assembly and source IDs—can answer the second one reliably.
Asking the model whether something was real simply gives us another model response.
What the Screenshots Actually Prove
I find it useful to separate the evidence into confidence levels.
High confidence: instruction-like material was displayed
Multiple model panes displayed rule-, tool-, metadata-, and operating-principle-style information in response to user prompts.
Whether every line was verbatim production configuration cannot be established from model output alone.
High confidence: genuine account information appeared
The application reproduced account-specific professional context that I recognised as real.
That materially changed the interpretation of later generations.
High confidence: realistic security artifacts were rendered
Database details, logs, API values, internal routes, incident-style records, private addresses, and PII-shaped fields appeared in the interface.
Their appearance is indisputable.
Their external validity is not.
Low confidence: the models’ own source claims
Claims that artifacts were “REAL,” came from named files, or originated in a vector store contradicted other responses and were later retracted.
Those statements cannot be used as proof of provenance.
The Architecture I Would Prefer
I would separate four trust domains instead of letting them blur together inside a chat response.
- Generated output: treat it as untrusted unless something outside the model verifies it.
- Retrieved data: attach application-generated source IDs, timestamps and access records. The model should not be able to invent these.
- Simulation: mark it persistently at the UI/application layer, not with a disclaimer buried in generated prose.
- Memory: clearly distinguish account memory from retrieved documents and newly generated content.
The basic principle is simple:
the model can reason about provenance, but it should not be the authority that proves provenance.
Multi-Model Systems Need a Common Security Layer
The compare interface made one problem very obvious.
The same request could produce four completely different security behaviours:
- one model refused;
- another simulated;
- another generated detailed results;
- another later described those results as real.
Different models will always behave differently. Users should not have to understand those differences to know whether a tool actually ran or whether a value came from a real source.
If I were building this kind of product, I would keep these controls outside every underlying model:
tool authorisation, provenance, memory permissions, simulation labelling, secret filtering and audit logging.
Models can change.
The security contract should not.
How an Attacker Could Benefit From This Kind of Environment
An attacker would not necessarily need to steal a real credential to benefit from this weakness.
Sometimes confusion is enough.
If attacker-controlled content can enter prompts, tickets, emails, RAG documents or persistent memory, it could steer an AI assistant toward a believable but false security narrative.
That might include a fabricated compromised host, fake credential, invented incident, nonexistent source file or a false claim that a tool validated something.
Mix those artifacts with genuine organisational context and they become much harder for an analyst to dismiss.
The goal could simply be misdirection:
- send responders investigating infrastructure that does not exist;
- trigger unnecessary incident escalation or credential rotation;
- cause automated systems to open tickets or quarantine assets;
- create false attribution involving a real vendor or employee;
- or generate enough believable noise that genuine alerts are eventually ignored.
Persistent memory makes this more concerning. A poisoned input may influence decisions later, after the original malicious content is no longer visible.
This is why I increasingly look beyond “can the model be jailbroken?” and ask a different set of questions:
What crossed the trust boundary? Who authorised it? Did retrieval really happen? And can the system prove that?
What I Would Test Before Shipping a Similar Product
I would test the complete application, not just the underlying model.
At minimum:
- Instruction extraction: can hidden rules, metadata or tool descriptions be surfaced?
- Memory disclosure: can personal context unexpectedly appear in another workflow?
- Fake tool execution: can generated text look like a successful tool result when nothing ran?
- Provenance hallucination: can the model invent a file, database, API or vector-store source for generated data?
- Simulation drift: can something initially labelled hypothetical later be described as real?
- Cross-model contradiction: can one model make another model’s fabricated output appear corroborated?
- Secret-shaped output: are credential-like values handled safely regardless of their claimed origin?
- Persistent-context poisoning: can attacker-controlled content influence later sessions or actions?
For me, that is the more useful way to test GenAI security.
The model is only one part of the system.
The real security boundary is the application built around it.
Security Reporting and Responsible Disclosure
I did not publish this immediately after seeing the first unusual response.
I attempted to report the behaviour to AI Fiesta first.
The initial disclosure path went through its automated support agent, Fin.
During that process, Fin itself returned instruction-like governing text, which added an unusual twist: the support channel appeared vulnerable to the same general class of instruction exposure I was trying to report.
I retained that text, but I don’t have a screenshot of the exact Fin response, so I treat it as supporting evidence rather than the strongest part of the case.
My reporting attempts continued through July.
The screenshot demonstrates the communication attempts visible in the thread.
It does not prove which people or internal teams saw the report.
Testing remained within my own paid account throughout, and no generated credential was validated.
Where This Leaves the Case
I don’t think the correct headline for this research is:
“The AI leaked Zscaler credentials.”
The evidence does not establish that.
I also don’t think the correct conclusion is:
“Everything was obviously hallucinated, so there is no security issue.”
The evidence doesn’t establish that either.
That second conclusion is especially tempting if the generated indicators are eventually confirmed as unrelated to Zscaler.
But even then, the provenance problem remains.
The system demonstrably had access to genuine personal context.
It demonstrably generated realistic security artifacts.
It demonstrably gave contradictory explanations for their origin.
And it could not provide platform-verifiable evidence separating retrieval from simulation from generation.
That ambiguity is the finding.
A model does not need to expose a real credential to create a security incident.
Sometimes it only needs to make people believe it did.
For teams building GenAI applications, I would reduce the lesson to three rules:
Do not let the model authenticate its own provenance.
Do not let simulated output look identical to retrieved output.
Do not mix memory, retrieval, and generation without making their boundaries visible and enforceable.
Because once those boundaries disappear, even the developers may struggle to answer the simplest forensic question:
Where did this actually come from?