A customer rarely writes:
Feature request: Add CSV export to the analytics dashboard.
They are more likely to write:
We prepare our monthly report in Excel, so right now I have to copy each number manually. Is there an easier way?
A human product manager immediately understands the underlying request.
The customer wants an export capability.
A software system sees something more complicated:
- A description of an existing workflow
- An implied feature request
- A preferred destination
- A repeated manual task
- A possible source of frustration
- No explicit use of the words “feature request”
This is why building an AI system for customer feedback involves more than summarizing messages.
The real task is converting unstructured conversations into structured product data without removing the context that makes the feedback useful.
The Input Is a Conversation, Not a Form
Traditional feedback boards receive relatively structured input:
Title: Export analytics to CSV
Description: We need to use the data in Excel.
Category: Analytics
Support systems receive conversations:
Customer: I am putting together our monthly report.
Agent: Which part is taking the most time?
Customer: Copying the numbers from your dashboard into Excel.
We need to do it for each client.
Agent: At the moment, the dashboard does not support exports.
Customer: That may become a problem as we add more clients.
The useful feedback is distributed across several messages.
Processing each message independently creates several problems.
The first customer message contains no product request. The second provides the operational context. The third reveals the missing capability. The final message adds a growth and retention signal.
The correct unit of analysis is therefore rarely one sentence. It is usually a conversational window.
Step 1: Normalize Every Source Into a Common Event Format
Feedback may originate from Intercom, Slack, email, a review platform, a survey, or an uploaded call transcript.
Each source has a different payload.
The first architectural decision is to avoid making the analysis pipeline understand every integration independently. Instead, transform source-specific payloads into one internal event format.
A simplified event might look like this:
{
"source": "intercom",
"external_conversation_id": "conv_123",
"account_id": "account_42",
"customer_id": "customer_91",
"occurred_at": "2026-07-10T14:05:00Z",
"messages": [
{
"speaker": "customer",
"text": "We prepare our reports in Excel..."
},
{
"speaker": "agent",
"text": "The dashboard does not currently support exports."
}
],
"metadata": {
"plan": "business",
"account_status": "active",
"source_url": "internal-source-reference"
}
}
This normalization layer has three responsibilities:
- Preserve the original content.
- Resolve the customer and account where possible.
- retain a reference to the source conversation.
The original message should never be discarded after an AI-generated insight is created.
Without provenance, the resulting data becomes a collection of claims that nobody can verify.
Step 2: Separate Feedback Detection From Feedback Extraction
A common implementation mistake is asking one model to perform every task at once:
Read this conversation, find the feedback, classify it, summarize it, decide its importance, merge it with existing requests, and recommend what we should build.
This produces impressive demos and fragile production systems.
A more reliable approach separates the workflow into stages.
The first stage answers a narrow question:
Does this conversation contain actionable product feedback?
Possible outputs include:
{
"contains_feedback": true,
"feedback_types": [
"feature_request",
"workflow_friction"
],
"confidence": 0.92
}
The second stage extracts one or more individual feedback items.
{
"items": [
{
"type": "feature_request",
"title": "Export analytics data",
"problem": "The customer manually copies dashboard data into Excel.",
"requested_outcome": "Download or transfer analytics data for reporting.",
"evidence": [
{
"message_index": 0,
"quote": "We prepare our reports in Excel..."
}
]
}
]
}
Detection and extraction are related, but they fail differently.
Detection failures cause feedback to be missed or irrelevant conversations to be processed.
Extraction failures distort what the customer actually requested.
Separating the stages makes each one easier to evaluate.
Step 3: Use Schemas as Product Contracts
An LLM response should not be treated as an essay.
It should be treated as untrusted input entering a software system.
That means every output requires:
- A defined schema
- Type validation
- Required fields
- Enumerated categories
- Length constraints
- Fallback behavior
- Versioning
For example:
type FeedbackItem = {
type:
| "feature_request"
| "bug"
| "usability_issue"
| "complaint"
| "integration_request"
| "churn_signal";
title: string;
problem: string;
requestedOutcome?: string;
evidence: EvidenceReference[];
confidence: number;
schemaVersion: "1.0";
};
A valid JSON response does not guarantee a correct interpretation.
It does, however, prevent malformed output from silently corrupting downstream systems.
The schema also forces the product team to define what it means by “feedback.”
That sounds obvious, but different teams often use the same word for very different things.
A support question is not automatically a feature request.
A customer saying “I cannot export the report” might mean:
- The export feature does not exist
- The feature exists but is difficult to find
- The feature is broken
- The customer lacks permission
- The customer expects a different format
The classification system must preserve those distinctions.
Step 4: Preserve Evidence at the Smallest Useful Level
Saving the conversation URL is not enough.
A product manager reviewing an insight should be able to see which exact messages support it.
One option is to store character ranges or message indexes:
{
"conversation_id": "conv_123",
"message_id": "msg_3",
"start_offset": 0,
"end_offset": 74
}
The interface can then display:
AI-generated insight: Customers need analytics exports for recurring reports.
Supported by:
“We prepare our monthly report in Excel, so right now I have to copy each number manually.”
This design changes how users interact with AI output.
Instead of asking users to trust a summary, it lets them inspect the evidence.
That becomes especially important when multiple conversations are later grouped into one product insight.
Step 5: Enrich the Feedback Without Polluting It
Customer and account metadata can make feedback more useful:
{
"plan": "enterprise",
"monthly_revenue_band": "1000-5000",
"lifecycle_stage": "renewal",
"customer_segment": "agency",
"region": "europe"
}
However, this metadata should remain separate from the extracted customer statement.
The customer’s feedback is evidence.
Revenue, plan, and lifecycle information are prioritization context.
Combining them too early can bias the extraction stage.
For example, the model should not reinterpret an ambiguous sentence as more important merely because it came from a high-value account.
A useful separation is:
What did the customer say?
↓
What product problem does it represent?
↓
Who is affected?
↓
How should the team prioritize it?
Each question deserves its own stage.
Step 6: Make the Pipeline Asynchronous and Idempotent
External integrations retry webhooks.
Users edit conversations.
Models time out.
A production pipeline must assume that the same event may be delivered more than once.
A simplified architecture looks like this:
Source webhook
↓
Raw event storage
↓
Normalization queue
↓
Feedback detection
↓
Structured extraction
↓
Validation
↓
Candidate deduplication
↓
Human-reviewable insight
Every stage should be independently retryable.
An idempotency key can be generated from the source, external event ID, and content version:
const idempotencyKey =
`${source}:${externalConversationId}:${contentVersion}`;
The raw payload should be stored before analysis begins. This provides a replay path when prompts, schemas, or models change.
Without replayability, improving the extraction system requires waiting for new feedback instead of reprocessing historical data.
Step 7: Store Model and Prompt Versions
AI output can change even when the input remains the same.
For every extraction, store:
{
"model": "model-name",
"prompt_version": "feedback-extraction-v7",
"schema_version": "1.0",
"created_at": "2026-07-10T14:06:00Z"
}
When a classification appears wrong, the team should be able to answer:
- Which model produced it?
- Which prompt was used?
- Which schema was expected?
- Was the conversation truncated?
- Was this the first attempt or a retry?
- Has a human corrected it?
Otherwise, debugging becomes guesswork.
The Hardest Failure Modes
The most dangerous errors are not malformed JSON responses. They are plausible interpretations that are subtly wrong.
Implied requests
“We still use a spreadsheet for this.”
This may imply a missing workflow, but the desired solution is unclear.
Multiple requests in one conversation
A customer might report a bug, request an integration, and describe a permissions problem in the same thread.
Solution versus problem
“Please add a Zapier integration.”
The underlying need may be sending data to a CRM. Treating the proposed solution as the complete problem can lead to poor product decisions.
Negation
“We do not need another dashboard.”
A weak classifier may detect “dashboard” as a request instead of recognizing the rejection.
Agent contamination
Support agents may suggest workarounds or speculate about future features. Those statements should not be attributed to the customer.
The Product Is the Evidence, Not the Summary
While building, we learned that the value of AI analysis does not come from generating polished summaries.
A polished but unverifiable summary is dangerous.
The useful output is a structured connection between:
- A product problem
- The original customer language
- The affected customers
- Related conversations
- The surrounding business context
That is what turns a collection of messages into product data.
The purpose of the system is not to replace product judgment.
It is to ensure that product judgment starts with complete, inspectable evidence.
Final Architecture
The complete flow can be represented as:
Intercom / Email / Slack / Reviews / Calls
↓
Source adapters
↓
Normalized conversation
↓
Feedback detection
↓
Structured item extraction
↓
Schema and evidence validation
↓
Similarity candidate generation
↓
Grouped product insight
↓
Human product decision
The most important principle is simple:
Never allow the generated insight to become detached from the conversation that produced it.
Once provenance is lost, customer feedback becomes another unreliable dataset.
When provenance is preserved, AI can make that dataset searchable, structured, and significantly easier to use.