Chunking Strategies for Structured Data in RAG Systems

While developing RAG pipelines across various enterprise use cases, a stakeholder asked me a question that stopped me mid conversation: “Can our database become a source for our knowledge base, just like our SharePoint documents? Can RAG give us answers by querying the database directly?“

The honest answer is yes, but only if you rethink how you chunk the data before it enters the knowledge base.

Most RAG chunking strategies are optimized for prose, i.e. documentation, articles, support tickets, web content, etc. The moment you ingest a CSV or metadata catalog export, they break down.

Here’s why: a table with 50 rows becomes a single chunk whose embedding captures a blurred average of all rows. When a user asks “What is the city with ID=5?”, the retriever can’t isolate that specific row because the chunk represents everything and nothing at once.

This article covers six chunking strategies for structured data, with honest trade offs for each and guidance on when to use which.

Strategy 6 touches on the agentic SQL routing approach but does not cover full agentic SQL implementation in depth. I will write a separate article for the same.

Why Default Chunking Fails on Tables

Fixed-size (512 tokens) and sentence-based chunking assume contiguous text carries contextual meaning. That a paragraph depends on surrounding paragraphs. Tables violate every one of these assumptions:

• Rows are independent facts — there’s no relational “flow” between row 4 and row 5
• Each row is self-contained — it doesn’t need neighboring rows for interpretation
• A table embedding averages N unrelated facts — it becomes a weak match for any single fact query
• Fixed-size splits cut mid-row — producing chunks where half a record is orphaned from its values
• Overlap creates duplicate retrievals, not helpful continuity

The result: your knowledge base confidently returns “I don’t have enough information” for data that’s sitting right there.

Strategy 1: Row-Level Chunking

Each row becomes its own chunk, serialized with schema context so it’s self-explanatory:

Table: prod_db.us_cities

Columns: id, city_name, state, population, region

---

Record: id=5, city_name=Seattle, state=WA, population=737015, region=Pacific Northwest

In table us_cities, for this record: the id is 5; the city_name is Seattle;

The state is WA, the population is 737015, the region is Pacific Northwest.

The dual representation maximizes retrieval across different query phrasings. structured key=value + natural language prose

Pros: Highest retrieval precision for point queries

• Simple to implement
• Pairs naturally with metadata filtering (.metadata.json sidecars for table/record filtering)

Cons: Chunk explosion at scale.

• 100K rows means 100K chunks with 100K embedding calls
• Cannot answer aggregation queries (COUNT, SUM, AVG) since retrieval fetches top-k, not all qualifying rows
• Redundant schema headers in every chunk add ~30% storage overhead

Suitable for: I have seen it getting used for lookup queries on small to medium tables (<10K rows), catalog/reference data, configuration tables with high cardinality keys

Strategy 2: Small-Group Chunking (N Rows per Chunk)

Group 3–10 rows per chunk, ideally by a shared attribute (region, category, time window) rather than arbitrary sequential order:

Table: prod_db.us_cities | Region: Pacific Northwest
| id | city_name | state | population |
|----|-----------|-------|------------|
| 5  | Seattle   | WA    | 737015     |
| 8  | Portland  | OR    | 652503     |
| 12 | Boise     | ID    | 235684     |

The grouping strategy determines the success or failure of this approach. Sequential grouping (rows 1–5, 6–10, etc.) can be arbitrary and often become useless. Category based grouping (all cities in a region, all orders from a customer) aligns chunks with likely query patterns.

Pros: 10x fewer chunks than row level

• Enables within group comparisons (“largest city in the Pacific Northwest?”)
• Lower embedding cost
• Works well when queries naturally target categories

Cons: Embedding dilution.

• The vector represents a blend of N rows, reducing precision for single-row lookups
• Requires deliberate chunk level data modelling effort. Wrong grouping can cause wrong rows to be co-located causing missed retrievals
• Cross group queries still fail (“Compare Seattle vs New York” if they’re in different chunks)

Suitable for: Medium tables (1K–100K rows) where row level creates unmanageable chunk counts, data with natural groupings, comparison queries within a category.

Strategy 3: Schema-Aware Semantic Chunking

Instead of treating all chunks equally, create specialized chunks at different levels of abstraction:

• Schema chunk: Table structure, column descriptions, data types, constraints
• Summary chunk: Row count, value ranges, distributions, top-N values
• Detail chunks: Individual rows (can combine with Strategy 1 or 2)

[SCHEMA] Table us_cities: 50 rows. Columns: id (PK, INT), city_name (VARCHAR),
         state (CHAR 2), population (INT, range 200K-8.3M), region (VARCHAR, 5 distinct)

[SUMMARY] 50 cities across 5 regions. Largest: New York (8.3M). Smallest: Boise (236K).
          Region breakdown: Northeast(12), South(15), Midwest(10), West(8), Pacific NW(5).

[DETAIL] id=5, city_name=Seattle, state=WA, population=737015, region=Pacific Northwest

This handles a class of questions the other strategies miss:

“What columns does the cities table have?”

“How many records are in us_cities?”

“What’s the population range?”

Pros: Answers metadata/overview questions that row-level chunks can’t

• The LLM understands the full dataset shape, which improves response grounding.
• Complements row level or group chunking as a layered approach

Cons: Summary chunks go stale when data changes (requires refresh pipeline)

• Computing meaningful summaries requires preprocessing logic
• For small tables, the summary may be larger than the data itself

Suitable for: Large tables needing both overview and detail retrieval, data catalog discovery (“what data do we have?”), combining as a layer on top of Strategies 1 or 2

Strategy 4: Entity-Centric Chunking

Restructure around entities — pre-join related tables at ingest time so one chunk contains everything known about a single entity:

Entity: Seattle (City ID: 5)

Source tables: us_cities, us_metro_economics, us_employers

State: WA | Population: 737,015 | Region: Pacific Northwest
Metro area GDP: $413B | Growth rate: 4.2% YoY
Major employers: Amazon, Boeing, Microsoft
Founded: 1851 | Area: 83.78 sq mi

This eliminates the multi-hop retrieval problem: instead of hoping the retriever fetches chunks from 3 different tables, all relevant data is pre-assembled.

Pros: One retrieval = complete entity context, no multi-hop needed

• Pre-joined data means faster, more complete answers
• Natural-language format embeds better than raw tabular notation

Cons: Complex preprocessing requires understanding FKs, join paths, and entity resolution

• Data duplication (shared attributes repeated across entity chunks)
• Any source table change triggers regeneration of all affected entity chunks
• Doesn’t handle analytical queries across entities

Suitable for: Customer/product/account data, multi table datasets with clear relational entity relationships, CRM style queries (“tell me everything about customer X”). I have seen it getting used for RAGs associated with Customer 360 kind of solutions.

Strategy 5: Hierarchical Chunking (Parent-Child)

Create a two-level hierarchy: parent chunks (partition-level summaries) that reference child chunks (individual rows):

[PARENT] Region: Pacific Northwest | 5 cities | Total pop: 2.1M
         Children: pnw_row_001 through pnw_row_005
[CHILD]  Parent: pacific_northwest | id=5, Seattle, WA, 737015

Retrieval first matches parent chunks (to understand scope and narrow the partition), then fetches relevant child chunks for specific details. This mimics how humans browse: scan the index, then drill into the section.

Pros: Efficient narrowing — partition first, then drill into rows

• Handles both “overview of region X” and “specific city in region X”
• Natural fit for data already partitioned by date, geography, or category

Cons: Requires multi-pass retrieval orchestration (not natively supported by most KB APIs)

• Parent-child references must stay in sync. Deletions/additions require parent updates
• More complex than flat chunking strategies

Suitable for: Partitioned datasets (by date, region, category), drill-down query patterns, tables with natural hierarchies

Comparison Matrix

Strategy	Lookup Precision	Aggregation	Scalability	Implementation Complexity
S1-Row-Level	Very High	Not Supported	Low	Low
S2-Small-Group	Medium	Not Supported	Medium	Low
S3 – Schema Aware	High	Very low. Summary only	High	Medium
S4 – Entity Centric	High	Not Supported	Medium	Medium
S5 – Hierarchical	High	Low	High	Medium
S6- Beyond RAG	Very High	Very High	Very High	High

Strategy 6: Beyond RAG

Accept that RAG alone cannot handle all structured data queries. RAG was not purpose built to answer database style queries. With the Agentic approach you can route each query to the engine best suited for it.

User Query → Bedrock Agent (Intent Classifier)

├── Lookup / factual       → RAG (row-level chunks in Knowledge Base)

└── Aggregation / analytical → Text-to-SQL (Lambda → Athena over Glue Catalog)

Route to RAG	Route to SQL
“City with ID=5?”	“How many cities have pop > 1M?”
“Describe the Seattle record”	“Average population by region”
Specific entity lookups	COUNT, SUM, AVG, GROUP BY, TOP-N, JOINs

The Agent’s instructions define routing logic, and the model distinguishes between “give me a specific record” from “compute something across records” without a separate classifier.

Pros: Best of both worlds retrieval precision of RAG + computational power of SQL

• Aggregation works natively (the database computes, not the LLM)
• No chunk explosion on the SQL path
• Scales to millions of rows

Cons: Two systems to build and maintain (KB + Athena + routing Agent)

• Athena latency adds 2–10s per query
• LLM-generated SQL can be syntactically valid but semantically wrong
• Ambiguous queries may route to the wrong engine
• Must keep schema context in the agent prompt synced with actual Glue Catalog

Suitable for: Production systems with mixed query patterns, large datasets (>100K rows), enterprise use cases requiring accurate numerical answers, data platforms built on Glue + Athena

Best Practices

• Always include schema context (table name, columns) in every chunk
• Use metadata filtering to narrow retrieval scope by table or partition
• Include both structured (key=value) and prose representations
• Don’t fight RAG’s limitations, route analytical queries to SQL
• Test with your actual user queries, not hypothetical ones

Practical Recommendations

After designing many RAG applications both at POC and Production scale, my recommendation is to start small to understand user query requirements at POC/MVP scale. Then move towards Agentic Approach.

Start here: Row-Level Chunking (Strategy 1) with schema headers. It solves the most common failure mode (point lookups returning nothing) and takes an afternoon to implement.

Graduate to Hybrid RAG + SQL (Strategy 6) when users start asking aggregation questions. RAG fundamentally cannot COUNT or SUM across all rows. It retrieves the top-k most similar, not all qualifying.

Layer Schema Aware chunks (Strategy 3) on top of whatever base strategy you choose they’re essentially free and dramatically improve the LLM’s understanding of your data.

Structured data is where most RAG implementations quietly fail, not because the technology is wrong, but because the chunking strategy was designed for prose, and not for structured data. The six approaches here aren’t a menu to pick one from. They are a layered toolkit. Start simple, measure where retrieval breaks down, and add complexity only where your data and query patterns demand it.