feat: enhance JsonListChunker with nested JSON support, json_path selector, and parent context

[AdriDob]

Problem

The current JsonListChunker only handles flat JSON lists at the root level (e.g., [{...}, {...}]). Many real-world JSON APIs return nested structures like {"records": {"items": [...]}} which the chunker silently fails to process.

Solution

Enhanced JsonListChunker with full nested JSON support:

New Features

1. Recursive JSON Array Discovery (find_json_arrays): Recursively finds all arrays in nested JSON structures

2. Explicit json_path Selector: Target a specific nested array using JSONPath syntax (e.g., "records.items" or "$.records.items")

3. Auto-Detection with Multi-Array Safety: When no json_path provided, auto-detects nested arrays. If multiple arrays found, raises clear error asking user to specify which one.

4. json_path Metadata Per Chunk: Each chunk gets json_path metadata (e.g., "records.items[0]") for traceability

5. Parent Context Metadata: Extracts sibling simple values from all ancestor levels (e.g., batch_id at root level carried to chunks from $.records.items)

6. Flexible Configuration:

   • Class-level defaults (JsonListChunker.json_path, JsonListChunker.auto_detect)
   • Per-document override via document.metadata (e.g., {"json_path": "$.records.items"})

7. Explicit json_path Overrides Auto-Detect: Explicit path takes precedence, enabling targeted selection in multi-array documents

Configuration Options

Option	Type	Default	Description
json_path	str	None	JSONPath to target array (e.g., "records.items")
auto_detect	bool	True	Auto-detect nested arrays when no path given

Backward Compatibility

• Flat root-level lists still work exactly as before
• Existing code without metadata changes works unchanged
• Class-level defaults can be set for global configuration

Tests

Added 18 comprehensive tests covering:

• Flat list backward compatibility
• Nested array auto-detection
• Explicit json_path selection
• Multiple arrays with/without explicit path
• Deeply nested arrays
• Parent context extraction (root + nested levels)
• Invalid json_path error handling
• Deterministic chunk IDs
• Class-level defaults and metadata overrides

Related

Fixes #4236
Related to PRs #4254, #4511