Skip to content

[fix](fe) Preserve path partition columns with CSV schema - #67407

Open
0AyanamiRei wants to merge 1 commit into
apache:masterfrom
0AyanamiRei:fix/doris-25094-csv-schema-path-partitions
Open

[fix](fe) Preserve path partition columns with CSV schema#67407
0AyanamiRei wants to merge 1 commit into
apache:masterfrom
0AyanamiRei:fix/doris-25094-csv-schema-path-partitions

Conversation

@0AyanamiRei

@0AyanamiRei 0AyanamiRei commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

What problem does this PR solve?

Issue Number: None

Related PR: None

Problem Summary: File table-valued functions returned only explicitly declared CSV columns when csv_schema was set, so columns derived from path_partition_keys disappeared from the table schema. Append non-conflicting path partition columns to the explicit CSV schema in their declared order and reject case-insensitive conflicts before they can create a duplicate schema.

Release note

CSV file table-valued functions now expose path partition columns when csv_schema is specified and reject conflicting column names.

Check List (For Author)

  • Test: FE unit test and regression test
    • FE unit test: ExternalFileTableValuedFunctionTest (5 tests)
    • Regression test: test_csv_schema_with_path_partition_keys, including conflict rejection
  • Behavior changed: Yes. Path partition columns remain available with an explicit CSV schema.
  • Does this need documentation: No

@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@0AyanamiRei

Copy link
Copy Markdown
Contributor Author

run buildall

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-H: Total hot run time: 16828 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 465e988df0ad58b2d652b3a56c94542759bce9b1, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17587	3035	3032	3032
q2	2101	259	217	217
q3	10226	847	526	526
q4	4674	246	199	199
q5	7680	563	385	385
q6	135	113	93	93
q7	527	505	389	389
q8	9242	951	913	913
q9	3496	2352	2356	2352
q10	6510	1022	709	709
q11	400	201	182	182
q12	610	257	202	202
q13	18119	1559	1162	1162
q14	165	151	144	144
q15	q16	434	393	367	367
q17	1394	885	770	770
q18	3115	2226	2227	2226
q19	1265	866	742	742
q20	369	288	208	208
q21	5585	1774	1810	1774
q22	337	264	236	236
Total cold run time: 93971 ms
Total hot run time: 16828 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	3395	3343	3341	3341
q2	515	391	368	368
q3	2221	2788	2198	2198
q4	1179	1158	884	884
q5	2150	2110	2097	2097
q6	169	126	85	85
q7	1047	932	853	853
q8	1597	1407	1420	1407
q9	3147	3136	3089	3089
q10	1886	1786	1618	1618
q11	353	266	250	250
q12	451	428	337	337
q13	1476	1536	1135	1135
q14	176	168	168	168
q15	q16	396	397	363	363
q17	3537	3285	3193	3193
q18	4762	4385	4715	4385
q19	874	899	886	886
q20	986	950	834	834
q21	3720	3004	3262	3004
q22	393	353	318	318
Total cold run time: 34430 ms
Total hot run time: 30813 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-DS: Total hot run time: 81714 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 465e988df0ad58b2d652b3a56c94542759bce9b1, data reload: false

query5	4250	411	334	334
query6	377	144	126	126
query7	4948	405	241	241
query8	292	135	114	114
query9	8677	2910	2892	2892
query10	383	219	177	177
query11	5365	1058	913	913
query12	120	70	67	67
query13	1190	431	332	332
query14	6029	2189	2097	2097
query14_1	1976	1946	1955	1946
query15	179	124	111	111
query16	922	363	357	357
query17	810	442	369	369
query18	2347	315	247	247
query19	164	138	107	107
query20	76	70	69	69
query21	202	101	86	86
query22	5389	5224	5251	5224
query23	6646	6263	6068	6068
query23_1	6001	6089	5798	5798
query24	7337	1144	801	801
query24_1	771	782	787	782
query25	404	293	238	238
query26	1232	233	127	127
query27	2800	397	251	251
query28	4706	1492	1506	1492
query29	930	416	345	345
query30	248	157	129	129
query31	823	399	322	322
query32	128	76	68	68
query33	455	206	166	166
query34	972	801	479	479
query35	403	397	334	334
query36	554	614	523	523
query37	124	80	66	66
query38	996	830	832	830
query39	494	480	487	480
query39_1	471	450	444	444
query40	204	93	72	72
query41	53	54	51	51
query42	74	75	74	74
query43	251	241	213	213
query44	1026	530	545	530
query45	110	101	99	99
query46	766	855	533	533
query47	770	761	702	702
query48	314	306	231	231
query49	554	240	194	194
query50	780	264	199	199
query51	8024	8036	8069	8036
query52	72	68	62	62
query53	196	196	153	153
query54	218	189	158	158
query55	75	57	58	57
query56	189	157	151	151
query57	691	713	614	614
query58	193	151	160	151
query59	1204	1239	1084	1084
query60	231	182	173	173
query61	122	112	134	112
query62	389	205	178	178
query63	165	141	142	141
query64	2856	680	602	602
query65	1576	1608	1588	1588
query66	1930	269	226	226
query67	9994	9658	9769	9658
query68	2752	1147	758	758
query69	341	222	190	190
query70	665	607	606	606
query71	239	175	162	162
query72	2307	1858	1559	1559
query73	669	602	344	344
query74	1586	1232	1128	1128
query75	1149	1080	949	949
query76	2283	734	556	556
query77	260	260	209	209
query78	3873	3649	3289	3289
query79	1199	814	583	583
query80	1046	331	270	270
query81	469	152	136	136
query82	593	128	98	98
query83	286	215	197	197
query84	289	113	87	87
query85	829	366	292	292
query86	353	178	173	173
query87	1012	951	882	882
query88	2744	2102	2108	2102
query89	279	195	177	177
query90	1835	127	126	126
query91	128	161	102	102
query92	78	65	74	65
query93	1152	1153	745	745
query94	553	256	222	222
query95	529	246	300	246
query96	787	572	270	270
query97	1110	1071	1003	1003
query98	147	132	140	132
query99	428	346	309	309
Total cold run time: 174644 ms
Total hot run time: 81714 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
ClickBench: Total hot run time: 14.61 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 465e988df0ad58b2d652b3a56c94542759bce9b1, data reload: false

query1	0.01	0.00	0.00
query2	0.07	0.03	0.03
query3	0.25	0.11	0.11
query4	1.61	0.10	0.09
query5	0.17	0.17	0.16
query6	1.22	0.69	0.70
query7	0.04	0.01	0.00
query8	0.05	0.03	0.03
query9	0.29	0.21	0.22
query10	0.35	0.35	0.34
query11	0.17	0.12	0.12
query12	0.15	0.12	0.12
query13	0.32	0.31	0.30
query14	0.45	0.45	0.44
query15	0.37	0.35	0.34
query16	0.21	0.23	0.22
query17	0.68	0.69	0.69
query18	0.18	0.16	0.17
query19	1.18	1.15	1.13
query20	0.02	0.01	0.01
query21	15.45	0.15	0.12
query22	5.07	0.05	0.04
query23	16.17	0.25	0.10
query24	3.01	0.33	0.25
query25	0.12	0.03	0.03
query26	0.73	0.17	0.13
query27	0.05	0.03	0.03
query28	3.62	0.57	0.29
query29	12.47	3.18	2.58
query30	0.25	0.12	0.13
query31	2.76	0.37	0.17
query32	3.51	0.32	0.23
query33	1.72	1.35	1.48
query34	15.36	2.17	1.76
query35	1.76	1.75	1.70
query36	0.46	0.30	0.29
query37	0.07	0.04	0.04
query38	0.05	0.03	0.03
query39	0.03	0.03	0.03
query40	0.11	0.07	0.08
query41	0.07	0.02	0.03
query42	0.03	0.02	0.02
query43	0.04	0.03	0.02
Total cold run time: 90.7 s
Total hot run time: 14.61 s

### What problem does this PR solve?

Issue Number: None

Related PR: None

Problem Summary: File table-valued functions returned only explicitly declared CSV columns when csv_schema was set, so columns derived from path_partition_keys disappeared from the table schema. Append non-conflicting path partition columns to the explicit CSV schema in their declared order and reject case-insensitive conflicts before they can create a duplicate schema.

### Release note

CSV file table-valued functions now expose path partition columns when csv_schema is specified and reject conflicting column names.

### Check List (For Author)

- Test: FE unit test and regression test
    - FE unit test: ExternalFileTableValuedFunctionTest (5 tests)
    - Regression test: test_csv_schema_with_path_partition_keys, including conflict rejection
- Behavior changed: Yes. Path partition columns remain available with an explicit CSV schema.
- Does this need documentation: No
@0AyanamiRei
0AyanamiRei force-pushed the fix/doris-25094-csv-schema-path-partitions branch from 465e988 to b895d8a Compare September 2, 2026 03:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants