Skip to content

[fix](fe) Reject conflicting path partition columns - #67408

Open
0AyanamiRei wants to merge 1 commit into
apache:masterfrom
0AyanamiRei:fix/doris-25098-path-partition-column-collision
Open

[fix](fe) Reject conflicting path partition columns#67408
0AyanamiRei wants to merge 1 commit into
apache:masterfrom
0AyanamiRei:fix/doris-25098-path-partition-column-collision

Conversation

@0AyanamiRei

Copy link
Copy Markdown
Contributor

What problem does this PR solve?

Issue Number: None

Related PR: None

Problem Summary: A file table-valued function could append a path partition column whose name already existed in the inferred file schema. The duplicate schema made direct column references ambiguous and allowed SELECT * to reach an internal backend error. Reject case-insensitive conflicts while constructing the schema so the invalid configuration fails during analysis.

Release note

File table-valued functions now reject path partition columns that conflict with existing file columns.

Check List (For Author)

  • Test: Regression test and manual test
    • Regression test: test_path_partition_column_collision
    • Manual test: DESC FUNCTION and SELECT * with a conflicting local CSV path partition
  • Behavior changed: Yes. Conflicting path partition columns now return a clear analysis error.
  • Does this need documentation: No

### What problem does this PR solve?

Issue Number: None

Related PR: None

Problem Summary: A file table-valued function could append a path partition column whose name already existed in the inferred file schema. The duplicate schema made direct column references ambiguous and allowed SELECT * to reach an internal backend error. Reject case-insensitive conflicts while constructing the schema so the invalid configuration fails during analysis.

### Release note

File table-valued functions now reject path partition columns that conflict with existing file columns.

### Check List (For Author)

- Test: Regression test and manual test
    - Regression test: test_path_partition_column_collision
    - Manual test: DESC FUNCTION and SELECT * with a conflicting local CSV path partition
- Behavior changed: Yes. Conflicting path partition columns now return a clear analysis error.
- Does this need documentation: No
@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@0AyanamiRei

Copy link
Copy Markdown
Contributor Author

run buildall

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-H: Total hot run time: 16892 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 567e775de2999d2ffb39f4d0a71c111312b824a6, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17566	3079	3066	3066
q2	2098	266	231	231
q3	10213	839	512	512
q4	4676	243	201	201
q5	7682	559	389	389
q6	139	111	95	95
q7	513	502	388	388
q8	9253	926	943	926
q9	3486	2395	2356	2356
q10	6515	868	741	741
q11	386	198	186	186
q12	615	261	192	192
q13	18118	1510	1150	1150
q14	156	152	135	135
q15	q16	427	392	365	365
q17	1315	884	804	804
q18	3102	2217	2222	2217
q19	1272	873	750	750
q20	385	277	209	209
q21	5666	1746	1845	1746
q22	328	276	233	233
Total cold run time: 93911 ms
Total hot run time: 16892 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	3438	3362	3337	3337
q2	514	387	372	372
q3	2229	2525	2131	2131
q4	1177	1151	909	909
q5	2175	2135	2116	2116
q6	170	126	89	89
q7	1050	955	831	831
q8	1608	1410	1410	1410
q9	3105	3126	3062	3062
q10	1835	1782	1590	1590
q11	371	268	254	254
q12	448	414	347	347
q13	1483	1520	1161	1161
q14	168	184	163	163
q15	q16	389	399	358	358
q17	3612	3332	3151	3151
q18	4839	4399	4727	4399
q19	845	869	875	869
q20	1001	954	830	830
q21	3722	3021	3205	3021
q22	398	346	318	318
Total cold run time: 34577 ms
Total hot run time: 30718 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-DS: Total hot run time: 81066 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 567e775de2999d2ffb39f4d0a71c111312b824a6, data reload: false

query5	4264	405	333	333
query6	394	137	139	137
query7	4934	428	238	238
query8	307	120	120	120
query9	8690	2860	2839	2839
query10	380	221	186	186
query11	5376	1036	911	911
query12	124	71	72	71
query13	1204	434	333	333
query14	6017	2180	2057	2057
query14_1	1959	1949	1936	1936
query15	170	118	111	111
query16	914	363	362	362
query17	797	455	358	358
query18	2327	316	235	235
query19	169	139	111	111
query20	70	69	67	67
query21	201	101	87	87
query22	5405	5338	5299	5299
query23	6645	6114	5886	5886
query23_1	5971	5987	6113	5987
query24	7355	1090	758	758
query24_1	766	773	781	773
query25	429	305	266	266
query26	1235	238	130	130
query27	2784	417	262	262
query28	4673	1492	1501	1492
query29	945	441	346	346
query30	257	155	129	129
query31	830	395	337	337
query32	141	81	84	81
query33	462	216	185	185
query34	1024	844	492	492
query35	398	409	350	350
query36	568	565	560	560
query37	120	81	73	73
query38	995	835	814	814
query39	491	478	471	471
query39_1	437	470	451	451
query40	209	93	78	78
query41	57	56	56	56
query42	75	73	78	73
query43	249	244	212	212
query44	1028	547	561	547
query45	111	109	106	106
query46	772	833	529	529
query47	748	750	690	690
query48	307	305	226	226
query49	530	235	185	185
query50	731	254	188	188
query51	7939	7827	8025	7827
query52	68	67	65	65
query53	194	194	143	143
query54	248	186	145	145
query55	77	59	58	58
query56	191	162	165	162
query57	648	617	692	617
query58	209	161	156	156
query59	1214	1241	1102	1102
query60	245	175	165	165
query61	116	113	113	113
query62	368	193	187	187
query63	203	142	143	142
query64	2682	682	648	648
query65	1560	1585	1587	1585
query66	1855	255	210	210
query67	9999	9699	9472	9472
query68	2922	1232	761	761
query69	338	225	187	187
query70	672	607	613	607
query71	257	175	163	163
query72	2335	1733	1568	1568
query73	660	612	351	351
query74	1988	1211	1115	1115
query75	1172	1080	936	936
query76	2355	723	529	529
query77	254	260	218	218
query78	4043	3620	3150	3150
query79	2790	767	584	584
query80	1581	328	270	270
query81	498	152	129	129
query82	792	120	97	97
query83	285	210	187	187
query84	301	107	90	90
query85	820	352	299	299
query86	408	177	174	174
query87	998	959	888	888
query88	2760	2088	2082	2082
query89	305	199	174	174
query90	1949	121	129	121
query91	134	122	94	94
query92	74	69	60	60
query93	1655	1122	709	709
query94	632	254	203	203
query95	552	314	224	224
query96	777	561	266	266
query97	1097	1022	968	968
query98	173	134	142	134
query99	440	340	308	308
Total cold run time: 178441 ms
Total hot run time: 81066 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
ClickBench: Total hot run time: 14.55 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 567e775de2999d2ffb39f4d0a71c111312b824a6, data reload: false

query1	0.00	0.00	0.00
query2	0.08	0.03	0.04
query3	0.27	0.11	0.11
query4	1.62	0.10	0.09
query5	0.18	0.15	0.15
query6	1.21	0.70	0.67
query7	0.04	0.00	0.00
query8	0.05	0.03	0.03
query9	0.28	0.21	0.22
query10	0.34	0.34	0.36
query11	0.18	0.11	0.12
query12	0.15	0.12	0.11
query13	0.30	0.30	0.31
query14	0.46	0.45	0.45
query15	0.38	0.35	0.35
query16	0.24	0.23	0.22
query17	0.67	0.69	0.66
query18	0.17	0.17	0.16
query19	1.12	1.14	1.19
query20	0.01	0.01	0.02
query21	15.44	0.16	0.12
query22	5.03	0.04	0.04
query23	16.15	0.26	0.11
query24	3.10	0.30	0.27
query25	0.10	0.04	0.04
query26	0.73	0.17	0.13
query27	0.05	0.04	0.03
query28	3.59	0.56	0.29
query29	12.40	3.16	2.56
query30	0.25	0.11	0.12
query31	2.75	0.36	0.16
query32	3.49	0.33	0.22
query33	1.38	1.37	1.35
query34	15.34	2.16	1.75
query35	1.74	1.72	1.72
query36	0.49	0.29	0.29
query37	0.07	0.04	0.04
query38	0.06	0.03	0.03
query39	0.04	0.02	0.03
query40	0.12	0.08	0.07
query41	0.08	0.02	0.02
query42	0.03	0.02	0.02
query43	0.03	0.02	0.03
Total cold run time: 90.21 s
Total hot run time: 14.55 s

@hello-stephen

Copy link
Copy Markdown
Contributor

FE UT Coverage Report

Increment line coverage 0.00% (0/3) 🎉
Increment coverage report
Complete coverage report

@hello-stephen

Copy link
Copy Markdown
Contributor

FE Regression Coverage Report

Increment line coverage 27.27% (3/11) 🎉
Increment coverage report
Complete coverage report

@0AyanamiRei

Copy link
Copy Markdown
Contributor Author

run buildall

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants