Skip to content

fix(runtime-filter): Preserve scan pruning filters and DATE bucket ha… - #67426

Open
foxtail463 wants to merge 1 commit into
apache:masterfrom
foxtail463:fix/runtime-filter-bucket-pruning-correctness
Open

fix(runtime-filter): Preserve scan pruning filters and DATE bucket ha…#67426
foxtail463 wants to merge 1 commit into
apache:masterfrom
foxtail463:fix/runtime-filter-bucket-pruning-correctness

Conversation

@foxtail463

Copy link
Copy Markdown
Contributor

Problem Summary:
When column statistics were missing or unknown, RuntimeFilterPruner could remove runtime filters that were still able to eliminate entire partitions or hash buckets. Remote TYPE_DATE IN filters were also deserialized as DATETIME values, producing a distribution hash different from the original DATE value and potentially selecting incorrect tablets.

Solution:
Introduce a unified scan-range pruning capability covering both partition and bucket pruning. When row-level selectivity cannot be estimated, retain joins containing runtime filters with this capability; decisions based on valid statistics remain unchanged.

Deserialize TYPE_DATE exact-set values with DATE semantics so their type and distribution hash survive remote runtime-filter serialization. Enable runtime-filter pruning in the existing bucket-pruning regression case so it exercises the real optimizer path.

@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@foxtail463

Copy link
Copy Markdown
Contributor Author

run buildall

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-H: Total hot run time: 16755 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 89b08b656f81ce1c4fbceb50768af6f2a0ebb9a6, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17619	3034	3038	3034
q2	2104	265	221	221
q3	10234	848	514	514
q4	4674	253	199	199
q5	7671	585	379	379
q6	137	114	95	95
q7	598	501	375	375
q8	9231	848	845	845
q9	3396	2397	2363	2363
q10	6567	876	716	716
q11	390	201	184	184
q12	606	257	194	194
q13	18129	1509	1145	1145
q14	162	153	138	138
q15	q16	430	397	364	364
q17	1389	948	832	832
q18	3132	2229	2228	2228
q19	1243	845	779	779
q20	375	290	199	199
q21	5603	1724	1873	1724
q22	333	270	227	227
Total cold run time: 94023 ms
Total hot run time: 16755 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	3429	3352	3341	3341
q2	524	385	386	385
q3	2184	2289	2128	2128
q4	1193	1174	887	887
q5	2169	2095	2123	2095
q6	172	119	88	88
q7	1043	910	898	898
q8	1595	1420	1407	1407
q9	3114	3095	3093	3093
q10	1878	1789	1622	1622
q11	353	268	255	255
q12	444	434	349	349
q13	1484	1563	1157	1157
q14	172	174	167	167
q15	q16	409	388	360	360
q17	3589	3275	3196	3196
q18	4773	4381	4686	4381
q19	875	800	931	800
q20	1011	971	852	852
q21	3861	3104	3249	3104
q22	392	342	320	320
Total cold run time: 34664 ms
Total hot run time: 30885 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-DS: Total hot run time: 81688 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 89b08b656f81ce1c4fbceb50768af6f2a0ebb9a6, data reload: false

query5	4272	406	329	329
query6	378	137	126	126
query7	4975	400	235	235
query8	303	124	116	116
query9	8688	2860	2873	2860
query10	409	215	173	173
query11	5378	1051	898	898
query12	116	68	70	68
query13	1193	417	325	325
query14	6131	2160	2070	2070
query14_1	1956	1957	1925	1925
query15	168	117	107	107
query16	914	376	346	346
query17	796	476	351	351
query18	2331	337	234	234
query19	162	134	102	102
query20	73	66	70	66
query21	198	100	86	86
query22	5284	5344	5347	5344
query23	6678	6276	5914	5914
query23_1	5980	6054	6274	6054
query24	7318	1095	751	751
query24_1	762	781	801	781
query25	406	284	231	231
query26	1232	220	131	131
query27	2798	404	252	252
query28	4693	1492	1513	1492
query29	910	438	338	338
query30	244	153	130	130
query31	823	400	331	331
query32	128	70	77	70
query33	454	224	173	173
query34	994	866	493	493
query35	386	424	336	336
query36	570	584	548	548
query37	119	82	72	72
query38	999	849	809	809
query39	521	485	478	478
query39_1	480	454	475	454
query40	208	95	80	80
query41	59	57	59	57
query42	75	75	73	73
query43	237	238	208	208
query44	1010	549	588	549
query45	112	106	105	105
query46	769	889	556	556
query47	765	757	706	706
query48	313	299	222	222
query49	540	247	193	193
query50	724	265	190	190
query51	8207	8034	8197	8034
query52	69	70	60	60
query53	201	214	154	154
query54	228	184	176	176
query55	79	65	57	57
query56	196	169	186	169
query57	692	783	664	664
query58	214	174	178	174
query59	1221	1211	1085	1085
query60	256	190	168	168
query61	114	118	124	118
query62	377	207	190	190
query63	170	146	137	137
query64	2790	659	623	623
query65	1641	1595	1598	1595
query66	1911	269	230	230
query67	10203	9604	9769	9604
query68	3027	1175	695	695
query69	337	225	191	191
query70	667	608	605	605
query71	244	178	169	169
query72	2356	1793	1576	1576
query73	660	580	338	338
query74	2003	1225	1131	1131
query75	1191	1091	949	949
query76	2381	736	506	506
query77	248	263	203	203
query78	3904	3579	3165	3165
query79	2745	876	576	576
query80	1619	337	314	314
query81	534	153	131	131
query82	640	120	96	96
query83	281	213	200	200
query84	300	113	91	91
query85	824	366	311	311
query86	478	178	182	178
query87	1002	985	897	897
query88	2915	2108	2095	2095
query89	286	194	175	175
query90	2160	132	128	128
query91	132	122	103	103
query92	99	70	73	70
query93	1881	1112	695	695
query94	629	251	215	215
query95	504	248	229	229
query96	819	565	277	277
query97	1075	1043	979	979
query98	180	133	131	131
query99	411	345	312	312
Total cold run time: 179579 ms
Total hot run time: 81688 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
ClickBench: Total hot run time: 14.56 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 89b08b656f81ce1c4fbceb50768af6f2a0ebb9a6, data reload: false

query1	0.01	0.01	0.00
query2	0.08	0.03	0.04
query3	0.25	0.11	0.11
query4	1.60	0.10	0.09
query5	0.17	0.15	0.17
query6	1.26	0.71	0.68
query7	0.04	0.01	0.00
query8	0.05	0.03	0.03
query9	0.28	0.22	0.21
query10	0.34	0.35	0.34
query11	0.16	0.12	0.11
query12	0.15	0.11	0.13
query13	0.31	0.30	0.31
query14	0.46	0.45	0.45
query15	0.37	0.36	0.34
query16	0.22	0.24	0.22
query17	0.67	0.70	0.72
query18	0.18	0.17	0.17
query19	1.23	1.14	1.08
query20	0.02	0.01	0.02
query21	15.44	0.17	0.13
query22	5.03	0.04	0.04
query23	16.15	0.26	0.10
query24	3.04	0.32	0.24
query25	0.14	0.04	0.03
query26	0.73	0.17	0.12
query27	0.04	0.03	0.02
query28	3.66	0.57	0.26
query29	12.42	3.15	2.57
query30	0.25	0.12	0.13
query31	2.75	0.35	0.18
query32	3.54	0.32	0.23
query33	1.45	1.38	1.38
query34	15.43	2.20	1.78
query35	1.75	1.72	1.73
query36	0.48	0.28	0.28
query37	0.06	0.04	0.04
query38	0.04	0.04	0.03
query39	0.03	0.02	0.03
query40	0.12	0.08	0.08
query41	0.08	0.02	0.02
query42	0.03	0.03	0.02
query43	0.03	0.02	0.02
Total cold run time: 90.54 s
Total hot run time: 14.56 s

@hello-stephen

Copy link
Copy Markdown
Contributor

BE Regression && UT Coverage Report

Increment line coverage 100% (0/0) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 76.23% (34688/45502)
Line Coverage 61.30% (391618/638822)
Region Coverage 57.45% (328941/572533)
Branch Coverage 58.21% (149974/257645)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants