Make sure SQL user-defined types are preserved in RecordMetadata

[hazefully]

This PR enhances the serialisation and deserialisation of a RecordLayerSchemaTemplate to/from a RecordMetaData by making sure that all user-defined SQL types (structs or enums) are preserved in the RecordMetadata and are available in the schema template when it is deserialised from the RecordMetaData.

Before this PR, only types which were used in a table were added, as a protobuf message type, to the FileDescriptor of the RecordMetaData. This means that if a user-defined type is only used as arguments of user-defined functions, or not used at all, it would not be preserved in the serialised FileDescriptor, making it not possible to use in temporary functions or the SQL query as an existing type. In addition, as explained in #4177 and #4317, defining compiled SQL functions or temporary macro functions with user-defined types as parameters didn't work as the deserialised schema template didn't maintain a way to access these user-defined types by name.

This PR fixes both problems, by making sure that all auxiliaryTypes in a RecordLayerSchemaTemplate are added to the FileDescriptor of the RecordMetaData whether they are used in a table or not. In addition, it adds a new generated message type called AuxiliaryTypeUnion the serialised FileDescriptor, which similar to the existing RecordTypeUnion message type provides a way to access these user-defined types by name in the deserialised RecordMetaData and RecordLayerSchemaTemplate without having to store the protobuf descriptor of the user-defined types more than once. The new generated type AuxiliaryTypeUnion has a field for each user-defined SQL type with the name of the user-defined type and the protobuf message type of the user-defined SQL type. This new message type is only added to RecordMetaDatas serialised from a RecordLayerSchemaTemplate.

This PR also includes the following minor fixes:

• When executing a RecordQueryIndexPlan or a RecordQueryCoveringIndexPlan in a continuation, where the index plan has comparisons which involve dynamic type, a runtime error would occur as the dynamic types were not added correctly to the type repository used for executing the continuation.
• In RoutineParser.java, when we parse SQL functions included in the metadata as raw SQL queries, we assert in the code that the parsed function is a compiled SQL function. There is no technical reason for this restriction, and relaxing this assertion allows us to attach user-defined macro functions as raw SQL queries in the metadata.

Resolves #4177, #4317.

[hazefully]

This is necessary because the old planner generates SimpleComparisons where the inner value is an enum and not a scalar value, which now throws when getDynamicTypes is called on an index plan.

[github-actions]

📊 Metrics Diff Analysis Report

Summary

• New queries: 3
• Dropped queries: 0
• Plan changed + metrics changed: 6
• Plan unchanged + metrics changed: 1

ℹ️ About this analysis

This automated analysis compares query planner metrics between the base branch and this PR. It categorizes changes into:

• New queries: Queries added in this PR
• Dropped queries: Queries removed in this PR. These should be reviewed to ensure we are not losing coverage.
• Plan changed + metrics changed: The query plan has changed along with planner metrics.
• Metrics only changed: Same plan but different metrics

The last category in particular may indicate planner regressions that should be investigated.

New Queries

Count of new queries by file:

• yaml-tests/src/test/resources/import-schema-template/with-auxiliary-types.metrics.yaml: 1
• yaml-tests/src/test/resources/user-defined-macro-function-tests.metrics.yaml: 2

Plan and Metrics Changed

These queries experienced both plan and metrics changes. This generally indicates that there was some planner change
that means the planning for this query may be substantially different. Some amount of query plan metrics change is expected,
but the reviewer should still validate that these changes are not excessive.

Total: 6 queries

Statistical Summary (Plan and Metrics Changed)

task_count:

• Average change: +76.3
• Average regression: +76.3
• Median change: +67
• Median regression: +67
• Standard deviation: 13.2
• Standard deviation of regressions: 13.2
• Range: +67 to +95
• Range of regressions: +67 to +95
• Queries changed: 6
• Queries regressed: 6

task_count % change distribution (6 queries, bin = 2%):

       Range                                  n
------------  ------------------------------  ---
[+46%, +48%)  ██████████████████████████████  4
[+48%, +50%)                                  0
[+50%, +52%)                                  0
[+52%, +54%)                                  0
[+54%, +56%)                                  0
[+56%, +58%)                                  0
[+58%, +60%)                                  0
[+60%, +62%)                                  0
[+62%, +64%)                                  0
[+64%, +66%)  ███████████████                 2

transform_count:

• Average change: +15.7
• Average regression: +15.7
• Median change: +14
• Median regression: +14
• Standard deviation: 2.4
• Standard deviation of regressions: 2.4
• Range: +14 to +19
• Range of regressions: +14 to +19
• Queries changed: 6
• Queries regressed: 6

transform_count % change distribution (6 queries, bin = 2%):

       Range                                  n
------------  ------------------------------  ---
[+34%, +36%)  ██████████████████████████████  4
[+36%, +38%)                                  0
[+38%, +40%)                                  0
[+40%, +42%)                                  0
[+42%, +44%)                                  0
[+44%, +46%)                                  0
[+46%, +48%)  ███████████████                 2

transform_yield_count:

• Average change: +10.7
• Average regression: +10.7
• Median change: +10
• Median regression: +10
• Standard deviation: 0.9
• Standard deviation of regressions: 0.9
• Range: +10 to +12
• Range of regressions: +10 to +12
• Queries changed: 6
• Queries regressed: 6

transform_yield_count % change distribution (6 queries, bin = 2%):

       Range                                  n
------------  ------------------------------  ---
[+76%, +78%)  ██████████████████████████████  4
[+78%, +80%)                                  0
[+80%, +82%)                                  0
[+82%, +84%)                                  0
[+84%, +86%)                                  0
[+86%, +88%)                                  0
[+88%, +90%)                                  0
[+90%, +92%)                                  0
[+92%, +94%)  ███████████████                 2

insert_new_count:

• Average change: +11.0
• Average regression: +11.0
• Median change: +10
• Median regression: +10
• Standard deviation: 1.4
• Standard deviation of regressions: 1.4
• Range: +10 to +13
• Range of regressions: +10 to +13
• Queries changed: 6
• Queries regressed: 6

insert_new_count % change distribution (6 queries, bin = 2%):

         Range                                  n
--------------  ------------------------------  ---
  [+82%, +84%)  ██████████████████████████████  4
  [+84%, +86%)                                  0
  [+86%, +88%)                                  0
  [+88%, +90%)                                  0
  [+90%, +92%)                                  0
  [+92%, +94%)                                  0
  [+94%, +96%)                                  0
  [+96%, +98%)                                  0
 [+98%, +100%)                                  0
[+100%, +102%)                                  0
[+102%, +104%)                                  0
[+104%, +106%)                                  0
[+106%, +108%)                                  0
[+108%, +110%)  ███████████████                 2

insert_reused_count:

• Average change: +2.0
• Average regression: +2.0
• Median change: +2
• Median regression: +2
• Standard deviation: 0.0
• Standard deviation of regressions: 0.0
• Range: +2 to +2
• Range of regressions: +2 to +2
• Queries changed: 6
• Queries regressed: 6

insert_reused_count % change distribution (6 queries, bin = 2%):

         Range                                  n
--------------  ------------------------------  ---
[+100%, +102%)  ██████████████████████████████  6

There were no queries with significant regressions detected.

Minor Changes (Plan and Metrics Changed)

In addition, there were 6 queries with minor changes.

Only Metrics Changed

These queries experienced only metrics changes without any plan changes. If these metrics have substantially changed,
then a planner change has been made which affects planner performance but does not correlate with any new outcomes,
which could indicate a regression.

Total: 1 query

Statistical Summary (Only Metrics Changed)

task_count:

• Average change: +213.0
• Average regression: +213.0
• Median change: +213
• Median regression: +213
• Standard deviation: 0.0
• Standard deviation of regressions: 0.0
• Range: +213 to +213
• Range of regressions: +213 to +213
• Queries changed: 1
• Queries regressed: 1

transform_count:

• Average change: +49.0
• Average regression: +49.0
• Median change: +49
• Median regression: +49
• Standard deviation: 0.0
• Standard deviation of regressions: 0.0
• Range: +49 to +49
• Range of regressions: +49 to +49
• Queries changed: 1
• Queries regressed: 1

transform_yield_count:

• Average change: +16.0
• Average regression: +16.0
• Median change: +16
• Median regression: +16
• Standard deviation: 0.0
• Standard deviation of regressions: 0.0
• Range: +16 to +16
• Range of regressions: +16 to +16
• Queries changed: 1
• Queries regressed: 1

insert_new_count:

• Average change: +26.0
• Average regression: +26.0
• Median change: +26
• Median regression: +26
• Standard deviation: 0.0
• Standard deviation of regressions: 0.0
• Range: +26 to +26
• Range of regressions: +26 to +26
• Queries changed: 1
• Queries regressed: 1

insert_reused_count:

• Average change: +1.0
• Average regression: +1.0
• Median change: +1
• Median regression: +1
• Standard deviation: 0.0
• Standard deviation of regressions: 0.0
• Range: +1 to +1
• Range of regressions: +1 to +1
• Queries changed: 1
• Queries regressed: 1

Significant Regressions (Only Metrics Changed)

There was 1 outlier detected. Outlier queries have a significant regression in at least one field. Statistically, this represents either an increase of more than two standard deviations above the mean or a large absolute increase (e.g., 100).

• yaml-tests/src/test/resources/user-defined-macro-function-tests.metrics.yaml:311: EXPLAIN select temp_constructor(r.u.w) from nested where id = 1
  • explain: SCAN([IS NESTED, EQUALS promote(@c14 AS LONG)]) | MAP ((_.R.U.W AS X, 5l AS _1) AS _0)
  • task_count: 204 -> 417 (+213)
  • transform_count: 57 -> 106 (+49)
  • transform_yield_count: 16 -> 32 (+16)
  • insert_new_count: 17 -> 43 (+26)
  • insert_reused_count: 1 -> 2 (+1)