Document ARRAY_AGG()

docs/sphinx/source/reference/Aggregates.rst

@@ -9,6 +9,7 @@ This section documents the system-defined **aggregate functions**. These functio
 .. toctree::
     :maxdepth: 1
 
+    Functions/aggregate_functions/array_agg
     Functions/aggregate_functions/avg
     Functions/aggregate_functions/bitmap_construct_agg
     Functions/aggregate_functions/count

docs/sphinx/source/reference/Functions/aggregate_functions/array_agg.diagram

@@ -0,0 +1,20 @@
+Diagram(
+    Terminal('ARRAY_AGG'),
+    Terminal('('),
+    Optional(
+        Terminal('ALL'),
+        'skip'
+    ),
+    NonTerminal('expression'),
+    Optional(
+        Sequence(
+            Choice(0,
+                Terminal('IGNORE'),
+                Terminal('RESPECT'),
+            ),
+            Terminal('NULLS'),
+        ),
+        'skip'
+    ),
+    Terminal(')'),
+)

docs/sphinx/source/reference/Functions/aggregate_functions/array_agg.rst

@@ -0,0 +1,184 @@
+=========
+ARRAY_AGG
+=========
+
+.. _array-agg:
+
+Collects the values of an expression across the rows of a group into a single array.
+
+Syntax
+======
+
+.. raw:: html
+    :file: array_agg.diagram.svg
+
+Parameters
+==========
+
+``expression``
+    The value collected from each row of the group. May be of any type except ``ARRAY``. The result is an array of the argument’s type.
+
+``ALL``
+    Collects every value of ``expression``, which is the default behavior when no set quantifier is present.
+
+``IGNORE NULLS``
+    Causes ``NULL`` values of ``expression`` to be omitted from the resulting array.
+
+``RESPECT NULLS``
+    Causes ``NULL`` values of ``expression`` to be collected as array elements. This is the default when no null-treatment clause is present. This behavior is subject to limitations; see the note on ``NULL`` handling under :ref:`Important Notes <array-agg-important-notes>`.
+
+Returns
+=======
+
+Returns an array whose elements are the values of ``expression`` in the group. The order of elements within the array is unspecified.
+
+The element type of the array is non-nullable when ``IGNORE NULLS`` is used, or when ``expression`` itself is non-nullable. Otherwise (that is, for a nullable ``expression`` with ``RESPECT NULLS`` behavior, which is the default) the element type is nullable. (However, see the note below regarding a current limitation on ``NULL`` elements in arrays.)
+
+The behavior on empty input depends on whether a ``GROUP BY`` clause is present:
+
+* Without ``GROUP BY``, aggregating over an empty input returns a single row whose array value is ``NULL``.
+* With ``GROUP BY``, aggregating over an empty input returns no rows.
+
+A group that does contain rows, but whose ``expression`` values are all ``NULL``, returns an empty array ``[]`` rather than ``NULL`` under ``IGNORE NULLS``. This holds whether or not a ``GROUP BY`` clause is present.
+
+Examples
+========
+
+Setup
+-----
+
+For these examples, assume we have a ``sales`` table:
+
+.. code-block:: sql
+
+    CREATE TABLE sales (
+        id BIGINT,
+        product STRING,
+        region STRING,
+        amount BIGINT,
+        PRIMARY KEY (id)
+    )
+
+    CREATE INDEX product_idx ON sales(product)
+
+    INSERT INTO sales VALUES
+        (1, 'Widget', 'North', 100),
+        (2, 'Widget', 'South', 150),
+        (3, 'Gadget', 'North', 200),
+        (4, 'Gadget', 'South', NULL),
+        (5, 'Widget', 'North', 120)
+
+The ``product_idx`` index is needed for the ``GROUP BY product`` query to be planned; see the note on required indexes under :ref:`Important Notes <array-agg-important-notes>`.
+
+Collecting all values
+---------------------
+
+The following query collects all amounts in the table into a single array.
+
+.. code-block:: sql
+
+    SELECT ARRAY_AGG(amount IGNORE NULLS) AS amounts FROM sales
+
+.. list-table::
+    :header-rows: 1
+
+    * - :sql:`amounts`
+    * - :json:`[200, 100, 150, 120]`
+
+Note that the ``NULL`` value in row 4 is omitted from the array.
+
+Note also that the elements do not appear in ``id`` order. They are collected in whatever order the rows happen to be read in, and here the query is served by a scan of ``product_idx``, which visits the ``Gadget`` row before the ``Widget`` rows. Adding or removing an index may therefore change the order of the elements within the array.
+
+ARRAY_AGG() with GROUP BY
+-------------------------
+
+The following query collects amounts per product.
+
+.. code-block:: sql
+
+    SELECT product, ARRAY_AGG(amount IGNORE NULLS) AS amounts
+      FROM sales
+     GROUP BY product
+
+.. list-table::
+    :header-rows: 1
+
+    * - :sql:`product`
+      - :sql:`amounts`
+    * - :json:`"Gadget"`
+      - :json:`[200]`
+    * - :json:`"Widget"`
+      - :json:`[100, 150, 120]`
+
+The ``Gadget`` group contains two rows, but the ``NULL`` amount is omitted, so its array has a single element.
+
+ARRAY_AGG() versus unnesting
+----------------------------
+
+Array aggregation can be viewed as the inverse operation of unnesting an array. The following example unnests an array literal into a stream of rows and then collects those rows back with ``ARRAY_AGG()``, reproducing the original array elements (although the order in which they come back is not guaranteed).
+
+.. code-block:: sql
+
+    SELECT ARRAY_AGG(x) AS numbers
+      FROM (SELECT a FROM VALUES ([2, 1, -2, 3, -2, 1, 2]) AS T(a)) AS sq,
+           sq.a AS x
+
+.. list-table::
+    :header-rows: 1
+
+    * - :sql:`numbers`
+    * - :json:`[2, 1, -2, 3, -2, 1, 2]`
+
+See :ref:`Unnesting <unnesting>` for the unnesting syntax used by the inner query.
+
+ARRAY_AGG() in a correlated subquery
+------------------------------------
+
+To collect a per-parent array of related child values, use a correlated subquery in the ``FROM`` clause. For the following example, assume a ``parent`` table and a ``child`` table joined on ``pid``:
+
+.. code-block:: sql
+
+    CREATE TABLE parent (pid BIGINT, name STRING, PRIMARY KEY (pid))
+
+    CREATE TABLE child (cid BIGINT, pid BIGINT, val BIGINT, PRIMARY KEY (cid))
+
+    CREATE INDEX child_by_pid ON child(pid)
+
+    INSERT INTO parent VALUES (1, 'a'), (2, 'b'), (3, 'c')
+
+    INSERT INTO child VALUES (1, 1, 100), (2, 1, 200), (3, 2, 300), (4, 2, NULL)
+
+The following query collects the ``val`` values of the children of each parent.
+
+.. code-block:: sql
+
+    SELECT p.pid, sq.vals
+      FROM parent p,
+           (SELECT ARRAY_AGG(c.val IGNORE NULLS) AS vals FROM child c WHERE c.pid = p.pid) sq
+
+.. list-table::
+    :header-rows: 1
+
+    * - :sql:`pid`
+      - :sql:`vals`
+    * - :json:`1`
+      - :json:`[100, 200]`
+    * - :json:`2`
+      - :json:`[300]`
+    * - :json:`3`
+      - :json:`null`
+
+Parent 2 has two children, but the ``NULL`` value of the second one is omitted, so its array has a single element. Parent 3 has no matching child rows at all, so its array is ``NULL`` rather than empty.
+
+.. _array-agg-important-notes:
+
+Important notes
+===============
+
+* **Required indexes**: In general, ``GROUP BY`` queries require an appropriate index to be executed. See :ref:`Indexes <index_definition>` for details on creating indexes that support ``GROUP BY`` operations.
+* **ARRAY_AGG() in indexes**: ``ARRAY_AGG()`` itself cannot currently be materialized in an index. Defining an index over it, as in ``CREATE INDEX idx AS SELECT ARRAY_AGG(val) FROM tab GROUP BY grp``, raises an ``UNSUPPORTED_OPERATION`` error.
+* **Element order**: The order of the elements within the returned array is unspecified. Elements are collected in whatever order the rows are read in, which depends on the plan used to execute the query—in particular on which index is used, if any. You therefore cannot rely on the order. There is currently no way to request a particular order, since an in-call ``ORDER BY`` clause is not supported yet. This limitation is tracked by `Issue #4498 <https://github.com/FoundationDB/fdb-record-layer/issues/4498>`_.
+* **NULL handling**: An array cannot currently hold ``NULL`` elements. This is due to a limitation at the level of the FDB Record Layer, tracked by `Issue #3646 <https://github.com/FoundationDB/fdb-record-layer/issues/3646>`_. A query that uses the default ``RESPECT NULLS`` behavior (including when no null-treatment clause is present) will fail at run time with an ``UNSUPPORTED_OPERATION`` error as soon as a ``NULL`` is encountered. To avoid this potential error, use ``IGNORE NULLS`` to omit ``NULL`` values from the array.
+* **Arrays of arrays**: An ``ARRAY``-typed argument would produce an array of arrays, which is not supported. ``ARRAY_AGG()`` over an ``ARRAY`` column raises an ``UNSUPPORTED_OPERATION`` error. This limitation is tracked by `Issue #4167 <https://github.com/FoundationDB/fdb-record-layer/issues/4167>`_. To collect nested collections, you can wrap the inner array in a struct, as in ``ARRAY_AGG((rid, tags))``.
+* **DISTINCT**: The ``DISTINCT`` set quantifier is not supported yet. The parser accepts ``ARRAY_AGG(DISTINCT «expression» …)`` but raises an ``UNSUPPORTED_QUERY`` error. This limitation is tracked by `Issue #4499 <https://github.com/FoundationDB/fdb-record-layer/issues/4499>`_.
+* **Subqueries**: ``ARRAY_AGG()`` may be used in a correlated ``FROM``-clause subquery, as shown in `ARRAY_AGG() in a correlated subquery`_ above, but not in a scalar subquery in the ``SELECT`` projection list. The latter, for example ``SELECT p.pid, (SELECT ARRAY_AGG(c.val IGNORE NULLS) FROM child c WHERE c.pid = p.pid) FROM parent p``, raises a ``SYNTAX_ERROR``. That is a general limitation of scalar subqueries in projections, not specific to ``ARRAY_AGG()``.

yaml-tests/src/test/java/DocumentationQueriesTests.java

@@ -34,6 +34,11 @@ void aggregateFunctionsDocumentationQueriesTests(YamlTest.Runner runner) throws
         runner.runYamsql(PREFIX + "/aggregate-functions-documentation-queries.yamsql");
     }
 
+    @TestTemplate
+    void arrayAggDocumentationQueriesTests(YamlTest.Runner runner) throws Exception {
+        runner.runYamsql(PREFIX + "/array-agg-documentation-queries.yamsql");
+    }
+
     @TestTemplate
     void betweenOperatorQueriesTests(YamlTest.Runner runner) throws Exception {
         runner.runYamsql(PREFIX + "/between-operator-queries.yamsql");

yaml-tests/src/test/resources/array-agg-tests.yamsql

@@ -28,10 +28,7 @@ schema_template:
     CREATE TABLE T1 (id BIGINT, grp BIGINT, val BIGINT, PRIMARY KEY (id))
     CREATE INDEX T1_grp AS SELECT grp FROM T1 ORDER BY grp
 
-    CREATE TABLE parent (pid BIGINT, name STRING, PRIMARY KEY (pid))
-    CREATE TABLE child (cid BIGINT, pid BIGINT, val BIGINT, PRIMARY KEY (cid))
     CREATE TABLE keywords (array_agg BIGINT, respect BIGINT, PRIMARY KEY (array_agg))
-    CREATE INDEX child_by_pid AS SELECT pid FROM child ORDER BY pid
 
     CREATE TYPE AS STRUCT Struct1 (a BIGINT, b BIGINT)
     CREATE TABLE T2 (id BIGINT, grp BIGINT, s Struct1, PRIMARY KEY (id))
@@ -58,15 +55,6 @@ setup:
                     (6, 30, NULL)
     - query: INSERT INTO keywords
              VALUES (1, 2)
-    - query: INSERT INTO parent
-             VALUES (1, 'a'),
-                    (2, 'b'),
-                    (3, 'c')
-    - query: INSERT INTO child
-             VALUES (1, 1, 100),
-                    (2, 1, 200),
-                    (3, 2, 300),
-                    (4, 2, NULL)
     - query: INSERT INTO T2
              VALUES (1, 10, (1, 2)),
                     (2, 10, (3, 4)),
@@ -248,21 +236,6 @@ test_block:
       - query: SELECT sq.agg[1] FROM (SELECT ARRAY_AGG(tags) AS agg FROM doc_rev) sq
       - error: UNSUPPORTED_OPERATION
 ---
-test_block:
-  name: array-agg-correlated-subquery
-  tests:
-    -
-      # A correlated ARRAY_AGG() subquery in a SELECT projection is currently not supported and fails with a parse-time
-      # syntax error. A scalar subquery is not allowed as a select-list expression (the grammar only permits EXISTS/IN
-      # subqueries there).
-      - query: SELECT p.pid, (SELECT ARRAY_AGG(c.val) FROM child c WHERE c.pid = p.pid) AS vals FROM parent p
-      - error: SYNTAX_ERROR
-    -
-      # Supported rewrite: A correlated ARRAY_AGG() subquery in the FROM clause. This is the recommended alternative
-      # for the projection form above. Row `pid=3` has no children, so its aggregate is NULL.
-      - query: SELECT p.pid, sq.vals FROM parent p, (SELECT ARRAY_AGG(c.val IGNORE NULLS) AS vals FROM child c WHERE c.pid = p.pid) sq
-      - result: [{1, [100, 200]}, {2, [300]}, {3, !null _}]
----
 # Tests involving element types whose runtime representation differs from their protobuf one. The accumulator has to
 # convert such types, both when it serializes its partial state and when it hands its result to the enclosing record
 # constructor.

yaml-tests/src/test/resources/documentation-queries/array-agg-documentation-queries.yamsql

@@ -0,0 +1,106 @@
+#
+# array-agg-documentation-queries.yamsql
+#
+# This source file is part of the FoundationDB open source project
+#
+# Copyright 2021-2026 Apple Inc. and the FoundationDB project authors
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+---
+options:
+  supported_version: !current_version
+---
+# The `sales` table and its `product_idx` index are the ones shown in the ARRAY_AGG() documentation page. The index is
+# what allows the GROUP BY query below to be planned, since a streaming aggregation needs its input sorted on the
+# grouping column.
+# The `parent` and `child` tables back the “ARRAY_AGG() in a correlated subquery” section. The `doc_rev` table backs the
+# “Arrays of arrays” note, whose examples use `rid` and `tags`.
+schema_template:
+    CREATE TABLE sales (id BIGINT, product STRING, region STRING, amount BIGINT, PRIMARY KEY (id))
+    CREATE INDEX product_idx ON sales(product)
+
+    CREATE TABLE parent (pid BIGINT, name STRING, PRIMARY KEY (pid))
+    CREATE TABLE child (cid BIGINT, pid BIGINT, val BIGINT, PRIMARY KEY (cid))
+    CREATE INDEX child_by_pid ON child(pid)
+
+    CREATE TABLE doc_rev (rid BIGINT, title STRING, tags STRING ARRAY, PRIMARY KEY (rid))
+---
+setup:
+  steps:
+    - query: INSERT INTO sales VALUES
+             (1, 'Widget', 'North', 100),
+             (2, 'Widget', 'South', 150),
+             (3, 'Gadget', 'North', 200),
+             (4, 'Gadget', 'South', NULL),
+             (5, 'Widget', 'North', 120)
+    - query: INSERT INTO parent VALUES (1, 'a'), (2, 'b'), (3, 'c')
+    - query: INSERT INTO child VALUES (1, 1, 100), (2, 1, 200), (3, 2, 300), (4, 2, NULL)
+    - query: INSERT INTO doc_rev VALUES (10, 'draft', ['internal']), (20, 'final', ['featured', 'public'])
+---
+test_block:
+  name: array-agg-documentation-tests
+  preset: single_repetition_ordered
+  tests:
+    -
+      - query: SELECT ARRAY_AGG(amount IGNORE NULLS) AS amounts FROM sales
+      - result: [{amounts: [200, 100, 150, 120]}]
+
+    -
+      - query: SELECT product, ARRAY_AGG(amount IGNORE NULLS) AS amounts
+               FROM sales
+               GROUP BY product
+      - result: [{product: 'Gadget', amounts: [200]},
+                 {product: 'Widget', amounts: [100, 150, 120]}]
+
+    -
+      - query: SELECT ARRAY_AGG(x) AS numbers
+                 FROM (SELECT a FROM VALUES ([2, 1, -2, 3, -2, 1, 2]) AS T(a)) AS sq,
+                      sq.a AS x
+      - result: [{numbers: [2, 1, -2, 3, -2, 1, 2]}]
+
+    -
+      - query: SELECT p.pid, sq.vals
+                 FROM parent p,
+                      (SELECT ARRAY_AGG(c.val IGNORE NULLS) AS vals FROM child c WHERE c.pid = p.pid) sq
+      - result: [{pid: 1, vals: [100, 200]},
+                 {pid: 2, vals: [300]},
+                 {pid: 3, vals: !null _}]
+
+    -
+      - query: SELECT p.pid, (SELECT ARRAY_AGG(c.val IGNORE NULLS) FROM child c WHERE c.pid = p.pid) FROM parent p
+      - error: SYNTAX_ERROR
+
+    -
+      # “Arrays of arrays”: Aggregating an ARRAY-typed argument is rejected, …
+      - query: SELECT ARRAY_AGG(tags) FROM doc_rev
+      - error: UNSUPPORTED_OPERATION
+
+    -
+      # … while wrapping the inner array in a struct works.
+      - query: SELECT ARRAY_AGG((rid, tags)) FROM doc_rev
+      - result: [{[{10, ['internal']}, {20, ['featured', 'public']}]}]
+
+---
+# “ARRAY_AGG() in indexes”: An index cannot be defined over ARRAY_AGG(). The schema template is created against the catalog,
+# as it is the DDL itself that has to be rejected.
+test_block:
+  name: array-agg-not-indexable-documentation-tests
+  connect: "jdbc:embed:/__SYS?schema=CATALOG"
+  preset: single_repetition_ordered
+  tests:
+    -
+      - query: CREATE SCHEMA TEMPLATE array_agg_doc_index_template
+               CREATE TABLE tab(id BIGINT, grp BIGINT, val BIGINT, PRIMARY KEY(id))
+               CREATE INDEX idx AS SELECT ARRAY_AGG(val) FROM tab GROUP BY grp
+      - error: UNSUPPORTED_OPERATION
+...