feat(clickhouse): Add support for clickhouse

README.md

@@ -70,6 +70,13 @@ You can use the provided `Taskfile.yaml` to easily set up a local development en
 | `global.kubectl.imageRegistry`   | Docker registry with kubectl image (for init containers)                                                | `docker.io`       |
 | `global.kubectl.imageRepository` | Docker repository with kubectl image (for init containers)                                              | `rancher/kubectl` |
 | `global.kubectl.imageTag`        | Tag of kubectl Docker image (for init containers) - if unset (default) it is set to k8s cluster version | `""`              |
+| `global.clickhouse.enabled`                | Enable ClickHouse integration within Lago                                      | `false`             |
+| `global.clickhouse.kafka.tls`              | Enable TLS support for Kafka when using ClickHouse                             | `true `             |
+| `global.clickhouse.kafka.saslMechanisms`   | Kafka SASL mechanism                                                           | `"SCRAM-SHA-512"`   |
+| `global.clickhouse.kafka.securityProtocol` | Kafka Security protocol                                                        | `"SASL_SSL"`        |
+| `global.clickhouse.kafka.consumerGroup`    | Kafka consumer group name                                                      | `"events_consumer"` |
+| `global.clickhouse.kafka.boostrapServers`  | *(Required)* Kafka list of endpoints                                           | `[]`                |
+| `global.clickhouse.kafka.topics`           | Dictionary of Kafka topics                                                     | {...}               |
 
 ### Frontend Configuration
 
@@ -140,6 +147,39 @@ You can use the provided `Taskfile.yaml` to easily set up a local development en
 | `worker.livenessProbe.timeoutSeconds`      | Liveness probe timeout                           | `1`       |
 | `worker.livenessProbe.failureThreshold`    | Liveness probe failure threshold                | `3`       |
 
+
+### Events Consumer Configuration
+
+| Parameter                                  | Description                                         | Default        |
+|--------------------------------------------|-----------------------------------------------------|----------------|
+| `eventsConsumer.databasePool`              | Size of the database connection pool                | `10`           |
+| `eventsConsumer.tolerations`               | Pod tolerations for Events Consumer pods            | `[]`           |
+| `eventsConsumer.nodeSelector`              | Node selector for Events Consumer pods              | `{}`           |
+| `eventsConsumer.affinity`                  | Affinity rules for Events Consumer pods             | `{}`           |
+| `eventsConsumer.rails.env`                 | Events Consumer environment                         | `production`   |
+| `eventsConsumer.rails.logStdout`           | Enable or disable logging to stdout                 | `true`         |
+| `eventsConsumer.rails.logLevel`            | Log level for the Rails app                         | `error`        |
+| `eventsConsumer.resources.requests.memory` | Memory request for Events Consumer                  | `1Gi`          |
+| `eventsConsumer.resources.requests.cpu`    | CPU request for Events Consumer                     | `1100m`        |
+| `eventsConsumer.podAnnotations`            | Annotations to add to the Events Consumer pod       | `{}`           |
+| `eventsConsumer.podLabels`                 | Labels to add to the Events Consumer pod            | `{}`           |
+| `eventsConsumer.extraEnv`                  | Additional environment variables for pods           | `{}`           |
+
+### Events Processor Configuration
+
+| Parameter                                   | Description                                         | Default        |
+|---------------------------------------------|-----------------------------------------------------|----------------|
+| `eventsProcessor.databasePool`              | Size of the database connection pool                | `10`           |
+| `eventsProcessor.env`                       | Events Processor environment                        | `production`   |
+| `eventsProcessor.tolerations`               | Pod tolerations for Events Processor pods           | `[]`           |
+| `eventsProcessor.nodeSelector`              | Node selector for Events Processor pods             | `{}`           |
+| `eventsProcessor.affinity`                  | Affinity rules for Events Processor pods            | `{}`           |
+| `eventsProcessor.resources.requests.memory` | Memory request for Events Processor                 | `1Gi`          |
+| `eventsProcessor.resources.requests.cpu`    | CPU request for Events Processor                    | `1100m`        |
+| `eventsProcessor.podAnnotations`            | Annotations to add to the Events Processor pod      | `{}`           |
+| `eventsProcessor.podLabels`                 | Labels to add to the Events Processor pod           | `{}`           |
+| `eventsProcessor.extraEnv`                  | Additional environment variables for pods           | `{}`           |
+
 ### Events Worker Configuration
 
 | Parameter                          | Description                                        | Default   |

Taskfile.yaml

@@ -38,3 +38,20 @@ tasks:
       - kubectl apply -f charts/lago/ci/postgres.yml
       - kubectl apply -f charts/lago/ci/redis.yml
       - ct install --config ct.yaml --namespace default
+
+  render:
+    cmd: >
+      helm template lago ./charts/lago
+      --set apiUrl=http://localhost:3000
+      --set frontUrl=http://localhost:3000
+      --set global.databaseUrl=postgres://postgres:postgres@localhost:5432/postgres
+      --set global.redisUrl=redis://localhost:6379
+      {{join " " .CLI_ARGS_LIST}}
+
+  render:clickhouse:
+    cmds:
+      - task: render
+        vars:
+          CLI_ARGS_LIST: >
+            --set global.clickhouse.enabled=true {{.CLI_ARGS}}
+            --set global.clickhouse.kafka.bootstrapServers[0]=kafka-server:9092

charts/lago/Chart.yaml

@@ -2,7 +2,7 @@ apiVersion: v2
 appVersion: "1.32.4"
 description: the Lago open source billing app
 name: lago
-version: 1.26.3
+version: 1.27.0
 dependencies:
   - name: minio
     version: "5.2.0"

charts/lago/ci/clickhouse.yaml

@@ -0,0 +1,127 @@
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: clickhouse
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app: clickhouse
+  template:
+    metadata:
+      labels:
+        app: clickhouse
+    spec:
+      containers:
+      - name: clickhouse
+        image: clickhouse/clickhouse-server:25.6-alpine
+        ports:
+        - containerPort: 9000
+        - containerPort: 8123
+        readinessProbe:
+          exec:
+            command:
+            - clickhouse-client
+            - --password
+            - default
+            - -q
+            - "SELECT 1"
+          initialDelaySeconds: 5
+          periodSeconds: 5
+          timeoutSeconds: 5
+          failureThreshold: 5
+        volumeMounts:
+        - name: clickhouse-config
+          mountPath: /etc/clickhouse-server/config.d/config.xml
+          subPath: config.xml
+        - name: clickhouse-config
+          mountPath: /etc/clickhouse-server/users.d/users.xml
+          subPath: users.xml
+      volumes:
+      - name: clickhouse-config
+        configMap:
+          name: clickhouse-config
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: clickhouse
+spec:
+  type: ClusterIP
+  selector:
+    app: clickhouse
+  ports:
+  - port: 9000
+    targetPort: 9000
+    name: tcp
+  - port: 8123
+    targetPort: 8123
+    name: http
+---
+apiVersion: v1
+kind: ConfigMap
+apiVersion: v1
+metadata:
+  name: clickhouse-config
+data:
+  config.xml: |
+    <clickhouse replace="true">
+        <logger>
+            <level>debug</level>
+            <log>/var/log/clickhouse-server/clickhouse-server.log</log>
+            <errorlog>/var/log/clickhouse-server/clickhouse-server.err.log</errorlog>
+            <size>1000M</size>
+            <count>3</count>
+        </logger>
+        <display_name>clickhouse_dev</display_name>
+        <listen_host>0.0.0.0</listen_host>
+        <http_port>8123</http_port>
+        <tcp_port>9000</tcp_port>
+        <user_directories>
+            <users_xml>
+                <path>users.xml</path>
+            </users_xml>
+            <local_directory>
+                <path>/var/lib/clickhouse/access/</path>
+            </local_directory>
+        </user_directories>
+    </clickhouse>
+  users.xml: |
+    <?xml version="1.0"?>
+    <clickhouse replace="true">
+        <profiles>
+            <default>
+                <max_memory_usage>10000000000</max_memory_usage>
+                <use_uncompressed_cache>0</use_uncompressed_cache>
+                <load_balancing>in_order</load_balancing>
+                <log_queries>1</log_queries>
+            </default>
+        </profiles>
+        <users>
+            <default>
+                <access_management>1</access_management>
+                <profile>default</profile>
+                <password>default</password>
+                <networks>
+                    <ip>::/0</ip>
+                </networks>
+                <quota>default</quota>
+                <access_management>1</access_management>
+                <named_collection_control>1</named_collection_control>
+                <show_named_collections>1</show_named_collections>
+                <show_named_collections_secrets>1</show_named_collections_secrets>
+            </default>
+        </users>
+        <quotas>
+            <default>
+                <interval>
+                    <duration>3600</duration>
+                    <queries>0</queries>
+                    <errors>0</errors>
+                    <result_rows>0</result_rows>
+                    <read_rows>0</read_rows>
+                    <execution_time>0</execution_time>
+                </interval>
+            </default>
+        </quotas>
+    </clickhouse>

charts/lago/ci/redpanda.yaml

@@ -0,0 +1,45 @@
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: redpanda
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app: redpanda
+  template:
+    metadata:
+      labels:
+        app: redpanda
+    spec:
+      containers:
+      - name: redpanda
+        image: docker.redpanda.com/redpandadata/redpanda:v23.2.9
+        args:
+        - redpanda start
+        - --smp 1
+        - --overprovisioned
+        - --kafka-addr internal://0.0.0.0:9092,external://0.0.0.0:19092
+        - --advertise-kafka-addr internal://redpanda:9092,external://localhost:19092
+        ports:
+        - containerPort: 9092
+        readinessProbe:
+          httpGet:
+            path: /v1/status/ready
+            port: 9644
+          initialDelaySeconds: 5
+          periodSeconds: 5
+          timeoutSeconds: 5
+          failureThreshold: 5
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: redpanda
+spec:
+  type: ClusterIP
+  selector:
+    app: redpanda
+  ports:
+  - port: 9092
+    targetPort: 9092

charts/lago/examples/events-consumer/README.md

@@ -0,0 +1,65 @@
+# Lago Events Processing - Example Configurations
+
+This directory contains example configurations for deploying Lago's events processing components.
+In this example, ClickHouse, Redpanda, Redis, and Postgres are deployed as standalone services within the Kubernetes cluster.
+This setup is intended solely for demonstration and testing purposes.
+**For any real-life or production use case, you must provision and host these dependencies in a production-ready manner**
+—using managed services or properly secured, highly available deployments—rather than relying on the example manifests provided here.
+
+ **Security Notice:**
+
+The example manifests in this directory are for demonstration only.
+**TLS encryption and secure passwords are NOT enabled by default.**
+
+- **TLS:** All external services (Postgres, Redis, ClickHouse, Redpanda) are deployed without TLS.
+  - In production, you **must** enable TLS for all external services to protect data in transit.
+  - Update the `values.yaml` and external manifests to enable and configure TLS certificates.
+- **Passwords:** Default credentials (e.g., `postgres:postgres`, `redis:default`) are insecure and should **never** be used in production.
+  - Always set strong, unique passwords for all services.
+  - Store secrets securely (e.g., in Kubernetes Secrets or a secret manager).
+- **ClickHouse & Kafka:** The example disables authentication and encryption.
+  - Enable user authentication and configure secure connections for ClickHouse and Redpanda/Kafka.
+
+**Before using in production:**
+- Review and update all manifests to enforce strong authentication and TLS.
+- Never expose these example services to the public internet.
+- Refer to the official documentation for secure deployment practices.
+
+## Example Scenarios
+
+### Complete Events Stack
+Deploy all external dependencies and the events processing components:
+
+```bash
+# Deploy dependencies first
+kubectl apply -f external/postgres.yml -f external/redis.yml -f external/redpanda.yaml -f external/clickhouse.yaml
+
+# Install the events processing chart
+helm install lago-events-example --values values.yaml ../../
+```
+
+### Custom Configuration
+
+You can mix and match these examples or create your own:
+
+```bash
+# Use specific example values
+helm install lago-events-example --values values.yaml ../../
+
+# Or combine multiple configurations
+helm install lago-events-example --values values.yaml --values custom.yaml ../../
+```
+
+## Upgrading
+
+Once installed, you can upgrade with new configurations:
+
+```bash
+helm upgrade --values values.yaml --values custom.yaml lago ../../
+```
+
+## Notes
+
+- This is an **example repository** for demonstration purposes
+- For production deployments, refer to the official Lago documentation
+- Adjust resource limits and configurations based on your environment needs