Skip to content

Commit e13f2e4

Browse files
IxDayIxDay
committed
feat(clickhouse): Add support for clickhouse
If clickhouse is enabled we must deploy additional resources. Namely the events processor and the events consumer. Also, those will use a Kafka instance and its endpoint must be provided.
1 parent af1b5d7 commit e13f2e4

30 files changed

Lines changed: 1123 additions & 89 deletions

‎README.md‎

Lines changed: 40 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -70,6 +70,13 @@ You can use the provided `Taskfile.yaml` to easily set up a local development en
7070
| `global.kubectl.imageRegistry` | Docker registry with kubectl image (for init containers) | `docker.io` |
7171
| `global.kubectl.imageRepository` | Docker repository with kubectl image (for init containers) | `rancher/kubectl` |
7272
| `global.kubectl.imageTag` | Tag of kubectl Docker image (for init containers) - if unset (default) it is set to k8s cluster version | `""` |
73+
| `global.clickhouse.enabled` | Enable ClickHouse integration within Lago | `false` |
74+
| `global.clickhouse.kafka.tls` | Enable TLS support for Kafka when using ClickHouse | `true ` |
75+
| `global.clickhouse.kafka.saslMechanisms` | Kafka SASL mechanism | `"SCRAM-SHA-512"` |
76+
| `global.clickhouse.kafka.securityProtocol` | Kafka Security protocol | `"SASL_SSL"` |
77+
| `global.clickhouse.kafka.consumerGroup` | Kafka consumer group name | `"events_consumer"` |
78+
| `global.clickhouse.kafka.boostrapServers` | *(Required)* Kafka list of endpoints | `[]` |
79+
| `global.clickhouse.kafka.topics` | Dictionary of Kafka topics | {...} |
7380

7481
### Frontend Configuration
7582

@@ -140,6 +147,39 @@ You can use the provided `Taskfile.yaml` to easily set up a local development en
140147
| `worker.livenessProbe.timeoutSeconds` | Liveness probe timeout | `1` |
141148
| `worker.livenessProbe.failureThreshold` | Liveness probe failure threshold | `3` |
142149

150+
151+
### Events Consumer Configuration
152+
153+
| Parameter | Description | Default |
154+
|--------------------------------------------|-----------------------------------------------------|----------------|
155+
| `eventsConsumer.databasePool` | Size of the database connection pool | `10` |
156+
| `eventsConsumer.tolerations` | Pod tolerations for Events Consumer pods | `[]` |
157+
| `eventsConsumer.nodeSelector` | Node selector for Events Consumer pods | `{}` |
158+
| `eventsConsumer.affinity` | Affinity rules for Events Consumer pods | `{}` |
159+
| `eventsConsumer.rails.env` | Events Consumer environment | `production` |
160+
| `eventsConsumer.rails.logStdout` | Enable or disable logging to stdout | `true` |
161+
| `eventsConsumer.rails.logLevel` | Log level for the Rails app | `error` |
162+
| `eventsConsumer.resources.requests.memory` | Memory request for Events Consumer | `1Gi` |
163+
| `eventsConsumer.resources.requests.cpu` | CPU request for Events Consumer | `1100m` |
164+
| `eventsConsumer.podAnnotations` | Annotations to add to the Events Consumer pod | `{}` |
165+
| `eventsConsumer.podLabels` | Labels to add to the Events Consumer pod | `{}` |
166+
| `eventsConsumer.extraEnv` | Additional environment variables for pods | `{}` |
167+
168+
### Events Processor Configuration
169+
170+
| Parameter | Description | Default |
171+
|---------------------------------------------|-----------------------------------------------------|----------------|
172+
| `eventsProcessor.databasePool` | Size of the database connection pool | `10` |
173+
| `eventsProcessor.env` | Events Processor environment | `production` |
174+
| `eventsProcessor.tolerations` | Pod tolerations for Events Processor pods | `[]` |
175+
| `eventsProcessor.nodeSelector` | Node selector for Events Processor pods | `{}` |
176+
| `eventsProcessor.affinity` | Affinity rules for Events Processor pods | `{}` |
177+
| `eventsProcessor.resources.requests.memory` | Memory request for Events Processor | `1Gi` |
178+
| `eventsProcessor.resources.requests.cpu` | CPU request for Events Processor | `1100m` |
179+
| `eventsProcessor.podAnnotations` | Annotations to add to the Events Processor pod | `{}` |
180+
| `eventsProcessor.podLabels` | Labels to add to the Events Processor pod | `{}` |
181+
| `eventsProcessor.extraEnv` | Additional environment variables for pods | `{}` |
182+
143183
### Events Worker Configuration
144184

145185
| Parameter | Description | Default |

‎Taskfile.yaml‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -39,3 +39,20 @@ tasks:
3939
- kubectl apply -f charts/lago/ci/postgres.yml
4040
- kubectl apply -f charts/lago/ci/redis.yml
4141
- ct install --config ct.yaml --namespace lago
42+
43+
render:
44+
cmd: >
45+
helm template lago ./charts/lago
46+
--set apiUrl=http://localhost:3000
47+
--set frontUrl=http://localhost:3000
48+
--set global.databaseUrl=postgres://postgres:postgres@localhost:5432/postgres
49+
--set global.redisUrl=redis://localhost:6379
50+
{{join " " .CLI_ARGS_LIST}}
51+
52+
render:clickhouse:
53+
cmds:
54+
- task: render
55+
vars:
56+
CLI_ARGS_LIST: >
57+
--set global.clickhouse.enabled=true {{.CLI_ARGS}}
58+
--set global.clickhouse.kafka.bootstrapServers[0]=kafka-server:9092

‎charts/lago/Chart.yaml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@ apiVersion: v2
22
appVersion: "1.32.4"
33
description: the Lago open source billing app
44
name: lago
5-
version: 1.26.0
5+
version: 1.27.0
66
dependencies:
77
- name: minio
88
version: "5.2.0"

‎charts/lago/ci/clickhouse.yaml‎

Lines changed: 127 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,127 @@
1+
apiVersion: apps/v1
2+
kind: Deployment
3+
metadata:
4+
name: clickhouse
5+
spec:
6+
replicas: 1
7+
selector:
8+
matchLabels:
9+
app: clickhouse
10+
template:
11+
metadata:
12+
labels:
13+
app: clickhouse
14+
spec:
15+
containers:
16+
- name: clickhouse
17+
image: clickhouse/clickhouse-server:25.4-alpine
18+
ports:
19+
- containerPort: 9000
20+
- containerPort: 8123
21+
readinessProbe:
22+
exec:
23+
command:
24+
- clickhouse-client
25+
- --password
26+
- default
27+
- -q
28+
- "SELECT 1"
29+
initialDelaySeconds: 5
30+
periodSeconds: 5
31+
timeoutSeconds: 5
32+
failureThreshold: 5
33+
volumeMounts:
34+
- name: clickhouse-config
35+
mountPath: /etc/clickhouse-server/config.d/config.xml
36+
subPath: config.xml
37+
- name: clickhouse-config
38+
mountPath: /etc/clickhouse-server/users.d/users.xml
39+
subPath: users.xml
40+
volumes:
41+
- name: clickhouse-config
42+
configMap:
43+
name: clickhouse-config
44+
---
45+
apiVersion: v1
46+
kind: Service
47+
metadata:
48+
name: clickhouse
49+
spec:
50+
type: ClusterIP
51+
selector:
52+
app: clickhouse
53+
ports:
54+
- port: 9000
55+
targetPort: 9000
56+
name: tcp
57+
- port: 8123
58+
targetPort: 8123
59+
name: http
60+
---
61+
apiVersion: v1
62+
kind: ConfigMap
63+
apiVersion: v1
64+
metadata:
65+
name: clickhouse-config
66+
data:
67+
config.xml: |
68+
<clickhouse replace="true">
69+
<logger>
70+
<level>debug</level>
71+
<log>/var/log/clickhouse-server/clickhouse-server.log</log>
72+
<errorlog>/var/log/clickhouse-server/clickhouse-server.err.log</errorlog>
73+
<size>1000M</size>
74+
<count>3</count>
75+
</logger>
76+
<display_name>clickhouse_dev</display_name>
77+
<listen_host>0.0.0.0</listen_host>
78+
<http_port>8123</http_port>
79+
<tcp_port>9000</tcp_port>
80+
<user_directories>
81+
<users_xml>
82+
<path>users.xml</path>
83+
</users_xml>
84+
<local_directory>
85+
<path>/var/lib/clickhouse/access/</path>
86+
</local_directory>
87+
</user_directories>
88+
</clickhouse>
89+
users.xml: |
90+
<?xml version="1.0"?>
91+
<clickhouse replace="true">
92+
<profiles>
93+
<default>
94+
<max_memory_usage>10000000000</max_memory_usage>
95+
<use_uncompressed_cache>0</use_uncompressed_cache>
96+
<load_balancing>in_order</load_balancing>
97+
<log_queries>1</log_queries>
98+
</default>
99+
</profiles>
100+
<users>
101+
<default>
102+
<access_management>1</access_management>
103+
<profile>default</profile>
104+
<password>default</password>
105+
<networks>
106+
<ip>::/0</ip>
107+
</networks>
108+
<quota>default</quota>
109+
<access_management>1</access_management>
110+
<named_collection_control>1</named_collection_control>
111+
<show_named_collections>1</show_named_collections>
112+
<show_named_collections_secrets>1</show_named_collections_secrets>
113+
</default>
114+
</users>
115+
<quotas>
116+
<default>
117+
<interval>
118+
<duration>3600</duration>
119+
<queries>0</queries>
120+
<errors>0</errors>
121+
<result_rows>0</result_rows>
122+
<read_rows>0</read_rows>
123+
<execution_time>0</execution_time>
124+
</interval>
125+
</default>
126+
</quotas>
127+
</clickhouse>

‎charts/lago/ci/redpanda.yaml‎

Lines changed: 45 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,45 @@
1+
apiVersion: apps/v1
2+
kind: Deployment
3+
metadata:
4+
name: redpanda
5+
spec:
6+
replicas: 1
7+
selector:
8+
matchLabels:
9+
app: redpanda
10+
template:
11+
metadata:
12+
labels:
13+
app: redpanda
14+
spec:
15+
containers:
16+
- name: redpanda
17+
image: docker.redpanda.com/redpandadata/redpanda:v23.2.9
18+
args:
19+
- redpanda start
20+
- --smp 1
21+
- --overprovisioned
22+
- --kafka-addr internal://0.0.0.0:9092,external://0.0.0.0:19092
23+
- --advertise-kafka-addr internal://redpanda:9092,external://localhost:19092
24+
ports:
25+
- containerPort: 9092
26+
readinessProbe:
27+
httpGet:
28+
path: /v1/status/ready
29+
port: 9644
30+
initialDelaySeconds: 5
31+
periodSeconds: 5
32+
timeoutSeconds: 5
33+
failureThreshold: 5
34+
---
35+
apiVersion: v1
36+
kind: Service
37+
metadata:
38+
name: redpanda
39+
spec:
40+
type: ClusterIP
41+
selector:
42+
app: redpanda
43+
ports:
44+
- port: 9092
45+
targetPort: 9092

‎charts/lago/examples/events-consumer/README.md‎

Lines changed: 65 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,65 @@
1+
# Lago Events Processing - Example Configurations
2+
3+
This directory contains example configurations for deploying Lago's events processing components.
4+
In this example, ClickHouse, Redpanda, Redis, and Postgres are deployed as standalone services within the Kubernetes cluster.
5+
This setup is intended solely for demonstration and testing purposes.
6+
**For any real-life or production use case, you must provision and host these dependencies in a production-ready manner**
7+
—using managed services or properly secured, highly available deployments—rather than relying on the example manifests provided here.
8+
9+
**Security Notice:**
10+
11+
The example manifests in this directory are for demonstration only.
12+
**TLS encryption and secure passwords are NOT enabled by default.**
13+
14+
- **TLS:** All external services (Postgres, Redis, ClickHouse, Redpanda) are deployed without TLS.
15+
- In production, you **must** enable TLS for all external services to protect data in transit.
16+
- Update the `values.yaml` and external manifests to enable and configure TLS certificates.
17+
- **Passwords:** Default credentials (e.g., `postgres:postgres`, `redis:default`) are insecure and should **never** be used in production.
18+
- Always set strong, unique passwords for all services.
19+
- Store secrets securely (e.g., in Kubernetes Secrets or a secret manager).
20+
- **ClickHouse & Kafka:** The example disables authentication and encryption.
21+
- Enable user authentication and configure secure connections for ClickHouse and Redpanda/Kafka.
22+
23+
**Before using in production:**
24+
- Review and update all manifests to enforce strong authentication and TLS.
25+
- Never expose these example services to the public internet.
26+
- Refer to the official documentation for secure deployment practices.
27+
28+
## Example Scenarios
29+
30+
### Complete Events Stack
31+
Deploy all external dependencies and the events processing components:
32+
33+
```bash
34+
# Deploy dependencies first
35+
kubectl apply -f external/postgres.yml -f external/redis.yml -f external/redpanda.yaml -f external/clickhouse.yaml
36+
37+
# Install the events processing chart
38+
helm install lago-events-example --values values.yaml ../../
39+
```
40+
41+
### Custom Configuration
42+
43+
You can mix and match these examples or create your own:
44+
45+
```bash
46+
# Use specific example values
47+
helm install lago-events-example --values values.yaml ../../
48+
49+
# Or combine multiple configurations
50+
helm install lago-events-example --values values.yaml --values custom.yaml ../../
51+
```
52+
53+
## Upgrading
54+
55+
Once installed, you can upgrade with new configurations:
56+
57+
```bash
58+
helm upgrade --values values.yaml --values custom.yaml lago ../../
59+
```
60+
61+
## Notes
62+
63+
- This is an **example repository** for demonstration purposes
64+
- For production deployments, refer to the official Lago documentation
65+
- Adjust resource limits and configurations based on your environment needs

0 commit comments

Comments
 (0)