dasomel
한국어
All systems
WORK / ACTIVE SYSTEM

Beluga

A local Kubernetes data platform connecting Kafka, CDC, Flink, Iceberg, Trino, Superset, and Airflow

Kubernetes · k3s · Data Platform · Kafka · Debezium · Flink · Iceberg · Trino · Airflow · GitOps
Source
01 / Problem02 / Architecture03 / Development04 / Proof05 / Knowledge06 / Record
01 / PROBLEM

Each modern data-platform component is easy to deploy in isolation, but reproducing CDC, streaming, lakehouse, query, BI, and orchestration boundaries consistently is difficult on a local machine

RESPONSE

A reproducible k3s-based reference platform using Vagrant, Helm, and Argo CD GitOps, validated through real end-to-end clickstream and PostgreSQL CDC flows

02 / ARCHITECTURE

Read the boundaries before the features.

Identity, delivery, network, storage and workload integration are shown as operational seams rather than a feature inventory.

Beluga data flow from Kafka and Debezium through Flink, Iceberg, Lakekeeper, Trino, Superset and Airflow
Beluga data platform flow from CDC and streaming to lakehouse, SQL, BI and orchestration.
ARCHITECTURE / HOW IT WORKS

Beluga Data Platform Pipeline

01
Kafka + Debezium
Events · CDC
02
Flink
Stream processing
03
Iceberg + Lakekeeper
Lakehouse · REST catalog
04
Trino + Superset
Distributed SQL · BI
05
Airflow
Orchestration
A GitOps-deployed data platform from streaming and CDC ingestion through stream processing and Iceberg storage to query, BI, and orchestration.
DATA FLOW / VISUAL EVIDENCE

Streaming to lakehouse to analytics

01
Kafka + Debezium
02
Flink processing
03
Iceberg + Lakekeeper
04
Trino + Superset + Airflow
A compact visual summary; the architecture diagram below explains the execution flow in more detail.
03 / DEVELOPMENT OVER TIME

Evolution is part of the evidence.

First commit, cumulative commits, releases and recent activity show whether the system is still being engineered.

DEVELOPMENT OBSERVATORY
dasomel/beluga

Observe a system evolving over time, not a repository frozen at one release.

Contributors: 2
First commit
Aug 9, 2026
Commits
285
Releases
0
Latest release
—
Last push
Sep 19, 2026
Active for
1 months
Recent development activity
257 commits / 20 weeks
PAST → NOW
Language: ShellLicense: Apache-2.0Stars: 0Forks: 0Open issues: 101
04 / PROOF, NOT BADGES
10
technology / domain signals

Declared scope and technical context

6
connected docs

Operational or implementation documentation

4
engineering notes

Knowledge produced by the project

0
digest links

External signals explicitly connected

05 / CONNECTED KNOWLEDGE

What this system produced beyond code

06 / ENGINEERING RECORD

Implementation details, operating notes and project-specific context.

Project Overview

Beluga is a self-hosted, learning-scale modern data platform for reproducing the end-to-end data lifecycle on local Kubernetes.

It intentionally focuses on integration evidence, not production-scale infrastructure. The platform demonstrates how CDC, streaming, stream processing, lakehouse storage, SQL analytics, BI, and orchestration can work together under one GitOps-managed environment.

Platform Components

AreaComponentsRole
Clusterk3s, Cilium, MetalLBLocal Kubernetes base
GatewayAPISIX, etcdHTTP entrypoint
GitOpsArgo CDApp-of-Apps deployment
IdentityKeycloak, OpenLDAPAuthentication and groups
PolicyOPA, OpenFGAPolicy and authorization
StreamingStrimzi Kafka, DebeziumEvents and CDC
ProcessingFlink Kubernetes OperatorStateful stream processing
CatalogLakekeeperIceberg REST catalog
StorageSeaweedFSS3-compatible object storage
DatabaseCloudNativePGPostgreSQL source and metadata
QueryTrinoDistributed SQL over Iceberg
BISupersetAnalytics and dashboards
OrchestrationAirflow 3DAG-based workflows
Optional governanceOpenMetadata, OpenSearchCatalog and lineage
ObservabilityPrometheus StackPlatform metrics

Integration Model

The platform is split into beluga-platform and beluga-data Helm layers, both deployed through Argo CD. This keeps platform services and data workloads independently understandable while maintaining a single bootstrap path.

Two real data demonstrations provide validation:

  • Synthetic clickstream — generated events through streaming and lakehouse processing
  • PostgreSQL CDC — source DB changes captured by Debezium and processed into the lakehouse

Architecture

Local Resource Profile

Beluga is not a tiny demo. It provisions four VMs and a full data stack.

  • 32GB host RAM minimum
  • 48GB+ enables larger profiles and OpenMetadata/Trino workers
  • 64GB+ adds further worker memory
  • VMware Fusion on ARM64 or VirtualBox on AMD64
  • profile selection is based on detected host resources

This makes Beluga a reference environment for studying the integration cost of a complete data platform, rather than a lightweight sample app.

Verification

The repository deliberately separates rendering from real runtime verification:

The scripts inspect real cluster/API state instead of treating successful Helm rendering as proof of a working platform.

Security and Credentials

Passwords are not committed to the repository. Bootstrap generates credentials and stores them in Kubernetes Secrets, while Helm values use placeholders/references rather than static credentials.

The repository also keeps policy declarations under policies/ so identity, authorization, and database policy generation can remain aligned.

Current Status

Beluga is a personal / learning-scale reference platform.

  • Core Kafka/CDC → Flink → Iceberg → Trino/Superset/Airflow flow is implemented toward clean-install E2E validation.
  • The local cluster is not continuously running, so not every latest change has live-cluster validation.
  • Governance/policy compiler integration continues as a separate evolution area.

Getting Started

git clone https://github.com/dasomel/beluga.git
cd beluga
make up
make status
make test

Documentation Index

TopicDocumentPurpose
OverviewPlatform OverviewScope and product model
ArchitecturePipeline ArchitectureComponent boundaries and data flow
Getting StartedCluster SetupVM → k3s → GitOps bootstrap
OperationsData OperationsLifecycle and operational runbooks
TroubleshootingTroubleshootingCluster, Kafka, Flink, and query issues

Project Relationship