Project Overview

Beluga is a self-hosted, learning-scale modern data platform for reproducing the end-to-end data lifecycle on local Kubernetes.

It intentionally focuses on integration evidence, not production-scale infrastructure. The platform demonstrates how CDC, streaming, stream processing, lakehouse storage, SQL analytics, BI, and orchestration can work together under one GitOps-managed environment.

Platform Components

The single source of truth for exact versions/images/licenses is the repository's VERSIONS.md. Notable pins: Kubernetes v1.36 (k3s channel), Strimzi Kafka Operator 1.1.0 (KRaft, Kafka 4.3.0), Debezium 3.6.1.Final, Flink Kubernetes Operator 1.15.0 (runtime Flink 1.20.0), Lakekeeper v0.13.1, Trino 483, Airflow 3.3.0, Superset 6.1.0, Keycloak 26.7.1, CloudNativePG (PostgreSQL 17.6).

Integration Model

The platform is split into beluga-platform and beluga-data Helm layers, both deployed through Argo CD. This keeps platform services and data workloads independently understandable while maintaining a single bootstrap path.

Two real data demonstrations provide validation:

• Synthetic clickstream — generated events through streaming and lakehouse processing
• PostgreSQL CDC — source DB changes captured by Debezium and processed into the lakehouse

Architecture

Local Resource Profile

Beluga is not a tiny demo. It provisions four VMs and a full data stack.

• 32GB host RAM minimum
• 48GB+ enables larger profiles and OpenMetadata/Trino workers
• 64GB+ adds further worker memory
• VMware Fusion on ARM64 or VirtualBox on AMD64
• profile selection is based on detected host resources

This makes Beluga a reference environment for studying the integration cost of a complete data platform, rather than a lightweight sample app.

Verification

The repository deliberately separates rendering from real runtime verification:

The scripts inspect real cluster/API state instead of treating successful Helm rendering as proof of a working platform.

Security and Credentials

Passwords are not committed to the repository. Bootstrap generates credentials and stores them in Kubernetes Secrets, while Helm values use placeholders/references rather than static credentials.

The repository also keeps policy declarations under policies/ so identity, authorization, and database policy generation can remain aligned.

Operations Agent (read-only PoC)

As the first safe-execution slice for issue #108, scripts/agent/operations_agent.py inspects the cluster, collects failure signals, and produces remediation proposals only, without mutating the cluster.

• The execution path follows OpenForge Agent Execution Security principles: request → resolve fixed tool/target → canonical invocation digest → risk classification → request-side authorization → fixed argv executor (read-only only) → result hashing/findings → evidence record.
• configs/operations-agent-policy.json declares a risk class per tool — cluster.nodes/cluster.pods/cluster.events/gitops.applications are read-only-diagnostic and executable, while data.mutate/external.http/platform.privileged are approvalRequired: true and executable: false in the PoC. An approval flag does not make a disabled class executable.
• Safety invariants: no free-form shell/model-generated kubectl arguments reach execution — only code-owned argv defined in the script does; an isolated kubeconfig is required instead of the shared ~/.kube/config; the Kubernetes context must always be beluga; evidence stores hashes/byte counts/status rather than raw output.
• The design has seven roles (cluster-inspector, data-service-inspector, identity-inspector, storage-inspector, observability-inspector, hypothesis-agent, verifier), but the current implementation covers only the shared read-only inspection boundary — per-role graph orchestration remains future work.
• make test-agent verifies the policy/security suite without a live cluster, backed by a permanent CI workflow (operations-agent-security.yml).

Current Status

Beluga is a personal / learning-scale reference platform.

• Core Kafka/CDC → Flink → Iceberg → Trino/Superset/Airflow flow is implemented toward clean-install E2E validation and marked done (docs/IMPLEMENTATION-STATUS.md, verified 2026-09-14).
• The local cluster is not continuously running (vagrant status shows all 4 VMs stopped), so not every latest change has live-cluster validation.
• The policy compiler now lives in manager/ and runs from the same tree; governance continues to evolve.
• The Operations Agent only runs its read-only diagnostic profile; data-mutation, external-egress, and privileged-operation execution remain contractually disabled.
• Cluster-stability fixes have continued since 2026-08-28: scoping MetalLB's L2Advertisement to the private network interface, unblocking the ArgoCD/OpenMetadata UIs behind APISIX, setting explicit key usages on internal-CA certificates, renaming the VMware provider to vagrant-vmware-desktop (vmware_desktop), and tuning the VM RAM profile for 64GB hosts.

Beluga Manager (manager/)

Beluga Manager is the unified control plane of the Beluga data platform. It used to be the separate repository dasomel/beluga-manager; on 2026-10-15, under ADR-0005, it was consolidated with full history into the manager/ directory of this repository, and the old repository was archived.

Kafka, Flink, Iceberg, Trino, and Airflow remain authoritative for their own resources. Manager does not clone their UIs; it connects their APIs through adapters to answer platform-level questions (which topic leads to which job and table, which service is degraded).

Repository layout — manager/ is an npm workspace (packages/domain-api, packages/policy-compiler, packages/web) with its own package.json and lockfile; .github/workflows/manager-ci.yml runs CI only for manager/** changes. It releases independently of the platform.

Policy compiler seam — manager/packages/policy-compiler compiles the YAML in policies/ into Keycloak configuration, Trino OPA Rego (trino.rego), and PostgreSQL DDL (db-roles.sql). Before consolidation, two workflows used external checkouts pinned to different SHAs; now test 14 and ci.yml/release.yml run the compiler from manager/ in the same tree, so a compiler change and its regenerated artifacts land in one PR.

Current maturity (checked 2026-10-15) — The Domain API, the web console (Overview, Services, Pipelines, Data Catalog, Query Workspace, Policy, and more), and the policy compiler are implemented and tested. However, the Domain API answers from in-repo fixtures (stubs) by default; the only opt-in read-only adapters are Flink, Trino query history, and Lakekeeper, and only the Flink adapter has a recorded live-cluster check. There are no Kafka, Debezium, or Airflow adapters, no authentication/authorization middleware, and no cluster deployment chart. Upstream hosts are hardcoded to *.local.beluga.internal. It is at a read-first stage, not a finished operations console.

The 22 open beluga-manager issues were transferred to #239-#260 in this repository (label area:manager). See the overview, architecture, development, and operations docs. The background is covered in the blog post Why we consolidated beluga-manager into the beluga monorepo.

Getting Started

git clone https://github.com/dasomel/beluga.git
cd beluga
make up
make status
make test

Documentation Index

Project Relationship