Data Operations Guide

Operational guidelines covering table compaction, snapshot maintenance, and telemetry.

Iceberg Table Maintenance & Compaction

Execute periodic optimization tasks to compact small Parquet files and purge stale snapshots:

ALTER TABLE iceberg.default.order_events EXECUTE optimize(file_size_threshold => '128MB');
ALTER TABLE iceberg.default.order_events EXECUTE expire_snapshots(retention_threshold => '7d');

Core Telemetry Metrics

• Kafka: kafka_consumergroup_lag
• Flink: flink_jobmanager_job_checkpointing_duration
• Trino: trino_execution_executor_runningqueries