lakehouse-federation

Design and manage multi-task Databricks Workflows with dependencies, parameters, retry logic, and scheduling.

Implement unit tests, integration tests, and data validation tests for Databricks notebooks and pipelines.

Share data securely across organizations using Delta Sharing and Unity Catalog sharing features.

Build and manage ML features using Databricks Feature Store and Unity Catalog feature engineering.

Build scalable Medallion (Bronze → Silver → Gold) pipelines using Delta Lake and Auto Loader.

Build, train, track, and deploy ML models using MLflow in Databricks.

Optimize Spark and Delta Lake performance through partitioning, caching, join strategies, and query tuning.

Securely manage credentials, API keys, and connection strings using Databricks Secret Scopes and external vaults.

Build real-time streaming pipelines using Kafka, Kinesis, Event Hubs, and Structured Streaming on Databricks.

Apply Unity Catalog governance including access control, masking, lineage, and auditing.

Optimize Databricks costs through cluster sizing, query tuning, storage optimization, and serverless strategies.

Implement CI/CD pipelines for Databricks using Databricks Asset Bundles (DABs), GitHub Actions, and Terraform.