cost-optimization

Build scalable Medallion (Bronze → Silver → Gold) pipelines using Delta Lake and Auto Loader.

Implement unit tests, integration tests, and data validation tests for Databricks notebooks and pipelines.

Design and manage multi-task Databricks Workflows with dependencies, parameters, retry logic, and scheduling.

Apply Unity Catalog governance including access control, masking, lineage, and auditing.

Build real-time streaming pipelines using Kafka, Kinesis, Event Hubs, and Structured Streaming on Databricks.

Share data securely across organizations using Delta Sharing and Unity Catalog sharing features.

Securely manage credentials, API keys, and connection strings using Databricks Secret Scopes and external vaults.

Optimize Spark and Delta Lake performance through partitioning, caching, join strategies, and query tuning.

Build and manage ML features using Databricks Feature Store and Unity Catalog feature engineering.

Build, train, track, and deploy ML models using MLflow in Databricks.

Query external databases (PostgreSQL, MySQL, SQL Server, Snowflake, BigQuery) directly from Databricks using Lakehouse Federation.

Set up SQL alerts, job health monitoring, pipeline observability, and custom alerting on Databricks.