Oct 2024 – May 2026 · 1 yr 7 mos · Bengaluru, Karnataka, India · On-site

• Client: PepsiCo :
• Automated Databricks cluster scaling and configuration using REST APIs and system tables, optimizing resource utilization
• and reducing idle time.
• Implemented Deletion Vectors and Liquid Clustering to enhance Delta Lake performance and reduce storage costs.
• Migrated data assets to Unity Catalog for centralized access control, lineage tracking, and secure workspace sharing.
• Transitioned workloads to Serverless SQL Warehouses and managed tables to reduce infrastructure overhead.
• Integrated Azure Logic Apps for automated email alerts on optimization reports and cluster health.
• Built Databricks dashboards for monitoring cost, job performance, and cluster utilization trends.
• Refactored PySpark and SQL scripts for partition pruning, caching, and job scheduling, achieving up to 40% performance
• gains.
• Impact: Achieved 25% reduction in compute costs and 35% faster query execution, improving governance and platform
• efficiency.
• Client: 7-Eleven
• Led end-to-end migration of Databricks workspaces to Unity Catalog for centralized governance and fine-grained access
• control.
• Automated Hive metastore migration using databricks-cli, REST APIs, and notebooks; validated workflows via Airflow.
• Integrated with Azure Purview for lineage tracking and audit logging; ensured zero data loss across 2000+ tables.
• Tech Stack: Databricks, Unity Catalog, Delta Lake, Airflow, Azure Purview, REST APIs, PySpark, SQL

Jan 2024 – Oct 2024 · 9 mos · On-site

• Developed and maintained PySpark scripts in S3 for loading purposes and transformation of data.
• End to End development of orchestration framework in managed airflow by aws also developing the capability to ingest data in series and in parallel thereby increasing the efficiency by almost 30%.
• Developed and optimized DBT models and macros, reducing data processing time by 40% and improving data accuracy and integrity within the data warehouse by 25%.
• Ability to troubleshoot various issues in airflow and pyspark.
• Optimization of airflow dags and pyspark code for better efficiency and speed.
• Tools & Technologies Used: PySpark, Airflow, DBT, Snowflake, AWS EMR, AWS s3, AWS Lambda, AWS Athena, Python, SQL, Unix Scripting.

Mar 2023 – Dec 2023 · 9 mos · On-site

• Developed and maintained PySpark scripts in databricks for loading purposes and transformation of data.
• End-to-End Data Pipeline Development: Designed and implemented real-time data pipelines using Databricks, integrating various sources such as Databases, Flat files Streaming, and Delta Lake to ensure efficient data ingestion, processing, and storage.
• Optimization of Spark Jobs: Optimized Spark jobs for performance tuning, including partitioning strategies, caching, and broadcast joins, resulting in a 40% reduction in processing time and improved resource utilization.
• Ability to troubleshoot common issues with Spark DataFrame, such as data processing errors, performance bottlenecks, and scalability limitations.
• Scalable ETL with Databricks Delta: Architected and implemented scalable ETL processes using Databricks Delta, handling petabyte-scale datasets efficiently while maintaining data quality and consistency.
• Delta Lake for Reliable Data Lakes: Spearheaded the adoption of Delta Lake on Databricks to ensure ACID transactions, schema enforcement, and time travel capabilities, enhancing data reliability and simplifying data lake management.
• Achieved proficiency in complex Spark-SQL scripts for the end-to-end load of data from source to data warehouse.
• Knowledge of Spark SQL optimization techniques, such as cost-based query optimization, column pruning, and predicate pushdown, and their impact on query performance and resource utilization.
• Responsible for managing the new tickets raised by customers and solving them.
• Tools & Technologies Used: Python, SQL, Azure Databricks, ADLS Gen2, PySpark, Jira, ADF, SQL, Azure SQL, Unix Scripting

Jun 2021 – Mar 2023 · 1 yr 9 mos · On-site

Apr 2021 – Jun 2021 · 2 mos · Bengaluru, Karnataka, India

Aug 2020 – Apr 2021 · 8 mos · Mysore, Karnataka, India