Aug 2021 – Sep 2025 · 4 yrs 1 mo · Bengaluru, Karnataka, India · Hybrid

• Designed ETL pipeline running on cluster dealing with different data sources (SQL Server, Flat
• Files, Excel) and loaded the data into snowflake target data sources by performing different
• kinds of transformations using Spark running on Databricks.
• Maintain version control of objects for every code change using Github.
• Developed an ETL pipeline using Databricks notebooks to extract data from relational
• database using spark jdbc connectivity.
• Utilized Spark for data transformation, including data cleaning, aggregation, and
• normalization. Loaded the transformed data into Snowflake using the Spark Snowflake
• connector
• Implemented advanced Spark optimization techniques such as Salting, broadcast join, and
• caching/persistence on Databricks platform to enhance query performance and reduce job
• execution time. Successfully debugged and resolved issues in failed Spark jobs, ensuring
• smooth workflow execution and maximizing data processing efficiency.
• Executed complex data transformations, including aggregation, joins, and filtering, aligned
• with business requirements, leading to a 20% reduction in data processing time.
• Experienced in the Spark ecosystem, utilizing Spark SQL and Scala for efficient data
• transformation and analysis. Skilled in processing various formats like Text, CSV, and JSON,
• and optimizing ETL pipelines. Proficient in leveraging Delta Lake for ACID transactions, time
• travel, and schema evolution. Expertise in performance tuning and ensuring data consistency
• throughout the processing pipeline.
• Experienced with various Ab-Initio parallelism techniques, Ab-Initio Graphs using Data,
• Pipeline & Component parallelisms. MFS, Normalize, Scan, Sort, PDL, Control Center.

Mar 2021 – Apr 2021 · 1 mo

Apr 2018 – Jun 2018 · 2 mos