Apr 2023 – Apr 2024 · 1 yr

Apr 2022 – Aug 2025 · 3 yrs 4 mos

• Experienced Data Engineer with a strong background in building scalable and efficient ETL pipelines using Databricks, PySpark, Spark SQL, and SQL. Designed and implemented robust data integration workflows to ensure quality, consistency, and reliability of data for downstream analytics. Proven expertise in optimizing Spark jobs and cluster configurations, resulting in a 30% improvement in runtime performance and 20% cost reduction.
• Skilled in developing reusable data frameworks, managing large-scale data lake architectures on AWS (S3, Athena, Lambda, Redshift, Glue), and applying advanced data partitioning techniques to enhance query performance. Automated data workflows using Apache Airflow and integrated CI/CD processes with GitHub to streamline deployment cycles and maintain code quality.
• Hands-on experience with API automation, dynamic job configuration using Databricks REST API, and implementing event-driven architectures using AWS Lambda. Ensured enterprise-grade data security by applying AWS KMS encryption for data stored and processed on the cloud.
• Key Skills:
• Spark | Python | Spark SQL | PySpark | AWS (S3, Athena, Lambda, Redshift, Glue) | Airflow | GitHub | SQL | ETL | Databricks | CI/CD | Data Lake Management | Performance Optimization

Apr 2022 – Apr 2023 · 1 yr

Sep 2014 – Apr 2022 · 7 yrs 7 mos

• Experienced Data Engineer with hands-on expertise in building scalable data pipelines and analytics solutions across domains such as Automobile Supply Chain, Workforce Analytics, Inventory Management, and Predictive Maintenance. Proficient in Databricks, PySpark, PySpark SQL, and tools in the Hadoop ecosystem including Hive, HDFS, and Sqoop.
• Designed and optimized robust ETL pipelines for ingesting and transforming large datasets from OLTP systems into distributed systems like HDFS and Databricks Lakehouse, leveraging partitioning, bucketing, and efficient file formats (ORC, Parquet). Applied complex Spark SQL, RDD, and DataFrame transformations to meet analytical and business needs.
• Collaborated closely with stakeholders to gather requirements and deliver actionable insights. Skilled in AWS services like Redshift, Athena, and Glue for cloud-based data engineering and analytics workflows. Experienced in supporting applications and ensuring high availability of data solutions.