Vedant Chimmad — Data Engineer

• Oversaw the execution and monitoring of Azure Data Pipelines, ensuring the timely and accurate processing of data. • Implemented file handling mechanisms to initiate data processing upon file arrival. • Developed and maintained file name pattern recognition logic to identify and validate incoming data files. • Created and optimized SQL queries to perform data validation, integrity checks, and transformation processes. • Enforced data quality standards and conducted thorough data quality checks to identify and rectify anomalies and discrepancies. • Maintained comprehensive documentation of data pipeline processes, file handling procedures, and rule definitions for future reference. Generated and presented regular reports on data processing activities and performance. • Collaborated effectively with cross-functional teams to address project challenges and ensure seamless data flow. • Actively participated in troubleshooting and resolution of data-related issues. • Actively contributed to process optimization and efficiency enhancements within the data pipeline, streamlining workflows and reducing processing times. • Proficient in translating complex SQL queries into PySpark code, ensuring seamless integration and execution within data pipelines. • Extensive experience with PySpark’s core functions and APIs, enabling efficient data manipulation and processing within distributed environments. • Familiarity with AWS Glue’s architecture and its integration with PySpark, leveraging Glue’s capabilities to manage ETL processes effectively. • Skilled in developing robust PySpark scripts for data transformation and analysis within AWS Glue, ensuring accuracy • Adept at identifying and resolving issues in PySpark code, utilizing debugging tools and techniques to ensure smooth data processing workflows. • Responsible for the continuous maintenance and enhancement of PySpark scripts within AWS Glue, ensuring they remain up-to-date with evolving data requirements and best practices. • Integrated AWS glue job and Databricks workflow job with Apache airflow

Stackforce AI infers this person is a Data Engineering specialist with expertise in cloud technologies and data pipeline optimization.

Location: Bengaluru, Karnataka, India

Experience: 4 yrs 6 mos