Oct 2022 – Oct 2024 · 2 yrs · Bengaluru, Karnataka, India · Hybrid

• Industry : Retail & Banking
• ● Planning, Migrating & Redesigning Entire Business Critical Data Pipelines from on-premises Oracle, MySQL, and MSSQL databases to Google BigQuery, facilitating enhanced data accessibility and scalability for advanced analytics.
• ● Architected and developed comprehensive dataflow pipelines for 58 distinct Source Files, encompassing rigorous data quality checks to ensure the accuracy and reliability of ingested data.
• ● Implemented proactive monitoring and alerting mechanisms to promptly identify and resolve issues in data pipelines, ensuring uninterrupted data flow and minimal downtime.
• ● Created and deployed a reusable ETL Framework utilizing Dataflow and Apache Beam for 3 different data sources. The framework efficiently handled data volumes ranging from 70 million records to 100 million records daily, transferring data from GCS to BigQuery.
• ● Developed a reusable framework using Python with Google Cloud Functions and Eventarc triggers for file validation. The framework automatically triggers Cloud Workflows when source files are uploaded to Google Cloud Storage, specifically initiating dataflow jobs after validation of 9 source files. This automation streamlines the ETL process and enhances operational efficiency.
• ● Orchestrated the execution of data pipelines based on both daily and monthly schedules, optimizing data processing and ingestion to align with business requirements and deadlines.
• ● Created Triggering Mechanisms for front end UI in BigQuery to generate reports from Stored Procedures by utilizing Rest API Calls, Cloud Functions and UDF in BQ.
• ● Designed a near-real-time pipeline to update retail transaction data from IOT sensors to BigQuery and Cloud SQL using Pub/Sub and Dataflow.
• ● Developed a Dataflow Pipeline for writing to Cloud SQL [PostgreSQL], incorporating a custom ParDo function to address Apache Beam logic gaps. Optimized Cloud SQL connections using JDBC Libraries, thus avoiding creating multiple connection pools.

Sep 2020 – Oct 2022 · 2 yrs 1 mo · Chennai, Tamil Nadu, India · Remote

• Industry : Telecommunications
• ● Building ETL data pipelines on Google Cloud for Vodafone which aggregate the data for the entire country(Hungary) in 9 sprints.
• ● Implemented CIM & EIM data models with a team of 7 along with Orchestrating Workflow on Cloud Composer(Airflow).
• ● Managed and Analysed Data Warehouse consisting of more than 50 million records in BigQuery.
• ● Developed and documented a Proof of Concept to transform Oracle-specific functionalities in BigQuery and Data Fusion.
• ● Collaborated with the Google Cloud team to implement a user-defined plugin based on the Hierarchy command in Oracle for business requirements.
• ● Configured drivers, memory, and executors cores of Dataproc clusters to improve the performance of the pipeline by 15%.
• ● Conducted numerous Knowledge Transfer Sessions for 7 new teammates and mentored 2 teammates from Germany.
• ● Performed Unit testing for data pipelines to ensure the quality of data remains at the peak.
• ● Worked beyond my responsibilities, as the roles of Data Analyst and Application Developer apart from being a data engineer.
• ● With extensive knowledge of file handing on Google Cloud Storage and Apache airflow, ameliorated numerous manual tasks to automated using python and shell scripts. Even from, source file schema generation in development to DDL [ Data Definition Language ] script deployment in production.
• ● Interacted with multi-regionalized people from Hungary, Portugal, Germany, Italy, UK and conveyed ideas effectively to ensure that delivered data pipelines meet the business requirements.
• ● Worked with Data architects to validate new designs on implementation to meet clients requirements.
• ● Collaborated cross-functionally according to the technical and business needs of the data engineering team.
• ● With Apache Beam replicated business logic in Cloud Data Flow a server-less tool for stream and Batch process.
• ● Followed the EU General Data Protection Regulation(GDPR) to ensure data compliance.

Jan 2020 – Apr 2020 · 3 mos · Chennai, Tamil Nadu, India

• Role - Flutter Application Developer - "Alexa Tablet App"
• ● The menu navigation of the app - Music, Bedtime Stories, General Knowledge, etc.,
• ● The app has search accessibility via Words and Voice recognition too.
• ● The app has been mainly designed for ease of general search via voice recognition.
• ● Established connectivity between Alexa Tablet App and Google Search Engine for search results.
• ● The results fetched would be playback through the Voice.