Nov 2024 – Jul 2025 · 8 mos · Dubai, United Arab Emirates · On-site

• Developed a real-time medallion architecture for SQL Server Change Data Capture leveraging Apache NiFi, Kafka, and MinIO with Docker and optimizing data capture processes.
• Led dashboard development initiatives using Apache Superset and Apache Trino and DuckDB, providing executives with critical insights for informed decision-making.
• Collaborated with cross-functional teams to align data solutions with business objectives.

Jun 2021 – Oct 2024 · 3 yrs 4 mos · Chennai, Tamil Nadu, India

• Migrated the entire on premise Hadoop architecture transfer for about 5000 pharmaceutical stores to GCS Data Lake, managing daily incremental and one-time dataflow
• Setup Auto deployment of pipeline with custom Jenkins pipelines from GIT, handling HIPAA and PII data
• Optimized Spark workflows resulting in reducing running time and effective cloud cost savings.
• Using Apache Airflow, organized the entire workflow and pipeline maintenance.
• Worked on Automation and SSL certificate renewal process using Venafi for updated NextGen PKI certs.
• Automated Unit Test execution and code coverage using Sonar.
• Connected with various business owners and met all the requirements within the design.

Dec 2019 – Jun 2021 · 1 yr 6 mos · Chennai Area, India

• Experienced in building a datalake - One time migration and change data capture using AWS Glue (Pyspark), DynamoDB, Athena
• Experienced in ELT pipelines for structured and semi-structured clickstream event data and setup BI environment with Quicksight
• Experience working in AWS Lake formation for migrating on-prem SQL server to S3 as datalake
• Built an entire Automated IP for data Migration from multiple RDBMS source to datalake on S3 with CloudFormation, Glue, DynamoDB
• Pre-processing of Time series data using Sagemaker with Pandas for AWS FORECAST, Built Forecast models with AWS forecast for travel domain and recognized as AWS Forecast partner
• Built recipes for automated pre-processing of time series data for AWS Forecast using Glue Databrew
• Parameterized and automated Amazon Forecast workflow with Papermill, AWS Lambda and Sagemaker notebook
• Extensively worked on Amazon Personalize - Per User recommendations for travel domain
• Worked extensively in Pandas including SHAP analysis, Pearson and spearman correlation
• Hosted a scikit machine learning model in Amazon Sagemaker by building a custom docker container
• Building a scalable datalake in Azure for one-time and CDC approach with Azure data factory and Azure Synapse analytics
• Worked with Unstructured PDF files and created a search engine with Amazon Kendra for a financial institution

Feb 2019 – Dec 2019 · 10 mos · Bengaluru, Karnataka, India

• Working in Apache Spark along with HiveQL on daily basis with AWS EMR as the production environment
• Worked extensively in AWS EMR in migrating the AWS on-premise production environment
• Involved in file movements between HDFS and AWS S3 and worked with AWS CLI. Sound knowledge in Amazon Web Services
• Worked in a team handling Adhoc requests and good knowledge on Git
• Worked in successive deployments along with regressive testing.
• Experience in data flow orchestration using Apache Airflow
• Worked on JSON, XML, CSV, Parquet file formats with snappy compression.
• Experience in shell scripting and wrote scripts to automate report generation and migration of reports to AWS S3
• Worked in building a data pipeline using Pandas and Flask Framework
• Good Familiarity with Anaconda and Jupyter notebook

Feb 2017 – Feb 2019 · 2 yrs · India

• Developer Role:
• Perform big data processing using Hadoop, MapReduce, Spark.
• Performed in-memory data processing using Apache spark core
• Experience in Handling Spark DataFrames in real time using case class and struct method using Scala
• Experience in writing and executing Spark DataFrames in SQL format
• Performed real-time twitter Analysis of counting the trending hashtags using Pyspark with tweepy package
• Good knowledge of Hadoop Architecture and various components such as HDFS's Job Tracker, Task Tracker, Name node, Data node
• Experience in importing and exporting data using Sqoop from relational database systems to HDFS and vice versa
• Created hive schemas using performance techniques like partitioning and bucketing.
• Worked in exporting data from sqoop directly into Hive tables
• Experience in analyzing data using Pig Latin scripts for custom MapReduce Jobs in Java
• Written Pig scripts to transform data into structured format
• Hands on experience in databases like Oracle SQL
• ADMIN Role:
• Setup, installed, and monitored 3-node CDH5 Hadoop cluster on Ubuntu Linux using cloudera express
• Hands on experience in setup, installing, configuring, and using Hadoop components like Hadoop Map Reduce, HDFS, Hive, Sqoop and Flume
• Deployed working CDH5 cluster for real time data analysis