Feb 2023 – Nov 2024 · 1 yr 9 mos

• At Lowe’s, I worked in designing and implementing an Enterprise Data Lake to support various analytics, processing, storage, and reporting needs for large volumes of rapidly changing data. Collaborating closely with stakeholders and solution architects, I ensured high-quality reference data through rigorous data cleaning, transformation, and integrity checks. My responsibilities included architecture and implementation assessments of AWS services like Amazon EMR, Redshift, and S3, as well as implementing machine learning algorithms using Python to predict user order quantities, leveraging Kinesis Firehose and S3 Data Lake. Additionally, I created Spark Streaming programs to process near real-time data from Kafka, developed Lambda functions to reduce EC2 costs, and managed database operations using SQL Server Integration Services (SSIS) and Data Transformation Services (DTS). My work included developing a reusable ETL framework, conducting ETL processes using Data Stage Open Studio, performing data analysis with Hive, and automating tasks related to Amazon SageMaker and Apache Airflow.

Jan 2018 – Jul 2021 · 3 yrs 6 mos · Hyderabad, Telangana, India · On-site

• At Nucleus Software, I leveraged Azure Data Factory to integrate data from both on-premises (MySQL, Cassandra) and cloud sources (Blob Storage, Azure SQL DB), applying necessary transformations and loading data into Azure Synapse. I managed resources with Azure Kubernetes Service and monitored Spark cluster performance using Log Analytics and Ambari Web UI, significantly improving query performance by transitioning log storage from Cassandra to Azure SQL Data Warehouse. I developed data ingestion pipelines on Azure HDInsight Spark cluster, worked extensively with Cosmos DB, and created dashboards using SSRS and Power BI to provide critical data insights. To enhance efficiency, I implemented automated procedures to split large files for FTP transfer, reducing execution time by 60%, and migrated large datasets to Databricks, configuring data pipelines from ADLS Gen2. Additionally, I wrote multiple MapReduce jobs using Java API, Pig, and Hive, and utilized Azure Logic Apps to automate batch jobs, integrating ADF pipelines with other services.