Manish Kumar Yadav

Data Engineer

India4 yrs 5 mos experience

Most Likely To SwitchHighly Stable

Key Highlights

Reduced SLA non-adherence by 35% through data transformation.
Achieved cost reductions of up to 70% in data processing.
Designed scalable ETL pipelines for real-time analytics.

Stackforce AI infers this person is a Data Engineer specializing in Fintech with expertise in scalable data solutions.

Contact

Skills

Core Skills

Data EngineeringEtl

Other Skills

Azure Data FactoryAzure DatabricksDelta LakeApache SparkPySparkUnity CatalogApache AirflowDBTAWS GlueAmazon RedshiftApache IcebergBigQuerySLA MonitoringFinancial Data PipelinesDatabricks

About

Microsoft-certified Data Engineer with a Bachelor of Technology in Computer Science from Madan Mohan Malaviya University of Technology. At LUMIQ, contributed to transformative data projects, leveraging Azure Data Factory, Databricks, and Delta Lake to enhance real-time analytics and reduce SLA non-adherence by 35%. Expert in designing scalable ETL pipelines, dynamic schema evolution, and cost-optimized data processing strategies, achieving cost reductions of up to 70%. Passionate about solving complex data challenges, improving decision-making, and collaborating on impactful, data-driven solutions.

Experience

4 yrs 5 mos

Total Experience

2 yrs 2 mos

Average Tenure

4 yrs 1 mo

Current Experience

Lumiq

2 roles

Data Engineer

Jul 2022 – Present · 3 yrs 10 mos

Migrated and transformed over 10 years of historical financial data for Voya Financial’s Wealth Central using Azure Data Factory, Databricks, and Delta Lake—enabled real-time analytics and improved SLA adherence by 35%.
Designed scalable and modular ETL pipelines with support for dynamic schema evolution and governance using Unity Catalog.
Reduced data processing costs by 60–70% through Spark job tuning, optimized cluster configurations, and efficient storage strategies.
Defined data contracts and collaborated with cross-functional teams to ensure seamless handoff across reporting and operational systems.
Built critical data pipelines for compliance reporting, retirement account aggregation, and performance dashboards.
Contributed to a financial cloud modernization initiative by implementing a data warehouse using Amazon Redshift, Apache Iceberg, and DBT.
Developed robust orchestration with Apache Airflow: included dynamic DAG generation, SLA monitoring, and integration with DBT and Redshift stored procedures.
Created BigQuery/Dataform-based reporting pipelines for CHL Group Insurance, monitoring KPIs and financial metrics for over 100K policyholders.
Designed and built centralized data lake for customer interactions (SMS, Email, Notifications) with near
real-time ingestion from MSK using PySpark Structured Streaming .
Implemented Raw and Stage layers using Apache Hudi; delivered curated datasets to DynamoDB for
low-latency UI consumption.
Orchestrated fault-tolerant pipelines using AWS Step Functions on EMR on EKS, enabling scalable and reliable event processing
Skills: Azure Data Factory · Azure Databricks · Delta Lake · Apache Spark · PySpark · Unity Catalog · Apache Airflow · DBT · AWS Glue · Amazon Redshift · Apache Iceberg · BigQuery · ETL · SLA Monitoring · Financial Data Pipelines