Ayush Shrivastava

Software Engineer

Kanpur Nagar, Uttar Pradesh, India4 yrs 3 mos experience

Highly Stable

Key Highlights

Over 4 years of experience in data engineering.
Certified Databricks Data Engineer Associate.
Expert in building scalable ETL pipelines.

Stackforce AI infers this person is a Data Engineer specializing in cloud-based data solutions and ETL processes.

Contact

Skills

Core Skills

Azure Data FactoryDatabricksSpark

Other Skills

Data WorkflowsYAML ConfigurationsAzure Logic AppsAutomationData ReliabilityPerformance OptimizationSQLCI/CDData QualityDatabricks Asset BundlesDatabricks WorkflowDelta LakeAzure DevopsMicrosoft AzureData Validation

About

I am a Data Engineer with over 4 years of experience in building scalable ETL pipelines and cloud-based data solutions. My expertise lies in data transformation, data modeling, and data quality validation, enabling me to deliver analytics-ready datasets for advanced assignments. I have developed a strong proficiency in leveraging Azure technologies, including Azure Data Factory and Databricks, to create reliable, production-grade data pipelines that meet the demands of modern data processing. Throughout my career, I have successfully orchestrated and automated data workflows, ensuring high performance and reliability. I am recognized for my ability to manage multiple projects simultaneously while maintaining strong client satisfaction. My recent certification as a Databricks Data Engineer Associate further solidifies my commitment to excellence in the field. I am passionate about utilizing my skills to drive impactful data solutions and contribute to the success of my team and organization. Key Achievements: - Optimized Spark jobs, resulting in a 20% reduction in processing time. - Successfully built and automated metadata-driven Azure Data Factory pipelines, significantly improving pipeline reliability and reducing failure rates. - Managed environment-based job deployments using Databricks Asset Bundles, enhancing deployment efficiency.

Experience

4 yrs 3 mos

Total Experience

4 yrs 3 mos

Average Tenure

Current Experience

Tiger analytics

2 roles

Senior Software Engineer

Promoted

Nov 2023 – May 2026 · 2 yrs 6 mos

In my current role as a Senior Software Engineer, I focus on building metadata-driven Azure Data Factory pipelines and automating scheduling with ADF Triggers, which has significantly reduced failure rates. I execute Spark workloads in Azure Databricks to process multi-format datasets and generate business KPIs. Additionally, I orchestrate production-grade data workflows using Databricks Workflows and manage environment-based job deployments with parameterized YAML configurations. My responsibilities also include implementing secrets management frameworks and structuring data pipelines using Medallion Architecture principles within Azure Data Lake Storage Gen 2 environments.
Achievements: I optimized Spark jobs through repartitioning and broadcast joins, achieving a 20% reduction in processing time. I also successfully configured monitoring and ingestion automation through Azure Logic Apps, enhancing pipeline alerts and data ingestion processes.

Azure Data FactoryDatabricksSparkData WorkflowsYAML ConfigurationsAzure Logic Apps

Software Engineer

Jan 2022 – Jan 2024 · 2 yrs

As a Software Engineer, I maintained and enhanced existing Azure Data Factory pipelines and Databricks codebases to support ongoing data processing workflows. I performed SQL-based data analysis for operational and analytical use cases and monitored pipeline executions to resolve runtime failures, ensuring continuity of scheduled data workflows. My role also involved supporting deployment cycles through CI/CD pipelines for data platform updates and validating ingested datasets for data quality and integrity using Azure Databricks.
Achievements: I played a key role in ensuring the reliability of data processing workflows and contributed to the successful deployment of updates to the data platform through effective CI/CD practices.

Azure Data FactoryDatabricksSQLCI/CDData Quality