Shivam Soni

Data Engineer

Crawley, England, United Kingdom11 yrs 8 mos experience

Most Likely To SwitchHighly Stable

Key Highlights

7+ years of experience in data engineering.
Expert in Azure Databricks and data governance.
Proven track record in cost optimization and automation.

Stackforce AI infers this person is a Senior Data Engineer specializing in Azure-based data solutions and governance in the SaaS industry.

Contact

Skills

Core Skills

Azure DatabricksAzure Data FactoryPython

Other Skills

Azure Logic AppsAzure FunctionsAPI ManagementPython (Programming Language)DatabricksAzure Service BusAzure Event HUBCloud ApplicationsBizTalkMicrosoft AzureMicrosoft Officedot netC (Programming Language)HTMLPower BI

About

🚀 Senior Data Engineer Lead (Platform & Engineering) | Azure Databricks Lakehouse | Data Governance | CI/CD | FinOps I’m a Senior Data Engineer Lead with 7+ years of experience building and scaling Azure Databricks lakehouse platforms—from ingestion to governance, automation and optimisation. I’m Microsoft Certified (DP-203) and Databricks Certified (Associate & Professional), with strong hands-on delivery across PySpark, SQL, Python, ADF, ADLS Gen2 and Power BI. 🔹 Platform Engineering & Governance I lead Databricks platform standards and governance, including Unity Catalog migrations (500+ tables), access controls, and workspace consolidation to simplify operations and improve reliability. 🔹 Automation, CI/CD & Infrastructure as Code I build repeatable delivery patterns using Azure DevOps CI/CD, Databricks Asset Bundles, and Terraform to automate deployments and provisioning in an auditable, scalable way. 🔹 Performance & Cost Optimisation (FinOps) I focus on making platforms efficient and sustainable—using monitoring, optimisation, and clean-up initiatives that have delivered significant monthly cost savings and improved job performance. 🔹 Leadership & Stakeholder Collaboration I currently lead an 8-member team, working closely with platform owners and architecture stakeholders to deliver secure, reliable data capabilities that teams can trust. I enjoy solving complex data platform problems, setting engineering standards, and enabling teams through well-governed, high-quality data. Let’s connect and talk modern data platforms, Databricks on Azure, governance, and automation. 🚀

Experience

11 yrs 8 mos

Total Experience

5 yrs 10 mos

Average Tenure

7 yrs 6 mos

Current Experience

Tata consultancy services

3 roles

Azure Data Engineer Lead

Promoted

Dec 2024 – Present · 1 yr 4 mos

Assistant Consultant
Leading a team of 6 members in the implementation and maintenance of Azure-based data solutions.
💡 Expertise in Azure Integration Services, including Azure Data Factory, Azure Databricks, Azure Logic Apps, Azure Functions, and API Management.
💡 Optimizing data pipelines, ETL processes, and real-time data streaming solutions.

Azure DatabricksAzure Data FactoryAzure Logic AppsAzure FunctionsAPI Management

Senior Data Engineer

Promoted

Oct 2021 – Dec 2024 · 3 yrs 2 mos

IT Analysts
💡 Developed and maintained enterprise-level Azure data solutions.
💡 Designed and implemented ETL pipelines and data transformation workflows.
💡 Collaborated with stakeholders to optimize data strategies.

Python (Programming Language)Azure Data FactoryPython

System Engineer

Sep 2018 – Sep 2021 · 3 yrs

💡 Built and optimized ETL pipelines for structured and unstructured data.
💡 Developed end-to-end data integration solutions using Azure Data Factory, SSIS, and APIs.
💡 Managed data migration projects, cloud data solutions, and real-time system integrations.

Azure Logic AppsAzure Functions