Abhishek Soni

Backend Engineer

Bengaluru, Karnataka, India4 yrs 6 mos experience

Highly Stable

Key Highlights

Expert in building scalable data pipelines.
Proficient in Apache Airflow and Google BigQuery.
Strong background in data integration and ETL processes.

Stackforce AI infers this person is a Data Engineering expert with a focus on cloud-based data solutions.

Contact

Skills

Core Skills

Data EngineeringEtl ProcessesData IntegrationData MigrationData Visualization

Other Skills

AWSAnalytical SkillsApache AirflowApache SparkAzure SQLBigQueryClouderaData Quality & Governance CloudEnterprise Data PreparationGCPLinuxLookMLLookerMachine LearningMySQL

About

As a dedicated and results-driven Data Engineer with three years of professional experience, I have developed a robust skill set that includes Python, SQL, Shell Scripting, Apache Airflow, Google BigQuery, Google Composer, Apache Hive, Pentaho, and PySpark. My expertise lies in building, optimizing, and maintaining scalable data pipelines and ETL processes to ensure seamless data flow and accessibility for various business needs. In my current role, I have successfully led multiple data integration projects, leveraging my proficiency in Airflow for orchestration and BigQuery for efficient querying and storage solutions. I have also implemented complex data transformations using PySpark, enhancing data quality and performance. My hands-on experience with Composer and Hive has enabled me to streamline workflows and manage big data environments effectively. Additionally, my proficiency in Pentaho has allowed me to design and deploy robust ETL solutions, catering to diverse data sources and formats. I am passionate about leveraging data to drive insights and support data-driven decision-making. I thrive in collaborative environments and am always eager

Experience

American express

Engineer III (Contractor via CBTS)

Dec 2024 – Present · 1 yr 3 mos · Bengaluru, Karnataka, India · Hybrid

I am working on developing and optimizing BigQuery SQL queries, ensuring efficient data transformation and analytics. I am designing and maintaining Apache Airflow DAGs to automate ETL pipelines, enhancing data processing efficiency and reliability. By implementing partitioning and clustering techniques in BigQuery, I am improving query performance and cost-effectiveness.

BigQuerySQLApache AirflowData EngineeringETL Processes

Quantiphi

3 roles

Senior Data Engineer

Promoted

Jul 2024 – Nov 2024 · 4 mos · Bengaluru, Karnataka, India

Optimized Azure SQL tables using clustering and partitioning techniques to enhance query performance and reduce costs.Improved stored procedures by implementing optimized SQL queries and adhering to Azure SQL best practices, ensuring efficient data processing and retrieval.Transitioned scheduling from Azure SQL to Apache Airflow for better workflow management and automation, utilizing
Python for data transformation and scripting tasks. Developed KPI dashboards in Power BI, providing actionable insights and visualizations by integrating Azure SQL data
to monitor performance metrics in real-time.

Azure SQLApache AirflowPythonPower BIData EngineeringData Integration

Data Engineer

Aug 2021 – Jun 2024 · 2 yrs 10 mos · Bengaluru, Karnataka, India

I worked on Cloudera to GCP migration projects, which involved ingestion jobs, transformation jobs, and export framework jobs. In addition, I worked on a Looker dashboard development project, which involved writing code for stored procedure run as scheduled queries using BigQuery Data Transfer, as well as writing LookML code. The dashboard was developed based on Retail and CPG data.

ClouderaGCPLookerBigQueryData MigrationData Visualization

Data Engineer Intern

Apr 2021 – Jul 2021 · 3 mos · Bengaluru, Karnataka, India

During this time i was introduced to fundamentals concept of cloud computing with GCP and AWS. Was trained on Looker BI Tool and some advance concept of SQL.

GCPAWSLookerSQL