Amitava Ghosh

Data Engineer

Kolkata, West Bengal, India5 yrs 4 mos experience

Most Likely To SwitchHighly Stable

Key Highlights

Designed a centralized Data LakeHouse architecture for seamless analytics.
Reduced data processing times by 30% through scalable pipelines.
Conducted over 500 data quality checks ensuring integrity.

Stackforce AI infers this person is a Data Engineering specialist in the SaaS industry.

Contact

Skills

Core Skills

Data EngineeringEtlData Ingestion

Other Skills

ANSI SQLAWS GlueAmazon AthenaAmazon CloudWatchAmazon DynamodbAmazon KinesisAmazon RedshiftAmazon Relational Database Service (RDS)Amazon S3Amazon Web Services (AWS)AnacondaAnalyticsApache AirflowApache HUDIAzure Data Factory

About

Amitava is a passionate Data Engineer with over 5 years of experience in designing and implementing ETL/Data Pipelines for analytics stakeholders (Data Scientists, Analysts, Product Managers). He is skilled in various Data Engineering technologies like SQL, Python, Spark, Airflow, Hive, Git, CICD, Azure and AWS Cloud services, and Data Engineering methodologies like Data/Dimensional Modelling, Star/Snowflake Schema, ETL, Incremental Loads, Slowly Changing Dimensions. His role involves interaction with stakeholders on a day-to-day basis to understand their requirements, use cases, and the scope of the analytics solutions like how the data will be consumed. He has some additional experience in the BI space where he has built few dashboards using tool like Power BI. He is very dedicated, a quick learner and eager to learn new tools and technologies.

Experience

Adidas

2 roles

Data Engineer - P1

Promoted

Jan 2024 – Present · 2 yrs 2 mos · Gurugram, Haryana, India · Hybrid

Designed and implemented a centralized Data LakeHouse architecture to consolidate data from multiple sources, enabling seamless integration and analytics.
Built scalable data pipelines using PySpark and AWS Glue, reducing processing times by 30%.
Developed dynamic DAGs in Apache Airflow, streamlining workflow management and replacing over 100 static DAGs with a single master code.
Led a cost-saving initiative by identifying pipelines consuming excessive time and resources. Generated comprehensive reports highlighting areas for optimization, leading to cost reductions.
Conducted thorough data quality checks using Great Expectations, configuring over 500 validation rules to ensure data integrity across teams.
Developed and implemented code to automate production release process, reducing release time by 70%.

PySparkAWS GlueApache AirflowGreat ExpectationsData EngineeringETL

Data Engineer - P2

Feb 2022 – Dec 2023 · 1 yr 10 mos · Gurugram, Haryana, India · Hybrid

Cognizant

3 roles

Programmer Analyst

Sep 2021 – Jan 2022 · 4 mos · Kolkata, West Bengal, India

Designed and enhanced metadata-driven data ingestion framework for an American multinational cosmetic product manufacturer leveraging Azure Data Factory(ADF) and Databricks.
Designed and created batch data pipelines to ingest structured and unstructured data from diverse sources, including Oracle, SQL Server, SAP HANA, SAP BW, CSV, Parquet, Hive tables, and REST APIs.
Implemented History Load, Incremental Load and Full Load Strategy using ADF Pipeline and Azure
Databricks Notebooks.
Ingested data to Blob Storage in Parquet format from SAP Source, further moved to Raw Layer as External Table in Azure Data Lake and then, to Process Layer in Delta Lake with applied transformations such as deduplication, null handling and data cleansing.
Integrated REST APIs to fetch Retail Store details, transformed JSON output into structured Parquet format using Databricks and Python, enabling seamless downstream analytics.

Azure Data FactoryDatabricksREST APIsData EngineeringData Ingestion

Programmer Analyst Trainee

Sep 2020 – Sep 2021 · 1 yr · Kolkata, West Bengal, India

Digital Analyst

Feb 2020 – Jun 2020 · 4 mos · Kolkata, West Bengal, India

Acquired technical training on Azure services such as Azure Data Factory, Blob Storage, Azure Data Lake, Azure SQL DB and Azure Databricks.
Completed skill development training in Python, SQL, and Power BI Desktop.
Enhanced behavioral and communication skills through dedicated training sessions.