Dipak Shaw

Data Engineer

Hyderabad, Telangana, India11 yrs 9 mos experience
Most Likely To SwitchHighly Stable

Key Highlights

  • Over 11 years of experience in data engineering.
  • Expertise in building data ingestion pipelines using Azure.
  • Strong background in multiple domains including Telecom and Retail.
Stackforce AI infers this person is a Data Engineering expert with extensive experience in cloud-based solutions and analytics.

Contact

Skills

Core Skills

Data EngineeringAzureData AnalysisBusiness IntelligenceBig Data EngineeringData Processing

Other Skills

AIAI Software DevelopmentAPI DevelopmentAgile DevelopmentAgile MethodologyAmazon AuroraAmazon S3Apache KafkaApache PigApache SparkApache Spark StreamingArchitectureAzure AIAzure AI StudioAzure Active Directory

About

I am driven by curiosity to learn, which keeps me moving to great heights. I started my career as a Big Data Engineer, which gave me a huge boost in my career. I have around 11+ years of experience in the data engineering and analytics fields. I learned so much from being able to work in multiple roles, right from solving technical problems to interacting with clients and building meaningful relationships. Every phase gave me an opportunity to learn, and I took ownership and have shown a positive approach. Currently, I am working with Microsoft. My role and responsibility are to build the ingestion pipeline to ingest and consume the data from multiple sources, clean, transform, and export the data into the target table and destination location for the reporting and make it available for further application using Azure services, Databricks, Power BI, SQL Server, etc by handling the several challenges with the data - privacy, unstructured and volume. Prior to Microsoft, I worked with HCL Technologies and Brillio Analytics, helping customers build data lakes, use an ETL pipeline to bring data into them from multiple sources, and analyze data to gain more insight into the business. I am also experienced in performing RCA on data and processes to answer specific business questions and identify opportunities for improvement. In addition to this, I am also having expertise experience in the Telecom, Retail, Logistics, and HR domains with multiple data-driven solutions, which has matured me as an IT professional with strong execution, methodology, and coding skills. Skills Highlight: Hadoop Components (HDFS, Hive, HBase, Sqoop, Map Reduce, Oozie, etc), Hortonworks, Cloudera, Spark, Scala, Python, AWS, Azure, Machine Learning, AI, Aurora, RabbitMQ, Shell Script, CICD, GitHub, Data Modeling & Designing, Data Warehousing and many more. I am an independent individual, and never stop at one thing, keep on upgrading myself, and believe in moving forward. He also loves motivating people and helping them achieve great things. As a wise man said once: "You are always a student, never a master. You have to keep moving forward!"

Experience

11 yrs 9 mos
Total Experience
2 yrs 11 mos
Average Tenure
6 yrs 2 mos
Current Experience

Microsoft

3 roles

Senior Data Engineer

Promoted

Sep 2023Present · 2 yrs 9 mos

  • Currently, I am working with Microsoft. My role and responsibility are to build the ingestion pipeline to ingest and consume the data from multiple sources, clean, transform, and load the data into the target table and destination location for reporting and make it available for further analysis.
  • Provide Timely Trusted & Relevant Data to HR Services Insights business (for the prioritized business process) with the best Engineering practices and Modern technologies adhering to Data Governance and Compliance. Lead migration of an on-premises Data Warehouse to Azure using Modern Tech
  • From the technical stack, it includes - Apache Spark, Azure Active Directory, Azure Data Lake Storage, Azure Blob Storage, Azure Databricks, Azure Synapse, Azure Data Flow, Azure DevOps, Azure Data Explorer, Event Hub, Function App REST API, Azure Data Factory, Power BI, SQL Server, Cosmos DB, Azure Open AI, Azure Purview, etc.Currently, I am working with Microsoft. My role and responsibility are to build the ingestion pipeline to ingest and consume the data from multiple sources, clean, transform, and load the data into the target table and destination location for reporting and make it available for further analysis. Provide Timely Trusted & Relevant Data to HR Services Insights business (for the prioritized business process) with the best Engineering practices and Modern technologies adhering to Data Governance and Compliance. Lead migration of an on-premises Data Warehouse to Azure using Modern Tech From the technical stack, it includes - Apache Spark, Azure Active Directory, Azure Data Lake Storage, Azure Blob Storage, Azure Databricks, Azure Synapse, Azure Data Flow, Azure DevOps, Azure Data Explorer, Event Hub, Function App REST API, Azure Data Factory, Power BI, SQL Server, Cosmos DB, Azure Open AI, Azure Purview, etc.
Apache SparkAzure Active DirectoryAzure Data Lake StorageAzure Blob StorageAzure DatabricksAzure Synapse+13

Data Engineer 2

Sep 2021Aug 2023 · 1 yr 11 mos

  • Provide Timely Trusted & Relevant Data to HR Services Insights business (for the prioritized business process) with the best Engineering practices and Modern technologies adhering to Data Governance and Compliance. Lead migration of an on-premises Data Warehouse to Azure using Modern Tech
  • From the technical stack, it includes - Apache Spark, Azure Active Directory, Azure Data Lake Storage, Azure Blob Storage, Azure Databricks, Azure Synapse, Azure Data Flow, Azure DevOps, Azure Data Explorer, Event Hub, Function App REST API, Azure Data Factory, Power BI, SQL Server, Cosmos DB, Azure Open AI, Azure Purview, etc.
Apache SparkAzure Active DirectoryAzure Data Lake StorageAzure Blob StorageAzure DatabricksAzure Synapse+13

Data Engineer

Mar 2020Sep 2021 · 1 yr 6 mos

  • Migrated the Legacy system to Modern technologies for reducing the data available time and more secure.

Brillio

Senior Data Engineer

Aug 2018Jan 2020 · 1 yr 5 mos · Bengaluru Area, India

  • In Brillio Analytics, I was helping customer, which is at the 1st rank in the world in the logistic domain, by creating a data lake, bringing the data into it from many sources, and analyzing data for Reducing the wastage (truck returning empty, energy sensor data, etc.) and implementing the platform for their customer visualization.
Data LakeData AnalysisData VisualizationData Engineering

Hcl technologies

3 roles

Business Intelligence Developer

Oct 2016Aug 2018 · 1 yr 10 mos · Kolkata Area, India

  • My role in the team was to develop and deliver modules like Consume messages from RabbitMQ, Ingesting data into Hadoop components, Processing a huge amount of data with business logic. I am developing/modifying existing feature based on the CR and deploying on the production. My experience and quick learner skills enabled me to get involved in the task which demands new technologies.
  • My learning curve involved the following technical skills and tools - Python, Spark, Hadoop Stack, Microsoft Azure Cloud, MongoDB, Gurobi, Shell Programming, CAWA, Jupiter, Jenkins, Toad, CICD, Confluence.
PythonSparkHadoopMongoDBRabbitMQShell Programming+2

Senior Big Data Engineer

Promoted

Sep 2015Oct 2016 · 1 yr 1 mo · Kolkata Area, India

  • As Senior Big Data Engineer, my role in the team was to take the responsibility to develop the job, deliver on the time with proper testing and code tuning performance. This includes interaction with clients, understand the tasks requirements, assign the task to the team member, keep the updates of the tasks, fix the issue raised by QA.
  • To enhance the skills for this role, I have acquired the following technical skills like Map Reduce, Hive, Pig, Pig unit Test, Spark, RabbitMQ, Oozie, Java, Maven, Shell Scripting, Agile development methodology. I have used following tools - Git, Jenkins, HP ALM Quality Center, JIRA, Eclipse.
  • I have completed some of the internal certification to keep enhanced my skills:
  • Data factory with hive
  • Spark
  • Data Repository with HDFS
  • Map Reduce Essential
Map ReduceHivePigSparkRabbitMQOozie+3

Big Data Engineer

Nov 2014Sep 2015 · 10 mos · Kolkata Area, India

  • I joined as a Big Data Engineer(Software Engineer) in HCL Technologies. As Hadoop Developer, my role among the team members was to take a project module ownership to develop end to end. This includes an understanding of the requirements and gathering all the relevant information for the same, develop the requirement, unit Testing, documentation if required, deployment, support, and maintenance in the production. In addition to this, have done peer review, code tuning etc. in order to optimize the code and performance.
  • In order to enhance the skills for this role, have acquired the following technical skills - Agile development Methodology, HDFS, Hive, HBase, Sqoop, Pig, Map Reduce, Oozie, Java, Maven, Teradata, Shell Scripting. In addition to this technical skill, have used the tools like Putty, WinSCP, Eclipse, HP ALM Quality Center, GIT, JIRA.
  • As Hadoop Developer, have completed some of the HCL internal certification:
  • Agile Development Methodology
  • Big Data
Agile DevelopmentHDFSHiveHBaseSqoopPig+3

Pratian technologies (india) private limited

Trainee Engineer

Jun 2014Nov 2014 · 5 mos · Bangalore

  • It was great learning during the period. Here, I had worked in a travel domain project with multiple skill sets including programming, design pattern, agile methodology, deployment, and visualization.
ProgrammingAgile Methodology

Education

MCKV Institute of Engineering

Bachelor’s Degree — Computer Science & Engineering

Stackforce found 100+ more professionals with Data Engineering & Azure

Explore similar profiles based on matching skills and experience