Harmeet Singh

Data Engineer

Canada0 mo experience

Key Highlights

  • Over 13 years of experience in data engineering and BI.
  • Expert in cloud technologies and ETL processes.
  • Proven ability to optimize data workflows and reporting.
Stackforce AI infers this person is a Data Engineer with expertise in Cloud Computing and Business Intelligence.

Contact

Skills

Core Skills

Data EngineeringEtlCloud MigrationWeb DevelopmentBusiness Intelligence

Other Skills

Apache AirflowApache SparkAutomationAzure Data FactoryAzure DatabricksBashContract ManagementDBTData Build Tool (DBT)Data ModelingData ScienceExtractGitHubGoogle BigQueryHive

About

• Over 13 years of experience in Cloud, Business Intelligence, Big Data and ETL domain. • CCA 175 - Cloudera Certified Hadoop and Spark Developer and Microsoft Azure Fundamentals Certified. • ETL Technologies Used: Azure Databricks/Data Factory, AWS Glue, NiFi, PySpark, SSIS, Talend, GCP Dataflow • BI Tools Used: IBM Cognos, Power BI, Tableau, Qlik, SSRS. • Database technologies used: MS SQL, PL/SQL, T-SQL, DB2, Oracle, Snowflake, Hive QL, Postgres, Bigquery • Programming Languages Used: Python (Spark, NumPy, Pandas, Matplotlib, tkinter, selenium), Java (in Talend), C#.net (for SSIS script tasks), JavaScript (Node JS and React JS). • Data Modelling Technologies Used: IBM Cognos Framework/Transformer for multi-dimensional modelling, MS SSAS, DAX and Power Query in Power BI. • Date Orchestration : Airflow, Databricks Workflows, SQL Server Agent • Shell Scripts: PowerShell, Bash, KSH • OS Used: Microsoft Windows, Linux, Unix • Scheduling Agents Used: Control-M, Autosys, Talend Administration Center (TAC), Oozie, SQL Server Agent. • Immense ability to learn new business quickly, having working experience in Banking, Contract Lifecycle Management (CLM), Healthcare, Insurance and Telecom domain. • Excellent communication, facilitation skills and proven ability to partner with senior leadership to solve complex problems.

Experience

0 mo
Total Experience
--
Average Tenure
--
Current Experience

Export development canada | exportation et développement canada

Data Engineer

Mar 2025Present · 1 yr 3 mos · Toronto, Ontario, Canada · Remote

Apache SparkAzure DatabricksAzure Data FactoryExtractTransformLoad (ETL)+10

Ford motor company

Senior Data Engineer

Apr 2021Mar 2025 · 3 yrs 11 mos · Waterloo, Ontario, Canada · Remote

  • Migrated on-premise ETLs to Google Cloud Platform (GCP) using cloud-native tools such as BigQuery, DBT, Google Cloud Storage, and Composer.
  • Utilized Python, SQL, and Spark for data transformation and ETL processes, implementing data cleansing, enrichment, and aggregation tasks.
  • Created and managed data storage solutions using GCP services such as BigQuery, Cloud Storage, and Cloud SQL
  • Created and currently maintaining Data Dictionary inhouse webapp using Pyspark as ETL , SQL Server as data storage , React JS as frontend and Node JS as backend technology.
  • Using Visualization tools such as Tableau, Power BI and Qlik Sense to showcase trends and utilization of each new feature released.
  • Created more pipelines using Talend V7.3 Enterprise to fetch data from multiple sources(flat files/oracle/sql server) to Snowflake and ADLS. Used Jenkins/Jfrog/Github for deployments to higher environments.
Apache SparkGoogle BigQueryData Build Tool (DBT)Microsoft Power BIAzure Data FactorySQL Server Integration Services (SSIS)+8

Td

Database / ETL Developer

May 2019Apr 2021 · 1 yr 11 mos · Toronto, Canada Area · On-site

  • Worked in the RISK department of TD.
  • Developed and maintained end-to-end operations of ETL data pipelines and worked with large data sets in Azure Data Factory.
  • Developed Python scripts to do file validations in Databricks and automated the process using ADF.
  • Analyzed data where it lives by mounting Azure Data Lake and Blob to Databricks.
  • Developed ETL pipelines using notebooks, Spark Data frames, SPARK SQL and python scripting
  • Developed UNIX scripts to automate different tasks involved as part of loading process
  • Developed SQL script for monthly archiving and purging process.
  • Created SQL Agent jobs for automating rollback and email notification.
  • Managed code repository using Github and Microsoft TFS.
  • Wrote ETL scripts in Talend and SSIS to fetch data from Cloudera cluster into SQL Server 2017.
  • Design and implement database solutions in Azure SQL Data Warehouse, Azure SQL
Bash

Great-west life

Senior Business Data Analyst

May 2018May 2019 · 1 yr · Toronto, Ontario, Canada

  • Design & implement work flows using Unix / Linux scripting to perform data ingestion and ETL on Big Data platforms.
  • Did various performance optimizations like using distributed cache for small datasets, Partition, Bucketing in hive and Map Side joins.
  • Wrote ETL scripts using Pig (Pig Latin), responsible for managing data from disparate sources.
  • Imported and exported data using Sqoop from Relational Databases(DB2) to HDFS and vice versa.
  • Involved in managing and reviewing Hadoop log files.
  • User Spark API over Hortonworks Hadoop YARN to perform analytics on data in Hive.
  • Designing and developing complex BI reports on unstructured data using Hive and IBM Cognos (version 10.2.1, 10.2.2, 11.0.X).
  • Writing ETL scripts using Pig (Pig Latin), responsible for managing data from disparate sources.
  • Importing and exporting data using Sqoop from Relational Databases(DB2) to HDFS and vice versa.
  • Developing Oozie workflow for scheduling and orchestrating the ETL process.
  • Delivered 3 reporting projects single handedly which involved creating 11 reports/dashboards/active reports in Cognos Analytics(V11.0.11) and Tableau.
  • Building relational and multi-dimensional data models in Cognos Framework and Transformer Cube.
  • Using Spark API over Hortonworks Hadoop YARN to perform analytics on data in Hive
  • Creating Cognos active report and interactive Crosstab report for the Analytics SAS team to provide a visual interface to the underlying data.
  • Performing unit testing (manual testing) for the reports that have been developed.
  • Conducting requirement gathering sessions with business teams for BI reporting.
  • Conducting demo and training sessions for the business on the new features of the reporting tool and how to run the reports.
  • Defining and enhancing the reporting architecture.
  • Following agile development practices for the development work.

Shell

Developer

Dec 2015Apr 2018 · 2 yrs 4 mos · Karnataka, India

  • Involved in loading process into Hadoop DFS and Pig to preprocess the data.
  • Developed Spark scripts by using Python commands as per the requirement.
  • Involved in creating Hive tables, and then applied HQL on those tables for data validation.
  • Designed and developed complex BI reports using IBM Cognos (version 10.2.1, 10.2.2, 11.0) for different lines of business.
  • Created a very complex report on quarterly actuals vs targets which helped in realizing profit worth 20 million to the company till date.
  • Built data models in Cognos Framework Manager.
  • Optimized existing BI reports and reduced the average response time by 75%(approx.).
  • Implemented java script in reports.
  • Involved and performed unit testing (manual testing) for the reports that have been developed.
  • Conducted requirement gathering sessions with business teams for BI reporting.
  • Introduced, designed and created technical specification documents and requirement gathering documents for BI reports.
  • Defined and enhanced the reporting architecture.
  • Acted as L3 support to resolve tickets related to reporting issues and enhancements.
  • Participated in onsite workshops as reporting specialist to select suitable vendor for Contract Management tool.
  • Followed agile development practices for the development work
  • Database:
  • Oracle : PL/SQL
  • Other tools used: IBM Emptoris, Saleforce

Foresight group international, ag

Consultant

Jul 2013Dec 2015 · 2 yrs 5 mos · Noida Area, India

  • Designed and developed simple to complex reports as well as dashboards like reports using Report Studio.
  • Applied various concepts and functionalities such as drill-through, conditional reporting, block, tables, charts, master-detail, maps, various calculations and joins using tabular sets and models.
  • Created several Ad-hoc (simple list, grouped list, section heading, crosstab, and nested crosstab) reports and represented data graphically by charts using Query Studio.
  • Implemented Framework Manager Security for row level access and package access based on the Client and Region.
  • Worked with ETL team to design Data Marts by understanding user requirements and increasing the reporting efficiency.
  • Scheduled and maintained reports using Cognos Connection and Microsoft SharePoint based support portals.
  • Built user status and Cognos license consumption reports using data from Cognos content manager database.
  • Performed installation, implementation and integration of Cognos BI Products.
  • Configuration of Web server and Cognos configuration.
  • Worked as a team for creating a product called PVQ for Pharmacovigilance domain which was developed using IBM Cognos SDK, Query Studio and JavaScript.
  • Worked with QA team to design test plan and test cases for User Acceptance Testing (UAT).
  • Also worked on reporting tools like Microsoft SSRS, Tableau and J Reports
  • Extracted data from internal data warehouse system to SSRS
  • Implemented complex features like drill through, drill down, drill up, hyperlinks in summary tabulations as part of report development.
  • Authored scripts to export report files and data sources from one server to other.
  • Hands on experience on Tableau Desktop versions 7/8.2/10.5, Tableau Reader and Tableau Server.
  • Developed Tableau data visualization using Cross tabs, Heat maps, Box and Whisker charts, Scatter Plots, Geographic Map, Pie Charts and Bar Charts and Density Chart.

Infosys

Systems Engineer

Sep 2011Jul 2013 · 1 yr 10 mos

  • Wrote test case scripts to test LTE(4G) call flows.
  • Developed, automated and generated Cognos based reports of call flows on LTE Scenario.
  • Stored and managed call flow data using PL/SQL procedures.
  • Debugging issues and modifications in already go-live reports
  • Involved in conducting demos and presentations to prospective clients visiting LTE Lab.
  • Modelled metadata using framework manager for implementing new projects
  • Participated in requirement gathering workshops as technical specialist to present POCs for the requirements.
  • Optimized PL/SQL queries using tools like TOAD, SQL Developer etc.

Education

Guru Tegh Bahadur Institute Of Technology

Bachelor of Technology (BTech) — Computer Science

Jan 2007Jan 2011

Guru Harkrishan Public School

High school — Non Medical

Jan 1994Jan 2007

Stackforce found 100+ more professionals with Data Engineering & Etl

Explore similar profiles based on matching skills and experience