Karan Singh

Software Engineer

New Delhi, Delhi, India3 yrs 9 mos experience

Key Highlights

Expert in Azure Data Factory and Databricks.
Proficient in building Power BI reports.
Strong background in data engineering and analysis.

Stackforce AI infers this person is a Data Engineer specializing in ETL processes and data analytics within SaaS environments.

Contact

Skills

Core Skills

Data EngineeringData Analysis

Other Skills

AlgorithmsAzure Data FactoryAzure DatabricksAzure SQLBig DataBlob StorageCPPDAXData IntegrationData Lake ServicesData MiningData StructuresData WarehousingDatabasesETL

Experience

Maq software

Software Engineer

Jun 2022 – Present · 3 yrs 9 mos · New Delhi, Delhi, India · On-site

Created Pipelines in ADF using Linked Services/Datasets/Pipeline to Extract, Transform and load data from diﬀerent sources like Azure SQL, Blob Storage, and Azure SQL Data warehouse, Azure Data Lake Services. Scheduled and monitored the jobs in Azure Data Factory for the automated process.
Developed and maintained end-to-end operations of ETL data pipelines and worked with large data sets in Azure Data Factory.
Implemented various components like pipeline, activity, data sets, parameters, linked services, jobs, and triggers in Data Factory.
Designed and implemented effective database solutions (Azure Blob storage, Azure Data Lake) to store and retrieve data.
Data Ingestion to one or more Azure services – Azure Data Lake, Azure Storage, Azure SQL, Azure DW and processing in Azure Databricks.
Have good understanding of understanding of provisioning of Databricks and Azure resources.
Created and developed databricks notebooks for data load, process, and publish using SQL, PySpark, Python modules numpy, pandas, SparkSQL and to implement business logic.
Implemented various components like clusters, notebooks, scope, keyvault, mount points, executing the notebooks via scheduled triggers in Data factory in Azure Databricks.
Ability to operationalize and apply CI/CD (Azure Devops) to the application on Databricks, Data Factory, Data Lake, Azure SQL database services.
Created Build and Release for modules in production environment using Visual Studio Team Services (Azure DevOps).
Experience with various sources to pull data into Power BI such as SQL Server, Excel, SQL Azure, Tabular, Cube to build reports.
Created SSAS cube/tabular with KPIs, aggregation, measures using DAX, relationships between Dim and Fact tables for the front-end analysis team and to build Power BI reports.
Product Backlog Items, User Stories, Tasks, and Bugs were maintained using Azure Devops.
Merge code to Azure Devops Git repository and to automate Azure Deployment of CI/CD process.