Rui Carvalho

Data Engineer

Porto, Porto, Portugal8 yrs 9 mos experience

AI EnabledAI ML Practitioner

Key Highlights

Expert in building scalable data pipelines.
Proficient in Azure cloud data architecture.
Speaker at data events and Medium writer.

Stackforce AI infers this person is a Data Engineer specializing in cloud-based data solutions for SaaS and Healthcare industries.

Contact

Skills

Core Skills

DatabricksAzure Cloud Data ArchitectureSparkData PipelinesEtlData WarehousingBusiness IntelligenceData VisualizationData Integration

Other Skills

Azure Data FactoryOptimizationAzureData ArchitectureMicrosoft FabricSpark SQLPysparkData FactorySynapsePowerBIDAXSQLGenAIAgent BricksMLflow

About

Data Engineer specializing in cloud-based data solutions with expertise in Databricks, Python, SQL, and Azure Data Factory. I focus on building efficient, scalable data pipelines and optimizing analytical workloads in cloud environments. My work centers on implementing best practices in data engineering, from notebook development to pipeline orchestration, ensuring code quality, performance, and maintainability. Core Competencies: Advanced Databricks development (PySpark, Delta Lake, Unity Catalog). Azure cloud data architecture and ETL/ELT pipelines. Data modeling, optimization and governance. Code quality and engineering best practices.

Experience

8 yrs 9 mos

Total Experience

2 yrs 6 mos

Average Tenure

2 yrs 6 mos

Current Experience

International workplace group plc

Senior Data Engineer

Mar 2025 – Present · 1 yr 2 mos · Remote

Advanced Spark Optimization: Engineered and optimized complex Spark workflows, achieving 30-40% overall reduction in processing times of different pipelines and infrastructure costs through strategic partitioning, caching and resource tuning.
Designed and implemented scalable data architectures on Azure using Databricks as the primary data processing and transformation layer, handling multi-TB datasets across bronze, silver and gold layers.
Created monitoring and optimization frameworks for identifying unused data assets and improving overall platform efficiency.
Followed coding standards and best practices for Python/PySpark development, including modular design patterns, error handling, comprehensive documentation, and unit testing with pytest to ensure code reliability and regression prevention.

DatabricksAzure Data FactoryAzure cloud data architecture

Devscope

3 roles

Senior Data & Analytics Engineer

Aug 2024 – Mar 2025 · 7 mos

Develop Data Pipelines using Spark SQL & Pyspark.
Using Databricks as a Data Process & Transformation Layer.
Developed and maintained ETL using Data Factory.
Implemented data warehousing solutions with Azure Databricks and SQL Server.
Optimized SQL queries for on-prem and cloud environments, enhancing system data performance and efficiency.
Maintenance of SSAS and SSIS ETL Pipelines.
Mentoring and guiding.
Collaborating with AI and BI teams to support data needs in different areas.
Projects following Agile methodology.

Azure Data FactoryMicrosoft FabricData PipelinesETL

Data & Analytics Engineer

Promoted

Jan 2020 – Aug 2024 · 4 yrs 7 mos

Developed and maintained ETL using Data Factory and Synapse, ensuring efficient and reliable data processing.
Implemented data warehousing solutions with Azure Databricks and SQL Server, focusing on performance and scalability.
Develop Data Pipelines using Spark SQL & Pyspark.
Optimized SQL queries for on-prem and cloud environments, enhancing system data performance and efficiency.
Maintenance of SSAS and SSIS ETL Pipelines.

Azure Data FactoryPySparkData WarehousingETL

BI Developer

Nov 2018 – Jan 2020 · 1 yr 2 mos

Focused on building PowerBI datasets and/or reports over data warehouses for projects in retail, finance, and leasing.
Developed PowerBI reports and semantic models for KPI analysis and financial control, supporting decision-making processes.
Building models and ETL using Power Query.
Used DAX language for performance indicators.
Used SQL to work and clean data from different sources.

PowerBIDAXBusiness IntelligenceData Visualization

The data therapy

Writer and Editor

Nov 2023 – Present · 2 yrs 6 mos

Write Data blog posts for The Data Therapy Publication.
Review publication posts from other users.

Altran

Junior BI Developer

Aug 2017 – Nov 2018 · 1 yr 3 mos · Porto e Região, Portugal

Supported ETL SSAS and SSIS (On-Prem), ensuring efficient data integration and processing.
Maintained and developed Tabular models, enhancing data analysis capabilities.
Developed PowerBI reports, facilitating accessible and insightful data visualization.
Created indicators for management and decision-making, focusing on healthcare sector.
Contributed to projects aimed at making medical information more accessible and centralized, significantly reducing the time for healthcare professionals to access hospital activity data.

PowerBIDAXBusiness IntelligenceData Visualization