Ingest
Land source data in a durable raw layer without losing the original structure.
Project 02
Senior Analytics Engineer
Cloud Data Engineering
Enterprise Lakehouseon AWS.
An end-to-end analytics platform that moved raw business data through governed Bronze, Silver and Gold layers before serving trusted insights through Athena and Power BI.
AWSData EngineeringLakehousePower BI
The Challenge
Raw data was available.Trusted analytics was not.
Business data often begins as disconnected files with inconsistent structures, repeated preparation work and no clear path from source to reporting.
The goal of this project was to design the full analytical journey—not only a dashboard—by creating a cloud-based foundation for ingestion, transformation, modeling, validation and business intelligence.
The architecture also needed to remain understandable, maintainable and easy to extend as new datasets and reporting requirements were introduced.
The Approach
Build the analytics foundation before building the reporting experience.
The platform used a medallion architecture to separate raw, validated and business-ready data. Python handled data preparation, Amazon S3 provided scalable storage, Athena enabled serverless SQL access and Power BI delivered the semantic and reporting layer.
Transform
Clean, validate and model the data into analytics-ready structures.
Serve
Expose governed datasets through Athena, semantic models and Power BI.
Architecture
From source files to business-ready insights.
Each layer had a clear responsibility, making the movement and transformation of data easier to understand and manage.
01
Source
Business Data
CSV files and operational datasets
02
Ingest
Amazon S3
Centralized cloud object storage
03
Bronze
Raw Layer
Original source data preserved
04
Silver
Validated Layer
Cleaned, standardized and quality checked
05
Gold
Business Layer
Analytics-ready dimensional models
06
Query
AWS Athena
Serverless SQL over optimized Parquet data
07
Experience
Power BI
Semantic models and business reporting
Engineering Decisions
The architecture was shaped by a few practical choices.
Separate storage from compute
Amazon S3 provided a durable and scalable storage foundation, while Athena enabled serverless querying without maintaining a traditional warehouse.
Preserve raw source data
The Bronze layer retained the original source files so transformations remained traceable, repeatable and easier to troubleshoot.
Build quality into the pipeline
Validation and standardization were handled before data reached the business layer, reducing inconsistencies in downstream reporting.
Model around business questions
The Gold layer used analytics-ready dimensional models designed for reporting performance, usability and consistent metric definitions.
Optimize for analytical workloads
Parquet storage and partition-aware querying helped create a more efficient foundation for Athena and Power BI.
Keep the architecture reusable
The pipeline was designed as a repeatable pattern that could support additional datasets and future reporting needs.
Technology Stack
Tools selected around the needs of the platform.
Amazon S3AWS AthenaPythonpandasboto3awswranglerSQLParquetPower BIStar SchemaGitMedallion Architecture
What It Enabled
A clearer and more dependable path from data to reporting.
+
Clearer data organization
Bronze, Silver and Gold layers created an understandable path from raw files to business-ready datasets.
+
More consistent reporting
Shared transformations and dimensional models reduced duplicated reporting logic and inconsistent calculations.
+
Scalable analytical foundation
The architecture provided a repeatable pattern for bringing additional datasets into the platform.
+
Simpler access to trusted data
Athena and Power BI made curated datasets easier for analysts and business users to explore.
Reflection
This project strengthened how I think about the full analytics lifecycle.
Reporting becomes more reliable when ingestion, transformation, validation and modeling are treated as one connected system. The dashboard is the visible layer, but the trust comes from everything built underneath it.