← Back to ForEach Academy dashboard Compare layouts (A/B/C)

AzureMX

Maximum Acceleration - Metadata-Driven Data Migration & Modernization Framework

1. Source Systems

Enterprise Data Sources
🗄️

SQL Server

Relational Databases

💾

Legacy Systems

On-Prem Databases

📁

Flat Files

CSV, TXT, Parquet

Metadata-Driven Ingestion

2. Azure Data Factory

Orchestration Layer
⚙️

Metadata-Driven Execution

  • Reads metadata tables
  • Determines load types
  • Handles dependencies & retries
  • Parameterizes workflows
Data Extraction

3. ADLS Gen2 - Raw Zone

Landing Zone
☁️
source_system/
schema/
table/
load_date/
Transformation

4. Azure Databricks

PySpark Processing
🔄

Full Load

Complete reload

📈

Incremental

Watermark-based

📸

Snapshot

Point-in-time

Schema Alignment
Deduplication
Data Standardization
Performance Optimization
Curated Output

5. Destination Systems

Analytics Ready
📊

ADLS Curated

Standardized datasets

🔷

Azure Synapse

Analytics platform

📈

Downstream Systems

Business intelligence

Audited & Validated Row counts, timestamps, status tracking

Key Features

🎯

Metadata-Driven

One framework supports hundreds of tables. New tables onboarded by adding metadata, not code.

Multiple Load Types

Supports full loads, incremental loads, and snapshot loads in a single framework.

🔍

Auditing & Control

Comprehensive audit framework tracking row counts, load duration, and statuses for every run.

🔄

Backdated Processing

Supports reprocessing historical date ranges without affecting current production loads.

📈

Performance Optimized

Optimized Databricks jobs with partitioning, efficient file sizing, and minimal shuffles.

🌐

Multi-Destination

Supports multiple destination targets including ADLS, Synapse, and downstream analytics systems.