Path 4 · Module 3
Design data integration
Read the module on Microsoft Learn, then come back here: test yourself with the quiz and apply it in the design exercise. Write down what you want to remember with the ✎ Note buttons.
Step 1 · Read the module
Open Design data integration on Microsoft Learn. The units:
- Introduction
- Design a data integration solution with Azure Data Factory
- Design a data integration solution with Azure Data Lake
- Design a data integration and analytic solution with Azure Databricks
- Design a data integration and analytic solution with Azure Synapse Analytics
- Design strategies for hot, warm, and cold data paths
- Design an Azure Stream Analytics solution for data analysis
- Module assessment
- Summary and resources
Close the Learn tab. Answering from memory is what makes it stick; looking it up just measures how fast you can scroll.
Quiz
20 exam-style questions, roughly in the order of the units. Click an answer to see the explanation. Your best score is saved and shown on the path page; use Retry at the end to go again.
1. Data from 30 on-premises and SaaS sources must be copied and transformed on a schedule, mostly without code. Which service?
D. Data Factory is the cloud ETL/ELT orchestration service with 90+ connectors.
2. Data Factory must read from a SQL Server inside the corporate network. What do you install?
C. The self-hosted IR runs on-premises (or in a VNet) to reach private data stores.
3. Existing SSIS packages must run in Azure with minimal changes. What do you use?
D. Azure-SSIS IR lifts and shifts SSIS packages.
4. In Data Factory, what defines a unit of work like "copy data" or "run a notebook"?
D. Pipelines group activities; linked services are connections; datasets describe data; triggers start pipelines.
5. What's the main benefit of Azure Data Lake Storage Gen2 for analytics?
C. The hierarchical namespace makes directory operations fast and enables fine-grained ACLs.
6. A data lake is organized as raw, cleansed and curated layers. What's this pattern often called?
B. Data is refined layer by layer from raw (bronze) to business-ready (gold).
7. Data scientists need collaborative notebooks with Apache Spark for machine learning and large-scale transformations. Which service?
C. Databricks is an Apache Spark analytics platform with collaborative notebooks.
8. Which open storage format adds ACID transactions and time travel to data lake files in Databricks?
D. Delta Lake is the default table format in Databricks.
9. A team wants one service for SQL data warehousing, Spark and data pipelines with a single workspace. Which service?
C. Synapse combines dedicated/serverless SQL pools, Spark pools and pipelines.
10. Analysts must query Parquet files in the data lake with T-SQL, paying only per query. Which option?
C. Serverless SQL pools query files in place and bill per TB processed.
11. A large enterprise data warehouse needs predictable performance with MPP. Which Synapse option?
A. Dedicated SQL pools provision compute (DWUs) for massively parallel processing.
12. In a hot/warm/cold data path design, what's the hot path for?
B. Hot path = low latency, e.g. streaming alerts. Cold path = batch over large historical data.
13. Telemetry must trigger alerts within seconds and also be stored for monthly trend analysis. What do you design?
A. This is the lambda-style split: real-time processing plus batch over full history.
14. Which service ingests millions of events per second as the entry point of a streaming pipeline?
A. Event Hubs is the big-data streaming ingestion service (also IoT Hub for device scenarios).
15. Which language does Azure Stream Analytics use to define jobs?
C. Stream Analytics uses a SQL-like language, extensible with JavaScript and C# functions.
16. Average temperature per device must be calculated every 5 minutes, with no overlap between windows. Which window?
B. Tumbling windows are fixed-size, non-overlapping and contiguous.
17. You need a 10-minute average reported every 2 minutes (overlapping windows). Which window?
C. Hopping windows have a size and a hop; they overlap when the hop is smaller than the size.
18. Which are typical Stream Analytics outputs?
C. Stream Analytics can write to many sinks, including real-time Power BI dashboards.
19. Data Factory must start a pipeline whenever a new file lands in a container. Which trigger?
A. Event-based triggers respond to blob created or deleted events via Event Grid.
20. Which service would you choose for low-code data transformations at scale inside Data Factory, without managing Spark?
C. Mapping data flows are visually designed transformations that run on managed Spark clusters.
Design exercise
On paper · about 25 minutes
Greenfield Logistics wants a data platform.
- Nightly loads from an on-premises Oracle ERP, a Salesforce CRM and CSV files from partners via SFTP.
- GPS data from 5,000 trucks every 10 seconds; a truck that stops for more than 30 minutes outside a depot must raise an alert within a minute.
- Data scientists build delay-prediction models in Python.
- Business users want Power BI reports on curated data; finance wants T-SQL access.
-
Task
Design ingestion
How does each source get into Azure, and where does it land?
Done when- Batch and streaming sources are covered.
- On-premises connectivity is solved.
Model answer
Data Factory with a self-hosted IR for Oracle, the Salesforce connector, storage event trigger for partner files (SFTP-enabled storage). GPS: IoT Hub / Event Hubs. Everything lands in ADLS Gen2 bronze.
-
Task
Design the hot and cold paths
Draw the GPS flow for alerts and for history.
Done when- The 1-minute alert is met.
- History is kept for analytics.
Model answer
Hot: Event Hubs → Stream Analytics (session or sliding window per truck, geofence reference data) → Function/Logic App alert + Power BI live tile. Cold: Event Hubs Capture → ADLS Gen2 → batch processing.
-
Task
Design processing and serving
Which engines transform data and serve it to data scientists, Power BI and finance?
Done when- Medallion layers are named.
- Each consumer has an access path.
Model answer
Databricks (or Synapse Spark) with Delta Lake for bronze → silver → gold and ML notebooks. Finance: Synapse serverless SQL over gold (or a dedicated pool for heavy use). Power BI on gold / SQL endpoints. Orchestration in Data Factory.