Power Query This option bears a great resemblance to what you might find in Power BI. The same engine that runs Power BI likely runs the Power Query plug‐in. The feature provides an interface for viewing the data from a selected dataset. You can then run through some transformation ideas and see how the data…
The Azure Resource Manager (ARM) exposes many capabilities you can use to work with the Azure platform. One such capability is an API that allows client systems to send ARM templates to API.These ARM templates contain JSON‐structured configurations that instruct the ARM API to provision and configure Azure products, features, and services. Instead of performing…
An integration runtime (IR) is the compute infrastructure used to run the work configured in your pipeline. Activities such as data flow management, SSIS package execution, data movement, and the monitoring of transformation activities are performed by the IR. There are three types of integration runtimes, which were introduced in Chapter 1. Azure is the…
What is a serving/data exploration layer? Don’t confuse it with something called the servicing layer, which is common in a service‐oriented architecture (SOA). For an illustration of the serving layer, see Figure 3.13. The serving layer is one component of a larger architecture that includes a speed layer and batch layer. The Big Data architecture…
FIGUER 3.42 Adding a workspace package in Azure Synapse Analytics FIGUER 3.43 Consuming a workspace package in Azure Synapse Analytics Configuring the workspace is a very powerful aspect of the platform. As long as your custom code runs with the default installed comments, you can run just about any computation. You are limited only by…
FIGUER 3.45 Azure Synapse Analytics configure GitHub FIGUER 3.46 Azure Synapse Analytics configure GitHub repository FIGUER 3.47 Azure Synapse Analytics configure GitHub saved You should not make public the repository where you store the Azure Synapse Analytics content, because it contains some sensitive information—for example, your Azure subscription number and the general configuration of your…
The Data, Develop, and Integrate hubs each have a link to the Gallery. When you click the Browse Gallery link after clicking the + to the right of either hub, you will see something similar to Figure 3.55. FIGUER 3.55 The Azure Synapse Analytics Browse Gallery This page includes templates for databases, datasets, notebooks, SQL…
An Apache Spark pool is the compute node that will execute the queries you write to pull data from, for example, Parquet files. You can provision a Spark pool from numerous locations. One such place is from the Manage page in Azure Synapse Analytics Studio. After clicking the Manage hub option, select Apache Spark pools….
Before reading any further, ask yourself what an analytical store is. If you struggle for the answer, refer to Table 3.2, which provides the analytical datastores available in Azure, as well as the data model that works optimally with those products. Table 3.1 provides a list of different ingestion types mapped to the most suitable…
Data is ingested in many forms and from a variety of different sources. From a streaming perspective, the data is being generated, sent to a subscriber, and ingested in real time. There is no incremental loading context in the streaming scenario when the data is coming directly from the data producer. Data that is captured…