The Big Data pipeline has been touched on numerous times; if you need a refresher, refer to Figure 2.30. Data ingestion is the first phase of that pipeline (refer to Figure 3.1). The pipeline itself is a series of purposefully sequenced activities. The activities can and often do span all the Big Data stages, from…
Before reading any further, ask yourself what an analytical store is. If you struggle for the answer, refer to Table 3.2, which provides the analytical datastores available in Azure, as well as the data model that works optimally with those products. Table 3.1 provides a list of different ingestion types mapped to the most suitable…
Azure Databricks generates a metastore by default with the provisioning of a cluster. After creating a cluster, you can query the metastore. Doing so produces a visualization of any table. Figure 3.22 illustrates the retrieval from the metastore using the show tables in <databaseName> command. Executing the command lists all the tables in the targeted…
The Manage hub includes the enablers to provision, configure, and connect many Azure Synapse Analytics features to other products. From the creation of SQL pools to the configuration of GitHub, read on to learn about each possibility. Analytics Pools When you run a SQL script in Azure Synapse Analytics Studio, some form of compute machine…
This feature is in preview at the time of writing, but it should be mentioned for future reference. A Data Explorer pool is a data analytics service for analyzing real‐time, large‐scale data. When you select the Data Explorer Pools link, the page shown in Figure 3.31 appears. FIGUER 3.31 Azure Synapse Analytics Data Explorer pool…
Azure Purview was introduced in Chapter 1. Azure Purview is very useful for governance, data discovery, and exploration. Keeping tabs on the data sources you have and what they contain is key to being able to securely manage them. There will be more on this in Chapter 8, which discusses data security and governance. When…
Data is ingested in many forms and from a variety of different sources. From a streaming perspective, the data is being generated, sent to a subscriber, and ingested in real time. There is no incremental loading context in the streaming scenario when the data is coming directly from the data producer. Data that is captured…
A dimensional hierarchy is set of related data tables that align to different levels (see Figure 3.19). The tables commonly have one‐to‐many or many‐to‐one relationships with each other. FIGUER 3.19 A dimensional hierarchy The hierarchy shown in Figure 3.19 could be named the “brainjammer dimension.” The frequencies roll up to electrodes, and the electrodes roll…
This SCD type provides a means for viewing historical records of the value prior to an update. This is commonly called versioning. Notice the columns in the top dimension table of Figure 3.16. The SK_ID column is a surrogate key, which is an internal key that identifies the version history of a unique dimensional member….
Managing changes to the data stored on dimension tables over time is an important factor to consider and be handled by designing slowly changing dimensions (SCD). Recognize that the data on the fact table will contain a large amount of data. Making an update to data on that scale is not a prudent approach to…