Azure Synapse Dedicated SQL Pool backup monitoring (part 3)

home

In my last post Azure Synapse Dedicated SQL Pool backup monitoring (part 2), we used Azure Monitor HTTP Data Collector API which is now deprecated. Updating the design required making use of Azure Monitor’s Log Ingestion API. On a high level, the Logs Ingestion API in Azure Monitor lets you send data to a Log Analytics workspace using either a REST API call or client libraries. The API allows you to send data to supported Azure tables or to custom tables that you create. You can also extend the schema of Azure tables with custom columns to accept additional data.

In this use case Log Ingestion API will be used inside the Azure Function which sends the queried backup logs to Custom Tables in Azure Log Analytics Workspace. Let’s list down the components that will be used when working with Log Ingestion API.

Data Collection Endpoint

A data collection endpoint (DCE) is a connection that the Logs ingestion API uses to send collected data for processing and ingestion into Azure Monitor. Azure Monitor Agent also uses data collection endpoints to receive configuration files from Azure Monitor and to send collected log data for processing and ingestion. The JSON source data exists in your API request payload. The API call performs the following three tasks.

  • Defines a data collection rule (DCR) that recognizes the source data structure
  • Refines and converts the source data to a structure valid in the target table
  • Sends the transformed data to the target table in the workspace

Data Collection Rule

Data collection rules (DCRs) are sets of instructions supporting data collection in Azure Monitor. They provide a consistent and centralized way to define and customize different data collection scenarios. Depending on the scenario, DCRs specify such details as what data should be collected, how to transform that data, and where to send it.

DCRs are stored in Azure so that you can centrally manage them. Different components of a data collection workflow will access the DCR for particular information that it requires. In some cases, you can use the Azure portal to configure data collection, and Azure Monitor will create and manage the DCR for you. Other scenarios will require you to create your own DCR. You may also choose to customize an existing DCR to meet your required functionality. More information DCR’s can be found here.

A revised design for this would require a centralized approach…