There is a wide range of monitoring capabilities for watching Azure services. When it comes to logging, Log Analytics workspaces are important instruments on Azure where we manage the logs as the first step of the monitoring lifecycle. With some major changes over the years, Log Analytics has evolved a lot in terms of log and query management. So, it's now easier than ever to query logs and export them to another location, such as a storage account. In this post, we'll have a look at how we can export Log Analytics logs using PowerShell.

There are several options for exporting Log Analytics logs, depending on what needs to be exported. If we want to export specific logs to a storage account in an automated way, then we can use a new feature called "Data Export," which allows automated export of logs to a storage account or an Event Hub. If a storage account is set, then Log Analytics sends the specified logs to the storage account hourly. It's done in near real-time when you set an Event Hub as the destination. To enable this option, we can either use Azure CLI or specific REST APIs. Fortunately, PowerShell makes it easier to use REST APIs. So, although indirectly, we can use PowerShell for this option.

The second option for pulling Log Analytics logs is to execute a PowerShell cmdlet to export the specified logs with custom Kusto Query Languages (KQL) queries. Optionally, we can automate the process by executing this PowerShell command according to a schedule as well.

Let's take a look at the first option now.

Exporting Log Analytics logs to a storage account using REST API calls in PowerShell ^

With this option, we will specify a storage account that must be in the same region as the Log Analytics workspace. Once configured, Log Analytics will start sending the specified logs to the storage account according to an hourly schedule. Here is the PowerShell script that we will use to leverage a REST API to manage data export operations:

function Manage-LogAnalyticsDataExport {

    [CmdletBinding()]
    param(
        [Parameter(Mandatory)]
        [ValidateSet('Get', 'Create', 'Disable', 'Enable', 'Delete')]
        [string]$Action,
        [Parameter(Mandatory)]  
        [string]$DataExportName
    )
    
    # Variables
    $TenantId = "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
    $SubscriptionId = "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
    $ClientId = "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
    $ClientSecret = "xxxxxxxxxxxxxxxxxxxxxxxxxxxx"
    $Resource = "https://management.core.windows.net/"
    $storageAccountName = "stracc000929273"
    $storageAccountRG = "stracc00001"
    $workspaceName = "DefaultWorkspace-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxx-WEU"
    $workspaceRG = "DefaultResourceGroup-WEU"
    $LogsToExport = @("AppServiceHTTPLogs", "Usage") | convertto-json
    ###
    
    
    $RequestAccessTokenUri = "https://login.microsoftonline.com/$TenantId/oauth2/token"
    $body = "grant_type=client_credentials&client_id=$ClientId&client_secret=$ClientSecret&resource=$Resource"
    $Token = Invoke-RestMethod -Method Post -Uri $RequestAccessTokenUri -Body $body -ContentType 'application/x-www-form-urlencoded'
    $ApiUriGetCreateDisableDelete = "https://management.azure.com/subscriptions/$SubscriptionId/resourceGroups/$workspaceRG/providers/Microsoft.OperationalInsights/workspaces/$workspaceName/dataexports/$dataexportName" + "?api-version=2020-08-01"
    $ApiUriList = "https://management.azure.com/subscriptions/$SubscriptionId/resourceGroups/$workspaceRG/providers/Microsoft.OperationalInsights/workspaces/$workspaceName/dataexports?api-version=2020-08-01"
    $Headers = @{}
    
    if ($action -eq "Disable") {
        $enableDisable = "false"
    }
    else {
        $enableDisable = "true"
    }
    
    $body = @"
    {
        "properties": {
            "destination": {
                "resourceId": "/subscriptions/$SubscriptionId/resourceGroups/$storageAccountRG/providers/Microsoft.Storage/storageAccounts/$storageAccountName"
            },
            "tablenames": $LogsToExport,
            "enable": $enableDisable
        }
    }
"@
    $Headers.Add("Authorization", "$($Token.token_type) " + " " + "$($Token.access_token)")
    
    
    switch ($Action)
    {
    
        "Get" { $command = Invoke-RestMethod -Method Get -Uri $ApiUriGetCreateDisableDelete -Headers $Headers; $command.properties }
        "Create" { $command = Invoke-RestMethod -Method Put -Uri $ApiUriGetCreateDisableDelete -Headers $Headers -Body $body -ContentType 'application/json'; $command.properties }
        "Disable" { $command = Invoke-RestMethod -Method Put -Uri $ApiUriGetCreateDisableDelete -Headers $Headers -Body $body -ContentType 'application/json'; $command.properties }
        "Enable" { $command = Invoke-RestMethod -Method Put -Uri $ApiUriGetCreateDisableDelete -Headers $Headers -Body $body -ContentType 'application/json'; $command.properties }
        "Delete" { $command = Invoke-RestMethod -Method Delete -Uri $ApiUriGetCreateDisableDelete -Headers $Headers; $command }
    }    
}

Once the function is loaded, we can start managing the Log Analytics Data Export feature in PowerShell. There are five options in this script function that allow us to get, create, disable, enable, and delete data exports very easily.

Manage date export feature operations in PowerShell

Manage date export feature operations in PowerShell

When a new data export is created, we should be able to see the logs in the storage account in an hour. There is a separate folder in the storage account for each log specified in the script.

Log Analytics logs are shipped to the storage account

Log Analytics logs are shipped to the storage account

The logs are in JSON lines format. Each record is delimited by a new line, which is much easier to read.

Logs shipped to storage accounts are stored in JSON format

Logs shipped to storage accounts are stored in JSON format

JSON lines are easier to read

JSON lines are easier to read

Exporting logs with custom KQL queries using PowerShell ^

This second option gives us the ability to define a KQL query, which is the default query language of Log Analytics, and get the results in a PowerShell object. Then we can either save the data to a local machine or push it to another place. In my scenario, I'll run a custom KQL query in my PowerShell script and then save the results to my local machine.

This script will simply use the "Invoke-AzOperationalInsightsQuery" cmdlet to send the queries to the Log Analytics workspace that I specify. To specify Log Analytics workspaces, we need to input the workspace ID (also known as "customer ID"). So, the following commands will first get the workspace ID and then send the custom query, which is also embedded in the script, to Log Analytics. As the TimeSpan is set as "3 days," the results for the past three days will be returned.

$workspaceName = "DefaultWorkspace-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxx-WEU"
$workspaceRG = "DefaultResourceGroup-WEU"
$WorkspaceID = (Get-AzOperationalInsightsWorkspace -Name $workspaceName -ResourceGroupName $workspaceRG).CustomerID
$query = 'AppServiceHTTPLogs 
| where ScStatus == "404"'

$result = Invoke-AzOperationalInsightsQuery -WorkspaceId $WorkspaceID -Query $query -Timespan (New-TimeSpan -days 3)

$result.results | export-csv c:\temp\LogAnalyticsLogs.csv -Delimiter "," -NoTypeInformation

$result.results | export-csv c:\temp\LogAnalyticsLogs.csv -Delimiter "," -NoTypeInformation

In the above example, AppServiceHTTPLogs will be searched and all the requests with 404 errors in the past three days will be put into the $result variable. Then the results will be exported to the local machine in CSV format.

Log Analytics logs downloaded in CSV format

Log Analytics logs downloaded in CSV format

With this method, you can go a bit further and automate the whole process using Azure functions or automation account runbooks in order to regularly get logs based on the custom queries that you define.

Subscribe to 4sysops newsletter!

Conclusion ^

The new data export feature in Log Analytics can easily be configured with Azure CLI and REST APIs in order to export the logs to Azure storage accounts and Event Hubs in an automated manner. Now, PowerShell can also easily be integrated with REST APIs to achieve the same with an easier user experience. Alternatively, you can use the "Invoke-AzOperationalInsightsQuery" cmdlet to execute on-demand queries against Log Analytics and get the results right away as a quick win.

0
1 Comment
  1. Rajesh Sura 10 months ago

    Baki Onur Okutucu thanks for nice notes. could you please share more about custom KQL querrs that would be very helpful to me

    0

Leave a reply

Please enclose code in pre tags

Your email address will not be published. Required fields are marked *

*

© 4sysops 2006 - 2021

CONTACT US

Please ask IT administration questions in the forums. Any other messages are welcome.

Sending

Log in with your credentials

or    

Forgot your details?

Create Account