Telemetry and Monitoring - AL-Go for GitHub

AL-Go for GitHub includes built-in telemetry capabilities to help you monitor workflow performance, troubleshoot issues, and gain insights into your DevOps processes.

Telemetry Overview

Partner Telemetry

Send AL-Go telemetry to your own Application Insights for custom analysis and dashboards

Microsoft Telemetry

Optionally share workflow data with Microsoft for product improvements and support

Quick Start

Enable telemetry by adding your Application Insights connection string to the AL-Go settings file.

Enable Partner Telemetry

Add the following setting to .github/AL-Go-Settings.json:

{
  "PartnerTelemetryConnectionString": "<your-application-insights-connection-string>"
}

Create Application Insights

If you don’t have an Application Insights resource:

Navigate to Azure Portal
Create a new Application Insights resource
Copy the connection string from the Overview page

Add Connection String

Update your AL-Go settings file with the connection string.

Commit and Run Workflow

Commit the settings change and run any AL-Go workflow to start collecting telemetry.

Your connection string is typically in the format: InstrumentationKey=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx;IngestionEndpoint=https://....

Microsoft Telemetry Options

AL-Go sends basic telemetry to Microsoft by default. You can control this behavior:

Opt Out
Extended Telemetry
Default (Basic)

Disable Microsoft telemetry completely:

{
  "microsoftTelemetryConnectionString": ""
}

Enable extended telemetry to help Microsoft investigate issues:

{
  "SendExtendedTelemetryToMicrosoft": true
}

Extended telemetry includes additional diagnostic information helpful for troubleshooting repository-specific issues.

Keep the default settings to send basic telemetry:

{}

No configuration needed - basic telemetry is enabled by default.

Telemetry Events

AL-Go emits four main types of telemetry events:

Workflow Completed

Emitted when an AL-Go workflow finishes successfully

Workflow Failed

Emitted when an AL-Go workflow fails or times out

Action Completed

Emitted when an AL-Go action completes successfully

Action Failed

Emitted when an AL-Go action encounters an error

Common Telemetry Dimensions

All telemetry events include these standard dimensions:

Dimension	Description
PowerShellVersion	Version of PowerShell executing the action
BcContainerHelperVersion	Version of BcContainerHelper (if imported)
WorkflowName	Name of the GitHub Actions workflow
RunnerOs	Operating system of the runner (Windows, Linux, macOS)
RunId	Unique GitHub Actions run ID
RunNumber	Sequential run number
RunAttempt	Retry attempt number
Repository	Repository identifier
RepositoryOwner	GitHub organization or user owning the repository
RepositoryName	Full repository name (owner/repo)
RefName	Git reference (branch or tag) that triggered the workflow
ALGoVersion	Version of AL-Go being used

Event Details

AL-Go Workflow Ran

Message: AL-Go workflow ran
Severity Level: 1 (Informational) Additional Dimensions:

Dimension	Description
WorkflowConclusion	Success or Cancelled
WorkflowDuration	Total workflow execution time (seconds)
RepoType	AppSource or PTE
GitHubRunner	Value of the gitHubRunner setting
RunsOn	Value of the runs-on setting

AL-Go Workflow Failed

Message: AL-Go workflow failed
Severity Level: 3 (Error) Additional Dimensions:

Dimension	Description
WorkflowConclusion	Failure or TimedOut
WorkflowDuration	Total workflow execution time (seconds)
RepoType	AppSource or PTE
GitHubRunner	Value of the gitHubRunner setting
RunsOn	Value of the runs-on setting

AL-Go Action Ran

Message: AL-Go action ran
Severity Level: 1 (Informational) Additional Dimensions:

Dimension	Description
ActionDuration	Action execution time (seconds)
RunnerEnvironment	GitHub-hosted or self-hosted

AL-Go Action Failed

Message: AL-Go action failed
Severity Level: 3 (Error) Additional Dimensions:

Dimension	Description
ErrorMessage	The error message that caused the failure
RunnerEnvironment	GitHub-hosted or self-hosted

Test Results

AL-Go logs telemetry for three types of test results:

AL-Go Test Results - Tests
AL-Go Test Results - Page scripting Tests
AL-Go Test Results - BCPT Tests

Severity Level: 1 (Informational) Test Dimensions:

Dimension	Description
TotalTests	Total number of tests executed
TotalPassed	Number of tests that passed
TotalFailed	Number of tests that failed
TotalSkipped	Number of tests skipped
TotalTime	Total test execution time (not available for BCPT)

The TotalTime dimension is only available for standard tests and page scripting tests. BCPT test results do not include timing information.

Querying Telemetry Data

Use Kusto Query Language (KQL) to analyze your AL-Go telemetry in Application Insights.

Workflow Completion Query

Get all completed workflow runs from the last 7 days:

traces
| where timestamp > ago(7d)
| project   timestamp,
            message,
            severityLevel,
            RepositoryOwner = tostring(customDimensions.RepositoryOwner),
            RepositoryName = tostring(customDimensions.RepositoryName),
            RunId = tostring(customDimensions.RunId),
            RunNumber = tostring(customDimensions.RunNumber),
            RunAttempt = tostring(customDimensions.RunAttempt),
            WorkflowName = tostring(customDimensions.WorkflowName),
            WorkflowConclusion = tostring(customDimensions.WorkflowConclusion),
            WorkflowDurationMinutes = round(todouble(customDimensions.WorkflowDuration) / 60, 2),
            ALGoVersion = tostring(customDimensions.ALGoVersion),
            RefName = tostring(customDimensions.RefName)
| extend HtmlUrl = strcat("https://github.com/", RepositoryName, "/actions/runs/", RunId)
| where message contains "AL-Go workflow"

Action Completion Query

Analyze individual action performance:

traces
| where timestamp > ago(7d)
| project   timestamp,
            message,
            severityLevel,
            RepositoryOwner = tostring(customDimensions.RepositoryOwner),
            RepositoryName = tostring(customDimensions.RepositoryName),
            RunId = tostring(customDimensions.RunId),
            RunNumber = tostring(customDimensions.RunNumber),
            RunAttempt = tostring(customDimensions.RunAttempt),
            WorkflowName = tostring(customDimensions.WorkflowName),
            WorkflowConclusion = tostring(customDimensions.WorkflowConclusion),
            WorkflowDuration = todouble(customDimensions.WorkflowDuration),
            ALGoVersion = tostring(customDimensions.ALGoVersion),
            RefName = tostring(customDimensions.RefName),
            RunnerOs = tostring(customDimensions.RunnerOs),
            RunnerEnvironment = tostring(customDimensions.RunnerEnvironment),
            ErrorMessage = tostring(customDimensions.ErrorMessage),
            ActionDurationSeconds = todouble(customDimensions.ActionDuration)
| extend HtmlUrl = strcat("https://github.com/", RepositoryName, "/actions/runs/", RunId)
| where message contains "AL-Go action"

Test Results Query

Track test execution and pass rates:

traces
| where timestamp > ago(7d)
| project   timestamp,
            message,
            severityLevel,
            RepositoryOwner = tostring(customDimensions.RepositoryOwner),
            RepositoryName = tostring(customDimensions.RepositoryName),
            RunId = tostring(customDimensions.RunId),
            RunNumber = tostring(customDimensions.RunNumber),
            RunAttempt = tostring(customDimensions.RunAttempt),
            RefName = tostring(customDimensions.RefName),
            TotalTests = todouble(customDimensions.TotalTests),
            TotalFailed = todouble(customDimensions.TotalFailed),
            TotalSkipped = todouble(customDimensions.TotalSkipped),
            TotalPassed = todouble(customDimensions.TotalPassed),
            TotalTime = todouble(customDimensions.TotalTime)
| extend HtmlUrl = strcat("https://github.com/", RepositoryName, "/actions/runs/", RunId)
| where message contains "AL-Go Test Results"

Azure Data Explorer Dashboard

AL-Go provides a template dashboard for Azure Data Explorer to visualize your telemetry.

Download Template

Download the telemetry dashboard template from the AL-Go repository.

Configure Dashboard

Open the downloaded telemetrydashboard.json file in an editor and replace:

database: Your Application Insights resource name

clusterUri: Your Application Insights Data Explorer URI:

https://ade.applicationinsights.io/subscriptions/YourSubscriptionId/resourcegroups/YourResourceGroup/providers/microsoft.insights/components/YourApplicationInsightsName

Import Dashboard

Navigate to https://dataexplorer.azure.com/dashboards
Click the arrow next to “New Dashboard”
Select “Import dashboard from file”
Upload your configured JSON file

Custom Dashboards and Alerts

Performance Monitoring Dashboard

Create a custom workbook to track workflow performance:

Workflow Duration Trends
Success Rate
Test Results Summary

traces
| where timestamp > ago(30d)
| where message contains "AL-Go workflow"
| project timestamp, 
          WorkflowName = tostring(customDimensions.WorkflowName),
          DurationMinutes = round(todouble(customDimensions.WorkflowDuration) / 60, 2)
| summarize avg(DurationMinutes), max(DurationMinutes), min(DurationMinutes) by bin(timestamp, 1d), WorkflowName
| render timechart

traces
| where timestamp > ago(7d)
| where message contains "AL-Go workflow"
| project timestamp,
          WorkflowName = tostring(customDimensions.WorkflowName),
          Conclusion = tostring(customDimensions.WorkflowConclusion)
| summarize Total = count(), 
            Successful = countif(Conclusion == "Success"),
            Failed = countif(Conclusion == "Failure")
          by WorkflowName
| extend SuccessRate = round(100.0 * Successful / Total, 2)
| project WorkflowName, Total, Successful, Failed, SuccessRate
| order by SuccessRate asc

traces
| where timestamp > ago(7d)
| where message contains "AL-Go Test Results"
| project timestamp,
          RepositoryName = tostring(customDimensions.RepositoryName),
          TotalTests = todouble(customDimensions.TotalTests),
          TotalPassed = todouble(customDimensions.TotalPassed),
          TotalFailed = todouble(customDimensions.TotalFailed)
| summarize TotalTests = sum(TotalTests),
            TotalPassed = sum(TotalPassed),
            TotalFailed = sum(TotalFailed)
          by bin(timestamp, 1d)
| extend PassRate = round(100.0 * TotalPassed / TotalTests, 2)
| render timechart

Configure Alerts

Set up alerts for critical events:

Workflow Failure Alert

Create an alert when workflows fail:

traces
| where timestamp > ago(5m)
| where message contains "AL-Go workflow failed"
| project timestamp,
          RepositoryName = tostring(customDimensions.RepositoryName),
          WorkflowName = tostring(customDimensions.WorkflowName),
          RunId = tostring(customDimensions.RunId)
| extend HtmlUrl = strcat("https://github.com/", RepositoryName, "/actions/runs/", RunId)

Set threshold to trigger when result count > 0.

Test Failure Alert

Alert when test pass rate drops below threshold:

traces
| where timestamp > ago(15m)
| where message contains "AL-Go Test Results"
| project TotalTests = todouble(customDimensions.TotalTests),
          TotalPassed = todouble(customDimensions.TotalPassed),
          RepositoryName = tostring(customDimensions.RepositoryName)
| extend PassRate = 100.0 * TotalPassed / TotalTests
| where PassRate < 90

Performance Degradation Alert

Alert when workflow duration exceeds normal range:

traces
| where timestamp > ago(1h)
| where message contains "AL-Go workflow ran"
| project WorkflowName = tostring(customDimensions.WorkflowName),
          Duration = todouble(customDimensions.WorkflowDuration)
| summarize AvgDuration = avg(Duration) by WorkflowName
| where AvgDuration > 1800  // Alert if average exceeds 30 minutes

Best Practices

Use Separate Workspaces

Create dedicated Application Insights resources for different environments (dev, test, prod)

Set Retention Policies

Configure appropriate data retention based on compliance requirements and cost considerations

Create Workbooks

Build custom Azure Monitor workbooks for your team’s specific metrics and KPIs

Regular Review

Schedule periodic reviews of telemetry data to identify trends and optimization opportunities

Security and Compliance

Important considerations for telemetry:

Telemetry data may contain repository names, workflow names, and error messages
Ensure your Application Insights workspace has appropriate access controls
Review data retention requirements for compliance
Do not include sensitive data in workflow or repository names

Troubleshooting

No telemetry data appearing

Check:

Connection string is correct in AL-Go settings
Application Insights resource is active
Workflows have run after enabling telemetry
No firewall blocking access to Application Insights ingestion endpoint

Incomplete telemetry data

Verify:

PowerShell execution policy allows AL-Go scripts to run
No network timeouts when sending telemetry
Application Insights sampling is not too aggressive

Query returns no results

Ensure:

Time range in query covers when workflows ran
customDimensions are queried correctly (case-sensitive)
Message filter matches exact telemetry messages
Data ingestion latency (may take 1-2 minutes)

Advanced Scenarios

Multi-Repository Dashboards

Track telemetry across all repositories in your organization:

traces
| where timestamp > ago(7d)
| where message contains "AL-Go workflow"
| summarize Runs = count(),
            Successful = countif(tostring(customDimensions.WorkflowConclusion) == "Success"),
            Failed = countif(tostring(customDimensions.WorkflowConclusion) == "Failure"),
            AvgDuration = round(avg(todouble(customDimensions.WorkflowDuration)) / 60, 2)
          by RepositoryName = tostring(customDimensions.RepositoryName)
| extend SuccessRate = round(100.0 * Successful / Runs, 2)
| order by Runs desc

Correlate with Azure DevOps

If using AL-Go with other DevOps tools, correlate data using custom dimensions:

{
  "PartnerTelemetryConnectionString": "<your-connection-string>",
  "CustomTelemetryDimensions": {
    "Team": "Platform",
    "Project": "ERP-Extension",
    "Environment": "Production"
  }
}

Getting Started

Setup & Configuration

Core Workflows

Development

Deployment

AppSource Publishing

Advanced Topics

Migration

​Telemetry Overview

Partner Telemetry

Microsoft Telemetry

​Quick Start

​Enable Partner Telemetry

​Microsoft Telemetry Options

​Telemetry Events

Workflow Completed

Workflow Failed

Action Completed

Action Failed

​Common Telemetry Dimensions

​Event Details

​AL-Go Workflow Ran

​AL-Go Workflow Failed

​AL-Go Action Ran

​AL-Go Action Failed

​Test Results

​Querying Telemetry Data

​Workflow Completion Query

​Action Completion Query

​Test Results Query

​Azure Data Explorer Dashboard

​Custom Dashboards and Alerts

​Performance Monitoring Dashboard

​Configure Alerts

​Best Practices

Use Separate Workspaces

Set Retention Policies

Create Workbooks

Regular Review

​Security and Compliance

​Troubleshooting

​Advanced Scenarios

​Multi-Repository Dashboards

​Correlate with Azure DevOps

​Next Steps

Azure KeyVault

Customization

Build docs developers (and LLMs) love

Telemetry Overview

Quick Start

Enable Partner Telemetry

Microsoft Telemetry Options

Telemetry Events

Common Telemetry Dimensions

Event Details

AL-Go Workflow Ran

AL-Go Workflow Failed

AL-Go Action Ran

AL-Go Action Failed

Test Results

Querying Telemetry Data

Workflow Completion Query

Action Completion Query

Test Results Query

Azure Data Explorer Dashboard

Custom Dashboards and Alerts

Performance Monitoring Dashboard

Configure Alerts

Best Practices

Security and Compliance

Troubleshooting

Advanced Scenarios

Multi-Repository Dashboards

Correlate with Azure DevOps

Next Steps