[Updated] Monitoring Azure resources with Zabbix

Monitoring Azure resources with Zabbix

June 25, 2020
During last year, I've been actively refactoring this solution and finally I've updated this article. All code moved to GitHub where you can clone it to your PC and update as needed.

Also, I've removed all comments as they're not relevant anymore.

 

Hey!

Today is the great day, as I'm finally going to share my approach to monitoring of Azure Cloud resources with Zabbix. There is no built-in solution for monitoring Azure with 3rd-party software (at least, I know nothing about that), so we'll need to build our own. Before you start configuring\scripting, please make sure you're familiar with most common Zabbix features, because the task is not trivial.

Azure monitoring

Why do we need all this? Well, I've encountered a number of issues, problems and limitations while trying to use native Azure tools - Log Analytics (to paint graphs) and Monitor (to configure alerting). Metrics could appear there with big delays, problems with writing queries, triggering - is also a problem.

So, we've decided to adopt Zabbix for monitoring of Azure...

 

Tools / prerequisites

What tools and APIs will we use in the article:

  • Azure Monitoring API - through this API we will fetch metrics data from cloud.
  • Zabbix :-)
  • Powershell 7.0 both for Linux and your local Windows machine.
  • Zabbix API
  • Azure Service Principal with permissions to read monitor metrics from the cloud.
  • VS Code is recommended

ALSO:
Zabbix Azure set of Powershell scripts will do all the work for you. You can get it from https://github.com/vicioussn/zabbix-azure. There are hard-coded values in the code and they should be replaced with your real data (like password, keys, etc). I'll talk more about this in the very end of the article.

ALSO:
You must be familiar with Powershell a bit to be able to modify scripts (replace strings, comment un-needed code, etc.).

Process overview

The following workflow\statements was designed to achieve the goal:

  1. Zabbix hosts represent Azure Resource Groups.
  2. Discover individual metrics of each required resources via Zabbix LLD and put them as Zabbix items to Zabbix host. Those items will be of 'trapper' type to allow external tools\scripts to push metrics in.
  3. Execute script on schedule basis (for example - every 5 minutes) per each Zabbix host (i.e. Resource Group). Script will use Azure Monitor API to pull metrics from Azure cloud and push them to Zabbix items.

Put the solution to your Zabbix server

In order to make this solution work, you must get the whole solution from GitHub repo (development sub-folder): https://github.com/vicioussn/zabbix-azure/tree/master/development. Put all files to the /usr/lib/zabbix/externalscripts on your Zabbix server (and ensure you and your Zabbix Server user account have permissions to create files there).

Along with the code, please create the /var/log/azure-script/ folder and assign permissions to yourself and your Zabbix server user. This folder is for script logging.

As described below in the Known Issues section, credentials to Zabbix Server and Azure are hard-coded in scripts. Thus you must change them to your data. Here's the list of files you should look into:

  • helpers/Get-ZaAzureMonitorToken.ps1 (parameters section)
  • helpers/Set-ZaAzureMonitorZabbixHostItemValue.ps1 (strings 8 and 10)

And the last thing - there are scripts in root folder with 'manual' suffix. You can use those to perform tests and debugging from your local Windows machine.

Zabbix host

To be more specific, let's set up monitoring for Azure SQL Database - to be able to continuously monitor it's performance and react if DTU consumption is too high.

First of all, let's create Zabbix hosts for our resources. My approach assume that we create single host for each Azure Resource Group you have. This host will contain items (metrics) for every Azure resource in the Resource Group. Here is sample screenshot:

Monitoring Azure resources with Zabbix

Note IP address field - it should point to Zabbix Server IP. 127.0.0.1 will be good.

Then go to Macros tab and set variables related to the Resource Group. Those will be used as parameters to LLD scripts and scheduled scripts.

  • {$RESOURCEGROUP} - is the name of the Resource Group.
  • {$SUBSCRIPTIONID} - is the Azure subscription ID.

Monitoring Azure resources with Zabbix

Now it's time to create Zabbix template. We will link it to our newly created host later.

Zabbix template

Cloud is very unstable thing, resources may be created dynamically and you can be sure - sometimes you'll not be informed about that. So we will use Zabbix discovery rules to have the most relevant info about Azure SQL databases.

  • Name is whatever you want.
  • Type is External check - we'll execute Powershell script to discover SQL databases (but yes - that's correct - Zabbix can run sh only scripts).
  • Key - is the name of Bash script to be executed with parameters. The last one for our particular case is sqlDatabase, but can be any and it's defined in the Get-ZaAzureResourceApiParameters.ps1 file of the solution (there is a switch statement there to handle possible types of resources).
  • Update interval - paste reasonable value there, or your server will be over-loaded.

Monitoring Azure resources with Zabbix

Here is the bash-script to run Powershell from Zabbix for this discovery rule:

 Here is sample output of script.

Monitoring Azure resources with Zabbix

 

NOTE
As you can see, this set of parameters passed to script will result that all available Azure Monitor metrics will be returned. You can limit the output by specifying additional parameter in the end - which is a metricName (not DisplayName). You can get the list of all supported metrics for all resource types at: Supported metrics with Azure Monitor.
Output example:

Monitoring Azure resources with Zabbix

 

Now you're ready to create Item prototypes in Zabbix discovery rule. Before you do that, make sure you've tested discovery script and it returns expected results.

  • Name - something you will understand and able to use in Triggers or Graphs.
  • Type - Zabbix trapper.
  • Key - azure.resource key with parameters (in order as follows), most of them will be discovered:
    • Resource name
      The screenshot below has mistake in this parameter for the case with Azure SQL - in the particular case it should be: {#PARENTRESOURCENAME}/{#RESOURCENAME}.
      This is the thing you should remember: if the resource type you're configuring is 'nested' (i.e. sqlServer/sqlDatabase) - then you will use additional property returned by discovery script {#PARENTRESOURCENAME}.
    • Resource type
    • Metric name
    • Time grain
    • Primary aggregation type

Monitoring Azure resources with Zabbix

  

Zabbix items

Ok, now you're ready to link template to host and see what happened. You should see a number of number discovered.

Monitoring Azure resources with Zabbix

 

There is another one thing to mention. Sometimes Azure Monitor will not work as expected. For example - if the API is broken and returning errors. Or maybe my solution will stop working (honestly - yes - there are some issues).

For the sake of being informed about those problems, you should create item with azureMonitor.lastUpdated key and create Trigger for it to be informed that the item was not updated for some time (use nodata() Zabbix function for that).

Pulling metrics from Azure Monitor API 

Here goes the most interesting part of my work. We've got items of Trapper type. That means that something must push metrics to Zabbix server to this item.

It will be done by script, executed every 5 minutes by cron. Why 5 minutes? As I wrote above, there is a timeGrain variable in Azure Monitor API, which defines, how often metric is updated in cloud. You might have tons of items in Zabbix, and pulling metrics from Azure every 1 minute will freeze your Zabbix server CPU. So I've decided some adequate interval to get updates from Azure.

Ok, let's move to the main script that pulls metrics from Monitor API. Don't forget to replace sensitive parameters to yours.

Main steps which script does:

  1. Create lock-file to indicate that the process is running. This is required to avoid running two or more instances of the script at the same time.
  2. Get Zabbix host name from parameters passed to script.
  3. Login to Zabbix API.
  4. Get Zabbix items from Zabbix host and parse their keys.
  5. Get Zabbix history for each item and identify - when was the last metric pushed to the server. Based on that info, script will generate the correct request to Azure Monitor API. If there are no values for the item in Zabbix, script will ask Monitor API metrics info for the last three days (Set-ZaAzureMonitorZabbixHostItemValue.ps1 file, string around 58).
  6. Login to Azure.
  7. For each Zabbix item get metrics from Monitor API for the calculated time period.
  8. Form a text string with metric info for each sample of metric from (5) and put string to result array.
  9. Write result array to a file.
  10. Push the file to Zabbix server with zabbix_sender tool.
  11. Push the 'success' string to the azureMonitor.lastUpdated item.
  12. Log out from Zabbix API.
  13. Remove lock-file.

 

 Here is how to set up cron to run script every 5 minutes. Don't ask me how to modify it to different interval - I copied it from stackoverflow :-).

  

Again - it's recommended that you perform test runs of the pulling script to identify issues.

Known problems

There are number of issues I know exist, but they're not critical and maybe I'll solve them in future. Or - you're welcome to contribute.

  • Sometimes the pulling script will fail and wont remove lock-file, blocking further script runs.
  • Sometimes, when two Powershell processes try to write temporary azure.json file, they will corrupt it. Again, this will lead to unexpected terminations.
  • Passwords\secrets are hard-coded in scripts. With Zabbix 5 released, they can be moved as secured Zabbix Host Macro. This will require some work.

script (en), powershell (en), azure (en), zabbix (en)

  • Hits: 110789
Add comment

Comments  
zabbix monitor from on-prem to cloud
Is it possible to monitor azure resources (i.e server VMs) from on premises zabbix server
RE: zabbix monitor from on-prem to cloud
Quoting Kingsley:
Is it possible to monitor azure resources (i.e server VMs) from on premises zabbix server


Hi!

Yep, it's possible, there is no any difference.
Zabbix Sender problem
Hi thanks for your response, i have problems with zabbix sender,
zabbix_sender -z localhost -i imports/Azure_2020-06-26T14-49-08Z.imports -T -vv
zabbix_sender [34459]: Warning: [line 1] invalid 'Timestamp' value detected
Sending failed.

The script has that line, but report that error, i did it manually and the same, what timestamp is reporting the problem?
RE: Zabbix Sender problem
Quoting JoseM:
Hi thanks for your response, i have problems with zabbix sender,
zabbix_sender -z localhost -i imports/Azure_2020-06-26T14-49-08Z.imports -T -vv
zabbix_sender [34459]: Warning: [line 1] invalid 'Timestamp' value detected
Sending failed.

The script has that line, but report that error, i did it manually and the same, what timestamp is reporting the problem?


Hi!

Saw that as well - fixed with this:
https://github.com/vicioussn/zabbix-azure/commit/675422002a06cf0b7a35ffa880f1622c6f5054c7#diff-937bce56bfde53feca9ca33ef7783efbR128
RE: Zabbix Sender problem
Quoting Super User:

Hi!

Saw that as well - fixed with this:
https://github.com/vicioussn/zabbix-azure/commit/675422002a06cf0b7a35ffa880f1622c6f5054c7#diff-937bce56bfde53feca9ca33ef7783efbR128


Hi, i replace the " in the script but the problem persist.

Also the template is reporting me that discovery rule is enabled but keep creating items and report the item key as already exists, it's trying to create them multiple times.

I'm working on zabbix sender error so i can move to template error, as soon i have fixed those i let you know.
RE: Zabbix Sender problem
Quoting JoseM:

Hi, i replace the " in the script but the problem persist.

Also the template is reporting me that discovery rule is enabled but keep creating items and report the item key as already exists, it's trying to create them multiple times.

I'm working on zabbix sender error so i can move to template error, as soon i have fixed those i let you know.


Can you please put first two lines from 'imports' file?

For template reporting duplicate items - remove already created items and re-run discovery couple of times.
RE: Zabbix Sender problem
Quoting Super User:

Can you please put first two lines from 'imports' file?

For template reporting duplicate items - remove already created items and re-run discovery couple of times.

HI i solved those incovenients.

Now the scripts are turning unstables, sometimes work and sometimes doesn't. i'm getting this error:

VERBOSE: Item 64/84
VERBOSE: Parsing item key 'azure.resource[PFSENSE-DEMO,virtualMachines,CPU Credits Consumed,PT1M,Average]' and getting variables to make REST request.
VERBOSE: Getting time span for item 'CPU Credits Consumed - PFSENSE-DEMO' (id=86754) to determine period to pull metrics from Azure.
VERBOSE: Getting Zabbix history for item '86754'.
VERBOSE: Successfully fetched Zabbix history for item '86754'. VERBOSE: Item 'CPU Credits Consumed - PFSENSE-DEMO' (id=86754) has no values (possibly new item?). VERBOSE: Timespan = '2020-06-28T14:10:44Z - 2020-07-01T14:07:44Z'. VERBOSE: Logging to Azure.
VERBOSE: Azure Monitor API header built.
VERBOSE: Getting metric values from Azure Monitor with parameters:
VERBOSE: - resourceName = 'PFSENSE-DEMO'
VERBOSE: - resourceType = 'virtualMachines'
VERBOSE: - subscriptionId = 'ece9a1a6-d2d7-4284-b2fd-865d967c7024'
VERBOSE: - resourceGroupName = 'FISA-TEST';
VERBOSE: - timeSpan = '2020-06-28T14:10:44Z - 2020-07-01T14:07:44Z'
VERBOSE: - timeGrain = 'PT1M'
VERBOSE: - metricName = 'CPU Credits Consumed'
VERBOSE: Building URI to pull metrics from Azure Monitor.
VERBOSE: URI: https://management.azure.com/subscriptions/ece9a1a6-d2d7-4284-b2fd-865d967c7024/resourceGroups/FISA-TEST/providers/Microsoft.Compute/virtualMachines/PFSENSE-DEMO/providers/microsoft.insights/metrics?timespan=2020-06-28T14:10:44Z/2020-07-01T14:07:44Z&interval=PT1M&api-version=2018-01-01&metricnames=CPU Credits Consumed&aggregation=Average
VERBOSE: Error while making REST request to Azure API.
VERBOSE: Retrying 1/5 in 3 seconds.
VERBOSE: Error while making REST request to Azure API.
VERBOSE: Retrying 2/5 in 3 seconds.
VERBOSE: Error while making REST request to Azure API.
VERBOSE: Retrying 3/5 in 3 seconds.
VERBOSE: Error while making REST request to Azure API.
VERBOSE: Retrying 4/5 in 3 seconds.
VERBOSE: Error while making REST request to Azure API.
VERBOSE: Retrying 5/5 in 3 seconds.
Invoke-ZaAzureMonitorApiQuery: /usr/lib/zabbix/externalscripts/helpers/Get-ZaAzureMonitorMetricValue.ps1:76
Line |
76 | … $response = Invoke-ZaAzureMonitorApiQuery -authHeader $authHeader -qu …
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
| Error while making REST request to Azure API. 5 retries were made.
Does this get metrics from Virtual Network Gateways?
Specifically, I am needing to monitor VPN connections in Azure. Will this pull full metrics from a Virtual Network Gateway?
VirtualNetworkGateway
Hi, just wanted to say I was able to add Microsoft.Network to get VNG metrics! Works great!
RE: VirtualNetworkGateway
Quoting Tony M:
Hi, just wanted to say I was able to add Microsoft.Network to get VNG metrics! Works great!


Happy to hear that! =)
Fixed?
Hi!

Take a look at that:
https://github.com/vicioussn/zabbix-azure/commit/92c41f55b1e1506d97746d0316ed81fe735f72e8#diff-915335a6fa377f86a14a6a6d62bfc3a8

With this you will get the detailed error output coming from Azure API.
RE: Fixed?
Quoting Super User:
Hi!

Take a look at that:
https://github.com/vicioussn/zabbix-azure/commit/92c41f55b1e1506d97746d0316ed81fe735f72e8#diff-915335a6fa377f86a14a6a6d62bfc3a8

With this you will get the detailed error output coming from Azure API.


This is the error:
{"code":"BadRequest","message":"End time: 7/2/2020 2:55:03 PM must be at least 1 minute later than start time: 7/2/2020
| 7:42:00 PM, traceId: {4c279006-2129-4293-b5c9-128090f12e9a}"}

i don't understand what's the problem. can you help me?
RE: Fixed?
Quoting JoseM:

This is the error:
{"code":"BadRequest","message":"End time: 7/2/2020 2:55:03 PM must be at least 1 minute later than start time: 7/2/2020
| 7:42:00 PM, traceId: {4c279006-2129-4293-b5c9-128090f12e9a}"}

i don't understand what's the problem. can you help me?


That's very strange ... Drop here please URL it's trying to reach for metrics (the line after the "VERBOSE: Building URI to pull metrics from Azure Monitor.")
RE: Fixed?
Quoting Super User:

That's very strange ... Drop here please URL it's trying to reach for metrics (the line after the "VERBOSE: Building URI to pull metrics from Azure Monitor.")


Here's the output log:

VERBOSE: Item 45/84
VERBOSE: Parsing item key 'azure.resource[LINUX-TEST,virtualMachines,Data Disk Queue Depth,PT1M,Average]' and getting variables to make REST request.
VERBOSE: Getting time span for item 'Data Disk Queue Depth (Preview) - LINUX-TEST' (id=86687) to determine period to pull metrics from Azure.
VERBOSE: Getting Zabbix history for item '86687'.
VERBOSE: Successfully fetched Zabbix history for item '86687'.
VERBOSE: Timespan = '2020-07-02T19:42:00Z - 2020-07-02T15:13:25Z'.
VERBOSE: Logging to Azure.
VERBOSE: Azure Monitor API header built.
VERBOSE: Getting metric values from Azure Monitor with parameters:
VERBOSE: - resourceName = 'LINUX-TEST'
VERBOSE: - resourceType = 'virtualMachines'
VERBOSE: - subscriptionId = 'ece9a1a6-d2d7-4284-b2fd-865d967c7024'
VERBOSE: - resourceGroupName = 'FISA-TEST';
VERBOSE: - timeSpan = '2020-07-02T19:42:00Z - 2020-07-02T15:13:25Z'
VERBOSE: - timeGrain = 'PT1M'
VERBOSE: - metricName = 'Data Disk Queue Depth'
VERBOSE: Building URI to pull metrics from Azure Monitor.
VERBOSE: URI: https://management.azure.com/subscriptions/ece9a1a6-d2d7-4284-b2fd-865d967c7024/resourceGroups/FISA-TEST/providers/Microsoft.Compute/virtualMachines/LINUX-TEST/providers/microsoft.insights/metrics?timespan=2020-07-02T19:42:00Z/2020-07-02T15:13:25Z&interval=PT1M&api-version=2018-01-01&metricnames=Data Disk Queue Depth&aggregation=Average
**********************
Command start time: 20200702101626
**********************
PS>TerminatingError(Invoke-RestMethod): "{"code":"BadRequest","message":"End time: 7/2/2020 3:13:25 PM must be at least 1 minute later than start time: 7/2/2020 7:42:00 PM, traceId: {3e799c75-afb7-4596-b85a-48967dd6d293}"}"
VERBOSE: Error while making REST request to Azure API.
VERBOSE: Retrying 1/5 in 3 seconds.
**********************
Command start time: 20200702101630
**********************
PS>TerminatingError(Invoke-RestMethod): "{"code":"BadRequest","message":"End time: 7/2/2020 3:13:25 PM must be at least 1 minute later than start time: 7/2/2020 7:42:00 PM, traceId: {b0d8ba88-c5f7-4f34-b40e-3452959e6fa6}"}"
VERBOSE: Error while making REST request to Azure API.
VERBOSE: Retrying 2/5 in 3 seconds.
**********************
Command start time: 20200702101634
**********************
PS>TerminatingError(Invoke-RestMethod): "{"code":"BadRequest","message":"End time: 7/2/2020 3:13:25 PM must be at least 1 minute later than start time: 7/2/2020 7:42:00 PM, traceId: {b71e949a-b0e1-48d6-aa59-89fd3fb47ce6}"}"
VERBOSE: Error while making REST request to Azure API.
VERBOSE: Retrying 3/5 in 3 seconds.
**********************
Command start time: 20200702101638
**********************
PS>TerminatingError(Invoke-RestMethod): "{"code":"BadRequest","message":"End time: 7/2/2020 3:13:25 PM must be at least 1 minute later than start time: 7/2/2020 7:42:00 PM, traceId: {169e0ec6-776f-428c-8252-831a7d7f4923}"}"
VERBOSE: Error while making REST request to Azure API.
VERBOSE: Retrying 4/5 in 3 seconds.
**********************
Command start time: 20200702101641
**********************
PS>TerminatingError(Invoke-RestMethod): "{"code":"BadRequest","message":"End time: 7/2/2020 3:13:25 PM must be at least 1 minute later than start time: 7/2/2020 7:42:00 PM, traceId: {351019bf-60e5-4838-9935-c071d266f223}"}"
VERBOSE: Error while making REST request to Azure API.
VERBOSE: Retrying 5/5 in 3 seconds.
**********************
Command start time: 20200702101645
**********************
PS>TerminatingError(Invoke-RestMethod): "{"code":"BadRequest","message":"End time: 7/2/2020 3:13:25 PM must be at least 1 minute later than start time: 7/2/2020 7:42:00 PM, traceId: {6964dfdc-9d84-4aad-a184-5a8a95e03271}"}"
VERBOSE: Error while making REST request to Azure API. 5 retries were made.
Removing lock file 'get-azure-metrics-values_Azure.lck'.
**********************
Command start time: 20200702101645
**********************
PS>TerminatingError(Invoke-RestMethod): "{"code":"BadRequest","message":"End time: 7/2/2020 3:13:25 PM must be at least 1 minute later than start time: 7/2/2020 7:42:00 PM, traceId: {6964dfdc-9d84-4aad-a184-5a8a95e03271}"}"
>> TerminatingError(Invoke-RestMethod): "{"code":"BadRequest","message":"End time: 7/2/2020 3:13:25 PM must be at least 1 minute later than start time: 7/2/2020 7:42:00 PM, traceId: {6964dfdc-9d84-4aad-a184-5a8a95e03271}"}"
>> TerminatingError(Invoke-RestMethod): "{"code":"BadRequest","message":"End time: 7/2/2020 3:13:25 PM must be at least 1 minute later than start time: 7/2/2020 7:42:00 PM, traceId: {6964dfdc-9d84-4aad-a184-5a8a95e03271}"}"
>> TerminatingError(Invoke-RestMethod): "{"code":"BadRequest","message":"End time: 7/2/2020 3:13:25 PM must be at least 1 minute later than start time: 7/2/2020 7:42:00 PM, traceId: {6964dfdc-9d84-4aad-a184-5a8a95e03271}"}"

Invoke-RestMethod: /usr/lib/zabbix/externalscripts/helpers/Invoke-ZaAzureMonitorApiQuery.ps1:40
Line |
40 | … $response = Invoke-RestMethod @params -Verbose:$false -ErrorAction St …
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
| {"code":"BadRequest","message":"End time: 7/2/2020 3:13:25 PM
| must be at least 1 minute later than start time: 7/2/2020
| 7:42:00 PM, traceId: {6964dfdc-9d84-4aad-a184-5a8a95e03271}"}

**********************
PowerShell transcript end
End time: 20200702101645
**********************
RE: Fixed?
Hi!

Here's your problem:
VERBOSE: Timespan = '2020-07-02T19:42:00Z - 2020-07-02T15:13:25Z'.
As you can see, the first time is later than the last block. There's some problem with time conversion.

I added more logging:
https://github.com/vicioussn/zabbix-azure/commit/00ea9e387ff20917da18dfd3724b254424705bcc

Please put here same logs after applying new code.
Btw, what's the time zone of your Zabbix host?
RE: Fixed?
Quoting Super User:
Hi!

Here's your problem:
VERBOSE: Timespan = '2020-07-02T19:42:00Z - 2020-07-02T15:13:25Z'.
As you can see, the first time is later than the last block. There's some problem with time conversion.

I added more logging:
https://github.com/vicioussn/zabbix-azure/commit/00ea9e387ff20917da18dfd3724b254424705bcc

Please put here same logs after applying new code.
Btw, what's the time zone of your Zabbix host?


Here's the output after add the lines:
VERBOSE: Item 24/147
VERBOSE: Parsing item key 'azure.resource[LINUX-TEST,virtualMachines,Data Disk Queue Depth,PT1M,Average]' and getting variables to make REST request.
VERBOSE: Getting time span for item 'Data Disk Queue Depth (Preview) - LINUX-TEST' (id=86687) to determine period to pull metrics from Azure.
VERBOSE: Getting Zabbix history for item '86687'.
VERBOSE: Successfully fetched Zabbix history for item '86687'.
VERBOSE: Latest timestamp for the item: 1594044780.
VERBOSE: Current Epoch time (-3 minutes): 1594046154.
VERBOSE: Timespan = '2020-07-06T19:13:00Z - 2020-07-06T14:35:54Z'.
VERBOSE: Logging to Azure.
VERBOSE: Azure Monitor API header built.
VERBOSE: Getting metric values from Azure Monitor with parameters:
VERBOSE: - resourceName = 'LINUX-TEST'
VERBOSE: - resourceType = 'virtualMachines'
VERBOSE: - subscriptionId = 'ece9a1a6-d2d7-4284-b2fd-865d967c7024'
VERBOSE: - resourceGroupName = 'FISA-TEST';
VERBOSE: - timeSpan = '2020-07-06T19:13:00Z - 2020-07-06T14:35:54Z'
VERBOSE: - timeGrain = 'PT1M'
VERBOSE: - metricName = 'Data Disk Queue Depth'
VERBOSE: Building URI to pull metrics from Azure Monitor.
VERBOSE: URI: https://management.azure.com/subscriptions/ece9a1a6-d2d7-4284-b2fd-865d967c7024/resourceGroups/FISA-TEST/providers/Microsoft.Compute/virtualMachines/LINUX-TEST/providers/microsoft.insights/metrics?timespan=2020-07-06T19:13:00Z/2020-07-06T14:35:54Z&interval=PT1M&api-version=2018-01-01&metricnames=Data Disk Queue Depth&aggregation=Average
**********************
Command start time: 20200706093855
**********************
PS>TerminatingError(Invoke-RestMethod): "{"code":"BadRequest","message":"End time: 7/6/2020 2:35:54 PM must be at least 1 minute later than start time: 7/6/2020 7:13:00 PM, traceId: {da29900e-f912-4260-98cc-a85eb64f5436}"}"
VERBOSE: Error while making REST request to Azure API.
VERBOSE: Retrying 1/5 in 3 seconds.
**********************
Command start time: 20200706093859
**********************
PS>TerminatingError(Invoke-RestMethod): "{"code":"BadRequest","message":"End time: 7/6/2020 2:35:54 PM must be at least 1 minute later than start time: 7/6/2020 7:13:00 PM, traceId: {bf6ba87d-dba4-482d-84c1-a2372a2eb2d3}"}"
VERBOSE: Error while making REST request to Azure API.
VERBOSE: Retrying 2/5 in 3 seconds.
**********************
Command start time: 20200706093902
**********************
PS>TerminatingError(Invoke-RestMethod): "{"code":"BadRequest","message":"End time: 7/6/2020 2:35:54 PM must be at least 1 minute later than start time: 7/6/2020 7:13:00 PM, traceId: {87317371-e4a4-492e-a9dd-9916ff146668}"}"
VERBOSE: Error while making REST request to Azure API.
VERBOSE: Retrying 3/5 in 3 seconds.
**********************
Command start time: 20200706093906
**********************
PS>TerminatingError(Invoke-RestMethod): "{"code":"BadRequest","message":"End time: 7/6/2020 2:35:54 PM must be at least 1 minute later than start time: 7/6/2020 7:13:00 PM, traceId: {6ca49caa-ad9a-4aaf-8104-bba7688c4c31}"}"
VERBOSE: Error while making REST request to Azure API.
VERBOSE: Retrying 4/5 in 3 seconds.
**********************
Command start time: 20200706093910
**********************
PS>TerminatingError(Invoke-RestMethod): "{"code":"BadRequest","message":"End time: 7/6/2020 2:35:54 PM must be at least 1 minute later than start time: 7/6/2020 7:13:00 PM, traceId: {7318dcad-b943-42b7-a54a-f56c172411f5}"}"
VERBOSE: Error while making REST request to Azure API.
VERBOSE: Retrying 5/5 in 3 seconds.
**********************
Command start time: 20200706093914
**********************
PS>TerminatingError(Invoke-RestMethod): "{"code":"BadRequest","message":"End time: 7/6/2020 2:35:54 PM must be at least 1 minute later than start time: 7/6/2020 7:13:00 PM, traceId: {e7f2e4bd-ad25-404c-a7cc-29af5d2540cc}"}"
VERBOSE: Error while making REST request to Azure API. 5 retries were made.
Removing lock file 'get-azure-metrics-values_Azure.lck'.
**********************
Command start time: 20200706093914
**********************
PS>TerminatingError(Invoke-RestMethod): "{"code":"BadRequest","message":"End time: 7/6/2020 2:35:54 PM must be at least 1 minute later than start time: 7/6/2020 7:13:00 PM, traceId: {e7f2e4bd-ad25-404c-a7cc-29af5d2540cc}"}"
>> TerminatingError(Invoke-RestMethod): "{"code":"BadRequest","message":"End time: 7/6/2020 2:35:54 PM must be at least 1 minute later than start time: 7/6/2020 7:13:00 PM, traceId: {e7f2e4bd-ad25-404c-a7cc-29af5d2540cc}"}"
>> TerminatingError(Invoke-RestMethod): "{"code":"BadRequest","message":"End time: 7/6/2020 2:35:54 PM must be at least 1 minute later than start time: 7/6/2020 7:13:00 PM, traceId: {e7f2e4bd-ad25-404c-a7cc-29af5d2540cc}"}"
>> TerminatingError(Invoke-RestMethod): "{"code":"BadRequest","message":"End time: 7/6/2020 2:35:54 PM must be at least 1 minute later than start time: 7/6/2020 7:13:00 PM, traceId: {e7f2e4bd-ad25-404c-a7cc-29af5d2540cc}"}"
{"code":"BadRequest","message":"End time: 7/6/2020 2:35:54 PM must be at least 1 minute later than start time: 7/6/2020 7:13:00 PM, traceId: {e7f2e4bd-ad25-404c-a7cc-29af5d2540cc}"}

Invoke-RestMethod: /usr/lib/zabbix/externalscripts/helpers/Invoke-ZaAzureMonitorApiQuery.ps1:40
Line |
40 | … $response = Invoke-RestMethod @params -Verbose:$false -ErrorAction St …
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
| {"code":"BadRequest","message":"End time: 7/6/2020 2:35:54 PM must be at least 1 minute later than start time: 7/6/2020
| 7:13:00 PM, traceId: {e7f2e4bd-ad25-404c-a7cc-29af5d2540cc}"}

**********************
PowerShell transcript end
End time: 20200706093914
**********************

My time zone is gmt -5 bogota.
RE: Fixed?
Looks like some problem with time conversion Epoch -> Windows.

I'll take a look tomorrow.
RE: Fixed?
Based on this...

VERBOSE: Latest timestamp for the item: 1594044780.
VERBOSE: Current Epoch time (-3 minutes): 1594046154.
VERBOSE: Timespan = '2020-07-06T19:13:00Z - 2020-07-06T14:35:54Z'.

...first conversion is incorrect
RE: Fixed?
RE: Fixed?
Quoting Super User:
Last commit should fix the problem for you:

https://github.com/vicioussn/zabbix-azure/commit/a0af60ea4fc047f89a5aab55b2bdc314e0c2d123


Thank you, it's working really good now. i will like to work with you in a couple of improvements if you're ok.

Right now when an item is deleted in azure causes the abnormal stop of get metrics script.

Second, do you think if there's a way to use macros so create a template with a discover host rule that discover items on azure and create the host with a new template linked. someting like vmware template?
RE: Fixed?
Glad it finally fixed =).

For the first one: yep, I was thinking to sort this problem out. Please create an issue on GitHub to track it. May require a few days - I'm out of the project which was using Zabbix, so now it's like a hobby.

For your second request. Unfortunately I've never been working with host discovery, so I dont really know how it works. Maybe if you describe a full process - I'll think about implementation. But currently not in plans.
RE: Fixed?
Quoting Super User:
Glad it finally fixed =).

For the first one: yep, I was thinking to sort this problem out. Please create an issue on GitHub to track it. May require a few days - I'm out of the project which was using Zabbix, so now it's like a hobby.

For your second request. Unfortunately I've never been working with host discovery, so I dont really know how it works. Maybe if you describe a full process - I'll think about implementation. But currently not in plans.


Ok i'll submit the issue.

I know or i have the idea about the host discovery configuration, i can help you with it.
Geek
Great piece of work there. Many thanks for this.
I'm having an issue with using the "Test Item" when attempt to run the discovery script in Zabbix. From the command line running :
./get.azure.metrics.sh vm

works perfectly.

When I run within Zabbix using "Test" on the discovery rule (I'm Zabbix 5.0) I'm getting a timeout message.
A manual execution of the script takes around 2 seconds and I've set the timeout in zabbix_server.conf to 10 seconds.

Any advice on how I can diagnose please?

Thanks in advance,
Dave
RE: Geek
Hi!

I would do the following:

1. Check the output of 'top' command to get CPU load during execution of LLD script from the Zabbix. What's going on there?
2. Verify that LLD script is not configured to run every 1m. Make it to run every 1h.

If that wont answer the question - then my next step will be - to enable Powershell transcript for LLD script - just like i did here: https://github.com/vicioussn/zabbix-azure/blob/master/development/get.azure.metrics.values.ps1#L6
RE: Geek
Has the same issue. Increased to 20 seconds in zabbix_server.conf and it solved it
Failed CoreCLR
Hello!

Congratulations on the initiative, very nice structure.

When performing a test, the following error is returned:
Failed to create CoreCLR, HRESULT: 0x80004005

And when it runs by scheduling the discovery this:
Invalid discovery rule value: cannot parse as a valid JSON object: invalid object format, expected opening character '{' or '[' at: 'Failed to create CoreCLR, HRESULT: 0x80004005'

Can you help me?
[Updated] Monitoring Azure resources with Zabbix
Thanks for finally writing about >[Updated] Monitoring Azure resources with Zabbix
RE: Failed CoreCLR
Hi!

This must be isolated to your VM - never saw this kind of issues.

Take a look at https://github.com/PowerShell/PowerShell/issues/13166 - maybe it will help.
Where is the Zabbix template?
Hello, where is the download for the Azure Zabbix template itself? I don't see it in the git repo or on the zabbix share site. Looks cool though, can't wait to try it!

Cheers!
RE: Failed CoreCLR
Had kind of similar issue. I was able to call scripts manually just fine, but when I run discovery from Zabbix it failed with "Invalid discovery rule value: cannot parse as a valid JSON object: invalid object format, expected opening character '{' or '[' at: 'Failed to create CoreCLR, HRESULT: 0x80004005'".

In the end, I fixed it by adding "#!/bin/bash" to the get.azure.metrics.sh which solved the problem inside Zabbix and discovery was performed just fine.
RE: Failed CoreCLR
Quoting isiroshtan:
Had kind of similar issue. I was able to call scripts manually just fine, but when I run discovery from Zabbix it failed with "Invalid discovery rule value: cannot parse as a valid JSON object: invalid object format, expected opening character '{' or '[' at: 'Failed to create CoreCLR, HRESULT: 0x80004005'".

In the end, I fixed it by adding "#!/bin/bash" to the get.azure.metrics.sh which solved the problem inside Zabbix and discovery was performed just fine.


Good catch!
Discovery Filtering
Firstly I'd like to thank you for the solution. It works great! Really can not imagine the amount of effort that was put into building it from scratch! Great work! Really hope that at some point I will be able to put together solutions of the same complexity :)

Next, I want to point out additional possibility that might be worth to put into the blog-post:
it does mention possibility to filter what metrics discovered by the script by providing "metricName" as an additional argument. While I did not tests out this option, based on description I assume it filters only based on this particular field.

Instead of pushing this task on script, it's possible to put task of filtering on Zabbix discovery mechanism. When creating discovery it has "filters" tab available. In my case of VM monitoring I added filter rule for macro "{#METRICDISPLAYNAME}" to "does not match" and the regexp "Credits|Deprecated|Preview". This effectively discovers all metrics with exception to metrics that contain any of the three words specified in their display name. In my case it brought down the amount of discovered metrics from 40+ to 19.

In similar way it's possible to filter discovery to single object (or set of objects) of specified type (filter by #PARENTRESOURCENAME for VMs, providing the names of machines that you want or don`t want to monitor for whatever reason) or specific metrics set using filtering based on "matches" criteria instead of my "does not match".

In general using filtering mechanism of discovery process in Zabbix gives you fine control of what you want to actually discovery and put into monitoring.
Admin
VERBOSE: Parsing item key 'azure.resource[ebt-db02,sqlDatabase,virtual_core_count,PT1M]' and getting variables to make REST request.
VERBOSE: Getting time span for item 'ebt-db02 / virtual_core_count' (id=46964) to determine period to pull metrics from Azure.
VERBOSE: Getting Zabbix history for item '46964'.
VERBOSE: Successfully fetched Zabbix history for item '46964'.
VERBOSE: Item 'ebt-db02 / virtual_core_count' (id=46964) has no values (possibly new item?).
VERBOSE: Current Epoch time (-3 minutes): 1606003574.
VERBOSE: Timespan = '2020-11-19T00:09:13Z - 2020-11-22T00:06:13Z'.
Get-ZaAzureMonitorMetricValue: /usr/lib/zabbix/externalscripts/helpers/Set-ZaAzureMonitorZabbixHostItemValue.ps1:80
Line |
80 | $metrics = Get-ZaAzureMonitorMetricValue @params;
| ~~~~~~~
| Cannot bind argument to parameter 'aggregation' because it is an empty string.

VERBOSE: Building result array for pushing to Zabbix server.
VERBOSE: Writing metrics to 'imports' file.
VERBOSE: Finished write to file.
VERBOSE: /usr/lib/zabbix/externalscripts//imports/Zabbix-server_2020-11-22T00-09-13Z.imports
VERBOSE: Pushing '/usr/lib/zabbix/externalscripts//imports/Zabbix-server_2020-11-22T00-09-13Z.imports' to Zabbix server.
sent: 0; skipped: 0; total: 0
VERBOSE: Sending 'azureMonitor.lastUpdated' semaphore to the Zabbix host.
Response from "localhost:10051": "processed: 0; failed: 1; total: 1; seconds spent: 0.000022"
sent: 1; skipped: 0; total: 1
Question/Possible change needed for other Resource Types
Thank you very much for working this out!

We put it in place and VMs worked great out of the box as did MS sqlDatabase but had more difficulty when attempting to add more Resource Types and supported metrics from - https://docs.microsoft.com/en-us/azure/azure-monitor/platform/metrics-supported

in particular AKS or Microsoft.ContainerService. We had issue (not on first part of Metric Discovery - Trapper item LLD) but the actual metrics values themselves. It does not appear to be honoring the $ResourceProviderApiVersion = stanza from Get-ZaAzureResourceApiParameters.ps1 into Get-ZaAzureMonitorMetricValue.ps1.

We found in the uri construction that the value appears to be hardcoded for API Version. It seems to matter for AKS.

You can see in the code
$uri = "https://management.azure.com/subscriptions/{0}/resourceGroups/{1}/providers/{2}/{3}/{4}/{5}/{6}/providers/microsoft.insights/metrics?timespan={7}&interval={8}&api-version={9}&metricnames={10}&aggregation={11}" -f `
$subscriptionId, ` # 0
$resourceGroupName, ` # 1
$resourceApiParams.provider, ` # 2
$parentProviderExtension, ` # 3
$parentResourceName, ` # 4
$childProviderExtension, ` # 5
$childResourceName, ` # 6
"$startTime/$endTime", ` # 7
$timeGrain, ` # 8
"2019-07-01", ` # 9
$metricName, ` # 10
$aggregation; # 11

as example. We had interesting issue that MS API for metric get to build trapper only worked with apiVersion = "2018-01-01"; set in Get-ZaAzureMonitorMetrics.ps1 but that value did not work in AKS metric value.

We changed the hardcoding to be 2019-07-01 which works for sqlDatabase, VM and AKS. Seems like it is close to being setup for acually using the API version correctly. Not proficient in powershell enough to be comfortable fixing.

Again just something we needed to do to get things working.
Invalid discovery rule value
Hi,
I have a following error in Zabbix:

Invalid discovery rule value: cannot parse as a valid JSON object: invalid object format, expected opening character '{' or '[' at: ''[91mParserError: '[0m/etc/zabbix/external_scripts/helpers/Set-ZaAzureAiLogZabbixHostItemValue.ps1:12
'[96mLine |
'[96m 12 | '[0m … redential ('[96

Any ideas?
RE: Where is the Zabbix template?
RE: Discovery Filtering
Thank you for the feedback!

Yes you're right that you can utilize the Zabbix LLD filters there. I'll put a note to write some sentences there.
VM metrics
First of all, great work, absolutly amazing. I'm also using this template for monitoring VM, but I don't have any value for items also if I have the item created for the VM that I have on Azure. I don't have any error when I run manually the pushing scripts but the items don't have any value. I have in zabbix logs the follow: get.azure.metrics.sh["{$RESOURCEGROUP}","{$SUBSCRIPTIONID}",vm]" became not supported: Timeout while executing a shell script. Failed to execute command "/usr/lib/zabbix/externalscripts/get.azure.metrics.sh 'resource' xxxxxxxxxxxxx' 'vm'": Timeout while executing a shell script. zabbix-server | 208:20210129:092100.038 discovery rule "resource:get.azure.metrics.sh["{$RESOURCEGROUP}","{$SUBSCRIPTIONID}",vm]" became supported I can see the item for the the hosts resource but with no value.
Backup Azure
Hello, can I monitor Azure storage (backup) using these scripts?
Authorization
good morning, i need some help in this error {"error": {"code":"AuthorizationFailed","message":"The client 'xxxxxxxxxxxxxx' with object id 'exxxxxxxxxxxxxxxxxxxx' does not have authorization to | perform action 'Microsoft.Sql/servers/read' over scope '/subscriptions/xxxxxxxxxxxxxx/resourceGroups/Zabbix-Api/providers/Microsoft.Sql' or the scope is invalid. If | access was recently granted, please refresh your credentials."}}
/usr/share/zabbix/externalscripts/get.azure.metrics.sh: [2] No such file or directory
Hi, I have some scripts stored in /usr/lib/zabbix/externalscripts. I created a Discovery Rule, type "External Check" that call the script get.azure.metrics.sh. In the host that I linked the Discovery Rule I get the follow error: /usr/share/zabbix/externalscripts/get.azure.metrics.sh: [2] No such file or directory. I don't understand the reaso of this path is using the "share" folder, being me created this file in "lib" folder. I checked in zabbix_server.conf and the ExternalScript variable is enable: ExternalScripts=/usr/lib/zabbix/externalscripts Please, help me!!
Invalid discovery rule value
The same error is going on with me
get.azure.metrics.values error
I am getting itÇ [email protected]:/usr/lib/zabbix/externalscripts# /usr/lib/zabbix/externalscripts/get.azure.metrics.values.sh OSDev sqlDatabase Transcript started, output file is /var/log/azure-script/PowerShell_transcript.zbxSrvDev.pBLdW6z_.20210326131552.txt Creating lock file 'get-azure-metrics-values_OSDev.lck'. Directory: /usr/lib/zabbix/externalscripts Mode LastWriteTime Length Name ---- ------------- ------ ---- ----- 03/26/2021 13:15 0 get-azure-metrics-values_OSDev.lck VERBOSE: Logging to Zabbix. VERBOSE: Successfully logged to Zabbix. VERBOSE: Getting Zabbix host object for 'OSDev'. VERBOSE: Successfully fetched host object 'OSDev' from Zabbix. Get-ZaZabbixHostMacro: /usr/lib/zabbix/externalscripts/helpers/Set-ZaAzureMonitorZabbixHostItemValue.ps1:14 Line | 14 | … tMacro -url $zabbixUrl -auth $zabbixToken -hostId $zabbixHost.hostid; | ~~~~~~~~~~~~~~~~~~ | Cannot bind argument to parameter 'hostId' because it is an empty string. VERBOSE: Getting items from Zabbix host 'OSDev'. VERBOSE: Successfully fetched items objects from 'OSDev'. VERBOSE: Writing metrics to 'imports' file. d---- 03/26/2021 13:15 imports VERBOSE: Finished write to file. VERBOSE: /usr/lib/zabbix/externalscripts//imports/OSDev_2021-03-26T13-15-53Z.imports VERBOSE: Pushing '/usr/lib/zabbix/externalscripts//imports/OSDev_2021-03-26T13-15-53Z.imports' to Zabbix server. Set-ZaAzureMonitorZabbixHostItemValue: /usr/lib/zabbix/externalscripts/get.azure.metrics.values.ps1:27 Line | 27 | Set-ZaAzureMonitorZabbixHostItemValue @params -Verbose; | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ | Error while pushing '/usr/lib/zabbix/externalscripts//imports/OSDev_2021-03-26T13-15-53Z.imports' to Zabbix server. Exception: /usr/lib/zabbix/externalscripts/helpers/Set-ZaAzureMonitorZabbixHostItemValue.ps1:147 Line | 147 | zabbix_sender -z localhost -i $fileName -T -v | ~~~~~~~~~~~~~ | The term 'zabbix_sender' is not recognized as a name of a cmdlet, function, script file, or executable program. Check the spelling of the name, or if a path was included, verify | that the path is correct and try again.
Monitoramento Azure Backup
Boa tarde, preciso monitorar o backup feito no storage do Azure, consigo com estes scripts?

Related Articles