Migrating From Azure Monitor to Prometheus and Grafana on Linode

Azure Monitor is Microsoft’s built-in observability platform. It is designed to monitor and analyze the performance and reliability of applications and infrastructure within the Azure ecosystem. It collects metrics, logs, and telemetry data from Azure resources, on-premises environments, and other cloud services. It also offer tools to optimize and maintain system health.

This guide explains how to migrate Azure Monitor service logs and metrics to Prometheus and Grafana running on a Linode instance.

Introduction to Prometheus and Grafana

Prometheus is a time-series database that collects and stores metrics from applications and services. It provides a foundation for monitoring system performance using the PromQL query language to extract and analyze granular data. Prometheus autonomously scrapes (pulls) metrics from targets at specified intervals, efficiently storing data through compression while retaining the most critical details. It also supports alerting based on metric thresholds, making it suitable for dynamic, cloud-native environments.

Grafana is a visualization and analytics platform that integrates with Prometheus. It enables users to create real-time, interactive dashboards, visualize metrics, and set up alerts to gain deeper insights into system performance. Grafana can unify data from a wide array of data sources, including Prometheus, to provide a centralized view of system metrics.

Prometheus and Grafana are often used together to monitor service health, detect anomalies, and issue alerts. Being both open source and platfrom-agnostic allows them to be deployed across a diverse range of cloud providers and on-premise infrastructures. Organizations often adopt these tools to reduce operational costs while gaining greater control over how data is collected, stored, and visualized.

Before You Begin

1. If you do not already have a virtual machine to use, create a Compute Instance. See our Getting Started with Linode and Creating a Compute Instance guides. The examples in this guide use the Linode 8 GB Shared CPU plan with Ubuntu 24.04 LTS.

   Provisioning Compute Instances with the Linode CLI

   The Linode CLI provides an alternative way to provision resources. For example, the following command creates a Linode 8 GB compute instance (g6-standard-4) running Ubuntu 24.04 LTS (linode/ubuntu24.04) in the Miami datacenter (us-mia):

   linode-cli linodes create \
       --image linode/ubuntu24.04 \
       --region us-mia \
       --type g6-standard-4 \
       --root_pass PASSWORD \
       --authorized_keys "$(cat ~/.ssh/id_rsa.pub)" \
       --label monitoring-server

   Note the following key points:

   • Replace PASSWORD with a secure alternative.
   • This command assumes that an SSH public/private key pair exists, with the public key stored as id\_rsa.pub in the user’s $HOME/.ssh/ folder.
   • The --label argument specifies the name of the new server (monitoring-server).

2. Follow our Setting Up and Securing a Compute Instance guide to update your system. You may also wish to set the timezone, configure your hostname, create a limited user account, and harden SSH access.

Install Prometheus as a Service

In order to install Prometheus, you must first SSH into the newly provisioned Linode.

1. Create a dedicated user for Prometheus, disable its login, and create the necessary directories for Prometheus:

   sudo useradd --no-create-home --shell /bin/false prometheus
   sudo mkdir /etc/prometheus
   sudo mkdir /var/lib/prometheus

2. Download the latest version of Prometheus from its GitHub repository:

   wget https://github.com/prometheus/prometheus/releases/download/v2.55.1/prometheus-2.55.1.linux-amd64.tar.gz

   This guide uses version 2.55.1. Check the project’s releases page for the latest version that aligns with your instance’s operating system.

3. Extract the compressed file and navigate to the extracted folder:

   tar xzvf prometheus-2.55.1.linux-amd64.tar.gz
   cd prometheus-2.55.1.linux-amd64

4. Move both the prometheus and promtool binaries to /usr/local/bin:

   sudo cp prometheus /usr/local/bin
   sudo cp promtool /usr/local/bin

   The prometheus binary is the main monitoring application, while promtool is a utility application that queries and configures a running Prometheus service.

5. Move the configuration files and directories to the /etc/prometheus folder you created previously:

   sudo cp -r consoles /etc/prometheus
   sudo cp -r console_libraries /etc/prometheus
   sudo cp prometheus.yml /etc/prometheus/prometheus.yml

6. Set the correct ownership permissions for Prometheus files and directories:

   sudo chown -R prometheus:prometheus /etc/prometheus
   sudo chown -R prometheus:prometheus /var/lib/prometheus
   sudo chown prometheus:prometheus /usr/local/bin/prometheus
   sudo chown prometheus:prometheus /usr/local/bin/promtool

Create a systemd Service File

A systemd service configuration file must be created to run Prometheus as a service.

1. Create the service file using a command line text editor such as nano.

   sudo nano /etc/systemd/system/prometheus.service

   Add the following content to the file:

   File: /etc/systemd/system/prometheus.Service

   1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17

   [Unit] Description=Prometheus Service Wants=network-online.target After=network-online.target [Service] User=prometheus Group=prometheus Type=simple ExecStart=/usr/local/bin/prometheus \ --config.file=/etc/prometheus/prometheus.yml \ --storage.tsdb.path=/var/lib/prometheus \ --web.console.templates=/etc/prometheus/consoles \ --web.console.libraries=/etc/prometheus/console_libraries [Install] WantedBy=multi-user.target

   When done, press CTRL+X, followed by Y then Enter to save the file and exit nano.

2. Reload the systemd configuration files to apply the new service file:

   sudo systemctl daemon-reload

3. Run the following systemctl commands to start the flash-app service and enable it to automatically start after a system reboot:

   sudo systemctl start prometheus
   sudo systemctl enable prometheus

4. Enter the following command to verify that Prometheus is running:

   systemctl status prometheus

   The output should display active (running), confirming a successful setup:

   ● prometheus.service - Prometheus Service
        Loaded: loaded (/etc/systemd/system/prometheus.service; enabled; preset: enabled)
        Active: active (running) since Thu 2024-12-05 16:11:57 EST; 5s ago
      Main PID: 1165 (prometheus)
         Tasks: 9 (limit: 9444)
        Memory: 16.2M (peak: 16.6M)
           CPU: 77ms
        CGroup: /system.slice/prometheus.service

   When done, press Q key to exit the status output and return to the terminal prompt.

5. Open a web browser and visit port 9090 ( Prometheus’s default port) of your instance’s IP address:

   http://IP_ADDRESS:9090

   The Prometheus UI should appear:

   Note

   Prometheus settings are configured in the /etc/prometheus/prometheus.yml file. This guide uses the default values. For production systems, consider enabling authentication and other security measures to protect your metrics.

Install the Grafana Service

Grafana provides an apt repository, reducing the number of steps needed to install and update it on Ubuntu.

1. Install the necessary package to add new repositories:

   sudo apt install software-properties-common -y

2. Import and add the public key for the Grafana repository:

   wget -q -O - https://packages.grafana.com/gpg.key | sudo apt-key add -
   sudo add-apt-repository "deb https://packages.grafana.com/oss/deb stable main"

3. Update package index and install Grafana:

   sudo apt update
   sudo apt install grafana -y

4. The installation process already sets up the systemd configuration for Grafana. Start and enable the Grafana service:

   sudo systemctl start grafana-server
   sudo systemctl enable grafana-server

5. Run the following command to verify that Grafana is active (running):

   systemctl status grafana-server

   ● grafana-server.service - Grafana instance
        Loaded: loaded (/usr/lib/systemd/system/grafana-server.service; enabled; preset: enabled)
        Active: active (running) since Thu 2024-12-05 13:57:10 EST; 8s ago
          Docs: http://docs.grafana.org
      Main PID: 3434 (grafana)
         Tasks: 14 (limit: 9444)
        Memory: 71.4M (peak: 80.4M)
           CPU: 2.971s
        CGroup: /system.slice/grafana-server.service

Connect Grafana to Prometheus

1. Open a web browser and visit port 3000 (Grafana’s default port) of your instance’s IP address to access the Grafana web UI:

   http://IP_ADDRESS:3000

2. Login using the default credentials of admin for both the username and password:

   

3. After logging in, you are prompted to enter a secure replacement for the default password:

   

   Now it’s time to add Prometheus as a data source. Expand the Home menu, navigate to the Connections entry, then click Add new connection:

   

4. Search for and select Prometheus.

5. Click Add new data source.

6. In the URL field, enter http://localhost:9090.

7. Click Save & Test to confirm the connection.

   If the test succeeds, your Grafana installation should now be connected to the Prometheus installation running on the same Linode.

Migrate from Azure Monitor to Prometheus and Grafana

Migrating from Azure Monitor to Prometheus and Grafana offers several advantages. These include increased control data storage and handling, cost reduction, and enhanced monitoring capabilities across multi-cloud or hybrid environments. However, the transition requires careful planning and a clear understanding of the differences between these tools:

While Azure Monitor includes native tools for creating dashboards, Grafana enhances visualization capabilities by supporting multiple data sources. This allows users to combine real-time and historical data from Azure Monitor, Prometheus, and other platforms in a single unified view.

Applications monitored by Azure Monitor might use the following tools:

• Azure OpenTelemetry Exporter: For intentional collection of application metrics using the OpenTelemetry Standard.
• Application Insights: For automatic collection of metrics and telemetry data from applications.

Configure Example Flask Server

1. Change into your user’s home directory and use git to clone the example Flask server’s GitHub repository to your compute instance:

   cd ~
   git clone https://github.com/nathan-gilbert/simple-ec2-cloudwatch.git

2. Change into the example-flask-prometheus folder in the new simple-ec2-cloudwatch directory:

   cd simple-ec2-cloudwatch/example-flask-prometheus

3. A virtual environment is required to run pip commands in Ubuntu 24.04 LTS. Use the following command to install python3.12-venv:

   sudo apt install python3.12-venv

4. Create a virtual environment named venv within the example-flask-prometheus directory:

   python3 -m venv venv

5. Activate the venv virtual environment:

   source venv/bin/activate

6. Use pip to install the example Flask servers’s dependencies:

   pip install -r requirements.txt

7. Exit the virtual environment:

   deactivate

8. Create a systemd service file for the example Flask app:

   sudo nano /etc/systemd/system/flask-app.service

   Provide the file with the following content, replacing USERNAME with your username:

   File: /etc/systemd/system/flask-app.service

   1 2 3 4 5 6 7 8 9 10 11 12

   [Unit] Description=Flask Application Service After=network.target [Service] User=USERNAME WorkingDirectory=/home/USERNAME/simple-ec2-cloudwatch/example-flask-prometheus ExecStart=/home/USERNAME/simple-ec2-cloudwatch/example-flask-prometheus/venv/bin/python /home/USERNAME/simple-ec2-cloudwatch/example-flask-prometheus/app.py Restart=always [Install] WantedBy=multi-user.target

9. Reload the systemd configuration files to apply the new service file:

   sudo systemctl daemon-reload

Assess Current Monitoring Requirements

Begin the migration process by auditing the current Azure Monitor configuration. Identify the following:

• Metrics: Catalog the metrics being monitored, their collection intervals, and how they are utilized in day-to-day operations.
• Logs: Review logs collected from Azure resources and applications, noting any patterns or specific data points needed for troubleshooting or analysis.
• Alerts: Note also what specific alerts are configured.

This assessment can help determine the equivalent monitoring setup needed for Prometheus and Grafana.

The screenshots below are examples of Azure Monitor metrics captured for a Python Flask application running on an Azure Virtual Machine. These metrics, such as server request counts and response latency, would need to be migrated to Prometheus for continued monitoring.

Azure Monitor also collects logs from Azure resources. The following example shows a log entry from the Python Flask application:

Export Existing Azure Monitor Logs and Metrics

There are two commonly used approaches to exporting logs and metrics from Azure Monitor for migration to Prometheus and Grafana:

Option 1: Azure Monitor Metrics Explorer

The Azure Monitor Metrics Exporter is an open source tool designed to scrape Azure metrics and expose them in a Prometheus-compatible format. It supports collecting metrics from various Azure resources, such as Virtual Machines, App Services, and SQL Databases.

Typically deployed as a container or agent, the exporter is configured with Azure service principal credentials to access resources. Metrics are exposed at an HTTP endpoint, which Prometheus can scrape at regular intervals.

While this method enables real-time metric collection, it may require tuning to stay within Azure API limits in high-volume environments. Additionally, some Azure Monitor metrics or custom metrics may not be fully compatible with this exporter, necessitating alternative configurations.

Option 2: Diagnostic Settings in Azure Monitor

Azure Monitor’s diagnostic settings provide another way to export logs and metrics. This approach uses Azure Event Hub, a managed data streaming service, to transfer data from Azure Monitor.

• Metrics Workflow:

  1. Configure diagnostic settings for the Azure resources you want to export metrics from.

  2. Specify Azure Event Hub as the destination.

  3. Use a streaming service, such as Kafka or a custom consumer, to route data from Event Hub to Prometheus-compatible storage (e.g. a time-series database). Grafana can ingest metrics directly if the format is supported.

• Logs Workflow:

  1. Set diagnostic settings to route logs from Azure Monitor to Azure Event Hub.

  2. Stream logs from Event Hub to a log platform such as Loki, commonly paired with Grafana for visualization.

  3. If needed, employ an ETL (Extract, Transform, Load) pipeline or serverless job to reformat logs for compatibility with Loki or another log storage system.

Expose Application Metrics to Prometheus

Prometheus works differently from Azure Monitor: instead of pushing data like Azure Monitor, Prometheus pulls metrics from the monitored application. After assessing or exporting metrics as needed, modify the application to enable Prometheus metric scraping so that it collects the same metrics previously sent to Azure Monitor.

The prometheus_flask_exporter library is a standard library for instrumenting Flask applications to expose Prometheus metrics.

1. Open the app.py file:

   nano app.py

   Ensure the following lines are present, adding or adjusting them if needed:

   File: ~/simple-ec2-cloudwatch/example-flask-prometheus/app.py

   1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34

   import logging import random import time from flask import Flask from prometheus_flask_exporter import PrometheusMetrics logging.basicConfig(filename="/home/USERNAME/simple-ec2-cloudwatch/example-flask-prometheus/flask-app.log", level=logging.INFO) logger = logging.getLogger(__name__) app = Flask(__name__) metrics = PrometheusMetrics(app) metrics.info("FlaskApp", "Application info", version="1.0.0") @app.route("/") def hello_world(): logger.info("A request was received at the root URL") return {"message": "Hello, World!"}, 200 @app.route("/long-request") def long_request(): n = random.randint(1, 5) logger.info( f"A request was received at the long-request URL. Slept for {n} seconds" ) time.sleep(n) return {"message": f"Long running request with {n=}"}, 200 if __name__ == "__main__": app.run(host="0.0.0.0", port=8080)

   These lines use the prometheus_flask_exporter library to:

   • Instrument the Flask app for Prometheus metrics.
   • Expose default and application-specific metrics at the /metrics endpoint.
   • Provide metadata such as version information via metrics.info.

2. Save and close the file, then start and enable the flask-app service:

   sudo systemctl start flask-app
   sudo systemctl enable flask-app

3. Verify that the flask-app service is active (running):

   systemctl status flask-app

   ● flask-app.service - Flask Application Service
        Loaded: loaded (/etc/systemd/system/flask-app.service; enabled; preset: enabled)
        Active: active (running) since Thu 2024-12-05 17:26:18 EST; 1min 31s ago
      Main PID: 4413 (python)
         Tasks: 1 (limit: 9444)
        Memory: 20.3M (peak: 20.3M)
           CPU: 196ms
        CGroup: /system.slice/flask-app.service

4. Make sure the Flask app is accessible by issuing the following cURL command:

   curl http://IP_ADDRESS:8080

   You should receive the following response:

   {"message": "Hello, World!"}

5. To view the metrics, open a web browser and visit the following URL:

   http://FLASK_APP_IP_ADDRESS:8080/metrics

   The metrics shown include http_request_duration_seconds (request latency) and http_requests_total (total number of requests).

Configure Prometheus to Ingest Application Metrics

1. Modify the Prometheus configuration at /etc/prometheus/prometheus.yml to include the Flask application as a scrape target:

   sudo nano /etc/prometheus/prometheus.yml

   Append the following content to the scrap_configs section of the file, replacing FLASK_APP_IP_ADDRESS with the actual IP address of your monitoring-server instance, or in this case, localhost:

   File: /etc/prometheus/prometheus.yml

   1 2 3

   - job_name: 'flask_app' static_configs: - targets: ['FLASK_APP_IP_ADDRESS:8080']

   This configuration tell Prometheus to scrape metrics from the Flask application running on port 8080.

2. Save the file and restart Prometheus to apply the changes:

   sudo systemctl restart prometheus

3. To verify that Prometheus is successfully scraping the Flask app, open a web browser and navigate to the Prometheus UI:

   http://INSTANCE_IP_ADDRESS:9090

4. In the Prometheus UI click the Status tab and select Targets. You should see the Flask application service listed as a target with a status of up, indicating that Prometheus is successfully scraping metrics from the application.

Create a Grafana Dashboard with Application Metrics

Grafana serves as the visualization layer, providing an interface for creating dashboards from Prometheus metrics.

1. Open a web browser and visit the following URL to access the Grafana UI:

   http://INSTANCE_IP_ADDRESS:3000

2. Navigate to the Dashboards page:

   

3. Create a new dashboard in Grafana by clicking Create dashboard:

   

4. Next, click Add visualization:

   

5. In the resulting dialog, select the prometheus data source:

6. To duplicate the Azure Monitor metrics for the Flask application, first click on the Code tab in the right-hand side of the panel editor:

7. Input the following PromQL query to calculate the average latency for an endpoint:

   flask_http_request_duration_seconds_sum{method="GET",path="/",status="200"} /
   flask_http_request_duration_seconds_count{method="GET",path="/",status="200"}

8. After entering the formula, click Run queries to execute the PromQL query. The chart should update with data pulled from Prometheus:

   This visualization replicates Azure Monitor’s latency metrics, detailing the average latency over time for a specific endpoint. Prometheus further enhances this by providing default labels, such as method, path, and status codes, for greater granularity in analysis.

Additional Considerations and Concerns

When migrating from Azure Monitor to Prometheus and Grafana, it’s important to address several key considerations to ensure a smooth and effective transition.

Cost Management

Migrating to Prometheus and Grafana eliminates the recurring licensing costs associated with Azure Monitor. However, infrastructure costs for running Prometheus and Grafana are still a consideration. Running Prometheus and Grafana requires provisioning compute and storage resources, with expenses for maintenance and handling network traffic. Additionally, because Prometheus is designed for short-term data storage, setting up a long-term storage solution may also increase costs.

Recommendation:

• Assess the current costs of Azure Monitor, including licensing fees, and compare them to the infrastructure costs for hosting Prometheus and Grafana.
• Use Prometheus’s default short-term storage for real-time monitoring, reserving long-term storage for critical metrics only.
• Configure Grafana alerts and dashboards to minimize high-frequency scrapes and unnecessary data retention.
• Regularly review and refine retention policies and scraping intervals to balance cost with visibility needs.

Data Consistency and Accuracy

Azure Monitor automatically standardizes metrics and logs within its centralized system. In contrast, Prometheus and Grafana require custom configurations, raising concerns about consistency in data formatting, collection intervals, and overall accuracy during migration.

Recommendation:

• Document and map existing Azure Monitor metrics. logs, and alerts to align collection intervals and query formats.
• Standardize scrape intervals and retention policies across all Prometheus exporters to ensure consistent and comparable data.
• Regularly audit and validate data accuracy between Azure Monitor and Prometheus to detect any discrepancies early in the migration process.
• Utilize Grafana dashboards to compare data from both systems during the migration to ensure equivalence and reliability.

Azure Monitor Aggregated Data Versus Prometheus Raw Data

Azure Monitor simplifies analysis by aggregating data, offering summary metrics that reduce data volume while providing a high-level view of system health. In contrast, Prometheus collects raw, fine-grained metrics. While this enables detailed analyses and granular troubleshooting, it can also increase storage requirements and complexity when interpreting data.

Recommendation:

• Leverage recording rules in Prometheus to create aggregated views of commonly queried metrics, thereby reducing storage while retaining essential insights.
• For historical trends or aggregated overviews, use a data pipeline to export and store high-level summaries in Grafana.
• Consider integrating an archival database for long-term storage of aggregated data, reducing reliance on raw metrics for historical monitoring.

Alert System Migration

Azure Monitor supports custom alerts based on queries or thresholds. These can be replicated in Prometheus using PromQL, with AlertManager handling alert routing and notifications. To ensure a seamless migration, it is critical to ensure that the both conditions and intent for each alert are accurately translated.

Recommendation:

• Audit all Azure Monitor alerts to understand their conditions and thresholds
• Replicate these alerts in Prometheus using PromQL, ensuring they match the original intent and refining where necessary.
• Integrate Alertmanager with any existing notification systems (e.g. email, Slack, etc.) to maintain consistency in how teams are alerted to critical events.

Security and Access Controls

Azure Monitor integrates with Azure Active Directory to provide built-in role-based access control (RBAC). In contrast, securing Prometheus and Grafana involves setting up user authentication (e.g. OAuth, LDAP, etc.) and ensure metrics and dashboards are only accessible to authorized personnel. To maintain security, data in transit should be encrypted using TLS.

Recommendation:

• Establish a strong security baseline by implementing secure access controls from the start.
• Configure Grafana with a well-defined RBAC policy.
• Integrate Grafana and Prometheus with an authentication system to centralize access management.
• Enable TLS for Prometheus to encrypt data in transit.
• Ensure that any sensitive metrics are restricted from unauthorized users.
• Regularly audit access logs and permissions to identify and mitigate vulnerabilities.

Separate Log and Metric Responsibilities

Azure Monitor provides a unified interface for managing both metrics and logs. Because Prometheus is primarily a metrics-based monitoring solution, it does not have built-in capabilities for handling logs in the way Azure Monitor does. Therefore, it’s important to decouple log management needs from metric collection when migrating.

Recommendation:

• Introduce a specialized log aggregation solution alongside Prometheus and Grafana for collecting, aggregating, and querying logs:
  • Grafana Loki is designed to integrate with Grafana. It provides log querying capabilities within Grafana’s existing interface, giving a unified view of metrics and logs in a single dashboard.
  • Fluentd is a log aggregator that can forward logs to multiple destinations, including object storage for long-term retention. It works with both Loki and ELK.

More Information

You may wish to consult the following resources for additional information on this topic. While these are provided in the hope that they will be useful, please note that we cannot vouch for the accuracy or timeliness of externally hosted materials.

• Azure Monitor Documentation
• Prometheus Documentation
• Grafana Installation Documentation
• Grafana Dashboard Documentation