Setting Up Prometheus with NGINX Proxy & Authentication + Configuring Agent Pools

Challenge: Monitoring System

Deploy a Prometheus monitoring system on an Ubuntu cloud instance. The system should be secured and accessible through a reverse proxy, with proper authentication, and connected to an external Node Exporter. You can follow these steps:

Install Prometheus and Node Exporter:

Update your package list and install Prometheus with the following commands:

sudo apt update
sudo apt install prometheus prometheus-node-exporter

Verify the installation by checking the status of Prometheus services:

sudo service prometheus status
sudo service prometheus-node-exporter status

Completion Criteria

Use NGINX as a reverse proxy to forward requests from your server's IP address to Prometheus' default ports as follows:

http://{YOUR-SERVER-IP}/ to Prometheus (<http://localhost:9090/>)
http://{YOUR-SERVER-IP}/metrics/ to the Node Exporter (<http://localhost:9100/>)

Basic authentication is set up, requiring credentials to access both endpoints.
Direct access to ports 9090 (Prometheus) and 9100 (Node Exporter) is blocked, and access is only allowed via NGINX.
Prometheus is scraping metrics from an external Node Exporter running the same saver

Prerequisites

Azure Devops Account
Cloud Azure Account
Created a vm1 on Azure

Step 1: Create Vm1 on Azure and configure the security group

Go to Azure Port al.
Sign in with your Azure account.
In the Azure Portal, search for Virtual Machines in the search bar
Click Create → Azure Virtual Machine.
1. Subscription: Select your Azure subscription.
2. Resource Group: Choose an existing one or create a new one.
3. Region: Select the closest or preferred Azure region.
4. Image: Select the OS (e.g., Ubuntu 20.04 LTS).
5. VM Size: Choose a size based on your workload (e.g., Standard_B2s for small workloads).
Open the NSG you just created.
Go to Inbound security rules under Settings.
Click + Add Rule to create a new rule.
1. First, go to this site to know what your public IP is - https://nordvpn.com/pt-br/what-is-my-ip/
2. Add an Inbound role with source your public IP and destination your VM, you will create two roles one for port 80 and the other for port 22.

Step 2: Connect in your VM with putty

Go to Putty Gen with your credentials, access Conversions → Import Key, and select .pem key
Select on creating your private key
Now you will open the Putty
Select the public IP of your VM and Go to SSH → Auth → Browser and select your .ppk key

Step 3: Configure and Install Agent Pools in Azure DevOps

Go to Azure DevOps.
Sign in with your account.
Select your organization
Click on Organization Settings (bottom left corner).
Under Pipelines, select Default.
Click + New Agent.
Select Linux and follow this installation
When you are executing the Configure Agent with ./config in one step will ask you about the token, so you will need to create this token:
1. Click on your Profile Icon (top-right corner).
2. Select Personal Access Tokens.
3. Click + New Token.
4. Put the name.
5. Scope: Full Access.
After you enter the token, just click enter with the default settings.
Start the Agent with ./run.sh

Step 4: Create, Configure, and test your Pipeline

Create file azure-pipelines.yml

Enter these configurations:

 trigger:
 - main

 pr:
   branches:
     include:
     -  main

 stages:
     - stage: Prometheus
     displayName: Prometheus
     pool:
       name: Default
       vmImage: 'vm1'
     jobs:
     - job: Prometheus
       displayName: "Prometheus"
       steps:
       - script: |
           sudo apt install -y prometheus prometheus-node-exporter nginx apache2-utils

           sudo systemctl enable --now prometheus
           sudo systemctl enable --now prometheus-node-exporter
           sudo systemctl enable --now nginx

           systemctl status prometheus --no-pager
           systemctl status prometheus-node-exporter --no-pager
           systemctl status nginx --no-pager

         displayName: "Prometheus Install"

       - script: |
           echo 'global:
             scrape_interval: 15s

           scrape_configs:
             - job_name: "prometheus"
               static_configs:
                 - targets: ["localhost:9090"]

             - job_name: "node_exporter"
               static_configs:
                 - targets: ["localhost:9100"]
           ' | sudo tee /etc/prometheus/prometheus.yml

           sudo systemctl restart prometheus
         displayName: "Configurar Prometheus para coletar métricas"

       - script: |
           sudo ufw allow ssh
           sudo ufw allow http
           sudo ufw allow https
           sudo ufw deny 9090
           sudo ufw deny 9100
           sudo ufw enable
         displayName: "Configurar Acessos"

       - script: |
           echo 'server {
               listen 80;
               server_name _;

               location / {
                   proxy_pass http://127.0.0.1:9090/;
                   proxy_set_header Host $host;
                   proxy_set_header X-Real-IP $remote_addr;
                   proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                   auth_basic "Restricted Access";
                   auth_basic_user_file /etc/nginx/.htpasswd;
               }

               location /metrics/ {
                   proxy_pass http://127.0.0.1:9100/;
                   proxy_set_header Host $host;
                   proxy_set_header X-Real-IP $remote_addr;
                   proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                   auth_basic "Restricted Access";
                   auth_basic_user_file /etc/nginx/.htpasswd;
               }
           }' | sudo tee /etc/nginx/sites-available/default

           sudo systemctl restart nginx
         displayName: "Configurar Nginx"

       - script: |
           echo "Setting up Nginx authentication..."

           # Create .htpasswd file and add user
           echo "senha123" | sudo htpasswd -c -i /etc/nginx/.htpasswd joaochiroli

           sudo systemctl restart nginx
         displayName: "Configure Basic Auth for Nginx"