Hugo Future Imperfect Slim

Tech Notes

Linux,MAC,Docker,Kubernetes,
CloudStack,AWS,GCP,AZURE,
Raspberrypi,VMWARE,GO,BOOKS,
NETFLIX,Movies,Seasons

Monitoring Apache Cloudstack with Prometheus and Grafana

3-Minute Read

Monitoring Cloudstack with Prometheus and Grafana

Cloudstack supports prometheus integration and it exports various metrics for the prometheus server to Scrape

Enable prometheus integragration on Cloudstack Management server

  1. Enable prometheus plugin in Cloudstack Management server

Login to the database or enable the following global setting parameters

prometheus.exporter.enable : Enable the prometheus exporter plugin, management server restart needed. prometheus.exporter.allowed.ips: List of comma separated prometheus server ips (with no spaces) that should be allowed to access the metrics endpoint.

mysql>update configuration set value="true" where name="prometheus.exporter.enable";

mysql>update configuration set value="127.0.0.1,10.0.33.227" where name="prometheus.exporter.allowed.ips";
  1. Restart the cloudstack mangement server
systemctl restart cloudstack-managment
  1. Check if the management server is exporting the metrics on port 9595

curl -vv 127.0.0.1:9595/metrics

cloudstack_host_is_dedicated{zone="ref-trl-4857-k-M7-kiran-chavala",hostname="ref-trl-4857-k-M7-kiran-chavala-kvm1",ip="10.0.33.130"} 0
cloudstack_host_cpu_usage_mhz_total{zone="ref-trl-4857-k-M7-kiran-chavala",hostname="ref-trl-4857-k-M7-kiran-chavala-kvm1",ip="10.0.33.130",overprovisioningfactor="2.0",filter="used",tags="kiran"} 2500.00
cloudstack_host_cpu_usage_mhz_total{zone="ref-trl-4857-k-M7-kiran-chavala",hostname="ref-trl-4857-k-M7-kiran-chavala-kvm1",ip="10.0.33.130",overprovisioningfactor="2.0",filter="total",tags="kiran"} 6300.00
cloudstack_host_memory_usage_mibs_total{zone="ref-trl-4857-k-M7-kiran-chavala",hostname="ref-trl-4857-k-M7-kiran-chavala-kvm1",ip="10.0.33.130",overprovisioningfactor="1.0",filter="used",dedicated="0",tags="kiran"} 2816.00
cloudstack_host_memory_usage_mibs_total{zone="ref-trl-4857-k-M7-kiran-chavala",hostname="ref-trl-4857-k-M7-kiran-chavala-kvm1",ip="10.0.33.130",overprovisioningfactor="1.0",filter="total",dedicated="0",tags="kiran"} 6940.01
cloudstack_host_vms_total{zone="ref-trl-4857-k-M7-kiran-chavala",hostname="ref-trl-4857-k-M7-kiran-chavala-kvm1",address="10.0.33.130"} 5
cloudstack_host_vms_cores_total{zone="ref-trl-4857-k-M7-kiran-chavala",hostname="ref-trl-4857-k-M7-kiran-chavala-kvm1",ip="10.0.33.130",filter="used",dedicated="0",tags="kiran"} 5
cloudstack_host_vms_cores_total{zone="ref-trl-4857-k-M7-kiran-chavala",hostname="ref-trl-4857-k-M7-kiran-chavala-kvm1",ip="10.0.33.130",filter="total",dedicated="0",tags="kiran"} 3
cloudstack_host_cpu_usage_mhz_total{zone="ref-trl-4857-k-M7-kiran-chavala",filter="allocated"} 2500.00
cloudstack_host_memory_usage_mibs_total{zone="ref-trl-4857-k-M7-kiran-chavala",filter="allocated"} 2816.00
cloudstack_host_vms_cores_total{zone="ref-trl-4857-k-M7-kiran-chavala",filter="allocated"} 5
cloudstack_hosts_total{zone="ref-trl-4857-k-M7-kiran-chavala",filter="online"} 1
cloudstack_hosts_total{zone="ref-trl-4857-k-M7-kiran-chavala",filter="offline"} 0
cloudstack_hosts_total{zone="ref-trl-4857-k-M7-kiran-chavala",filter="total"} 1
cloudstack_hosts_total_by_tag{zone="ref-trl-4857-k-M7-kiran-chavala",filter="total",tags="kiran"} 1
cloudstack_hosts_total_by_tag{zone="ref-trl-4857-k-M7-kiran-chavala",filter="online",tags="kiran"} 1
cloudstack_hosts_total_by_tag{zone="ref-trl-4857-k-M7-kiran-chavala",filter="offline",tags="kiran"} 0
cloudstack_host_vms_cores_total_by_tag{zone="ref-trl-4857-k-M7-kiran-chavala",filter="allocated",tags="kiran"} 0
cloudstack_host_cpu_usage_mhz_total_by_tag{zone="ref-trl-4857-k-M7-kiran-chavala",filter="allocated",tags="kiran"} 0.00
cloudstack_host_memory_usage_mibs_total_by_tag{zone="ref-trl-4857-k-M7-kiran-chavala",filter="allocated",tags="kiran"} 0.00
cloudstack_vms_total_by_tag{zone="ref-trl-4857-k-M7-kiran-chavala",filter="starting",tags="kiran"} 0
cloudstack_vms_total_by_tag{zone="ref-trl-4857-k-M7-kiran-chavala",filter="running",tags="kiran"} 0
cloudstack_vms_total_by_tag{zone="ref-trl-4857-k-M7-kiran-chavala",filter="stopping",tags="kiran"} 0
cloudstack_vms_total_by_tag{zone="ref-trl-4857-k-M7-kiran-chavala",filter="stopped",tags="kiran"} 0
cloudstack_vms_total_by_tag{zone="ref-trl-4857-k-M7-kiran-chavala",filter="destroyed",tags="kiran"} 0
cloudstack_vms_total_by_tag{zone="ref-trl-4857-k-M7-kiran-chavala",filter="expunging",tags="kiran"} 0
cloudstack_vms_total_by_tag{zone="ref-trl-4857-k-M7-kiran-chavala",filter="migrating",tags="kiran"} 0
cloudstack_vms_total_by_tag{zone="ref-trl-4857-k-M7-kiran-chavala",filter="error",tags="kiran"} 0
cloudstack_vms_total_by_tag{zone="ref-trl-4857-k-M7-kiran-chavala",filter="unknown",tags="kiran"} 0
cloudstack_vms_total_by_tag{zone="ref-trl-4857-k-M7-kiran-chavala",filter="shutdown",tags="kiran"} 0
cloudstack_vms_total{zone="ref-trl-4857-k-M7-kiran-chavala",filter="starting"} 0
cloudstack_vms_total{zone="ref-trl-4857-k-M7-kiran-chavala",filter="running"} 5
cloudstack_vms_total{zone="ref-trl-4857-k-M7-kiran-chavala",filter="stopping"} 0
cloudstack_vms_total{zone="ref-trl-4857-k-M7-kiran-chavala",filter="stopped"} 0
cloudstack_vms_total{zone="ref-trl-4857-k-M7-kiran-chavala",filter="destroyed"} 0
cloudstack_vms_total{zone="ref-trl-4857-k-M7-kiran-chavala",filter="expunging"} 0
cloudstack_vms_total{zone="ref-trl-4857-k-M7-kiran-chavala",filter="migrating"} 0
cloudstack_vms_total{zone="ref-trl-4857-k-M7-kiran-chavala",filter="error"} 0
cloudstack_vms_total{zone="ref-trl-4857-k-M7-kiran-chavala",filter="unknown"} 0
cloudstack_vms_total{zone="ref-trl-4857-k-M7-kiran-chavala",filter="shutdown"} 0
cloudstack_volumes_total{zone="ref-trl-4857-k-M7-kiran-chavala",filter="ready"} 5
cloudstack_volumes_total{zone="ref-trl-4857-k-M7-kiran-chavala",filter="destroy"} 0
cloudstack_volumes_total{zone="ref-trl-4857-k-M7-kiran-chavala",filter="total"} 5
cloudstack_storage_pool_gibs_total{zone="ref-trl-4857-k-M7-kiran-chavala",name="ref-trl-4857-k-M7-kiran-chavala-kvm-pri1",address="10.0.32.4:/acs/primary/ref-trl-4857-k-M7-kiran-chavala/ref-trl-4857-k-M7-kiran-chavala-kvm-pri1",type="primary",overprovisioningfactor="2.0",filter="used"} 1252.33
cloudstack_storage_pool_gibs_total{zone="ref-trl-4857-k-M7-kiran-chavala",name="ref-trl-4857-k-M7-kiran-chavala-kvm-pri1",address="10.0.32.4:/acs/primary/ref-trl-4857-k-M7-kiran-chavala/ref-trl-4857-k-M7-kiran-chavala-kvm-pri1",type="primary",overprovisioningfactor="2.0",filter="allocated"} 20.88
cloudstack_storage_pool_gibs_total{zone="ref-trl-4857-k-M7-kiran-chavala",name="ref-trl-4857-k-M7-kiran-chavala-kvm-pri1",address="10.0.32.4:/acs/primary/ref-trl-4857-k-M7-kiran-chavala/ref-trl-4857-k-M7-kiran-chavala-kvm-pri1",type="primary",overprovisioningfactor="2.0",filter="unallocated"} 4073.12
cloudstack_storage_pool_gibs_total{zone="ref-trl-4857-k-M7-kiran-chavala",name="ref-trl-4857-k-M7-kiran-chavala-kvm-pri1",address="10.0.32.4:/acs/primary/ref-trl-4857-k-M7-kiran-chavala/ref-trl-4857-k-M7-kiran-chavala-kvm-pri1",type="primary",overprovisioningfactor="2.0",filter="total"} 2047.00
cloudstack_storage_pool_gibs_total{zone="ref-trl-4857-k-M7-kiran-chavala",name="NFS://10.0.32.4/acs/secondary/ref-trl-4857-k-M7-kiran-chavala/ref-trl-4857-k-M7-kiran-chavala-sec1",address="NFS://10.0.32.4/acs/secondary/ref-trl-4857-k-M7-kiran-chavala/ref-trl-4857-k-M7-kiran-chavala-sec1",type="secondary",filter="used"} 1252.33
cloudstack_storage_pool_gibs_total{zone="ref-trl-4857-k-M7-kiran-chavala",name="NFS://10.0.32.4/acs/secondary/ref-trl-4857-k-M7-kiran-chavala/ref-trl-4857-k-M7-kiran-chavala-sec1",address="NFS://10.0.32.4/acs/secondary/ref-trl-4857-k-M7-kiran-chavala/ref-trl-4857-k-M7-kiran-chavala-sec1",type="secondary",filter="total"} 2047.00
cloudstack_private_ips_total{zone="ref-trl-4857-k-M7-kiran-chavala",filter="allocated"} 2
cloudstack_private_ips_total{zone="ref-trl-4857-k-M7-kiran-chavala",filter="total"} 20
cloudstack_public_ips_total{zone="ref-trl-4857-k-M7-kiran-chavala",filter="allocated"} 3
cloudstack_public_ips_total{zone="ref-trl-4857-k-M7-kiran-chavala",filter="total"} 20
cloudstack_shared_network_ips_total{zone="ref-trl-4857-k-M7-kiran-chavala",filter="allocated"} 0
cloudstack_shared_network_ips_total{zone="ref-trl-4857-k-M7-kiran-chavala",filter="total"} 0
cloudstack_vlans_total{zone="ref-trl-4857-k-M7-kiran-chavala",filter="allocated"} 1
cloudstack_vlans_total{zone="ref-trl-4857-k-M7-kiran-chavala",filter="total"} 20
cloudstack_active_domains_total{zone="ref-trl-4857-k-M7-kiran-chavala"} 1
cloudstack_active_accounts_total{zone="ref-trl-4857-k-M7-kiran-chavala"} 1
cloudstack_vms_total_by_size{zone="ref-trl-4857-k-M7-kiran-chavala",cpu="1",memory="512"} 2
cloudstack_domain_limit_cpu_cores_total 0
cloudstack_domain_limit_memory_mibs_total 0
cloudstack_domain_resource_count{domain="/", type="memory"} 1024
cloudstack_domain_resource_count{domain="/", type="cpu"} 2
cloudstack_domain_resource_count{domain="/", type="primary_storage"} 17179869184
cloudstack_domain_resource_count{domain="/", type="secondary_storage"} 13832814592
* Connection #0 to host 127.0.0.1 left intact

Steps to install Prometheus server

The steps include to start the prometheus as a systemd service

  1. Download the latest binary of Prometheus from
https://prometheus.io/

https://prometheus.io/download/


wget "https://github.com/prometheus/prometheus/releases/download/v2.43.1/prometheus-2.43.1.linux-amd64.tar.gz"
  1. Untar the package
tar -xvf prometheus-2.43.1.linux-amd64.tar.gz
cd prometheus-2.43.1.linux-amd64
  1. Create a prometheus user and change the directory permissions and also copy the necessary permissions
sudo useradd --no-create-home --shell /bin/false prometheus
sudo mkdir /etc/prometheus
sudo mkdir /var/lib/prometheus
sudo chown prometheus:prometheus /etc/prometheus
sudo chown prometheus:prometheus /var/lib/prometheus

sudo cp prometheus /usr/local/bin
sudo cp promtool /usr/local/bin

sudo chown prometheus:prometheus /usr/local/bin/prometheus
sudo chown prometheus:prometheus /usr/local/bin/promtool

sudo cp -r consoles /etc/prometheus/
sudo cp -r console_libraries/ /etc/prometheus/

sudo chown -R prometheus:prometheus /etc/prometheus/consoles
sudo chown -R prometheus:prometheus /etc/prometheus/console_libraries/

sudo cp prometheus.yml /etc/prometheus/

sudo chown  prometheus:prometheus /etc/prometheus/prometheus.yml
  1. Create the systemd file
sudo vi /etc/systemd/system/prometheus.service

[Unit]
Description=Prometheus
Wants=network-online.target
After=network-online.target

[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/local/bin/prometheus \
--config.file /etc/prometheus/prometheus.yml \
--storage.tsdb.path /var/lib/prometheus/ \
--web.console.templates=/etc/prometheus/consoles \
--web.console.libraries=/etc/prometheus/console_libraries

[Install]
WantedBy=multi-user.target

sudo systemctl daemon-reload

sudo systemctl start prometheus
sudo systemctl status prometheus
sudo systemctl enable prometheus
  1. Check if prometheus is listening on port 9090

Go to the browser

http://:9090/graph

curl -vv :9090/metrics

  1. Edit the prometheus.yaml to scrape the metrics from
  - job_name: "Cloudstack-metrics"
    static_configs:
      - targets: ["<ip of cloudstack managment server>:9595"]

You should be able to get the metrics in the prometheus dashboard

Steps to install Grafana

  1. Add the grafana repo
vi /etc/yum.repos.d/grafana.repo

[grafana]
name=grafana
baseurl=https://rpm.grafana.com
repo_gpgcheck=1
enabled=1
gpgcheck=1
gpgkey=https://rpm.grafana.com/gpg.key
sslverify=1
sslcacert=/etc/pki/tls/certs/ca-bundle.cr
  1. Install Grafana
yum install grafana
systemctl start grafana-server
systemctl status grafana-server
  1. Check is Grafana is working on Port 3000

http://:3000

  1. Home > Administration > Add data sources> select Prometheus
  1. Now you can explore the metrics via Grafana and create nice dashboard

Ref:

https://www.youtube.com/watch?v=m8mYdWHoxLY&t=2033s
https://cwiki.apache.org/confluence/display/CLOUDSTACK/CloudStack+Metrics+Exporter+for+Prometheus
comments powered by Disqus

Recent Posts

Categories

About

I am a Software Engineer at Persistent Systems, working on Cloud Stack Orchestration and various tech related to Cloud Infra, container technology like Docker, Kubernetes