Installing Zabbix into Azure using a MySQL PaaS

For those of you that don't know Zabbix is an open source monitoring tool very much like Nagios (which i think is slightly better known).I blogged a while ago about how to set this up using Oracle as the back end system here but now I'm having to set this up in Azure I've gone a different route.

There is no Oracle PaaS solution in Azure (and likely never will be) however there are a couple of PaaS solutions for MySQL - one of these has recently been added by Microsoft themselves and actually seems like much better value than the other offerings so I wanted to try setting up Zabbix using that as the backend so i don't have to worry about managing MySQL (which i still know very little about).

More details on the Microsoft offering https://azure.microsoft.com/en-us/services/mysql/

In this post I'll take you through the steps i went though to set up zabbix server from scratch on a Redhat 7.3 IaaS box connecting to a MySQL PaaS backend repository database. Even if you have no interest in Zabbix there are some useful bits of information in this post for general use - i certainly learnt quite a lot doing it.

I'll start off by provisioning the MySQL PaaS - so here we go.

1) Go to the azure portal and search on the marketplace until you find "Azure Database for MySQL" and click 'create'

2) Fill in the basic details required for the service - see basic example below - there is not too much that is really selectable

3) Then click the create on this page and a couple of minutes later the db server is provisioned and we can navigate to its main maintenance page which looks something like this - i've purposely highlighted the connection settings page to show that if gives you the basic connect string types for most common tools.

4) now it's provisioned i want to connect to it to make sure all is OK - now this is where my lack of MySQL skills combined with some subtle differences in Azure connectivity caused me to stumble for a while when connecting - here is what happened:

First attempt:

[root@redhat conf.d]# mysql -u myadminaccount -p -h myazurepaasservername.mysql.database.azure.com
Enter password:
ERROR 2001 (28000): The connection string may not be right. Please visit portal for references.

OK this took me a while to get further with - turns out the username format has to be specified slightly differently

Second attempt:

[root@redhat conf.d]# mysql -u myadminaccount@myazurepaasservername.mysql.database.azure.com -p -h myazurepaasservername.mysql.database.azure.com
Enter password:
ERROR 2001 (28000): SSL connection is required. Please specify SSL options and retry.

OK - good progress but network encryption with ssl is required- so how to do that?

Third attempt:

Download the cert (this is the same for any MySQL Paas)

wget  https://www.digicert.com/CACerts/BaltimoreCyberTrustRoot.crt
--2017-07-11 10:19:25--  https://www.digicert.com/CACerts/BaltimoreCyberTrustRoot.crt
Resolving www.digicert.com (www.digicert.com)... 64.78.193.234
Connecting to www.digicert.com (www.digicert.com)|64.78.193.234|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 891 [application/x-x509-ca-cert]
Saving to: ‘BaltimoreCyberTrustRoot.crt’

100%[==============================================================================================================================>] 891         --.-K/s   in 0s

2017-07-11 10:19:26 (230 MB/s) - ‘BaltimoreCyberTrustRoot.crt’ saved [891/891]

Now go in to openssl and create a client cert based on that

[root@redhat ~]# openssl
OpenSSL> x509 -inform DER -in BaltimoreCyberTrustRoot.crt -out MyServerCACert.pem

That create the .pem file that i can now reference in my connection - so lets try again

[root@redhat ~]# mysql -u myadminaccount@myazurepaasservername.mysql.database.azure.com -p -h myazurepaasservername.mysql.database.azure.com  --ssl-ca=./MyServerCACert.pem
Enter password:
ERROR 2003 (28000): Client with IP address x.x.x.x is not allowed to access the server.

And now the firewall blocks it (slight aside here before we cover that - the certificate info can be specified in a global config file (/etc/my.cnf) so we don't have to pass it each time - sample content here

cat /etc/my.cnf
[client]
ssl-ca=/root/MyServerCACert.pem

Right back to the firewall - lets add my ip that was reported in the error into the MySQL PaaS config - that's done on this screen

And now Fourth attempt:

[root@redhat ~]# mysql -u myadminaccount@myazurepaasservername.mysql.database.azure.com -p -h myazurepaasservername.mysql.database.azure.com  --ssl-ca=./MyServerCACert.pem            Enter password:
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MySQL connection id is 65020
Server version: 5.6.26.0 MySQL Community Server (GPL)

Copyright (c) 2000, 2016, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MySQL [(none)]>

And we're in!


Right stage 1 complete - now to go and do the zabbix server install - then we'll come back and hook them up together.

So for the zabbix server install on RHEL 7.3 here are the steps

1) Load the basic software

 yum install httpd httpd-devel mysql php php-cli php-common php-devel php-pear php-gd php-mbstring php-mysql php-xml

2) enable apache to autostart and also start it

chkconfig httpd on
service httpd start

3) Add the zabbix repo to the existing set

rpm -Uvh http://repo.zabbix.com/zabbix/3.0/rhel/7/x86_64/zabbix-release-3.0-1.el7.noarch.rpm

4) install zabbix components from that repo

yum install zabbix-server-mysql zabbix-web-mysql zabbix-agent zabbix-java-gateway

5) update the date.timezone parameter in /etc/httpd/conf.d/zabbix.conf

<Directory "/usr/share/zabbix">
    Options FollowSymLinks
    AllowOverride None
    Require all granted

    <IfModule mod_php5.c>
        php_value max_execution_time 300
        php_value memory_limit 128M
        php_value post_max_size 16M
        php_value upload_max_filesize 2M
        php_value max_input_time 300
        php_value always_populate_raw_post_data -1
        php_value date.timezone Europe/Berlin
    </IfModule>
</Directory>

6) restart apache to pick that up

service httpd restart

And at this point we have the basic software in and running but we need to now link the two components together

So first step now is to create the database objects in the database - the file containign the commands comes as part of what we just installed - so lets go and load it

cd /usr/share/doc/zabbix-server-mysql-3.0.9
gunzip create.sql.gz

Now we have the create.sql file which contains everything we need to create - now we just need a database to put that in. So lets login to the PaaS and create a new zabbix database/schema (whatever the term is in MySQL :-))

MySQL [(none)]> CREATE DATABASE zabbixdb CHARACTER SET UTF8;
Query OK, 1 row affected (0.26 sec)

MySQL [(none)]> GRANT ALL PRIVILEGES on zabbixdb.* to myadminaccount@localhost;
Query OK, 0 rows affected, 2 warnings (0.61 sec)


MySQL [(none)]> FLUSH PRIVILEGES;
Query OK, 0 rows affected (0.77 sec)

MySQL [(none)]> quit
Bye

Now lets load the objects in - loads of output here so i removed most of it

MySQL [zabbixdb]> source create.sql
Query OK, 0 rows affected (1.03 sec)

Query OK, 0 rows affected (0.52 sec)
Records: 0  Duplicates: 0  Warnings: 0

Query OK, 0 rows affected (0.48 sec)
Records: 0  Duplicat

etc
etc 

OK - all seems good so far - now lets start zabbix server and hook these things together


chkconfig zabbix-server on

[root@redhat zabbix-server-mysql-3.0.9]# service zabbix-server start
Redirecting to /bin/systemctl start  zabbix-server.service
Job for zabbix-server.service failed because a configured resource limit was exceeded. See "systemctl status zabbix-server.service" and "journalctl -xe" for details.

And this is where it got tricky again - lets try and get some more info what is wrong..

[root@redhat zabbix-server-mysql-3.0.9]# systemctl status zabbix-server.service
? zabbix-server.service - Zabbix Server
   Loaded: loaded (/usr/lib/systemd/system/zabbix-server.service; enabled; vendor preset: disabled)
   Active: activating (auto-restart) (Result: resources) since Tue 2017-07-11 11:00:28 UTC; 3s ago
  Process: 38203 ExecStop=/bin/kill -SIGTERM $MAINPID (code=exited, status=1/FAILURE)
  Process: 38211 ExecStart=/usr/sbin/zabbix_server -c $CONFFILE (code=exited, status=0/SUCCESS)
 Main PID: 38202 (code=exited, status=1/FAILURE)

Jul 11 11:00:28 redhat systemd[1]: zabbix-server.service never wrote its PID file. Failing.
Jul 11 11:00:28 redhat systemd[1]: Failed to start Zabbix Server.
Jul 11 11:00:28 redhat systemd[1]: Unit zabbix-server.service entered failed state.
Jul 11 11:00:28 redhat systemd[1]: zabbix-server.service failed.

That's pretty useless but a quick google revealed this was something selinux related so i installed the setroubleshoot software (which had about 80 pre-reqs.....)

yum install setroubleshoot

From one of the tools in this set i could then run this to tell me what was wrong

 ausearch -m avc,user_avc,selinux_err |grep zabbix |grep denied

which showed this

type=AVC msg=audit(1499771577.661:620): avc:  denied  { setrlimit } for  pid=38708 comm="zabbix_server" scontext=system_u:system_r:zabbix_t:s0 tcontext=system_u:system_r:zabbix_t:s0 tclass=process

To allow selinux to allow this i can use this other utility audit2allow to create a new policy addition that allows it

ausearch -m avc,user_avc,selinux_err -ts recent|grep zabbix |grep denied | audit2allow -M 

zabbixpol

******************** IMPORTANT ***********************

To make this policy package active, execute:

semodule -i zabbixpol.pp

I then run the command line it mentions and the zabbix server starts OK.

Now i need to open the firewall to allow me to connect to the zabbix website

[root@redhat zabbix]# firewall-cmd --zone=public --add-port=80/tcp --permanent

success

[root@redhat zabbix]# firewall-cmd --reload

success

Now we can switch back to GUI mode to complete the setup - so lets navigate to the zabbix website - which is just http://servername/zabbix

We move to the next screen for db connection properties - and at this point i realized i didn't really want to store the admin account credentials for the actual application - so i went ahead and created a new zabbix account to be used for logging in to the database.

This was just a case or running this

MySQL [zabbixdb]> grant all privileges on zabbixdb.* to zabbix@'%' identified by 'password';
Query OK, 0 rows affected, 1 warning (0.75 sec)

MySQL [zabbixdb]> flush privileges;
Query OK, 0 rows affected (0.74 sec)

I then used these credentials in the config screen.

And this is where i get into more issues - it started with this - which was an easy fix (similar to the issue i had on command line so i just had to extend to the other username format.

Even after that though i couldn't connect with the message error (13) - this again was a very tricky one to solve - in the end it was some combo of these 3 changes that got me past that

setsebool -P zabbix_can_network=1
setsebool -P httpd_can_network_connect_db=1
semanage port -m -t http_port_t -p tcp 10051

After that though i hit the ssl connection error that i saw from command line - and after a lot of digging it seems that the php code supplied uses a mysql api that doesn't support ssl connections -it's possible this could be changed by someone who knows what they are doing but i left it for the moment and changed the PaaS to allow non ssl connections

After that it connected fine and i moved on to the next screen

And the next

Then it seems to have worked

Let's try and login

And we're good!

So there you have it - Zabbix works fine with a PaaS database - little fiddly in some parts but some of that was me just getting to grips with how things work. selinux threw a few spanners in - so if you don't care for that switch it off and it will save you some pain.

By the way the picture at the top is a monitor on a platform and this is a monitoring tool being installed on a platform - see what i did there......? :-)

Sorry there is more to this as i just discovered when trying to get the agent on the server to work....

Quick addendum on additional steps........

chkconfig zabbix-agent on
service zabbix-agent start

This fails to start due to selinux - so we do same trick as before

ausearch -m avc,user_avc,selinux_err -ts recent|grep zabbix |grep denied | audit2allow -M zabbixagentpol
******************** IMPORTANT ***********************
To make this policy package active, execute:

semodule -i zabbixagentpol.pp

Now we start OK

service zabbix-agent start

Now we enable some more ports in the firewall

firewall-cmd --add-port=10050/tcp --permanent
firewall-cmd --add-port=10051/tcp --permanent

Then we have to update the config file /etc/zabbix/zabbix_server.conf with the mysql location - i had thought the GUI had done this - but it seems that's just for the php web config and not the backend server config - so the following lines need to be updated

DBHost=yourpaasname.mysql.database.azure.com
DBName=zabbixdb
DBUser=zabbix@yourpaasname.mysql.database.azure.com
DBPassword=yourpassword

And for the final twist in the tale there is yet another seliux thing to allow as once the connection gets further port 10051 has issues :-) - so the fix for that is this

ausearch -m avc,user_avc,selinux_err -ts recent|grep zabbix |grep denied | audit2allow -M zabbixserver

******************** IMPORTANT ***********************

To make this policy package active, execute:

semodule -i zabbixserver.pp

After that and as long as you have the correct server/ip/active server in the agent config the dman thing finally appears OK and goes green.....