Setup Zabbix monitoring with Oracle back end repository

We currently have a number of monitoring tools in place in the organisation (about 5 at last count - and there may be more....). None of them seems to do exactly what the organisation wants as different areas require different functionality, we also have licence and access restrictions in place for some of those tools.

Anyway we are now evaluating Zabbix to see how that can help us and how it compares with other tools already in place.

For those of you not in the know (whom i was one of until very recently), Zabbix is an open source monitoring tool - you can read all the spiel about it here: http://www.zabbix.com/

So lets get down to it then - i want to install this and as i don't have root access and getting anything done by the central admin function is pretty painful - so i don't want to use any prebuilt packages as these can only be loaded by the root user.

Instead i'm doing this the hardcore way and downloading all the sources and i'm going to compile and build all of this myself - and if that's not enough i'm going to use Oracle as the back end repository database where normally this would be mySQL (I don't want to have to learn mySQL - at least not right now anyway....)

My environment in SLES11sp3 and the repo database I'm planning to use is Oracle 11.2.0.4.3

So first things first - and this is the only point i have to involve the unix team - i need a new group creating called zabbix and a new user created called zabbix (with zabbix as it's primary group)

so just

#groupadd zabbix
#useradd -g zabbix zabbix

Now that's out of the way i can get down to the serious task of installing the software - now as i went through this i discovered that more and more sources were needed to be able to set all this up (particularly in the apache/php area) - I'm listing them all here for completeness - i would advise you to read all of this post before you start doing anything as i didn't do things in the most logical order and there were problems along the way.

So here is the complete list of sources

zabbix-2.4.4rc1.tar.gz
httpd-2.2.29.tar.gz
php-5.6.7.tar.gz
libxml2-2.9.2.tar.gz
freetype-2.5.5.tar.gz
libpng-1.6.16.tar.gz
zlib-1.2.8.tar
jpegsrc.v6b.tar.gz

Depending on what your current system has you may need more or less than this - but this did it for me.

First up was the zabbix source itself - i needed to unzip and extract that to get to the files i needed to create the database schema inside oracle - to get to these i did the following

gunzip zabbix-2.4.4rc1.tar.gz
tar -xvf zabbix-2.4.4rc1.tar
cd ./zabbix-2.4.4rc1/database/oracle

In this directory are 3 files

 ls
data.sql  images.sql  schema.sql

The install docs tell you to create a schema to hold the objects but they give no clue as to what privileges it should have - so i took a look at the files and ran the following steps

create tablespace zabbix;
create user zabbix identified by zabbix default tablespace zabbix;
alter user zabbix quota unlimited on zabbix;
grant resource,create session to zabbix;
grant create any directory to zabbix;

So now we have somewhere to create the objects lets login and run the supplied script

sqlplus zabbix/zabbix
SQL>@schema

Now first run through this spewed out loads of errors basically because the column length is too long for oracle (which makes me wonder if this was tested...)

Anyway i did a global replace of

nvarchar2(2048)

with

nvarchar2(2000)

in the schema.sql file and then dropped and recreated the user - i then re-ran it and all was fine.

Next stage is to load the images into the database

To enable this the images.sql file must be edited to change the path to where the images are located - in my case this involved changing this line

CREATE OR REPLACE DIRECTORY image_dir AS '/home/zabbix/zabbix/create/output_png'
/

to this

CREATE OR REPLACE DIRECTORY image_dir AS '/export/home/zabbix/zabbix-2.4.4rc1/misc/images'
/

Then the script runs ok - it just throws a single error at the end which we can ignore as its just a tidy up

DROP DIRECTORY image_dir
*
ERROR at line 1:
ORA-01031: insufficient privileges


So that's images done - now onto some basic config data that has to be loaded

This was the easiest of the three and ran without error - it just needed a commit at the end.

OK - now the database is ready to go - now onto the Zabbix software (and the supporting cast.....)

So first up i got to the directory where i untar'd zabbix and i run the configure command specifying explicit install paths, the fact i want server and agent and the fact i want to use oracle - this seems very happy and gives me a nice output at the end

cd /export/home/zabbix/zabbix-2.4.4rc1
./configure --prefix=/export/home/zabbix --enable-server --enable-agent --with-oracle=/oracle/11.2.0.4.3.DB


Configuration:

  Detected OS:           linux-gnu
  Install path:          /export/home/zabbix
  Compilation arch:      linux

  Compiler:              gcc
  Compiler flags:        -g -O2   -I/oracle/11.2.0.4.3.DB/rdbms/public -I/oracle/11.2.0.4.3.DB/rdbms/demo

  Enable server:         yes
  Server details:
    With database:         Oracle
    WEB Monitoring:        no
    Native Jabber:         no
    SNMP:                  no
    IPMI:                  no
    SSH:                   no
    ODBC:                  no
    Linker flags:          -rdynamic      -L/oracle/11.2.0.4.3.DB/lib
    Libraries:             -lm -ldl -lrt  -lresolv     -lclntsh -lnnz11

  Enable proxy:          no

  Enable agent:          yes
  Agent details:
    Linker flags:          -rdynamic
    Libraries:             -lm -ldl -lrt  -lresolv

  Enable Java gateway:   no

  LDAP support:          no
  IPv6 support:          no

***********************************************************
*            Now run 'make install'                       *
*                                                         *
*            Thank you for using Zabbix!                  *
*              <http://www.zabbix.com>                    *
***********************************************************

I then do

make install

and it's all gone very smoothly....

I now update
serveractive = server shortname
server = server shortname

 in the

/export/home/zabbix/etc/zabbix_agentd.conf

and in

 zabbix_server.conf

i set

dbname=TNSNAMES entry of repo database
dbuser=zabbix
dbpassword=zabbix

And that is zabbix configured (at least from a backend point of view) - now it got more tricky - the front end needed a load of other components to make it work. I went through a load of trial and error to get this to work - some of which is detailed below.....

For the front end then first of all we need apache - lets get that done

Don't fall into the trap of just trying to build apache directly - there are some other bits to do first from the downloaded file - first up is apr

cd /export/home/zabbix/httpd-2.2.29/srclib/apr
./configure --prefix /export/home/zabbix/apr
make install

then apr-util (which is dependant on apr)

cd ../apr-util
./configure --prefix=/export/home/zabbix/apr-util --with-apr=/export/home/zabbix/apr
make
make install

then apache itself (dependent on the 2 we already built)

cd ../..
 ./configure --prefix /export/home/zabbix/apache --with-apr=/export/home/zabbix/apr  --with-apr-util=/export/home/zabbix/apr-util
make
make install

In the apache config file (/export/home/zabbix/apache/conf/httpd.conf ) i switch the port to 3333 (rather than the root only port 80 and to keep it away from other used ports)

So i change

Listen 3333

I then copy the Zabbix front end files under the apache tree

cd /export/home/zabbix/apache/htdocs
mkdir zabbix
cd /export/home/zabbix/zabbix-2.4.4rc1/frontends/php
cp -a * /export/home/zabbix/apache/htdocs/zabbix

Now we can start apache

/export/home/zabbix/apache/bin> ./apachectl start

The basic test page works fine

However as soon as i try the zabbix pages i just get a directory listing and they are not processed correcly. After some basic trial and error it emerges that you have to install php (I had foolishly thought that this somehow came with apache....)

So lets get php - for brevity I'm just going to show the working steps - but this is where i spent most of my effort......

So i get all of these.....

php-5.6.7.tar.gz
libxml2-2.9.2.tar.gz
freetype-2.5.5.tar.gz
libpng-1.6.16.tar.gz
zlib-1.2.8.tar
jpegsrc.v6b.tar.gz

and do the following

gunzip libxml2-2.9.2.tar.gz
tar -xvf libxml2-2.9.2.tar
./configure --prefix=/export/home/zabbix/libxml2 --without-python
make
make install

gunzip freetype-2.5.5.tar.gz
tar -xvf freetype-2.5.5.tar
./configure --prefix=/export/home/zabbix/freetype
make
make install

gunzip zlib-1.2.8.tar.gz
 tar -xvf zlib-1.2.8.tar
 cd zlib-1.2.8
 ./configure --prefix=/export/home/zabbix/zlib
 make
 make install

export LDFLAGS="-L/export/home/zabbix/zlib/lib"
export CPPFLAGS="-I/export/home/zabbix/zlib/include"

gunzip libpng-1.6.16.tar.gz
 tar -xvf libpng-1.6.16.tar
./configure --prefix=/export/home/zabbix/libpng

gunzip jpegsrc.v6b.tar.gz
tar -xvf jpegsrc.v6b.tar
cd jpeg-6b
   mkdir /export/home/zabbix/libjpeg/lib
mkdir  /export/home/zabbix/libjpeg/
mkdir /export/home/zabbix/libjpeg/bin
 mkdir -p /export/home/zabbix/libjpeg/man/man1
./configure --prefix=/export/home/zabbix/libjpeg --enable-shared

Edit the makefile for this one to change ./libtool to just libtool

make 
make install

 gunzip php-5.6.7.tar.gz
tar -xvf php-5.6.7.tar
cd php-5.6.7
./configure --prefix=/export/home/zabbix/php --with-apxs2=/export/home/zabbix/apache/bin/apxs --with-libxml-dir=/export/home/zabbix/libxml2 --enable-bcmath --enable-mbstring --enable-sockets --with-gd --with-gettext --with-oci8=/oracle/11.2.0.4.3.DB --with-png-dir=/export/home/zabbix/libpng --with-freetype-dir=/export/home/zabbix/freetype --with-zlib-dir=/export/home/zabbix/zlib --with-jpeg-dir=/export/home/zabbix/libjpeg
make
make install

Now after all that messing about i now have a working apache/php setup (however there is still a little bit of config i need to set for php itself). Now i wasn't sure where this config file should be located - but fortunately it's easy enough to find out

you just create a simple file info.php in htdocs/zabbix with this content

<?php phpinfo(); ?>

When you call this page from a browser it tells you where the config file should be

In this file (/export/home/zabbix/php/lib/php.ini ) i paste the following content (which is case sensitive i found out....)

max_execution_time = 600
expose_php = off
date.timezone = CET
post_max_size = 32M
upload_max_filesize = 16M
max_input_time = 600
memory_limit = 256M
always_populate_raw_post_data = -1

Now all that is done i just need to check/amend some entries in the httpd.conf for apache so that the php stuff is correctly working

LoadModule php5_module        modules/libphp5.so

<FilesMatch "\.ph(p[2-6]?|tml)$">
    SetHandler application/x-httpd-php
</FilesMatch>

<IfModule mime_module>

    AddType application/x-httpd-php .php

</IfModule>

And in mime.types (not sure if this is actually needed)

application/x-httpd-php                         php

Right now all that lot is in place i can start up everything (note the following 2 env variables must be defined)

 export ORACLE_HOME=/oracle/11.2.0.4.3.DB
export LD_LIBRARY_PATH=$ORACLE_HOME/lib

(oh and make sure your tnsnames entry is set up correctly for your repo DB....)

we start apache
/export/home/zabbix/apache/bin/apachectl start

zabbix server
/export/home/zabbix/sbin/zabbix_server

zabbix agent
/export/home/zabbix/sbin/zabbix_agentd

And everything starts without error......... (no really it does - i've spared you some of the pain)


OK - thats all the back end stuff done - now to go through the web gui to set that lot up - this is just a 'wizard' so is pretty painless once everything is set up OK

So navigating to the web gui is just http://zabbix-server:3333/zabbix/index.php - skipping through the welcome screen - the main screen to get past is this one (here is one of my earlier attempts before everything was set up) - you can't move on until everything is green

Next up is db connection

Now server details - just specify hostname again here

Then we get a summary

And away we go

Now we can navigate to the login screen

And there we go it's working - we just need to update the agent config for the zabbix server host to be the hostname and not localhost and then tick the enable box - see the 3 fields highlighted below

And there we have a fully working system just monitoring a single host - and we can see some graphs and things already - like this one

So now its a case of add more agents and configure monitoring as appropriate.

Note again - no root rights are needed to run or configure this (other than the user creation at the start)

Lets see how the battle of the monitoring tools is won (or lost....)