This past week we've been trying to enhance our monitoring of java application servers (in our case Tomcat) using zabbix. This actually proved to be a lot more fiddly than we had hoped - and actually this is largely due to two reasons:
1) The documentation is not clear on the architecture
2) The error messages are somewhat misleading
Let me start off by talking about the first point - from the docs I have read and what the screens kind of lead you to believe the zabbix server is communicating directly with the jmx interface on the monitored server to pull out the java information - this is not true. If you look at the host screen for example you will see something like this:
This really seems to imply that it's talking directly to the jmx component (in the same way it talks to the standard agent directly)
However this is not the case - the architecture actually looks like the below setup - it would be really useful if the actually just had a picture like this to better explain it.....
The key bit of understanding is that you need an additional bit of software to make this work - the 'java gateway' - this acts like a proxy server of sorts and all requested go through this to get to the actual endpoint server we are actually interested in monitoring.
So when i fill in a config that says the jmx interface is on 10.10.10.10 on port xxxxx - the zabbix server is not connecting directly to this - it actually sends those connection details to the java gateway and this then connects to that ip on that port number. This is the bit that speaks java (though to be honest i'm not sure why this is just not directly integrated into the product.....)
In my environment (i guess like most setups) i installed the java gateway on the same server as zabbix itself (though you dont need to) - so the cloudy shaped bit in the picture above is actually a single server for me.
The port thing is also confusing - 10052 is the default java gateway port - not the jmx port - so don't confuse those things up. In the zabbix jmx config for each tomcat config you actually need to specify the normal jmx port (12345) - unless you want to change off the defaults. The 10052 port is just mentioned in the zabbix internal config and nowhere else.
I ended up moving all of the jmx server ports to 10052 as i was confused about what ports this was really working on - but that is not necessary - and actually makes things more confusing.....
In terms of the server config that had to be configured to make it aware of the java gateway and where it was - all that is needed are these two lines of config
JavaGateway=10.x.x.x
JavaGatewayPort=10052
The java gateway itself has a config file too but i didn't need to amend anything in that.
The java gateway also needs to be started to enable it to run as this isn't automatic - you'll need to somehow add that in to machine startup routines to make sure it is there when you need it.
So once you've got the architecture clear and have configured things it will just work right?
Well in my case not (but this will depend on how you have your infra setup). It seems like the only comms that should be going on is zabbix server to java gateway on port 10052 and then java gateway to server running tomcat on whatever the configured jmx port is (default 12345) - however if you just open those ports things won't work. I fact you get the message ' no route to host' which is incredibly misleading - in fact it seems that jmx opens random ports all over the place that cant be tied down. If you switch off the firewall on the tomcat server then everything works fine....
This isn't great security wise but i guess you can tie it down to only be from the zabbix server which helps a little......
Actually just for reference here are the lines from my tomcat startup file showing the jmx part just so you have the full picture.
This is the content of my setenv.sh file
export CATALINA_OPTS="$CATALINA_OPTS -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=10052 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Djava.rmi.server.hostname=10.x.x.x -Djava.net.preferIPv4Stack=true"
1) jmsremote.port - i have 10052 but that is confusing - this can be left as default (12345)
2) jmsremote.ssl and jmsremote.authenticate should be set as above (unless you know what you are doing and can configure this :-))
3) server.hostname should be the primary ip of the tomcat server (nothing to do with zabbix here)
4) The ipv4 thing shouldn't be necessary but i had it set while trying to debug.
OK so now all of that is working we can now deploy the zabbix template for tomcat and everything is great right?
Well wrong - it seems that the current template is designed for old versions of tomcat and simply doesn't work properly. However the generic jmx one does work and we can now implement that to retrieve some metrics
Here is what the generic one looks like:
And here is the broken tomcat specific one
So anyway we have some monitoring now for jmx - it's not perfect but it's give us what we want for now.
I think the zabbix docs need to be clarified and someone needs to write a template for newer versions of tomcat - i guess there is just not the number of people asking for it that means its been created yet?
Hopefully this post can help other that may be struggling to get this working.....
Comments
Post a Comment