Zabbix Server: Some issues and some tips

In addition to the implementation details, I wrote few days back, I would like the add up some precautions and resolution to some issues, which would sure make life with Zabbix little easier for newbie .

Keep the date time configuration correct: I really faced much trouble over this one. Before installation of any new server, make sure that you have chosen correct Timezone settings. For most of the version of Linux, you need to type setup on console and then go to timezone configuration and set as per yours (e.g. Asia/Kolkata) and then make the same entry in /etc/php.ini for variable date.timezone as well (e.g. data.timezone=Asia/Kolkata).

This is really an irritating bug at Zabbix side as client and server both the sides should use Server Timezone entry. But its not resolved till now and might be reason for trouble to many.

Problem I faced with it: I went really a lot of trouble with time settings with one of my zabbix servers. While all others were working fine, one was not giving any data on graphs. Although resolution worked in some other way, but I would like to give a few basics —

1. Verify all the settings like agents are referencing the right server or not

2. Check that whether ports are open or not.

The Server listens on 10051 and client listens on 10050. Now, How to Check open Ports in Linux? Although there are many ways to do the same, but simpler is using netstat

# netstat –ntulp

It will show all open ports and also that which process is using them. You must find 10051 port opened on Server.

3. Check the latest data under monitoring tab (My case) This provides you info about clients that when you last moment received data from them and what data. In my case, I found that my clients received data in future time that’s why there was nothing on the graph as it was checking data in past time.

I struggled a lot, but found only one solution that was to delete all the clients from Zabbix and then import the same configuration again. When you delete one host, then it delete all the logs related to it and that’s why when you re-import the configuration then it will start-a-fresh. Thats why I insist on making a backup of all configuration once, coz after then for recreating whole thing, you need just an import and then recreating screens. 🙂

Increase the upload size in php.ini: For using the above feature, its a necessary step as  many times the size of xml file might exceed the default 2mb. Find the maximum_upload_size and increase to be at least 8mb from 2mb (default).

Copying Configuration is better in place of creating new: For Linux Servers, copy all the  configurations/ items/ graphs etc from the Linux Based Zabbix Server itself, while for Windows, you need to configure one first and then copying all the entry to other in similar manner as was in Linux Servers.

Replace CPU Utilization Screen with Memory Utilization: Although it depends on your requirements, but for me in general, its better to replace CPU Utilization screen with a custom made Memory Utilization screen made of items Total Memory and Free Memory as CPU Load data is sufficient to monitor CPU Health in most of the cases.

Make a configuration backup once you are done with adding your servers: This is the most important thing as it helps you out in quickly restoring the things, if want to migrate the whole thing to new server in case of any problem in original server. You need to use Export feature of existing server to export a xml file (might be >2 mb) and then import it once you done with installing a fresh server or existing server. Then you will be needed to change the Server IP in all client’s zabbix agent configurations (/etc/zabbix/zabbix_agentd.conf in Linux and c:\program files\zabbix agent\zabbix_agentd.conf in windows) and restarted their zabbix services.

Create a slideshow for all the servers: This is a feature, which makes it look more cool. Create a slideshow of all the screens, you create and it will keep it changing in specified time duration (default is 5 secs), which is quite better, comfortable and effective than changing the screens yourself to take a look. You could make very good use of a projector with this feature 😉



Making Zabbix Monitoring Server from Scratch: The Dummies Manual

[tweetmeme source=”mrnitishkumar” only_single=”false”]

Update: Wrote this post long back.. just was meant to show that how easy it could be…many things needs to be Add to Google Buzzupdated in reference to new features and UI.. will update soon. Recently I received a book over Zabbix from Packt Publicing. Amazing book and probably the only available for the same. Worth buying….

“ How we system administrators work in case of any trouble? If something goes down or anything wrong would have happened and we get asked about how it happened? We just run into logs or try to guess what would have been happened, was it high CPU, low disk space or tremendous network traffic? Sometimes we get our answers, sometime not and we keep on saying ourselves that we can’t be everywhere to monitor each and everything all the time. But it’s the time to update yourself and get optimized. I went through a few of monitoring solutions and with their pros and cons, finally I got settled with the Open Source Solution developed by Zabbix SIA. ”


What is Zabbix? Before the configuration and installation part, I am starting with a little history.zabbix1_thumb Zabbix is a network management system application created by Alexei Vladishev in 1998 (public release in 2001) to monitor and track the status of various network services, servers, and other network hardware. Zabbix uses MySQL, PostgreSQL, SQLite or Oracle to store data. Its backend is written in C and the web frontend is written in PHP and javascript.

Zabbix offers several monitoring options. Simple checks can verify the availability and responsiveness of standard services such as SMTP or HTTP without installing any software on the monitored host. A Zabbix agent can also be installed on UNIX and Windows hosts to monitor statistics such as CPU load, network utilization, disk space, etc. As an alternative to installing an agent on hosts, Zabbix includes support for monitoring via SNMP, TCP and ICMP checks, IPMI and custom parameters. Zabbix supports a variety of real-time notification mechanisms, including XMPP.

Installation and Configuration to monitor Windows\ Linux Servers:

Although I tried and found it extremely efficient , but even then I didn’t went along with the wonderful easy install procedure made by Brendon Baumgartner, not due to some fault in that one, but due to inherent problems with I guess not stable yet latest version of zabbix. But I sure be waiting for BB to write other magic scripts in the same way to make things one Enter only.

Here comes the way, I followed to install zabbix in manual way over the servers.

Step by Step Implementations with details:

Base machine: As for most of Linux Development I choose, even here I am choosing CentOS as the base system to build the solution upon. One should go for at least CentOS 5.x as there are a few options not available with older CentOS 4.x. As per recommendation for CentOS, machine should have at least 512 MB RAM and decent processor, LAN card etc. and although not kind of requirement but the machine should has ample space in it to store logs for as many days.

As usually I do, installation was customized with 1GB swap, 200 MB of boot partition, all PHP/ MySQL related packages checked, all libraries checked, SendMail package checked. In next, as the installation will pull updates, packages from internet directly, so Ethernet configurations should be ready and internet should be working fine over the same machine. Also note that it would be around 27-28 mb download size in total while installation.

Pre-requisites for the installation:

First of all, we need to finish installation/ update of all required packages in reference with zabbix installation.

yum install httpd php mysql mysql-server mysql-devel php-gd php-mysql php-bcmath gcc net-snmp net-snmp-libs net-snmp-utils net-snmp-devel curl-devel

Once finished with this, one need to check whether the crucial components are configured to start with the system and are working well.

# service httpd start
# chkconfig –add httpd
# chkconfig httpd on
# service mysqld start
# chkconfig –add mysqld
# chkconfig mysqld on

Possible issues and resolutions first:

Need to check whether things are working fine.

# mysql –u root

This should leave one to MySQL prompt, if not then need to check. In next, need to check whether httpd is working fine (Apache is properly installed and configured). Type the IP Address of the target Zabbix Installation Server in browser from some other computer, if you are able to see Apache page, then its fine otherwise you will be needed to check things like SELINUX.

How to check selinux status?

Execute the following command:

# sestatus

If its showing it enabled, then could disable it by

# setenforce 0

To permanently disabled the same, we need to do some editing in configuration file, otherwise it will be back with the next boot

# vi /etc/selinux/config

The file must look like (after setting SELINUX=disabled)

# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
# enforcing – SELinux security policy is enforced.
# permissive – SELinux prints warnings instead of enforcing.
# disabled – SELinux is fully disabled.
# SELINUXTYPE= type of policy in use. Possible values are:
# targeted – Only targeted network daemons are protected.
# strict – Full SELinux protection.

Check if the browser started opening apache page after this, if yes, then we are done, otherwise should look down in /var/log/httpd/error.log for other possible reasons.

Installation and configuration steps:

Here I am assuming that you are using CentOS 5.x as base, otherwise the configurations might be altered.

# cd /usr/src
# tar zxf zabbix-1.4.5.tar.gz
# cd zabbix-1.4.5
# mysql -u root
# mysql> create database zabbix;
# mysql> quit
# cat create/schema/mysql.sql | mysql -u root zabbix
# cat create/data/data.sql | mysql -u root zabbix
# cat create/data/images_mysql.sql | mysql -u root zabbix
# ./configure –enable-server –enable-agent –with-mysql –with-net-snmp –with-libcurl
# make install
# mkdir /etc/zabbix
# cp misc/conf/* /etc/zabbix
# groupadd zabbix
# useradd -g zabbix zabbix
# chown zabbix:zabbix /usr/local/sbin/zabbix*
# cp misc/init.d/redhat/8.0/zabbix* /etc/init.d
# vi /etc/init.d/zabbix_agentd

change BASEDIR=/opt/zabbix to BASEDIR=/usr/local/sbin


# vi /etc/init.d/zabbix_server

Make the same changes as above.

# chkconfig –add zabbix_agentd
# chkconfig –add zabbix_server
# chkconfig –level 3 zabbix_server on
# chkconfig –level 3 zabbix_agentd on

Now, installing the web application….

# mkdir /var/www/html/zabbix
# cp -R frontends/php/* /var/www/html/zabbix
# chown apache:apache /var/www/html/zabbix/conf
# vi /etc/php.ini

Find resource limits section and change max_execution_time to 300

;;;;;;;;;;;;;;;;;;; ; Resource Limits ; ;;;;;;;;;;;;;;;;;;;

max_execution_time = 300 ; Maximum execution time of each script, in seconds

Also make changes for date.timezone entry, it should be commented, uncomment it and set it accordingly like in case of India ..


At the end, start zabbix services

# service zabbix_server start
# service zabbix_agentd start

Add zabbix ports to be listened to the configuration of services.

# vi /etc/services

And put the following two lines at the very end of the file.

zabbix_agent 10050/tcp # Zabbix ports
zabbix_trap 10051/tcp

Zabbix Server console installation is ready to and web installation is ready to go from here.

After finishing all the above steps, now go to url http://IP_ADDRESS/zabbix (Here IP_ADDRESS is what you have given to the server Ethernet port)

It should ask for installation steps, if you have followed my steps exactly, then it must go like next, next only. After finishing all the steps, you will get a login prompt, user name will be admin here and password is blank. After login, go to profile link and change the password to whatever suits to you. Logout and login again. Click monitoring and then go to screens in second row. You should get five screens on the same page for the zabbix server itself. One screen must be having map only. Check that rest all showing you graphs or not. If yes, then server side work is done and you are ready to move towards adding clients.

Zabbix Agent Installation over Linux Clients:

First part is to download the appropriate zabbix agent package from the official page. I assume, your clients (Servers to monitor) are updated and have kernel at least 2.6.x (If not then choose the suitable package).

# yum install net-snmp net-snmp-libs net-snmp-utils net-snmp-devel curl-devel (Optional)
# groupadd zabbix
# useradd -g zabbix zabbix
# cd /usr/src
# tar zxvf zabbix-1.8.tar.gz
# cd zabbix-1.8
# chmod 7555 *
# ./configure -enable-agent -with-mysql -with-net-snmp -with-libcurl
# make install
# chown zabbix:zabbix /usr/local/sbin/zabbix*
# mkdir /etc/zabbix
# cp misc/conf/* /etc/zabbix
# cp misc/init.d/redhat/8.0/* /etc/init.d
# chmod +x /etc/init.d/zabbix_*

Now, Change the binary location to /usr/local/sbin

# vi /etc/init.d/zabbix_agentd

In next, make the zabbix agent point to your Zabbix Server by putting Server’s IP Address in configuration file of agent

# vi /etc/zabbix/zabbix_agentd.conf

And as final step on client side, set up the services

# chkconfig –add zabbix_agentd
# chkconfig –level 3 zabbix_agentd on
# service zabbix_agentd start

Getting the graphs from data coming from agents:

You will see that Zabbix Server itself added in the WebUI. In place of creating a new monitoring scenario, first it’s easier to copy the same entries to the newly added Linux Servers.

Go to Hosts, create a new host with the name of your agent (whatever name you choose), select group as Linux Server, then put the right IP Address and set the tab below to it on IP Address and save.

After this, go to Items, from combo box at right top, select zabbix server, it will show up items already configured for zabbix server, you need to select all the items and press copy selected items to button at the bottom and then when it will show the list of all templates and hosts configured, choose the one, you just created.

From here, you have setup items to fetch data from agent, now how to use it. Go to graphs, from right top select zabbix server and copy all the graphs to newly created server. Graphs are ready from here.

Finally, make a screen to monitor. Make a screen of 2×2 and give it the name as per your choice. Once screen created, click edit. Then configure CPU Loads, CPU Utilization, Disk Space and Network Utilization graphs for the agent, you setup and you are ready to monitor your Linux Server.

Zabbix Agent Installation over Windows Clients:

I find Windows Installation easier in first part and little bit complicated in second part. First need to download the windows agent from here

Then install it and it will ask for server IP Address, give zabbix server’s IP Address. Now, go to WebUI of Zabbix Server, Create host/items/graphs/screens in similar ways and you are ready to go.

There are many features that are there in Linux Agent but nowhere in Windows one. One workaround is to use data fetched from Windows Performance Monitor (permon.exe).

For example: Network Utilization

Go to cmd, type typeperf –qx |find “Network” >c:\network.txt

Now, check the network.txt file created in C: drive, it would be containing the exact network resource/ variable name for bytes sent and bytes received.

Now, how to call it?

Go to C:\program files\zabbix agent\zabbix_agentd.conf

Add something like following lines at the end…

PerfCounter= Server1NetIn,"\Network Interface(Intel Pro Server Adapter)\Bytes Received/sec",60
PerfCounter= Server1NetOut,"\Network Interface(Intel Pro Server Adapter)\Bytes Sent/sec",60

Note that the value placed in inverted commas is exactly the same, which you must have obtained from that network.txt file. Restart the Zabbix Service from services.msc. Now, go to Zabbix Server WebUI, create an item in target Windows Server area with any name for Network and put Server1NetIn or Server1NetOut as key. Save item and you are ready to use it in your graphs 🙂

Add to FacebookAdd to DiggAdd to Del.icio.usAdd to StumbleuponAdd to RedditAdd to BlinklistAdd to TwitterAdd to TechnoratiAdd to Yahoo BuzzAdd to Newsvine

How to: (Linux router utilizing multiple uplink providers)

Utilizing Multiple DSL Gateways to get combined into Single Gateway

Multiple DSL connection often exist in corporate scenario due to higher gap in prices between DSL plan and Leased Line plans and we also find people struggling with connectivity issues due to downlink with any of the DSL connection, while other connections of them might be working due to failover shifting etc. After struggling with the same issue for a while, few internet resources helped us to devise a solution for the same. A Linux based Software router utilizing all the connections at the same time. Before the implementation details, let’s put a FAQ over here.

Que: Why we need a Linux based router, when already our DSLs are acting as hardware based router from our service provider?

Answer: There are multiple reasons for the same:

  1. Connection Reliability & Failover handling: You might be having multiple connections to internet, but still might be struggling due to downlink with any of the uplink as switching mechanism from one uplink is sure going to cumbersome in any environment. Automatic failover handling makes it extremely attractive solution for any demanding environment. Somewhere it scores higher over a dedicated leased due to its failover handling.
  2. Firewall/ Proxy: The same solution could provide us a mechanism to utilize our uplink connection as no more transparent to our inner network (as each request with pass through Linux router) henceforth could control the way traffic get utilized. It would be a central gateway for outgoing request.
  3. Better throughput & Load balancing: Theoretically and practically, if one has say three modems of strength 512 Kbps each, then this solution doesn’t provide the combined strength of 1.5 Mbps, but one could attain a better throughput for multiple connections like torrents and multiple applications utilizing and enjoy connections through each of the DSL at the same time.

Hardware/ Software requirements:

Minimum requirements might vary, but here I am providing the requirements, which I felt as somewhat standard.

  • Processor: P4 2.4 GHz or more.
  • RAM:
    512 MB or more.

There are additional benefit of engaging two or more Ethernet card over the same machine, say two; one handling the requests and the other one communicating with DSLs.

For security reasons, one may wish to isolate the DSLs from the inner network and in that case, you will need a separate Switch having only DSLs and this router connected to the same and no more connection to rest of the network. Rest of the network will be connected to other Ethernet card configured in preferably different subnet.

Step-By-Step Procedure :-

Here I am presenting the way, we did the things in most simplest and layman way, without going into far technical details. Sure there might be better or alternate or customized ways suiting your own environment. One may always tweak over any solution.

Here are the steps:

  1. Machine: Get a machine ready with Either of Latest Fedora, RHEL or latest of CentOS with standard installation covering all the important packages.
  2. IP Forwarding: The OS need to has IP forwarding enabled. Its disabled by default. For checking the same, you may type the following:

    cat /proc/sys/net/ipv4/ip_forward

    If output is 1, then nothing to do and if output is 0, then it needs to be ON.

    For permanently putting IP Forwarding as ON, you need to change the value of net.ipv4.ip_forward to 1 from 0 in the file

    /etc/sysctl.conf. The changes could take affect by either a reboot or by the command

    sysctl –p /etc/sysctl.conf

  3. Configure IPs for both the Ethernet cards. I am keeping mine in the same subnet. Keeping all the IPs in same subnet.

    Say my DSL IPs are [MY_DSL1_IP], [MY_DSL2_IP], [MY_DSL3_IP] and two IPs of this router are [Ethernet1_IP] and [Ethernet2_IP]. I am keeping my subnet as 172.16.x.x/16

  4. In next, you need to create multiple profiles of one Ethernet card (take eth1) to communicate with each of DSL (in our case three different profiles with three different IPs).

    Make copies of file ifcfg-eth1 as ifcfg-eth1:0, ifcfg-eth1:1 and ifcfg-eth1:2. [Location /etc/sysconfig/network-scripts]

    cp ifcfg-eth1 ifcfg-eth1:0
    cp ifcfg-eth1 ifcfg-eth1:1
    cp ifcfg-eth1 ifcfg-eth1:2

    Make changes in files as given below:

    Content of ifcfg-eth1

    # Accton Technology Corporation SMC2-1211TX



    Content of ifcfg-eth1:0

    # Accton Technology Corporation SMC2-1211TX



    Content of ifcfg-eth1:1

    # Accton Technology Corporation SMC2-1211TX


    Content of ifcfg-eth1:2

    # Accton Technology Corporation SMC2-1211TX



  5. Add the entries in your rt_tables file for your DSLs

    Just do vi /etc/iproute2/rt_tables and put enties, so that the file contents will look like something as given below:

1 dsl1
2 dsl2
3 dsl3

# reserved values
255    local
254    main
253    default
0    unspec
# local
#1    inr.ruhep

Now most important step, place the routes in the rc.local file. Just do vi /etc/rc.d/rc.local and put enties, so that the file contents will look like something as given below:



# This script will be executed *after* all the other init scripts.
# You can put your own initialization stuff in here if you don’t
# want to do the full Sys V style init stuff.
touch /var/lock/subsys/local
/sbin/ip route add dev eth1:0 src [Profile0_IP] table dsl1
/sbin/ip route add dev eth1:1 src [Profile1_IP] table dsl2
/sbin/ip route add dev eth1:2 src [Profile2_IP] table dsl3
/sbin/ip route add default via [MY_DSL1_IP] table dsl1
/sbin/ip route add default via [MY_DSL2_IP] table dsl2
/sbin/ip route add default via [MY_DSL3_IP] table dsl3
/sbin/ip rule add from [MY_DSL1_IP] table dsl1
/sbin/ip rule add from [MY_DSL2_IP] table dsl2
/sbin/ip rule add from [MY_DSL3_IP] table dsl3
/sbin/ip route add default scope global nexthop via [MY_DSL1_IP] dev eth1:0 weight 1 nexthop via [MY_DSL2_IP] dev eth1:1 weight 1 nexthop via [MY_DSL3_IP] dev eth1:2 weight 1
/usr/bin/nohup usr/local/bin/gwping >>/var/log/gwping.log 2>&1

As about the last line and gwping, then it will come in later step.

  1. The next part is to place a script, which will take care of failover shifting and logging. What it basically do is to ping some public website through each of DSL, determining that when any of DSL went down and switching routing rules accordingly.

    Here is our customized script saved as /usr/local/bin/gwping:

    GWCMD="/sbin/ip route replace default scope global"

    # Conventionally 0 indicates success in this script.

    # Time between checks in seconds


    #IP Address or domain name to ping. The script relies on the domain being

    #pingable and always available


    #Ping timeout in seconds


    # External interfaces




    #IP address of external interfaces. This is not the gateway address.




    #Gateway IP addresses. This is the first (hop) gateway, could be your router IP

    #address if it has been configured as the gateway




    # Relative weights of routes. Keep this to a low integer value.




    # Broadband providers name; use your own names here.




    #No of repeats of success or failure before changing status of connection



    # Do not change anything below this line

    # Last link status indicates the macro status of the link we determined. This is down initially to force routing change upfront. Don’t change these values.




    # Last ping status. Don’t change these values.




    # Current ping status. Don’t change these values.




    # Change link status indicates that the link needs to be changed. Don’t change these values.




    # Count of repeated up status or down status. Don’t change these values.




    while : ; do

    /bin/ping -W $TIMEOUT -I $IP1 -c 1 $TESTIP > /dev/null 2>&1


    if [ $RETVAL -ne 0 ]; then

    echo $NAME1 Down





    if [ $LPS1 -ne $CPS1 ]; then

    echo Ping status changed for $NAME1 from $LPS1 to $CPS1



    if [ $LPS1 -ne $LLS1 ]; then

    COUNT1=`expr $COUNT1 + 1`



    if [[ $COUNT1 -ge $SUCCESSREPEATCOUNT || ($LLS1 -eq 0 && $COUNT1 -ge $FAILUREREPEATCOUNT) ]]; then

    echo Uptime status will be changed for $NAME1 from $LLS1



    if [ $LLS1 -eq 1 ]; then









    /bin/ping -W $TIMEOUT -I $IP2 -c 1 $TESTIP > /dev/null 2>&1


    if [ $RETVAL -ne 0 ]; then

    echo $NAME2 Down





    if [ $LPS2 -ne $CPS2 ]; then

    echo Ping status changed for $NAME2 from $LPS2 to $CPS2



    if [ $LPS2 -ne $LLS2 ]; then

    COUNT2=`expr $COUNT2 + 1`



    if [[ $COUNT2 -ge $SUCCESSREPEATCOUNT || ($LLS2 -eq 0 && $COUNT2 -ge $FAILUREREPEATCOUNT) ]]; then

    echo Uptime status will be changed for $NAME2 from $LLS2



    if [ $LLS2 -eq 1 ]; then









    /bin/ping -W $TIMEOUT -I $IP3 -c 1 $TESTIP > /dev/null 2>&1


    if [ $RETVAL -ne 0 ]; then

    echo $NAME3 Down





    if [ $LPS3 -ne $CPS3 ]; then

    echo Ping status changed for $NAME3 from $LPS3 to $CPS3



    if [ $LPS3 -ne $LLS3 ]; then

    COUNT3=`expr $COUNT3 + 1`



    if [[ $COUNT3 -ge $SUCCESSREPEATCOUNT || ($LLS3 -eq 0 && $COUNT3 -ge $FAILUREREPEATCOUNT) ]]; then

    echo Uptime status will be changed for $NAME3 from $LLS3



    if [ $LLS3 -eq 1 ]; then









    # Changeover Logic here

    if [[ $CLS1 -eq 0 || $CLS2 -eq 0 || $CLS3 -eq 0 ]]; then

    if [ $LLS1 -eq 0 ]; then

    echo Adding $NAME1

    GWCMD="$GWCMD nexthop via $GW1 dev $EXTIF1 weight 1"


    if [ $LLS2 -eq 0 ]; then

    echo Adding $NAME2

    GWCMD="$GWCMD nexthop via $GW2 dev $EXTIF2 weight 1"


    if [ $LLS3 -eq 0 ]; then

    echo Adding $NAME3

    GWCMD="$GWCMD nexthop via $GW3 dev $EXTIF3 weight 1"


    echo "Change execute now \n $GWCMD"


    GWCMD="/sbin/ip route replace default scope global"


    sleep $SLEEPTIME


  2. The last place is to make DNS entries. Just do a vi /etc/resolv.conf and type the following:


    I am using OpenDNS IP as DNS here. One may wish to have any other reliable one.

  3. Oops!! forgot one more thing, do a chmod 777 /var/log/gwping.log

    Now, you just need to reboot the same machine and then use [Ethernet1_IP] as Gateway in your production machines/ Proxy/ Firewall and here you go.

Waiting for your comments. Soon will post the HowTo, if one wishes to get the same thing done without putting one additional ethernet card in action means with one card only.

PDF Version