Blog-Post

Making fabric query zabbix for host selection

Oct 11, 2013

For some of our devop tasks we use fabric. As this is not a full blown infrastructure management stack like chef or SaltStack it lacks some nice features for e.g. target host selection. As we also use zabbix to monitor our services, a logical approach to this problem was to use zabbix hosts and group configurations for host selection in fabric.

A simple call to fabric would look like this:

fab -H localhost,linuxbox task_name

What we wanted was more like this:

fab -H 'zabbix_hosts:regex' task_name
fab -H 'zabbix_groups:group1+group2+group3:regex' task_name

For this to work, we needed to intercept the default host selection of fabric. When calling a fabric task in a given fabfile, fabric automagically sets a list of hosts to connect to in fabric.api.env.hosts. This list is populated with the value passed in via the -H command line parameter. So this looked like a good candidate to inject our code.

For all of the following scripts we used this python zabbix api lib.

A generic approach to replace these hosts was to scan for a trigger string in the hostname and try to dynamically call the corresponding handler function to find the needed host names:

def replace_special_hosts():
"""Iterate over the hosts in fabric.api.env.hosts and replace special host names, e.g. zabbix_hosts:regex"""
for i, host in enumerate(fabric.api.env.hosts):
    # Complex pattern, e.g. zabbix_groups:Infrastructure:monitoring.* or zabbix_hosts:nginx*
    if host.find(":") > -1:
        function, remaining_parameters = host.split(":", 1)
        try:
            function_name = "replace_" + function
            hostList = globals()[function_name](remaining_parameters)
            del fabric.api.env.hosts[i]
            fabric.api.env.hosts += hostList
        except Exception, e:
            log(str(e), "ERROR")
            log("Could not find function to parse special host key replace_%s. Please check." % function, "ERROR")
            sys.exit(255)

The following functions will take care of selecting the correct hosts via a zabbix call:

"""TODO: Filter list by host status to make sure we will not try to deploy to an unreachable host"""
def replace_zabbix_groups(parameter_string):
    """ Replace host entries that refer to zabbix groups with the actual zabbix group members.
        The expected pattern in parameter_string is groupname1+groupname2+groupnameX:regexpattern.
        The optional regexpattern will be used to filter the host list after all groupmembers have been determined."""
    regex = ""
    if parameter_string.find(":") == -1:
        groups = parameter_string.split("+")
    else:
        groups, regex = parameter_string.split(":")
        groups = groups.split("+")
        regex = re.compile(regex)
    hostList = zabbixGetHostsByGroupNames(groups)
    if regex == "":
        return hostList
    hostList = [host for host in hostList if regex.search(host)]
    return hostList

"""TODO: Filter list by host status to make sure we will not try to deploy to an unreachable host"""
def replace_zabbix_hosts(parameter_string):
    """ Replace host entries that refer to zabbix hosts filtered by a regex pattern.
        The expected pattern in parameter_string is a regexpattern.
        Regexpattern will be used to filter the host list after all hosts have been determined."""
    regex = re.compile(parameter_string)
    getParameters = {'output': ['hostid', 'host']}
    hostList = [host['host'] for host in zabbixGetHosts(getParameters) if regex.search(host['host'])]
    return hostList 
    
#=============================================================================
#
# ZABBIX UTILITY FUNCTIONS
#
# =============================================================================
def _zabbixLogin():
    zapi = ZabbixAPI(server="http://" + config.config["zabbix"]['Host'], log_level=0)
    try:
        zapi.login(config.config["zabbix"]['User'], config.config["zabbix"]['Pass'])
        return zapi
    except ZabbixAPIException, e:
        log(str(e), "ERROR")
        return False
        
def zabbixGetHostsByGroupNames(zabbix_groups):
    """Get xen for a given group from zabbix server."""
    if type(zabbix_groups) is str:
        zabbix_groups = [zabbix_groups]
    zapi = _zabbixLogin()
    if not zapi:
        log("Could not connect to zabbix server %s. Please check." % config.config["zabbix"]['Host'], "ERROR")
        sys.exit(255)
    groupIds = zapi.hostgroup.get({'output': 'groupids', 'filter': {'name': zabbix_groups}})
    if len(groupIds) == 0:
        log("Could not find any zabbix groups with passed names.", "ERROR")
        sys.exit(255)
    getParameters = {'output': ['hostid', 'host'], 'groupids': [id['groupid'] for id in groupIds]}
    hosts = zabbixGetHosts(getParameters)
    hostList = [host['host'] for host in hosts]
    return hostList

def zabbixGetHosts(getParameters):
    """Get host by given filter."""
    zapi = _zabbixLogin()
    if not zapi:
        log("Could not connect to zabbix server %s. Please check." % config.config["zabbix"]['Host'], "ERROR")
        sys.exit(255)
    hosts = zapi.host.get(getParameters)
    return hosts

With this in place we are now able to execute commands like:

fab -H zabbix_groups:nginx_server exec_command:'/etc/init.d/nginx reload'