Page Comparison

This section covers adding cluster nodes to a Squirro installation. Refer to the Setup on Linux documentation on the base installation.

Table of Contents

...

1. Install the cluster node package, as described in the section Cluster Node Installation of Setup on Linux.

2. Install the additional required package for cluster coordination (Zookeeper library): yum install squirro-python-squirro.lib.zookeeper

3. Ensure that each of the cluster node can talk to the Elasticsearch cluster (Squirro Storage Node). Change the nginx config at /etc/nginx/conf.d/upstream-elastic.inc to 

   Code Block
   language bash
   upstream elastic { server <storagenode1 ip>:9200; server <storagenode2 ip>:9200; server <storagenode3 ip>:9200; }

4. Whitelist all the Squirro Cluster nodes in the following nginx ipfilter files:

   • /etc/nginx/conf.d/ipfilter-cluster-nodes.inc

   • /etc/nginx/conf.d/ipfilter-api-clients.inc

   • /etc/nginx/conf.d/ipfilter-monitoring.inc.

     In each of these files include each IP address as follows:

     Code Block
     language bash
     allow <clusternode1 ip>; allow <clusternode2 ip>; allow <clusternode2 ip>;

     Alternatively the allow also allows the specification of network addresses, e.g. 10.1.4.0/24 to whitelist an entire network.

5. Reload nginx at each of the cluster nodes.

   Code Block
   language bash
   $ systemctl reload nginx

6. Create the MariaDB / MySQL control users required by the cluster service. Because these users have elevated permissions, they are not created automatically by the Squirro installer. To create these users, invoke the mysql command as root user, then enter the following two commands:

   Code Block
   language sql
   grant select, reload, super, lock tables, replication slave, create, drop, alter, insert, process on *.* to "cluster"@"localhost" identified by "CLUSTER_CLUSTER_PASSWORD"; grant replication slave, replication client on *.* to "repl"@"%" identified by "CLUSTER_REPL_PASSWORD";

   Generate secure passwords for both users (needs to be the same on all cluster nodes, though), they will be added to the configuration file later.

7. Stop all the services by executing the following command:

   Code Block
   language bash
   cd /usr/lib/systemd/system for service in $(ls sq*d.service); do echo "Stopping $service"; systemctl stop $service; done for service in redis-server redis-server-cache mariadb zookeeper; do echo "Stopping $service"; systemctl stop $service; done

8. Squirro cluster service

   1. Edit the /etc/squirro/cluster.ini configuration file.

      1. Ensure that the mgmt_iface setting under cluster section specifies a valid network interface on which all the cluster nodes can communicate with each other (you need to look it up using network settings of your cluster node, i.e. with ifconfig, and set appropriately:

         Code Block
         [cluster] # If the appropriate network interface is ens142 mgmt_iface = ens142

      2. Also inside the [cluster] section, edit the following settings:

         1. id: change this to the same value as on the previous cluster nodes - ensuring it's the same value for all cluster nodes.

         2. redis_controller: set this to true so that Redis replication is managed by the Squirro cluster service.

         3. mysql_controller: set this to true so that MySQL replication is managed by the Squirro cluster service.

      3. Add the database passwords to the [mysql] section (changing the passwords to the generated values):

         Code Block
         [mysql] db = mysql+pymysql://cluster:CLUSTER_CLUSTER_PASSWORD@127.0.0.1:3306 repl_password = CLUSTER_REPL_PASSWORD

      4. Add the list of all the zookeeper nodes (including this new node) to the hosts list in the [zookeeper] section:

         /etc/squirro/cluster.ini

         Code Block
         language text
         [zookeeper] hosts = <clusternode1 ip>:2181,<clusternode2 ip>:2181,<clusternode3 ip>:2181

   2. Turn on endpoint discovery for all Redis and database connections. This ensures that the services consult the cluster service to know which cluster node is currently the master.
      
      Changing this requires all the endpoint_discovery (for Redis) db_endpoint_discovery (for MySQL) configuration entries and in every /etc/squirro/*.ini file to be set to true. This can be automated with the following sed commands:

      Code Block
      sed -i -e 's/^endpoint_discovery = false/endpoint_discovery = true/' /etc/squirro/*.ini sed -i -e 's/^db_endpoint_discovery = false/db_endpoint_discovery = true/' /etc/squirro/*.ini

9. MySQL

   1. Enable MySQL replication. This requires two changes in /etc/mysql/conf.d/replication.cnf - both of these values are commented out by default:

      1. server_id: this integer value needs to be a unique value over the whole cluster. For example use 10 for the first server in the cluster, 11 for the second, etc.

      2. report_host: set this to the human-readable name of the server, as it should be reported to the other hosts - for example node01.

   2. Raise the MySQL limits on open files and maximum connections.

      /etc/mysql/conf.d/maxconnections.cnf

      Code Block
      language text
      [mysqld] open_files_limit = 8192 max_connections = 500

      The max_connections setting should be set higher depending on number of cluster nodes. We recommend at least 150 connections for each cluster node.

10. Redis

    1. Extend the list of listening addresses for redis-server and redis-server-cache services, by editing /etc/redis/redis.conf and /etc/redis/cache.conf and listing all the cluster nodes (including this new server):

       /etc/redis/redis.conf and /etc/redis/cache.conf

       Code Block
       language text
       bind 1270.0.0.0.1 <clusternode1 ip> <clusternode2 ip> <clusternode3 ip>

       
       Note: this will have Redis listen on all network interfaces and IP addresses configured on the server.
       

11. Zookeeper

    1. Set the unique Zookeeper node identifier. This ID needs to start at 1, and then for each node incremented by 1. Write this identifier to /var/lib/zookeeper/data/myid.

    2. Add a list of all cluster nodes to Zookeeper. Edit /etc/zookeeper/zoo.cfg and list all the cluster nodes (including this new server):

       /etc/zookeeper/zoo.cfg

       Code Block
       language text
       server.1=<clusternode1 ip>:2888:3888 server.2=<clusternode2 ip>:2888:3888 server.3=<clusternode3 ip>:2888:3888

    3. Start services necessary for a leader election. Do NOT start the cluster service yet to avoid promoting the new node to master. We will only start the cluster service on this node after making sure that a leader has been elected from one of the existing nodes by following the next step.

       Code Block
       systemctl start zookeeper systemctl start redis-server systemctl start redis-server-cache systemctl start mariadb

    4. At this point follow the Squirro Cluster Expansion#Process on all the other cluster nodes section.

12. Starting

    1. Start the cluster node:

       Code Block
       systemctl start sqclusterd

    2. Wait for the cluster node to come up. Make sure the election leader is the same one as on the previous nodes.

       Code Block
       curl -s http://127.0.0.1/service/cluster/v0/leader/cluster.json | python -mjson.tool | grep electionLeader

       This command may have be repeated a few times until a result is returned.

    3. Start all other services:

       Centos 7

       Code Block
       cd /usr/lib/systemd/system for service in $(ls sq*d.service); do echo "Starting $service"; systemctl start $service; done

...