In some cases, e.g. when deploying Squirro to Amazon AWS or MS Azure, you may elect to rely on a remote MySql Server installation (e.g. RDS in case of AWS) and/or remote Redis Servers (e.g. RedisLabs). This page describes how you would configure such remote servers for use by Squirro and how to configure Squirro Cluster Nodes accordingly.
Table of Contents
Preparation
Before you install Squirro RPMs, ensure that you put in place the file /etc/squirro/backends.ini with the following content:
To set up Squirro with remote MySql server and Redis server "Backends", please create a readable file /etc/squirro/backends.ini :
mkdir -p /etc/squirro touch /etc/squirro/backends.ini
And add the following content depending on whether you want to run MySql server or Redis servers remotely or both. The example below show the content for both MySql and Redis servers to be configured remotely:
# Rely on a remote MySql Server installation is_mysql_server_remote = true # Rely on remote Redis Servers is_redis_server_remote = true
MySql Server Setup
On the MySql Server front there are server-level configurations, database-level configurations, and Squirro Cluster Node-level configurationsServer-level configuration
MySql Server-level Configuration
Configure MySql on User level scope, make sure your ~/.my.cnf file content looks like this:
[client] host="remote mysql server" user="user for the remote mysql server" password="password for the remote mysql server" ssl_ca="location of your SSL certificate"
If there is a requirement, follow the instruction here to secure MySql connection with SSL.
Following configuration are required by Squirro on MySql server level.
[mysqld] character_set_server=utf8 collation_server=utf8_unicode_ci innodb_flush_log_at_trx_commit=1
Make sure you configure the MySql client on all Squirro Cluster Nodes to point to the server installation. Also note the name of the MySql administrator account - often called root.
MySql Database and User Creation
Run the following on a Squirro Cluster Node on which /etc/squirro/backends.ini has been set up and squirro-cluster-node has been installed per Setup on Linux:
MYSQL_ADMIN=root # or the name of the MySql Administrator account able to create users and databases
DATABASE_PASSWORD="password of your choosing"
squirro_mysql_databases=`cd /etc/squirro/; grep "^db *=" *.ini | sed -e "s/\..*$//"`
for mysql_db in $squirro_mysql_databases; do
. /opt/squirro/setup/ensure_mysql_user_and_database_exist.sh $MYSQL_ADMIN $mysql_db $DATABASE_PASSWORD
done
Squirro Cluster Node Configuration
Before starting Squirro services, the configuration files under /etc/squirro/ need to be made to point to the remote installation:
MYSQL_SERVER_ADDRESS="address or name of MySql installation" sed -e "s|squirro/4u|$DATABASE_PASSWORD|" -e "s|localhost|$MYSQL_SERVER_ADDRESS|" -i /etc/squirro/*.ini
Note that the script above depends on the same DATABASE_PASSWORD variable from the "MySql Database and User Creation" section above.
Redis Server Setup
Redis Server-level Configuration
Squirro writes into about a dozen Redis databases. Ideally your Redis Server setup would have multiple Redis databases although somewhat experimentally it should be possible to use a single Redis Server. We do however recommend at least having two Redis-Servers one for key/value storage and other for caching. If required, follow the instruction to Secure Redis instance over SSL.
Squirro Cluster Node Configuration
You would point the Squirro services to the two Redis Server installation like so:
REDIS_STORAGE_HOST="Redis Server Storage host"
REDIS_STORAGE_PORT="Redis Server Storage port number"
REDIS_STORAGE_PASSWORD="Redis Server Storage password"
REDIS_CACHE_HOST="Redis Server Cache host"
REDIS_CACHE_PORT="Redis Server Cache port number"
REDIS_CACHE_PASSWORD="Redis Server Cache password"
redis_storage_dbs=`grep "^\[redis.*]" /etc/squirro/*.ini | grep -v "_cache.*]" | grep -v "redis_key_value" | sed -e "s/^.*\[//" -e "s/] *$//" | sort -u`
redis_cache_dbs=`grep "^\[redis.*_cache.*]" /etc/squirro/*.ini | grep -v "redis_key_value" | sed -e "s/^.*\[//" -e "s/] *$//" | sort -u`
redis_key_value_dbs=`grep "^\[redis_key_value_store.*]" /etc/squirro/*.ini | sed -e "s/^.*\[//" -e "s/] *$//" | sort -u`
redis_key_value_cache_dbs=`grep "^\[redis_key_value_cache.*]" /etc/squirro/*.ini | sed -e "s/^.*\[//" -e "s/] *$//" | sort -u`
# remove old redis passwords:
sed -e "/^password *=/d" -i /etc/squirro/*.ini
sed -e "/^redis_password *=/d" -i /etc/squirro/*.ini
for storage_db in $redis_storage_dbs; do
sed "s|^\(\[${storage_db}] *\)$|\1\nhost = $REDIS_STORAGE_HOST\nport = $REDIS_STORAGE_PORT\npassword = $REDIS_STORAGE_PASSWORD|" -i /etc/squirro/*.ini
done
for cache_db in $redis_cache_dbs; do
sed "s|^\(\[${cache_db}] *\)$|\1\nhost = $REDIS_CACHE_HOST\nport = $REDIS_CACHE_PORT\npassword = $REDIS_CACHE_PASSWORD|" -i /etc/squirro/*.ini
done
for storage_db in $redis_key_value_dbs; do
sed "s|^\(\[${storage_db}] *\)$|\1\nredis_host = $REDIS_STORAGE_HOST\nredis_port = $REDIS_STORAGE_PORT\nredis_password = $REDIS_STORAGE_PASSWORD|" -i /etc/squirro/*.ini
done
for cache_db in $redis_key_value_cache_dbs; do
sed "s|^\(\[${cache_db}] *\)$|\1\nredis_host = $REDIS_CACHE_HOST\nredis_port = $REDIS_CACHE_PORT\nredis_password = $REDIS_CACHE_PASSWORD|" -i /etc/squirro/*.ini
done
if grep -Fxq "[queues_local]" /etc/squirro/common.ini
then
sed "s|^\(\[queues_local] *\)$|\1\nredis_host = $REDIS_STORAGE_HOST\nredis_port = $REDIS_STORAGE_PORT\nredis_password = $REDIS_STORAGE_PASSWORD|" -i /etc/squirro/common.ini
else
cat >>/etc/squirro/common.ini <<FOO
[queues_local]
redis_host = $REDIS_STORAGE_HOST
redis_port = $REDIS_STORAGE_PORT
redis_password = $REDIS_STORAGE_PASSWORD
FOO
fi
Return to the Linux Installation Steps
Congratulations. You are now ready to continue with the Squirro Cluster Node installation steps and starting (or restarting) Squirro services.