Squirro offers a number of ways to debug what is happening in the system. Each service maintains its log file, which helps with finding the source of an error. Additionally pre-installed daemon Monit helps by automatically restarting services that fail completely. A few basic operating system commands are also explained here, which can be used to find out if the base system is running smoothly.
...
| Log File | Service | OS | Additional Notes |
|---|---|---|---|
/var/log/squirro/squirro-*.log | Squirro services | All | Detailed log file about each service. Symlink to /var/log/squirro/SERVICE_NAME/SERVICE_NAME.log. |
/var/log/squirro/nginx-*-access.log | Nginx (Squirro services) | All | Every request to the web services is recorded in this log file in a line-by-line format. Symlink to /var/log/squirro/SERVICE_NAME/nginx-access.log. |
/var/log/squirro/nginx-*-error.log | Nginx (Squirro services) | All | Records errors on the HTTP level. When a service is stopped, errors may show up here indicating that the service is not reachable. Symlink to /var/log/squirro/SERVICE_NAME/nginx-error.log. |
/var/log/messages | - | All | General system log. Serious system failures will be recorded here. |
/var/log/elasticsearch/squirro.log | Elasticsearch | All | Records cluster information and major failures. |
/var/log/redis/*.log | Redis | All | |
/var/log/mysqld.log | MySQL | RHEL/CentOS 6 | |
/var/log/mariadb/mariadb.log | MariaDB | RHEL/CentOS 7 | |
/var/log/cron | Cron | ||
/var/log/monit/monit.log | Monit | RHEL/CentOS 6 | Records service state changes and actions. |
| /var/log/secure /var/log/audit/audit.log | System | RHEL/CentOS 6/7 | Used to debug connection issues |
Monit
With GNU/Linux Centos 6, Squirro comes pre-installed with Monit, a service that is used to automatically restart services that have failed.
Monit is configured by configuration files in /etc/monit.d. By default, all the main Squirro services are monitored.
...
Monit has a lot of options that can be added by configuration. For example, there is a web interface that can be used to see the current status of the machine. For security reasons, that interface is restricted to local access only by default. Consult the Monit manual, specifically the section MONIT HTTPD for information on how to enable this web interface for wider access. When doing that, beware that the local Firewall also needs to be opened for the corresponding port.
To retrieve the status fo the Squirro services via the Monit command line interface, the following command can be used:
| Code Block | ||
|---|---|---|
| ||
monit summary |
System commands
The Squirro services are standard Unix daemons. Standard Linux utilities can be used to debug any issues that may arise.
...
The grep command searches files for occurrences of a specific text. For example, if Squirro is reporting errors, but you are unsure where they might be coming from, the following command helps pinning down the responsible service:
...
This will output a list of all Squirro log files that contain the text "ERROR" together with the lines that contain this text.
Squirro Logs
All the Squirro logs are stored in /var/log/squirro. Each service has its own directory that can be queried as follow:
| Code Block | ||
|---|---|---|
| ||
tail -f /var/log/squirro/SERVICE/*.log |
For instance, if we are interested in the topic service:
| Code Block | ||
|---|---|---|
| ||
tail -f /var/log/squirro/topic/*.log |
In case we want to check all the services log:
| Code Block |
|---|
tail -f /var/log/squirro/*/*.log |
The ingester service
Due to its complexity, the ingester service has a different log structure. To do its job the service manages a set of processes (named processor_X where X in (1,2 ... N)). Each process maintains its unique log in its unique directory. The easiest way to debug consists in merging their content via the following command:
| Code Block |
|---|
tail -f /var/log/squirro/ingester/processor_*/*.log |