Crash Recovery With Beaver v.31, SQLite3 and Logstash v.1.4.2

As anyone who has worked with log data can tell you, getting that data stored and sorted in a searchable fashion in anything approaching real-time is a set of tasks best left to automated processes for the good of both efficiency and sanity.  However, every so often, processes hang, software crashes and somebody has to go in and clean up the mess.  Our team here at Bytecode IO encountered this just recently on a system using a combination of Beaver v. 31, Sqlite3 and logstash v. 1.4.2.

Crash Recovery

In the event that Beaver crashes, Beaver’s Sincedb support using Sqlite3 allows Beaver to pick up events written while crashed. In most situations, Beaver could simply be restarted with nothing more than a quick systems check to verify that nothing was lost. Please see Sincedb support using Sqlite3 in the documentation.

Logrotate Runs While Beaver Is Not

If logrotate has run while Beaver has not been running, additional steps are required.  The following is an example for the Nginx access log.

1) Get the fid of the log file that was rotated (the last fid listed)

grep nginx /var/log/beaver.log
[2014-08-06 22:01:39,194] INFO    [ca01g868d] - watching logfile /var/log/nginx/access.log
[2014-08-06 22:03:48,612] INFO    [ca01g868d] - watching logfile /var/log/nginx/access.log
[2014-08-06 22:03:51,675] INFO    [ca01g868d] - file rotated /var/log/nginx/access.log
[2014-08-06 22:03:51,676] INFO    [ca01g868d] - un-watching logfile /var/log/nginx/access.log
[2014-08-06 22:03:51,676] INFO    [ca01g87c1] - watching logfile /var/log/nginx/access.log


/var/log# sqlite3 /etc/beaver/since.db
sqlite> select * from sincedb;
ca01g56ff|/var/log/nginx/access.log|3
ca01g47fbd|/var/www/shared/log/unicorn.log|1038
ca01g7569|/var/log/nginx/access.log|0
ca01g868d|/var/log/nginx/access.log|22
ca01g87c1|/var/log/nginx/access.log|12

2) Send all events after that line to the current access log

/var/log# cat /var/log/nginx/access.log.1 | awk 'NR>12' >> /var/log/nginx/access.log

3) Get the fid of the newly created log file.

/var/log# printf "%xg%x" $(stat /var/log/nginx/access.log --format="%d %i")
ca01g9305

4) Use sqlite3 to access sincedb and insert a new record for the newly created file

/etc/logrotate.d# sqlite3 /etc/beaver/since.db
SQLite version 3.8.2 2013-12-06 14:53:30
Enter ".help" for instructions
Enter SQL statements terminated with a ";"
sqlite> insert into sincedb (fid, filename, position) values ("ca01g9305", "/var/log/nginx/access.log", 1);
sqlite> select * from sincedb;
ca01g56ff|/var/log/nginx/access.log|3
ca01g47fbd|/var/www/shared/log/unicorn.log|1038
ca01g7569|/var/log/nginx/access.log|0
ca01g868d|/var/log/nginx/access.log|22
ca01g87c1|/var/log/nginx/access.log|12
ca01g9305|/var/log/nginx/access.log|1

5) Start Beaver

/etc/logrotate.d# start beaver
beaver start/running, process 21177

If there are multiple log files that were rotated, the position for each should be updated.  Upon starting, Beaver should pick up all events from the beginning of the new file created by logrotate.

Leave a Reply