Handling logs on Ubuntu

Logs can leave a big mess on Ubuntu (or on any OS for that matter) when left unattended. Let me show you how I tipically configure Ubuntu to avoid a possible disk fillup.

System log defaults

The below table shows my usual defaults when it comes to logrotate configuration.

File	Type	Default rotation frequency	My suggestion	Rollups to keep
apt/*	text	monthly	weekly	1
atop/*	binary	28 days	daily	3
auth.log	text	weekly	weekly	3
cloud-init.log, cloud-init-output.log	text	-	can be deleted after 3 days	-
dpkg.log	text	monthly	weekly	1
faillog	binary	weekly	weekly	3
kern.log	text	weekly	3 days	1
syslog.log	text	daily	daily	3
wtmp	binary	monthly	monthly	3

I generally set a 10M size limit on each logfile so when the size is reached, it rotates. Most of the time I am forwarding important (auth.log, faillog, etc...) logs to central log storage, hence I am quite brave with keeping only 1-3 rollups at once.

Be advised that rotating logs based on file size is not always the best idea!

Rotating atop logs

Rotating atop logs can get rather exotic. One way is to copy /usr/share/atop/atop.daily script to /etc/cron.daily/ and modify this line:

                
#Default rotate frequency is 28 days
( (sleep 3; find $LOGPATH -name 'atop_*' -mtime +28 -exec rm {} \;)& )
                
              

Another way is to write logrotate config that is ugly because of the atop log file format (atop_YYYYMMDD), but I prefer it much more.

              
# taken from https://unix.stackexchange.com/a/466709
/var/log/atop/atop_20[0-9][0-9][0-9][0-9][0-9][0-9] {
  missingok
  daily
  nodateext
  rotate 5
  ifempty
  nocreate
  postrotate
    /usr/bin/find /var/log/atop/ -maxdepth 1 -mount -name atop_20\[0-9\]\[0-9\]\[0-9\]\[0-9\]\[0-9\]\[0-9\]\* -mtime +40 -exec /bin/rm {} \;
  endscript
}
              
            

Syslog & journal

Before anything else, create a /var/log/journal so that journal can persist logs between reboots.

On Ubuntu, both syslog & journal are enabled by default, and journal's ForwardToSyslog set to True. That can lead to plentiful "duplicated" logs, especially if numerous systemd services are running on the node with verbose logging properties. In case /var/log/syslog is flooded with systemd service logs, you can try a few things to keep syslog clean:

I don't have a silver bullet when it comes to journal. Most of the time I keep the default settings and it just works. Occasionally, in case I need to rotate journal logs by hand, these commands come in handy:

              
# put recent files from active to archive state
journalctl --rotate

# retain only the last 1 day
journalctl --vacuum-time=1d
# or
# retain only the last 100M worth of logs
journalctl --vacuum-size=100M
              
            

Docker logs

By default Docker does not perform ANY logrotation, so produced logs can eat up the disk space really fast. To come up with sane defaults, create the following file, then restart docker daemon:

                  
#/etc/docker/daemon.json
{
  "log-driver": "json-file",
  "log-opts": {
      "max-size": "10m",
      "max-file": "5"
  }
}
                  
              

You can find container logs at /var/lib/docker/containers/*/*-json.log

Forwarding logs

I highly recommend you to forward the most important OS / service logs to a central storage for safe keeping. 9 out of 10 times I use the fluent family or Datadog agent (yes, this is a commerical product) for log forwarding because of their ease of use & the tons of plugins they operate with. An other good candidate is Filebeat. Of course using a "3rd-party" tool is not always an option, that is when rsyslog comes to the rescue. You can use the following template go get going:

                             
$ModLoad imfile
$InputFileName "location of the logfiles"
$InputFileTag "custom tags"
$InputFileStateFile "stat-servicename"
$InputFileSeverity "severity"
$InputFileFacility local3
$InputRunFileMonitor
local3.* @@hostname:port
                   
              

To activate a config:

Log types to forward

Extra measures

Reserved space

Sometimes, when the system disk gets full, opening a new ssh session becomes impossible. By utilizing that most fileystems reserve a small amount of space for the root account, the ssh access still can be maintained to the OS:

Partitioning

I recommend storing the entire /var directory on a different disk or partition

Learning

Lastly, I suggest you to read UNIX and Linux System Administration Handbook which is the definitive source for learning Linux.

---