The opinions expressed on this blog are purely mine

Handling logs on Ubuntu

Logs can leave a big mess on Ubuntu (or on any OS for that matter) when left unattended. Let me show you how I tipically configure Ubuntu to avoid a possible disk fillup.

System log defaults

The below table shows my usual defaults when it comes to logrotate configuration.

File Type Default rotation frequency My suggestion Rollups to keep
apt/* text monthly weekly 1
atop/* binary 28 days daily 3
auth.log text weekly weekly 3
cloud-init.log, cloud-init-output.log text - can be deleted after 3 days -
dpkg.log text monthly weekly 1
faillog binary weekly weekly 3
kern.log text weekly 3 days 1
syslog.log text daily daily 3
wtmp binary monthly monthly 3

I generally set a 10M size limit on each logfile so when the size is reached, it rotates. Most of the time I am forwarding important (auth.log, faillog, etc...) logs to central log storage, hence I am quite brave with keeping only 1-3 rollups at once.

Be advised that rotating logs based on file size is not always the best idea!

Rotating atop logs

Rotating atop logs can get rather exotic. One way is to copy /usr/share/atop/atop.daily script to /etc/cron.daily/ and modify this line:

#Default rotate frequency is 28 days
( (sleep 3; find $LOGPATH -name 'atop_*' -mtime +28 -exec rm {} \;)& )

Another way is to write logrotate config that is ugly because of the atop log file format (atop_YYYYMMDD), but I prefer it much more.

# taken from
/var/log/atop/atop_20[0-9][0-9][0-9][0-9][0-9][0-9] {
  rotate 5
    /usr/bin/find /var/log/atop/ -maxdepth 1 -mount -name atop_20\[0-9\]\[0-9\]\[0-9\]\[0-9\]\[0-9\]\[0-9\]\* -mtime +40 -exec /bin/rm {} \;

Syslog & journal

Before anything else, create a /var/log/journal so that journal can persist logs between reboots.

On Ubuntu, both syslog & journal are enabled by default, and journal's ForwardToSyslog set to True. That can lead to plentiful "duplicated" logs, especially if numerous systemd services are running on the node with verbose logging properties. In case /var/log/syslog is flooded with systemd service logs, you can try a few things to keep syslog clean:

set ForwardToSyslog=no in /etc/systemd/journald.conf and reload systemd (this is NOT a great idea)
You can prevent a service to forward logs to stdout/stderr by adding StandardOutput=null under the [Service] section of the service's systemd config file. You should ONLY do this if the service logs to an other location anyways.

I don't have a silver bullet when it comes to journal. Most of the time I keep the default settings and it just works. Occasionally, in case I need to rotate journal logs by hand, these commands come in handy:

# put recent files from active to archive state
journalctl --rotate

# retain only the last 1 day
journalctl --vacuum-time=1d
# or
# retain only the last 100M worth of logs
journalctl --vacuum-size=100M

Docker logs

By default Docker does not perform ANY logrotation, so produced logs can eat up the disk space really fast. To come up with sane defaults, create the following file, then restart docker daemon:

  "log-driver": "json-file",
  "log-opts": {
      "max-size": "10m",
      "max-file": "5"
You can find container logs at /var/lib/docker/containers/*/*-json.log

Forwarding logs

I highly recommend you to forward the most important OS / service logs to a central storage for safe keeping. 9 out of 10 times I use the fluent family or Datadog agent (yes, this is a commerical product) for log forwarding because of their ease of use & the tons of plugins they operate with. An other good candidate is Filebeat. Of course using a "3rd-party" tool is not always an option, that is when rsyslog comes to the rescue. You can use the following template go get going:

$ModLoad imfile
$InputFileName "location of the logfiles"
$InputFileTag "custom tags"
$InputFileStateFile "stat-servicename"
$InputFileSeverity "severity"
$InputFileFacility local3
local3.* @@hostname:port

To activate a config:

Create a config file under /etc/rsyslog.d
Restart rsyslog

Log types to forward

Audit logs (auth.log, faillog, wtmp, tallylog)
system/kernel stuff (syslog.log, kern.log)
custom application / container logs

Extra measures

Reserved space

Sometimes, when the system disk gets full, opening a new ssh session becomes impossible. By utilizing that most fileystems reserve a small amount of space for the root account, the ssh access still can be maintained to the OS:

Enable the root account on Ubuntu
Configure ssh daemon to allow root login
Disable password login for root, use cert


I recommend storing the entire /var directory on a different disk or partition


Lastly, I suggest you to read UNIX and Linux System Administration Handbook which is the definitive source for learning Linux.