Managing KVM Console Logs for Nova

Update

It turns out that this DOES NOT work around bug 832507. Please do not use this thinking that it does.

There's a problem with console logs. That was a hard sentence to word. I wanted to say a number of versions of it but each made it sound like the problem was with either OpenStack, Nova, libvirt / qemu-kvm. I didn't particularly feel like pointing the finger as the solution appears to need to come from a number of directions...

So the problem is that it is entirely possible when running KVM hypervisors with OpenStack Nova to have the compute nodes disks fill up when instance(s) console logs get a little chatty.

It's not a desirable event and the source will catch you by surprise.

There's currently no way to manage these KVM console logs via either qemu-kvm or via OpenStack / Nova so I wrote manage_console_logs.sh (github) (bitbucket) to do this.

manage_console_logs.sh operates as follows:

• Creates a lock file using flock to ensure that the script is not already running.
• Checks the size of each console log
• If it's greater than the nominated size it's truncated using tail.

That's it. A pretty straight forward method for ensuring your compute node disks do not fill up.

You should schedule this via cron at you desired frequency and add a monitoring check to ensure it's doing it's job as expected.