When deleting a volume in OpenStack you may sometimes get an error message stating that Cinder was unable to delete the volume because the volume was busy:
2015-05-21 23:31:41.160 16911 ERROR cinder.volume.manager [req-6f77ef4d-bbff-4ff4-8a3e-4c6b264ac5ca \
04b7cb61dd3f4f2f8f80bbd9833addbd 5903e3bda1e840d492fe79fb840acacc - - -] Cannot delete volume \
f8867d43-bc82-404e-bcf5-6d345c32269e: volume is busy
There are a number of reasons why a volume may be reported by Ceph as busy, however the most common reason in my experience has been that a Cinder client connection has not yet been closed, possibly because a client crashed.
If you were to look at the volume in Cinder, that status is usually available, the record looks in order. When you check Ceph, you'll see that the volume still exists there too.
% cinder show f8867d43-bc82-404e-bcf5-6d345c32269e | grep status
| status | available |
# rbd -p my.ceph.cinder.pool ls | grep f8867d43-bc82-404e-bcf5-6d345c32269e
volume-f8867d43-bc82-404e-bcf5-6d345c32269e
Perhaps there's a lock on this volume. Let's check for locks and then remove them if we find one:
# rbd lock list my.ceph.cinder.pool/volume-f8867d43-bc82-404e-bcf5-6d345c32269e
If there are any locks on the volume, you can use lock remove using the id and locker from the previous command to delete the lock:
# rbd lock remove <image-name> <id> <locker>
What if there are no locks on the volume but you're still unable to delete it from either Cinder or Ceph? Let's check for snapshots:
# rbd -p my.ceph.cinder.pool snap ls volume-f8867d43-bc82-404e-bcf5-6d345c32269e
SNAPID NAME SIZE
2072 snapshot-33c4309a-d5f7-4ae1-946d-66ba4f5cdce3 25600 MB
When you attempt to delete that snapshot you will get the following:
# rbd snap rm my.ceph.cinder.pool/volume-f8867d43-bc82-404e-bcf5-6d345c32269e@snapshot-33c4309a-d5f7-4ae1-946d-66ba4f5cdce3
rbd: snapshot 'snapshot-33c4309a-d5f7-4ae1-946d-66ba4f5cdce3' is protected from removal.
2015-05-22 01:21:52.504966 7f864f71c880 -1 librbd: removing snapshot from header failed: (16) Device or resource busy
This reveals that it was the snapshot that was busy and locked all along.
Now we need to unprotect the snapshot:
# rbd snap unprotect my.ceph.cinder.pool/volume-f8867d43-bc82-404e-bcf5-6d345c32269e@snapshot-33c4309a-d5f7-4ae1-946d-66ba4f5cdce3
You should now be able to delete the volume and it's snapshot via Cinder.
Enjoy :-)