Data Center restart (or chef-solo) should unmount the NFS share before mounting it again if case of changes

Problem Definition

When changes are done to the NFS share IP or DNS in configuration.json. After running:

hipchat datacenter configure --file configuration.json
hipchat datacenter restart

The restart command fails with

subprocess.CalledProcessError: Command '['sudo', '/opt/atlassian/hipchat/sbin/cs']' returned non-zero exit status 1

Upon looking at /var/log/chef.log, you will see this error:

STDERR: mount.nfs4: /file_store/shared is busy or already mounted

This is occurring because /file_store/shared has to be manually unmounted first before it's mounted again by Chef.

Suggested Solution

When Chef tries to mount the NFS share and fails with error code 32:

root@hipchat1:/home/admin# mount -t nfs4 -o rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2 172.16.116.152:/share/nfs4/hipchat /file_store/shared
mount.nfs4: /file_store/shared is busy or already mounted
root@hipchat1:/home/admin# echo $?
32

Chef should unmount /file_store/shared and mount the NFS share again.

Workaround

Umount /file_store/shared manually as root then run the restart again:

sudo dont-blame-hipchat -c 'umount /file_store/shared'