I just noticed some strangeness in a NFS mounted device on my Ubuntu machine.
I have duplicity running on my machines to back up to a few Synology Diskstation RAID devices. Recently I noticed that Duplicity was complaining about an assertion error. I now realize that it was complaining about having two files with the same name on the drive.
I did some digging this morning and noticed that two files were sharing the same name / inode. I did a ls -i and got the inode of the files (they were the same):
root@lister:/mnt/backup_music/backup_files/music# find . -inum 9306120
./duplicity-full.20171011T090009Z.vol4554.difftar.gz
./duplicity-full.20171011T090009Z.vol4554.difftar.gz
Huh. What's going on here?
I immediately thought "well, I'll need to fsck the disk" but then I rmembered that you can't fsck an NFS mount. So I logged into the Synology Diskstation to see if it was showing two files. To my surprise the Synology didn't show anything was awry.
After a little poking and prodding online I wondered what would happen if I renamed the file:
mv duplicity-full.20171011T090009Z.vol4554.difftar.gz duplicity-full.20171011T090009Z.vol4554.difftar.gz.old
and then rename it back:
mv duplicity-full.20171011T090009Z.vol4554.difftar.gz.old duplicity-full.20171011T090009Z.vol4554.difftar.gz
That seemed to work. ls -i shows one file.
But here's the strange part: That find command? Still shows two files pointing at the same file.
I'm wondering if it'll survive a re-mount / reboot.
[Time passes]
Yep. Looks like there's still something in NFS that thinks there's still two files with the same inode, but only one file in ls.
I'm a little stumped on how to ultimately fix this (save for a complete "nuke it from orbit" reformatting. And that's no guarantee that it won't come back.
If you have some ideas on what's going on (I'm pretty sure it's a buggy NFS issue as this has shown up before) I'd love to know. Better yet: if you know how to fix this that wold be awesome as well.
Update: Found a work-around for this:
- Copy the file to another location (I used
rsync to copy it to /tmp, which removed it from the device completely)
- Run
rm on the file to remove it.
- Copy it back using rsync
- Unmount / remount the drive.
Seems things are back to normal. Will report back if something changes. Thanks to daddo in the #linuxjournal channel for the idea on how to work-around this.
Update: No, it didn't work: it created a new file / new inode and then exhibited the same issue with duplicate inodes for that file.
What the @#$%#%^#!
Update: Tried it again, but this time I touched a file, removed the file, unmounted the drive, killed nfs-kernel-server (which likely didn't do much) and then re-copied the file back over to the drive. That seems to have worked, but will see if it manifests itself in other weirdness.