how to move tons of data using rsync
So: I bought a new 500 GB NAS from NewEgg to consolidate the 160 GB NAS + 250 GB USB2 drive Frankensteinian setup I’ve been using as a Media Dump — I love the linux-based SimpleShare as a data source for the XBox Media Center, but plugged in external USB drives tend not to want to spin down, generating heat and wasting electricity, so I got a large one to consolidate the big mess into something more manageable, quieter, and hopefully less of an electricity hog.
But here’s the problem — how does one move 400 GB of audio and video around from SMB share to SMB share, anyhow? First I (naively) tried Windows’ “copy” method — this actually got through quite a bit through before dying at a random point, leaving me no clue what was copied and what was left. Then, I tried Mac OS X, which also died midway through the transfer at a seemingly random and unknowable point. So that sucked.
So then I decided to pull out my Cruddy Ubuntu laptop(tm) and try rsync. After I installed the linux smb client:
sudo apt-get install smbfs smbclient
…and figured out how to map my SMB shares to local folders:
mkdir new_media
mkdir old_media
sudo mount -t smbfs //10.0.1.5/Media /home/jeff/old_media
sudo mount -t smbfs //10.0.1.10/Media /home/jeff/new_media
…all it took was one single command (and about 36 hours!) to move about 400 GB worth of data:
sudo rsync --progress -r /home/jeff/old_media/ /home/jeff/new_media/
I’m sure there’s a better way to do this (and if so, let me know!), but I was pretty psyched at how effectively it worked once everything was in place; rsync even moves things in alphabetical order, versus Mac and Windows, which both appear to copy files using some inscrutable, spindly method based on reasons long forgotten.
~Jeff
Pingback: University Update - Linux - how to move tons of data using rsync
Pingback: Rsync To Smb Share - Dogpile Web Search