splitting and joining huge files

Today I wanted to sftp a 10Gb file over the wire. Knowing that the transfer would fail in the middle of it, I decided to split this file into small pieces beforehand. Below I think is the easiest way on osx.

// to split my huge file into 100mb files, having the prefix "bak_"
split -b 100m my_huge_file.ext bak_

// to re-construct my huge file
cat `ls bak_*` > my_huge_file2.ext

After running the split command, you will see some files named bak_aa, bak_ab, etc in your directory. These are the chucks of your original huge file. So just transfer these small bak_* files over the wire.

sftp doesn’t have mput… sigh…

Also the original file name is gone missing in the process… sigh… That’s unfortunate. What I decided to do is to write up a small readme file to store the above two commands, so later on I know what output filename to use when running the second command.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s