Tricks of the trade: Figuring out progress of a large upload
I found myself today needing to upload a file to S3, the upload size is a few hundred GBs in size. I expected the appropriate command, like so:
aws s3api put-object --bucket twitter-2020-rvn-dump --key mydb.backup --body ./mydb.backup
But then I realized that this is uploading a few hundred GB file to S3, which may take a while. The command doesn’t have any progress information, so I had no way to figure out where it is at.
I decided to see what I can poke around to find, first, I ran this command:
ps -aux | grep s3api
This gave me the PID of the upload process in question.
Then I checked the file descriptors for this process, like so:
$ ls -alh /proc/84957/fd
dr-x------ 2 ubuntu ubuntu 0 Mar 30 08:10 .
dr-xr-xr-x 9 ubuntu ubuntu 0 Mar 30 08:00 ..
lrwx------ 1 ubuntu ubuntu 64 Mar 30 08:10 0 -> /dev/pts/8
lrwx------ 1 ubuntu ubuntu 64 Mar 30 08:10 1 -> /dev/pts/8
lrwx------ 1 ubuntu ubuntu 64 Mar 30 08:10 2 -> /dev/pts/8
lr-x------ 1 ubuntu ubuntu 64 Mar 30 08:10 3 -> /backups/mydb.backup
As you can see, we can tell that file descriptor#3 is the one that we care about, then we can ask for more details:
$ cat /proc/84957/fdinfo/3
pos: 140551127040 flags: 02400000 mnt_id: 96 ino: 57409538
In other words, the process is currently at ~130GB of the file or there about.
It’s not ideal, but it does give me some idea about where we are at. It is a nice demonstration of the ability to poke into the insides of a running system to figure out what is going on.
Your consistency in blogging is admirable
Thank you, it's usually a lot of fun.
of course, to me, it is also a learning hobby: ehsanghanbari.com anyway, I've been reading your blog posts since 2008 and always enjoyed thank you
You may enjoy reading about the 'pv' (pipe viewer) unix command too, its designed to sit between a transfer of some kind and provide you some estimates!
That is a really useful tool, thanks. In this case, there aren't actually any pipes (handled internally by the aws cli), though
Join the conversation...