Note, a plain dd will report progress on stderr if sent a SIGUSR1. That can sometimes be enough to avoid the overhead of pv(1) and sending huge amounts of data through another pipe.
About the pv overhead, I sent a note to the pv author saying that on Linux at least the transfer function could take advantage of the splice system call. Splice "copies" data from one file descriptor to another file descriptor, avoiding having to copy the data from the kernel to userspace and back.
On FreeBSD (and maybe Mac OS X) you can press Control-T to send a SIGINFO signal (non-standard extension. This prints out how far dd has gotten (and perhaps other FreeBSD utilities too).
N.B. also that dd on osx will EXIT IMMEDIATELY if sent a SIGUSR1. It double-sucks because one tends to do this on long-running dd processes that you're wondering are done yet...
Right. I usually use pv in place of 'cat' at the start of a long set of piped commands. Anywhere you could use 'cat' you can use pv. If I'm not mistaken, I think you can also use multiple 'pv' commands at a time too.
> If I'm not mistaken, I think you can also use multiple 'pv' commands at a time too.
I use multiple pv instances to track compression/decompression rates, and overall progress, for tarballs. Using the "-c" code for console codes keeps the instances from stomping over each other's output.
If I'm dealing with [compressed] files on disk or across a network, what's the practical overhead of pv? A few extra percent CPU, or some material decrease in through put?
Nifty command. Been using this for a month or so. One use case is measuring if a downstream process can consume data as fast the previous process is producing it (by alternating second process with /dev/null or some such).
http://news.ycombinator.com/item?id=3198572
http://news.ycombinator.com/item?id=2567186
It's also great that pv supports rate limiting, so you can choose if you want to saturate your network when you use it with netcat.
Somewhat related, trickle is a good user-space bandwidth shaper if you aren't sending data over a pipe: http://monkey.org/~marius/pages/?page=trickle
It's amazingly useful to have realtime throughput rates
tail -f query.log | pv -tri 1 > /dev/null
Note, a plain dd will report progress on stderr if sent a SIGUSR1. That can sometimes be enough to avoid the overhead of pv(1) and sending huge amounts of data through another pipe.
About the pv overhead, I sent a note to the pv author saying that on Linux at least the transfer function could take advantage of the splice system call. Splice "copies" data from one file descriptor to another file descriptor, avoiding having to copy the data from the kernel to userspace and back.
On FreeBSD (and maybe Mac OS X) you can press Control-T to send a SIGINFO signal (non-standard extension. This prints out how far dd has gotten (and perhaps other FreeBSD utilities too).
You may need to setup your terminal to do that with:
fsck also responds to SIGINFO by printing its status. I had to hack up rsync to do it (http://jcs.org/patches/rsync-show_current_file.diff), and I wish more utilities supported it by default.
N.B. also that dd on osx will EXIT IMMEDIATELY if sent a SIGUSR1. It double-sucks because one tends to do this on long-running dd processes that you're wondering are done yet...
How does it know the progress / ETA?
I was under the assumption that pipes do not know how much total data is on its way?
If you give it a file, or specify the amount on the commandline it knows, otherwise it doesn't
Right. I usually use pv in place of 'cat' at the start of a long set of piped commands. Anywhere you could use 'cat' you can use pv. If I'm not mistaken, I think you can also use multiple 'pv' commands at a time too.
> If I'm not mistaken, I think you can also use multiple 'pv' commands at a time too.
I use multiple pv instances to track compression/decompression rates, and overall progress, for tarballs. Using the "-c" code for console codes keeps the instances from stomping over each other's output.
This is indeed visible in the example, only source has progress while bzcat and gzip only have flow.
You have to supply it the stream sizes if you want completion percentages, otherwise it displays rates only.
Example: https://gist.github.com/2783910
If I'm dealing with [compressed] files on disk or across a network, what's the practical overhead of pv? A few extra percent CPU, or some material decrease in through put?
very very little, it's just doing read(), counting the bytes, then write(), with some screen output every few seconds.
(hell it could even use splice() and never even see the bytes)
love pv, mostly use it for percentage bar on db restores:
pv -f *.sql.gz | zcat | mysql db_instance_name
Nifty command. Been using this for a month or so. One use case is measuring if a downstream process can consume data as fast the previous process is producing it (by alternating second process with /dev/null or some such).
Filed under "How did I not know this?!", with credit to @hmason on Twitter.
It's been here before:
http://news.ycombinator.com/item?id=3626020
http://news.ycombinator.com/item?id=3198572
I'm sure there were older stories too. IIRC I saw it on HN longer than 6 months ago, but I couldn't find that submission in a cursory search.
Nice - that'll spice up my rather boring svnadmin dump sessions - thanks for posting!
yes | pv > /dev/null
awesome, i was actually wondering if there was a "yes-esque" (yesque?) /dev/ file.
my throughput from yes is 49MB/s whereas from /dev/zero it's 12GB/s
Found out about pv ~year ago when working with large log files. Very useful. I also like it for testing how quickly a script can consume data.