This is huge. Not only does it let you get entropy without having to open a device file (which has problems with chroots and file descriptor exhaustion), it also provides the much-needed middle ground between /dev/random and /dev/urandom. Assuming this patch is applied (it's from Theodore Ts'o, so chances seem good), Linux will finally have a decent way to get randomness, which would make me very happy.
I'm familiar with what fork detection is (for those who aren't: a process can have the same PID as its grandparent, potentially exposing you to the risk of generating the same random data as the grandparent process), but can you elaborate on how you'd go about making it happen?
OpenBSD lets you use the minherit() syscall with the INHERIT_ZERO flag to specify that a page of virtual memory should be replaced with an all-zero page in the child process after a fork. During initialization of the PRNG, you map an anonymous "canary" page, put a non-zero byte in it, and tag it with INHERIT_ZERO. To check if a fork has occurred, you read the byte. If it's zero, a fork has occurred and you need to reseed the PRNG. This is airtight, unlike comparing getpid() against the last PID seen by the PRNG.
Another approach would be giving every process an ID that never repeats and is unique across all namespaces, and use that for comparison instead of the PID.
Further on in that same thread, Bob Beck just asked about adding minherit() functionality[1] to Linux as well. LibreSSL might end up adding some very useful syscalls to Linux, which helps with their portability and helps future Linux userland programs in the same situation. Win-win!
MADV_DONTFORK ends up working a bit different, when the forked process tries to read the page it ends up as a segmentation fault because the page wasn't carried over. It can be made to work but that ends up more difficult since you have to trap that signal and be able to prove you aren't going to catch a real one. Adding something like INHERITZERO would end up giving you a new blank page which would be much nicer to deal with in the userspace.
> MADV_DONTFORK ends up working a bit different, when the forked process tries to read the page it ends up as a segmentation fault because the page wasn't carried over. It can be made to work but that ends up more difficult since you have to trap that signal and be able to prove you aren't going to catch a real one.
Doubly difficult from within library code that needs to work in arbitrary programs, and thus cannot mess with signal handling.
> Another approach would be giving every process an ID that never repeats and is unique across all namespaces, and use that for comparison instead of the PID.
This seems like a nice choice. PRNG seeds aside, it's always bothered me that processes aren't uniquely identified, which makes monitoring them kind of a pain.
Well, currently. (~585 years to overflow at 1 billion PIDs/sec. Seems absurd, but then again, there are so many cases where we've gone "this should be good enough"...)
Ugh. Whatever happened to not wasting an excessive amount of space? Allocating 4KB for what is essentially a single bit flag seems like a needless waste of space, especially given that we already have syscalls that set up triggers to manipulate individual words in userspace (most prominently, the futex syscall, but there are others).
I suppose that if the state of your PRNG neatly fits into (almost) a multiple of 4KB, it makes sense to allocate that space explicitly and put a guard bit in there, then mark the entire region as INHERIT_ZERO. This has the added benefit of making it less like that PRNG state is leaked somewhere.
That's the OP's point. His/her question is: why allocate an entire page if, instead, you could have a single bit 'process did not initialize its source of randomness yet' in kernel space per process?
The only advantage I see is that the current solution allows one to implement the random number generator independently of the kernel. Introducing that bit creates a tight coupling.
Actually, there's an even better solution: simply give the kernel a userspace address range which will be zeroed on fork. There's simply no reason at all why this address range should be restricted to exactly 4KB or a multiple thereof (one could imagine the kernel doing some page-table tricks to avoid a full memset() for large areas, but that's an optimization that can be added transparently to an API that supports arbitrary address ranges).
The futex API (including set_tid_address) is precedence for this kind of syscall.
The problem with that is that a library can't count on pthreads: the application might call clone directly, without going through the libc wrapper, which calls the atfork callbacks.
/dev/urandom never blocks, even if it means returning randomness before the system has gathered enough entropy to safely seed a CSPRNG.
/dev/random blocks whenever the system's "entropy count" is too low, even if the system has previously gathered enough entropy to produce secure randomness. The frequent blocking of /dev/random makes it unsuitable for practical use while providing questionable security benefit.
getrandom() blocks only until the system has gathered enough entropy to safely seed a CSPRNG, and then never blocks again. (Assuming the GRND_BLOCK flag and not the GRND_RANDOM flag.)
/dev/urandom is insecure. Always use /dev/random for cryptographic purposes.
Fact: /dev/urandom is the preferred source of cryptographic randomness on UNIX-like systems.</i>
This seems to contradict your first point, which is that /dev/urandom is insecure in some situations. (Returning randomness without sufficient entropy is a security problem.)
Further down, under "Not everything is perfect," the author addresses the problem with /dev/urandom never blocking.
Seeding /dev/urandom from /dev/random is a trick that can be used early in the boot sequence to ensure that /dev/urandom is henceforth secure. (In practice, distros carry over a seed from the previous boot which accomplishes the same effect.)
I'm not really thrilled with the idea that you should use urandom to seed a CSPRNG; urandom should be your only CSPRNG. This most recent libressl kerfuffle is an illustration of why userspace CSPRNGs are a bad idea.
When I said "seed a CSPRNG" I was talking about the kernel seeding its own CSPRNG. I agree that just using the kernel's CSPRNG is the sanest solution to this mess.
By "this mess" I meant the LibreSSL fork/chroot mess. I think crypto libraries should just use whatever the OS provides and not try to implement their own CSPRNGs. But git-crypt is a cross-platform application, not a crypto library. Implementing my own cross-platform wrapper around OS CSPRNGs would be decidedly less sane (and more error-prone) than just using the crypto library's CSPRNG, even if I disagree with how the library has done things.
I know you said "CS-", but seed-able PRNGs in user-space can be immensely useful in tons of other situations, allowing things to be debugged, replayed, and synchronized.
So even if you convince to use urandom for their crypto needs on *nix platforms, it'll still have to be easy to seed one in userspace if someone really wanted to.
What about a non-global blocks-until-seeded CS-PRNG character device? I.e. you open an rw file descriptor to /dev/csprng, write(2) your seed to it, and then read(2) bytes from it. Open another file descriptor, and you get another CSPRNG that wants a separate seed.
This way, you could, say, somewhat simulate the behaviour of ZeroVM with Docker containers, by having the CSPRNG seed be a configuration parameter passed to `docker run`.
That's not what tptacek means when he says "[user space] CSPRNG", and is more appropriately called a keystream generator, not a CSPRNG, and the input is more appropriately called a key, not a seed.
I know that we need to support old programs that expect these behaviours of these device nodes. However, how about a workaround like "udev only creates /dev/urandom once /dev/random has been seeded"?
/dev/random is too severe. It's basically designed to be an information-theoretic random source, which means you could use its output as a one-time pad even if your adversary were time-travelling deities with countless universes full of quantum computers at their disposal. It blocks when you try to use it if there aren't enough bits to satisfy this.
/dev/urandom is too loose. It's designed to be computationally secure, which means that you could use it if your adversaries were stuck in our universe and had to make their computers out of matter and power them with energy. However, it never blocks, even if there isn't enough entropy in the pool.
We just need a PRNG that will spit out numbers as long as, say, 256 random bits were added to the pool at some point. Once you have that many bits, just keep on spitting out numbers.
urandom is not too loose. There is exactly one Linux issue with urandom: it will service requests before the urandom pool is initialized. Linux distros work around this by trying to make sure urandom is initialized securely very early in the boot process.
Once urandom is initialized, the idea that it can ever "run out of entropy" is nonsensical; urandom is structurally the same as a stream cipher keystream generator, and we generally don't fret about whether AES-CTR keystreams will "run out of key". Nonetheless, the belief that urandom will sporadically "run out of entropy" is virulent.
The motivation behind the new system call has more to do with chroot environments, where the device might not be available at all. It's a good change; randomness should be provided by a system call and not a device. Unfortunately, by adding the flag, they've basically managed to add two system calls: getrandom and geturandom. >HEADDESK<
There's been a lot of papers about urandom being a broken pool. They need to adopt something like Fortuna or one of the more modern formalisms of Fortuna.
The Linux urandom pool is managed almost identically to the random pool. The fact that random is "safe" (if blocking makes you safe) immediately after cold boot is actually just a side effect.
What those papers (or at least the ones I think you're talking about) deal with is something slightly more nuanced than what is being discussed here. They analyze how the pool recovers from a compromise, i.e., if you somehow manage to dump its entire state how quickly can you gather entropy again to minimize the damage.
It turns out Fortuna and variants thereof score very well in this metric, but this does not have any bearing on the quality of the output, or whether it loses entropy by generating more bytes.
>Once urandom is initialized, the idea that it can ever "run out of entropy" is nonsensical; urandom is structurally the same as a stream cipher keystream generator, and we generally don't fret about whether AES-CTR keystreams will "run out of key". Nonetheless, the belief that urandom will sporadically "run out of entropy" is virulent.
If the seed value for urandom were compromised or you were unsure of its provenance (some systems carry over from last boot), would you not be safer calling /dev/random for something sensitive like key generation? What if you did not trust the PRNG that urandom used?
But /dev/random has the additional constraint that it will stop outputting if that PRNG hasn't been reseeded in [some reasonable amount of time/output bits]. So you are guaranteed to get bits that have come from a recently-seeded PRNG, rather than any old PRNG. I have had this argument on here before and the conclusion was that it's not a real issue, but I think this is what the parent poster is talking about.
EDIT: Thanks for the clarification tptacek. I don't mean to disagree -- in fact it was you who explained this to me last time as well.
"Recently" seeded isn't a meaningful distinction. It has either been "seeded" or it hasn't. The recency of the seed --- a point Linux's interface worries a great deal about --- has nothing to do with security.
>The recency of the seed --- a point Linux's interface worries a great deal about --- has nothing to do with security.
Unless you are concerned about where the seed came from (e.g. not storage like /var/run/random-seed) or have any concerns that there is a flaw in the PRNG that could leak information.
> Once urandom is initialized, the idea that it can ever "run out of entropy" is nonsensical; urandom is structurally the same as a stream cipher keystream generator, and we generally don't fret about whether AES-CTR keystreams will "run out of key". Nonetheless, the belief that urandom will sporadically "run out of entropy" is virulent.
Something doesn't add up. Supposedly the reasons for creating urandom were to avoid blocking - but at the same time all the docs warn about entropy depletion. Are you now saying this is meaningless? If so, then why ever use /dev/random?
Given the lack of the "middle ground" introduced by this patch, you have (had) a choice between the failure mode of "block for a very long time" and the failure mode of "might generate easily predicted cryptographic keys under certain extremely rare circumstances". You use /dev/random if you prefer the first failure mode.
Edit: The whole situation didn't make sense from a system design point of view (unless you don't believe in CSPRNGs, but then you're basically screwed anyway), but given the unreasonable interface as a constraint, it's conceivable that somebody might have reasonably made the choice to use /dev/random.
> "might generate easily predicted cryptographic keys under certain extremely rare circumstances"
no, he's not saying that - he's saying that there's no such thing as entropy depletion - and so urandom is secure. Which makes me ask after urandom was created - why EVER bother using /dev/random with it's blocking flaw/deficiency?
He said there's no such thing as entropy depletion for urandom after it's been seeded. But the seed still has to come from somewhere, and one possible source for that seed is /dev/random.
/dev/random actually points to the same RNG as /dev/urandom, according to http://www.2uo.de/myths-about-urandom/ (and other sources I recall reading but can't find). So you wouldn't use /dev/random to seed /dev/urandom, but you (or the kernel) might use something else to seed both.
There is no reason to use /dev/random other than cargo culting devs that believe that /dev/urandom can supposedly run out of entropy.
Use /dev/urandom. Don't use /dev/random. On sane systems (FreeBSD for example), /dev/urandom is a symlink to /dev/random and /dev/random only blocks once, upon startup to gather entropy, after that it never blocks!
There's no such thing as entropy depletion, but there is such a thing as an insufficiently seeded CSPRNG - which means that /dev/urandom is not secure by design: it does not protect you against that failure mode, and in fact people rely on fragile hacks implemented by distributions to try to seed /dev/urandom properly as soon as possible in the bootup sequence. These hacks could easily break if somebody does not know exactly what they're doing while touching the boot sequence.
/dev/random is also stupid, but it does protect you against that particular failure mode.
Exactly. The warnings about "entropy depletion" are only relevant to theoretical attacks which require either impossibly large computational resources or a flaw in the PRNG algorithm. Since either assumption breaks all our crypto for other, unrelated reasons, we use /dev/urandom.
That, and if you use /dev/urandom before it's seeded, you expose yourself to real attacks.
Excuse me for asking a stupid question, I am not too deep into linux kernel randomness generation:
Why is /dev/urandom spitting out anything before it has acquired enough entropy for the initial seed? Wouldn't it be a good idea for it to initially block?
The contract when /dev/random and /dev/urandom came out was that urandom would never, ever block.
On a system with a recent Intel processor, there's a instruction (RDSEED) that uses on on-die hardware RNG. I'm not familiar with the standard linux boot-up process, but it could in principle seed urandom using RDSEED arbitrarily early in the process. That should work on VMs too unless the hypervisor is blocking access (can't imagine a good reason for that).
Via has on-die RNG considerably longer, though it's accessed slightly differently. I don't believe AMD or ARM has anything similar.
You make it sound like you are saying something new in your first couple paragraphs, but you aren't: the initial entropy gap issue seems to be exactly what klodolph was talking about. You hedge with "not really focused on you", but I'm going to say that you probably then have just not posted most of your post, as even if none of it were related to klodolph (which isn't the case anyway, due to the quoting of "too loose") it is still written in a way as to make people believe he made the mistake you are trying to be pedantic about :(. Just because many, even most, people make a particular mistake does not mean everyone does. Note, very carefully, the wording "at some point": and it wasn't then "until it runs out", it was "just keep on spitting out". To some people, this is a real issue stemming from a completely unreasonable kernel default that the developers of libraries and applications have absolutely no control over, which occasionally comes up on computers built with custom distributions by non-experts under the assumption "Linux will do something sane", and which in a prior thread we found actual Hacker News users who had been first burned and then didn't have good options for a fix as it was a customer's computer. I think if you would be willing to be more open to the premise that not everyone is wrong in trivial, predictable ways, you'd be surprised by how often they aren't :/. Your last paragraph is great, and I'm really glad you contributed it, but the first two were unwarranted.
I described in my comment how the comment by tptacek is not correct; I can state again: it "corrects" klodolph's comment, and yet klodolph's comment was correct and did not deserve correction. tptacek tries to use this comment as an example, turning to the side to address the audience to voice this correction towards everyone with the "not really focussed on you" hedge, but uses the "not too loose" quoting to make it clear that klodolph is still the example.
This particular way in which tptacek's comment is "mistaken" is related to the tone and attitude tptacek takes in his comments. If this were the only instance, even the only instance for this particular topic, it would be one thing, but this is a common issue with tptacek's comments. The pattern is that there are certain "common misconceptions" that everyone has, and tptacek is not generous to the poster of any specific comment that they might not possess them.
I further believe that it is a serious problem on Hacker News that people do not address these kinds of tone issues: that it is perfectly fine to "HEADDESK", claim that certain beliefs are "virulent", and to even use the word "nonsensical" to describe someone's idea, and yet attempts to point out issues in the tone of peoples' comments is somehow a problem: something where you feel the need to say "don't make this conversation personal". More people need to stand up to this.
I, myself, have done some of these things in my own comments. I feel like most of these cases were situations where I was responding to someone else doing it to me, but I've found at least a few instances where that is not the case. It makes me very unhappy that I contributed to this problem: someone should have also complained about my tone in those instances. It needs to be ok to exit the topic and address how someone is saying something, not just what was said.
To be clear, tptacek's position is correct: we don't need a new interface; klodolph's argument largely ends up arguing for fixing /dev/urandom. However, tptacek doesn't say this, as he has assumed that klodolph doesn't understand the difference between /dev/urandom and /dev/random, and then argued based on that assumption. If you read the other comments from klodolph, it is very very clear that he understands perfectly. tptacek could at least apologize.
I don't think 'klodolph took offense. If he did, I would feel bad, and would certainly apologize. For now, I'm going to presume he read my comment in the spirit it was intended: that the virulent, nonsensical idea I was describing was not being attributed directly to him, hence the disclaimer at the top of the comment.
(If you re-read my comment, you'll also find that the >HEADDESK< isn't addressed to 'klodolph at all, but to the designers of the Linux randomness system call).
You could probably mail me further thoughts you have about my comments on HN, if you wanted to keep talking about it.
We are commenting on a thread about the Linux kernel random developers making exactly the mistake you think I should assume people won't so readily make.
To add to tptacek's excellent comment, it's important to remember that some of the characteristics of /dev/random and /dev/urandom discussed here only apply to Linux; not other operating systems such as Solaris:
...the NSA would love that most of all. The vast majority of programmers lack the knowledge, skill, or both required to correctly implement anything related to cryptography.
Who's "everybody?" If you mean userspace application/library developers, they don't have a good source of entropy, so they have to get it from the kernel. That means userspace CSPRNGs end up depending on the kernel CSPRNG. Presto, two single points of failure!
I see, so prior to this random() and entropy() addition to Linux nobody could ever write any security related application. Yes, this is true. We should change the pull request to "Enable security for Linux, 2014". God, never too late...
The third paragraph in the DESCRIPTION section of the manpage contains a lot of errors:
- "then the /dev/raundom pool will be used" -> should be /dev/urandom
- "Unlike reading from the /dev/urandom" -> author meant /dev/random
- "EGAIN" -> should be EAGAIN
- In fact the entire sentence "Unlike reading from /dev/random, [it either blocks or returns EAGAIN]" should be removed. This blocking/EAGAIN behavior is the same regardless if you read from random or urandom.
> - "Unlike reading from the /dev/urandom" -> author meant /dev/random
Pretty sure he doesn't. He's describing the differences between using the /dev/urandom pool (through getrandom(2)) and actually reading from /dev/urandom.
> The /dev/random pool is limited based on the entropy that can be obtained from environmental noise, so if there is insufficient entropy, the requested number of bytes may not be returned. If there is no entropy available at all, getrandom(2) will either return an error with errno set to EAGAIN, or block if the GRND_BLOCK flags bit is set.
Man, that's an ugly interface. So, if there are 8 bits of entropy available and I ask for 16, it will return 8, but if there are none available and I ask for 16 it will block and then return 16.
Also, it's not possible to know how much entropy is available. I'd really like to emphasise that: entropy estimation is nonsense.
Don't block; seriously, don't block. Just spit out the output of a properly-seeded PRF (e.g. AES in CTR mode).
No, /dev/urandom fails the 'properly-seeded' criterion during early boot.
The real answer is to just change the interface and be done with it, and to hell with backwards-compatibility, but that'll never happen.
The really real answer is to make /dev/random and /dev/urandom the same CSPRNG, seeded as early in boot as possible (and continually refreshed with more entropy, of course), and never blocking thereafter, but that'll never happen either.
"Any userspace program which uses this new
functionality must make sure that if it is used in early boot, that it
will not cause the boot up scripts or other portions of the system
startup to hang indefinitely."
How can the userspace program "make sure that it doesn't hang indefinitely" if it is started early in the boot process and the call blocks?
In the synopsis, it says "It should not be used [by] Monte Carlo simulations or for other probabilistic sampling applications."
Does anyone know why not?
"deplete the entropy pool" isn't really a meaningful phrase given the way the Linux /dev/{random,urandom} work. AFAIK /dev/random and /dev/urandom use the same CSPRNG (different instances) which is periodically reseeded with entropy gathered from the system (hw, interrupt timings, etc.), the only difference being that the /dev/random device blocks if an entropy estimator says that there's "too little entropy".
It's not that /dev/random has been "depleted" (which is meaningless given how a CSPRNG works), it's just that some entropy estimation code thinks that it might not be a source of perfect entropy/randomness at a particular time. (This practice is pretty questionable, hence the proposed new syscall.)
At a guess, those applications routinely require large amounts of mostly random (not CSPRNG random) numbers. This syscall returns data from the kernel's entropy pool, presumably at the speed of /dev/urandom.
The recommendation is probably a way to say "please don't abuse, and please don't make the kernel spin".
Here are some guesses about the reasoning for this admonition:
* Suppose /dev/urandom is not receiving randomness at all, and the Monte Carlo application keeps drawing bits from it like crazy. Then this device is reduced, effectively, to a PRNG. Maybe, as such, it is not a particularly good PRNG; it is not intended to be used that way. It is supposed to take some real entropy bits, and "stretch" them to cover a larger request---but not ridiculously so.
* Simulation applications benefit from repeatability: the same seed fetches exactly the same results. You lose that with urandom.
Monte Carlo simulations need to be repeatable. So you can take a seed from entropy, but then you seed your own deterministic RNG. You log the seed so that you can backtest.
Doesn't appear many people on this thread are considering such use cases, but it appears the authors have.
I stumbled across a possible answer to this question just now in a discussion regarding a proposed interface for a randomness call where the callee would declare what they are using the randomness for - with Monte Carlo as one of the options. I immediately recalled your comment, so, here is your answer:
> I would hope that people would agree there should be a difference between what is needed for a Monte Carlo simulation and other various cryptographic use cases. (I've gotten bug reports from people who insisted on using /dev/urandom for Monte Carlo simulations, and who then complained when it was too slow for their purposes....)
> If the GRND_RANDOM flags bit is set, then draw from the
/dev/random pool instead of /dev/urandom pool. The
/dev/random pool is limited based on the entropy that can be
obtained from environmental noise, so if there is insufficient
entropy, the requested number of bytes may not be returned.
If there is no entropy available at all, getrandom(2) will
either return an error with errno set to EAGAIN, or block if
the GRND_BLOCK flags bit is set.
> If the GRND_RANDOM flags bit is not set, then the /dev/raundom
pool will be used. Unlike reading from the /dev/urandom, if
the urandom pool has not been sufficiently initialized,
getrandom(2) will either return an error with errno set to
EGAIN, or block if the GRND_BLOCK flags bit is set.
Is this complexity necessary? I'm concerned that this will be hard to use properly as there's lots of different behaviours based on the parameters, and different types of failure modes. Security code that is hard to call right or hard to understand tends to get misused, causing security bugs in application code.
Yay! However, a few things that are a bit unfortunate:
- It suggests that I use that entropy to seed a PRNG. No. It should be the PRNG.
- Fill the buffer with "up to buflen random bytes"? I asked for n bytes. Please give me actually n bytes. If there's a way to get this wrong, people will get it wrong: the BSD socket API does the same thing and at least it has good reason. However, the wording seems to imply that if you use the "urandom" source, it won't do that?
- "It should not be used Monte Carlo simulations or for other probabilistic sampling applications.". Doesn't mention why. Sounds like "run out of entropy" FUD :-(
Slightly off-topic, but I couldn't help but notice as a programmer. I find the coding style of the patch kind of random. The if statements, some of them have braces and some don't. Without a pattern it seems.
Oh right, I see the second if statement at block @@ -1533,6 +1544,26 @@ indeed threw me off. I would value consistency over necessity. The else-if at the same block looks weird and the use of new lines could also help the code to be more legible and maintainable.
The rationale for this system call is poor. If you're out of descriptors, so that you cannot open another one, then bail loudly! Do not fall back on some ill-conceived code that uses poor quality pseudo-random bits in a situation where such bits must not be used!
Also, an descriptor to /dev/urandom can be acquired early and retained for the life of the process. When you need bits, you just read from it. This gets around subsequent chroots and descriptor exhaustion.
You missed the 3rd reason for this. Calls to /dev/urandom can not block, so it a script or program requests random bytes before entropy has been set up, it can get back bad random bytes. The new system call getrandoms does block, so early scripts will not get bad entropy data.
The newly proposed system call, as implemented in the mailing list patch, provides the functionality of both random and urandom: the behaviors are distinguished by a bitmask.
Take a look at the code. If the flag GRND_RANDOM is used, then it uses the random device, otherwise urandom.
Independently of this, blocking behavior is requested with GRND_BLOCK. If this is omitted, then the random device bails with errno == EAGAIN if it doesn't have enough entropy, otherwise it blocks.
If GRND_BLOCK is omitted with the urandom method (GRND_RANDOM is omitted), then it will bail with -EAGAIN if the urandom device is not initialized; otherwise it just calls urandom_read. With GRND_BLOCK, it will block for urandom to initialize, if necessary.
The new system call only blocks when given GRND_BLOCK, and it uses urandom unless given the flag GRND_RANDOM. If it is given GRND_RANDOM, but not GRND_BLOCK, and not enough entropy is available, then the system call with bail with errno == EAGAIN. If there is no GRND_RANDOM then it falls back on urandom_read: that is usually non-blocking.
I see; if the point is that /dev/urandom doesn't even block when it is uninitialized, that is valid.
This system call (when used to access urandom) will either block on urandom to be initialized, or else fail loudly with -EAGAIN, which is an improved interface.
I think its because linux got there first and designed these interfaces
To quote Ted Tso (author of the above patch and the original writer of /dev/random in the first place)
The two other interfaces are two character devices /dev/random and
* /dev/urandom. /dev/random is suitable for use when very high
* quality randomness is desired (for example, for key generation or
* one-time pads), as it will only return a maximum of the number of
* bits of randomness (as estimated by the random number generator)
* contained in the entropy pool.
*
* The /dev/urandom device does not have this limit, and will return
* as many bytes as are requested. As more and more random bytes are
* requested without giving time for the entropy pool to recharge,
* this will result in random numbers that are merely cryptographically
* strong. For many applications, however, this is acceptable.
X11 has always been a bit of a weird beast. And things like /proc are definitely in everything-is-a-file tradition, so it wasn't quite dead. (Maybe just pining for the fjords?)
/dev/random and /dev/urandom have security attributes: ownership and permission. The newly proposed syscall makes no security checks whatsoever to protect the entropy resource: it punches a hole through these perms, effectively.
A security case for protecting a source of random bits is that they are important and that cryptographic applications and middleware can malfunction if these bits are not available. A world-readable /dev/random means that anyone can drain out random bits, making them unavailable to these applications.
Someone somewhere could be setting up permissions in their Linux-based system based on this type of reasoning.
It's a security regression if a character device can be suddenly accessed through a back-channel that ignores permissions.
If you don't trust a CSPRNG, then you pretty much might as well turn off all your crypto because you can't trust it.
If you can trust a CSPRNG, then you should be able to trust urandom, as long as it's been properly initialized once. It won't drain out. It doesn't run out of bits. One application reading it doesn't make it unavailable for others. The same is true of the syscall.
Short version is, on Solaris, you should use /dev/random for long-term keys and other high-value keying material. /dev/urandom should be used for everything else such as less demanding or shorter-term cryptographic uses like short-term session keys, paddings, and challenge strings.
1. It's not safe to use a weak RNG for session keys or even padding. Cryptosystems often fail catastrophically when their most obscure parameters are weak --- in fact, more exploitable cryptosystem flaws derive from issues like padding and IVs than keys.
2. FIPS 186-2 is a SHA-based DRBG. It doesn't "run out of entropy". What am I missing? The situation on Solaris w/r/t urandom appears to be identical to that of Linux.
1. The Solaris crypto designers / developers believe their rng is is appropriate for short-term session keys and padding. Read the post I linked before as to why the believe that.
2. /dev/random basically only uses bits from entropy sources while /dev/urandom just uses a few bits from /dev/random as a seed. Read the post I linked again.
Again, I would reiterate, the situation is somewhat different for Solaris. Usage of /dev/random vs. /dev/urandom is actually considered different.
I read this post, both when you linked to it just now, and, prior to that, while I was researching the urandom blog post I wrote a few months ago. I'm being repetitive, but:
* I think the distinction this post is drawing between Solaris random and urandom is functionally very similar to the distinction between Linux random and urandom (I think the distinction on Solaris might be that random has a higher quality rekeying system than urandom).
* I can read a straightforward description of a SHA1 PRNG keyed by kernel entropy, which is what this blog post describes, and unless I'm missing something in the article, I don't see how it is any less cryptographically secure than Solaris /dev/random. It seems to be less forward-secure than /dev/random. That's about it.
Since you appear to have grokked this article better than I did in either of the two cases I read it, could you give a short summary of the argument it presents for avoiding urandom for long-term secrets?
1) You're not completely off here; the primary difference is currently higher quality re-keying for /dev/random, but there are implementation differences.
2) From a cryptography standpoint, the precise implementation of the current framework makes /dev/urandom suitable for short-term session keys and padding. This may not hold true for Linux, but for Solaris it has been reviewed and deemed appropriate usage. Fundamentally, it's about constraints placed on the implementation that currently make /dev/random more "secure".
In conversing with one of the crypto developers for Solaris, they are considering whether to lessen the distinction between /dev/random and /dev/urandom further in the future.
I think the concern is that a process could then exhaust all available entropy, and DOS any other processes depending on /dev/random. Currently, a chroot could protect against that.
I didn't understand what you meant by you commentary.
The above excerpt shows they assuming upfront that they got the idea from BSD. As a system call is not an application that may be ported, they need to recode it.
If the point was the name change, I guess this is only to keep consistency within the system, and far from a problem.
They took the sane design from BSD ("here's a single system call that gives you crypto-safe entropy") and effectively made a family of system calls that replicate all the pointless drama of /dev/random and urandom. The NIH criticism is valid.
It seems pointless indeed. On the other hand, I'm almost glad the overengineering only went that far. Actually I'm expecting much worse after reading Theodore's ideas on the IETF list.. we'll see what kind of a library interface they'll come up with, if they make something besides getentropy().
But if we get all applications to use the same library, we can
abstract away not only differences in operating system but also
security policies vis-a-vis DRBG/NDRBG blocking/nonblocking. So what
*I* would prefer is a library interface where the application declares
what it wants the random numbers for:
* Monte carlo simulations
* Padding
* IV
* Session key
* long-term key
etc. The library can then decide whether or not the overall system
policy should force application to block when generating long-term
keys, ala gpg, or whether it should getting non-blocking entropy
directly from the kernel, or whether using a userspace DRBG which is
initialized from kernel-supplied entropy is the right answer.
What is a case where a random IV needs a different entropy source than a session key?
And if we're assuming the entropy pool is being compromised (full state read by attacker) from time to time, isn't it foolish to be generating keys on such a machine? Why would new state not be compromised in he same way the previous state was? I understand the system design may want to provide a robust RNG, but further than that seems slightly pointless.
This is huge. Not only does it let you get entropy without having to open a device file (which has problems with chroots and file descriptor exhaustion), it also provides the much-needed middle ground between /dev/random and /dev/urandom. Assuming this patch is applied (it's from Theodore Ts'o, so chances seem good), Linux will finally have a decent way to get randomness, which would make me very happy.
Hopefully proper fork detection will be next up.
I'm familiar with what fork detection is (for those who aren't: a process can have the same PID as its grandparent, potentially exposing you to the risk of generating the same random data as the grandparent process), but can you elaborate on how you'd go about making it happen?
OpenBSD lets you use the minherit() syscall with the INHERIT_ZERO flag to specify that a page of virtual memory should be replaced with an all-zero page in the child process after a fork. During initialization of the PRNG, you map an anonymous "canary" page, put a non-zero byte in it, and tag it with INHERIT_ZERO. To check if a fork has occurred, you read the byte. If it's zero, a fork has occurred and you need to reseed the PRNG. This is airtight, unlike comparing getpid() against the last PID seen by the PRNG.
Another approach would be giving every process an ID that never repeats and is unique across all namespaces, and use that for comparison instead of the PID.
That is really slick. Thank you.
Further on in that same thread, Bob Beck just asked about adding minherit() functionality[1] to Linux as well. LibreSSL might end up adding some very useful syscalls to Linux, which helps with their portability and helps future Linux userland programs in the same situation. Win-win!
[1] http://lists.openwall.net/linux-kernel/2014/07/17/707
That is really awesome. Though it will be 2 more years before an Ubuntu LTS will support these features :(
Not necessarily. Ubuntu have "enablement stacks" kernel updates for the LTS releases. These are more than just upstream patches.
For example, 12.04 LTS release had 12.04.4 with a backported kernel as a standard update.
https://wiki.ubuntu.com/PrecisePangolin/ReleaseNotes/ChangeS...
Oh, and here I thought they just patched security holes but kept the same old kernel for 2 years. [insert "the more you know" graphic here]
Seems like that functionality would better be included as separate madvise() advice than a new syscall, but probably a good addition either way.
The next email in that thread suggests using madvise() as well.
I'm a noob concerning all of this, but how would it be different to MADV_DONTFORK? Or why can't MADV_DONTFORK be used for the same task?
MADV_DONTFORK ends up working a bit different, when the forked process tries to read the page it ends up as a segmentation fault because the page wasn't carried over. It can be made to work but that ends up more difficult since you have to trap that signal and be able to prove you aren't going to catch a real one. Adding something like INHERITZERO would end up giving you a new blank page which would be much nicer to deal with in the userspace.
> MADV_DONTFORK ends up working a bit different, when the forked process tries to read the page it ends up as a segmentation fault because the page wasn't carried over. It can be made to work but that ends up more difficult since you have to trap that signal and be able to prove you aren't going to catch a real one.
Doubly difficult from within library code that needs to work in arbitrary programs, and thus cannot mess with signal handling.
There's a real need for per-memory-range segfault handlers.
I see.
> Another approach would be giving every process an ID that never repeats and is unique across all namespaces, and use that for comparison instead of the PID.
This seems like a nice choice. PRNG seeds aside, it's always bothered me that processes aren't uniquely identified, which makes monitoring them kind of a pain.
A 64-bit process ID would allow that.
Well, currently. (~585 years to overflow at 1 billion PIDs/sec. Seems absurd, but then again, there are so many cases where we've gone "this should be good enough"...)
But at least if it lasts longer than your lifespan, you can be sure it will be someone else's problem when it goes wrong.
Ugh. Whatever happened to not wasting an excessive amount of space? Allocating 4KB for what is essentially a single bit flag seems like a needless waste of space, especially given that we already have syscalls that set up triggers to manipulate individual words in userspace (most prominently, the futex syscall, but there are others).
I suppose that if the state of your PRNG neatly fits into (almost) a multiple of 4KB, it makes sense to allocate that space explicitly and put a guard bit in there, then mark the entire region as INHERIT_ZERO. This has the added benefit of making it less like that PRNG state is leaked somewhere.
The page size is a minimum of 4KB, so you can't really allocate anything smaller from the OS.
That's the OP's point. His/her question is: why allocate an entire page if, instead, you could have a single bit 'process did not initialize its source of randomness yet' in kernel space per process?
The only advantage I see is that the current solution allows one to implement the random number generator independently of the kernel. Introducing that bit creates a tight coupling.
Actually, there's an even better solution: simply give the kernel a userspace address range which will be zeroed on fork. There's simply no reason at all why this address range should be restricted to exactly 4KB or a multiple thereof (one could imagine the kernel doing some page-table tricks to avoid a full memset() for large areas, but that's an optimization that can be added transparently to an API that supports arbitrary address ranges).
The futex API (including set_tid_address) is precedence for this kind of syscall.
Would it be possible to do something a bit more modern-feeling such as register a callback address that gets executed on fork?
Just don't fork in that callback... Or call anything that potentially forks.
If you can count on pthreads, you can use pthread_atfork for that.
The problem with that is that a library can't count on pthreads: the application might call clone directly, without going through the libc wrapper, which calls the atfork callbacks.
Very good point; pthread_atfork does not suffice. Thanks for the correction.
What's the reason for not keeping the PRNG data in a page of its own which is marked INHERIT_ZERO?
it also provides the much-needed middle ground between /dev/random and /dev/urandom.
What do you mean?
/dev/urandom never blocks, even if it means returning randomness before the system has gathered enough entropy to safely seed a CSPRNG.
/dev/random blocks whenever the system's "entropy count" is too low, even if the system has previously gathered enough entropy to produce secure randomness. The frequent blocking of /dev/random makes it unsuitable for practical use while providing questionable security benefit.
getrandom() blocks only until the system has gathered enough entropy to safely seed a CSPRNG, and then never blocks again. (Assuming the GRND_BLOCK flag and not the GRND_RANDOM flag.)
An in-depth explanation: http://www.2uo.de/myths-about-urandom/
From your link:
/dev/urandom is insecure. Always use /dev/random for cryptographic purposes.
Fact: /dev/urandom is the preferred source of cryptographic randomness on UNIX-like systems.</i>
This seems to contradict your first point, which is that /dev/urandom is insecure in some situations. (Returning randomness without sufficient entropy is a security problem.)
Can urandom ever be used insecurely? Tptacek has said many times that you should just seed /dev/urandom from /dev/random, then never use /dev/random again: http://hn.algolia.com/#!/comment/forever/prefix/0/author%3At...
Is LibreSSL's situation different? If so, why?
Further down, under "Not everything is perfect," the author addresses the problem with /dev/urandom never blocking.
Seeding /dev/urandom from /dev/random is a trick that can be used early in the boot sequence to ensure that /dev/urandom is henceforth secure. (In practice, distros carry over a seed from the previous boot which accomplishes the same effect.)
I'm not really thrilled with the idea that you should use urandom to seed a CSPRNG; urandom should be your only CSPRNG. This most recent libressl kerfuffle is an illustration of why userspace CSPRNGs are a bad idea.
When I said "seed a CSPRNG" I was talking about the kernel seeding its own CSPRNG. I agree that just using the kernel's CSPRNG is the sanest solution to this mess.
> I agree that just using the kernel's CSPRNG is the sanest solution to this mess.
well, you agree up to the point you actually need to convert your words into facts, at that point you don't agree anymore :)
https://github.com/AGWA/git-crypt/commit/34432e915e8415b112c...
Oh boy, I should have seen this one coming :-)
By "this mess" I meant the LibreSSL fork/chroot mess. I think crypto libraries should just use whatever the OS provides and not try to implement their own CSPRNGs. But git-crypt is a cross-platform application, not a crypto library. Implementing my own cross-platform wrapper around OS CSPRNGs would be decidedly less sane (and more error-prone) than just using the crypto library's CSPRNG, even if I disagree with how the library has done things.
I know you said "CS-", but seed-able PRNGs in user-space can be immensely useful in tons of other situations, allowing things to be debugged, replayed, and synchronized.
So even if you convince to use urandom for their crypto needs on *nix platforms, it'll still have to be easy to seed one in userspace if someone really wanted to.
What about a non-global blocks-until-seeded CS-PRNG character device? I.e. you open an rw file descriptor to /dev/csprng, write(2) your seed to it, and then read(2) bytes from it. Open another file descriptor, and you get another CSPRNG that wants a separate seed.
This way, you could, say, somewhat simulate the behaviour of ZeroVM with Docker containers, by having the CSPRNG seed be a configuration parameter passed to `docker run`.
What if you want reproducible random numbers, such as in, I dunno, AES? That limitation would prevent any sort of decrypting things.
That's not what tptacek means when he says "[user space] CSPRNG", and is more appropriately called a keystream generator, not a CSPRNG, and the input is more appropriately called a key, not a seed.
I know that we need to support old programs that expect these behaviours of these device nodes. However, how about a workaround like "udev only creates /dev/urandom once /dev/random has been seeded"?
/dev/random is too severe. It's basically designed to be an information-theoretic random source, which means you could use its output as a one-time pad even if your adversary were time-travelling deities with countless universes full of quantum computers at their disposal. It blocks when you try to use it if there aren't enough bits to satisfy this.
/dev/urandom is too loose. It's designed to be computationally secure, which means that you could use it if your adversaries were stuck in our universe and had to make their computers out of matter and power them with energy. However, it never blocks, even if there isn't enough entropy in the pool.
We just need a PRNG that will spit out numbers as long as, say, 256 random bits were added to the pool at some point. Once you have that many bits, just keep on spitting out numbers.
Pedantry, not really focused on you:
urandom is not too loose. There is exactly one Linux issue with urandom: it will service requests before the urandom pool is initialized. Linux distros work around this by trying to make sure urandom is initialized securely very early in the boot process.
Once urandom is initialized, the idea that it can ever "run out of entropy" is nonsensical; urandom is structurally the same as a stream cipher keystream generator, and we generally don't fret about whether AES-CTR keystreams will "run out of key". Nonetheless, the belief that urandom will sporadically "run out of entropy" is virulent.
The motivation behind the new system call has more to do with chroot environments, where the device might not be available at all. It's a good change; randomness should be provided by a system call and not a device. Unfortunately, by adding the flag, they've basically managed to add two system calls: getrandom and geturandom. >HEADDESK<
There's been a lot of papers about urandom being a broken pool. They need to adopt something like Fortuna or one of the more modern formalisms of Fortuna.
The Linux urandom pool is managed almost identically to the random pool. The fact that random is "safe" (if blocking makes you safe) immediately after cold boot is actually just a side effect.
What those papers (or at least the ones I think you're talking about) deal with is something slightly more nuanced than what is being discussed here. They analyze how the pool recovers from a compromise, i.e., if you somehow manage to dump its entire state how quickly can you gather entropy again to minimize the damage.
It turns out Fortuna and variants thereof score very well in this metric, but this does not have any bearing on the quality of the output, or whether it loses entropy by generating more bytes.
Linux kernel random interface designs are themselves not a good source of entropy.
"The motivation behind the new system call has more to do with chroot environments, where the device might not be available at all. "
Why wouldn't one simply create the appropriate device node inside the chroot? This argument strikes me as nonsense.
it is normal for a vast amount of software to fork privsep children into /var/empty, which as name suggests, is empty.
Forcing urandom to exist also precludes mounting partions 'nodev' which has secuirty implications in a chroot enviroment.
Lastly, relying on a devicefile makes you prone to things like fd exhaustion attacks.
>Once urandom is initialized, the idea that it can ever "run out of entropy" is nonsensical; urandom is structurally the same as a stream cipher keystream generator, and we generally don't fret about whether AES-CTR keystreams will "run out of key". Nonetheless, the belief that urandom will sporadically "run out of entropy" is virulent.
If the seed value for urandom were compromised or you were unsure of its provenance (some systems carry over from last boot), would you not be safer calling /dev/random for something sensitive like key generation? What if you did not trust the PRNG that urandom used?
Both /dev/random and /dev/urandom use the same PRNG...
But /dev/random has the additional constraint that it will stop outputting if that PRNG hasn't been reseeded in [some reasonable amount of time/output bits]. So you are guaranteed to get bits that have come from a recently-seeded PRNG, rather than any old PRNG. I have had this argument on here before and the conclusion was that it's not a real issue, but I think this is what the parent poster is talking about.
EDIT: Thanks for the clarification tptacek. I don't mean to disagree -- in fact it was you who explained this to me last time as well.
"Recently" seeded isn't a meaningful distinction. It has either been "seeded" or it hasn't. The recency of the seed --- a point Linux's interface worries a great deal about --- has nothing to do with security.
>The recency of the seed --- a point Linux's interface worries a great deal about --- has nothing to do with security.
Unless you are concerned about where the seed came from (e.g. not storage like /var/run/random-seed) or have any concerns that there is a flaw in the PRNG that could leak information.
If you are concerned that your PRNG is broken, then you can't trust crypto in general. Doesn't matter how often you reseed your PRNG.
Right, and if it is flawed to long term analysis in some way, you want it to be frequently re-seeded.
> Once urandom is initialized, the idea that it can ever "run out of entropy" is nonsensical; urandom is structurally the same as a stream cipher keystream generator, and we generally don't fret about whether AES-CTR keystreams will "run out of key". Nonetheless, the belief that urandom will sporadically "run out of entropy" is virulent.
Something doesn't add up. Supposedly the reasons for creating urandom were to avoid blocking - but at the same time all the docs warn about entropy depletion. Are you now saying this is meaningless? If so, then why ever use /dev/random?
If so, then why ever use /dev/random?
Given the lack of the "middle ground" introduced by this patch, you have (had) a choice between the failure mode of "block for a very long time" and the failure mode of "might generate easily predicted cryptographic keys under certain extremely rare circumstances". You use /dev/random if you prefer the first failure mode.
Edit: The whole situation didn't make sense from a system design point of view (unless you don't believe in CSPRNGs, but then you're basically screwed anyway), but given the unreasonable interface as a constraint, it's conceivable that somebody might have reasonably made the choice to use /dev/random.
> "might generate easily predicted cryptographic keys under certain extremely rare circumstances"
no, he's not saying that - he's saying that there's no such thing as entropy depletion - and so urandom is secure. Which makes me ask after urandom was created - why EVER bother using /dev/random with it's blocking flaw/deficiency?
He said there's no such thing as entropy depletion for urandom after it's been seeded. But the seed still has to come from somewhere, and one possible source for that seed is /dev/random.
/dev/random actually points to the same RNG as /dev/urandom, according to http://www.2uo.de/myths-about-urandom/ (and other sources I recall reading but can't find). So you wouldn't use /dev/random to seed /dev/urandom, but you (or the kernel) might use something else to seed both.
There is no reason to use /dev/random other than cargo culting devs that believe that /dev/urandom can supposedly run out of entropy.
Use /dev/urandom. Don't use /dev/random. On sane systems (FreeBSD for example), /dev/urandom is a symlink to /dev/random and /dev/random only blocks once, upon startup to gather entropy, after that it never blocks!
I don't think it's a "cargo cult" if the official docs mention and warn about entropy pool exhaustion.
The official docs are wrong.
Then submit a patch to fix them.
Instead of telling other people what to do, maybe you should just do it yourself.
Corey Csuhta filed an issue for this four months ago, but no one has responded to it: https://bugzilla.kernel.org/show_bug.cgi?id=71211
There's no such thing as entropy depletion, but there is such a thing as an insufficiently seeded CSPRNG - which means that /dev/urandom is not secure by design: it does not protect you against that failure mode, and in fact people rely on fragile hacks implemented by distributions to try to seed /dev/urandom properly as soon as possible in the bootup sequence. These hacks could easily break if somebody does not know exactly what they're doing while touching the boot sequence.
/dev/random is also stupid, but it does protect you against that particular failure mode.
> If so, then why ever use /dev/random?
Exactly. The warnings about "entropy depletion" are only relevant to theoretical attacks which require either impossibly large computational resources or a flaw in the PRNG algorithm. Since either assumption breaks all our crypto for other, unrelated reasons, we use /dev/urandom.
That, and if you use /dev/urandom before it's seeded, you expose yourself to real attacks.
Excuse me for asking a stupid question, I am not too deep into linux kernel randomness generation:
Why is /dev/urandom spitting out anything before it has acquired enough entropy for the initial seed? Wouldn't it be a good idea for it to initially block?
Yes. But it would break userspace to change it now (because some init script might block trying to read /dev/urandom)
The contract when /dev/random and /dev/urandom came out was that urandom would never, ever block.
On a system with a recent Intel processor, there's a instruction (RDSEED) that uses on on-die hardware RNG. I'm not familiar with the standard linux boot-up process, but it could in principle seed urandom using RDSEED arbitrarily early in the process. That should work on VMs too unless the hypervisor is blocking access (can't imagine a good reason for that).
Via has on-die RNG considerably longer, though it's accessed slightly differently. I don't believe AMD or ARM has anything similar.
You make it sound like you are saying something new in your first couple paragraphs, but you aren't: the initial entropy gap issue seems to be exactly what klodolph was talking about. You hedge with "not really focused on you", but I'm going to say that you probably then have just not posted most of your post, as even if none of it were related to klodolph (which isn't the case anyway, due to the quoting of "too loose") it is still written in a way as to make people believe he made the mistake you are trying to be pedantic about :(. Just because many, even most, people make a particular mistake does not mean everyone does. Note, very carefully, the wording "at some point": and it wasn't then "until it runs out", it was "just keep on spitting out". To some people, this is a real issue stemming from a completely unreasonable kernel default that the developers of libraries and applications have absolutely no control over, which occasionally comes up on computers built with custom distributions by non-experts under the assumption "Linux will do something sane", and which in a prior thread we found actual Hacker News users who had been first burned and then didn't have good options for a fix as it was a customer's computer. I think if you would be willing to be more open to the premise that not everyone is wrong in trivial, predictable ways, you'd be surprised by how often they aren't :/. Your last paragraph is great, and I'm really glad you contributed it, but the first two were unwarranted.
If he's mistaken, simply point out the mistake. There's no reason to make this conversation personal.
Is there a specific quote that isn't correct?
I described in my comment how the comment by tptacek is not correct; I can state again: it "corrects" klodolph's comment, and yet klodolph's comment was correct and did not deserve correction. tptacek tries to use this comment as an example, turning to the side to address the audience to voice this correction towards everyone with the "not really focussed on you" hedge, but uses the "not too loose" quoting to make it clear that klodolph is still the example.
This particular way in which tptacek's comment is "mistaken" is related to the tone and attitude tptacek takes in his comments. If this were the only instance, even the only instance for this particular topic, it would be one thing, but this is a common issue with tptacek's comments. The pattern is that there are certain "common misconceptions" that everyone has, and tptacek is not generous to the poster of any specific comment that they might not possess them.
I further believe that it is a serious problem on Hacker News that people do not address these kinds of tone issues: that it is perfectly fine to "HEADDESK", claim that certain beliefs are "virulent", and to even use the word "nonsensical" to describe someone's idea, and yet attempts to point out issues in the tone of peoples' comments is somehow a problem: something where you feel the need to say "don't make this conversation personal". More people need to stand up to this.
I, myself, have done some of these things in my own comments. I feel like most of these cases were situations where I was responding to someone else doing it to me, but I've found at least a few instances where that is not the case. It makes me very unhappy that I contributed to this problem: someone should have also complained about my tone in those instances. It needs to be ok to exit the topic and address how someone is saying something, not just what was said.
To be clear, tptacek's position is correct: we don't need a new interface; klodolph's argument largely ends up arguing for fixing /dev/urandom. However, tptacek doesn't say this, as he has assumed that klodolph doesn't understand the difference between /dev/urandom and /dev/random, and then argued based on that assumption. If you read the other comments from klodolph, it is very very clear that he understands perfectly. tptacek could at least apologize.
I don't think 'klodolph took offense. If he did, I would feel bad, and would certainly apologize. For now, I'm going to presume he read my comment in the spirit it was intended: that the virulent, nonsensical idea I was describing was not being attributed directly to him, hence the disclaimer at the top of the comment.
(If you re-read my comment, you'll also find that the >HEADDESK< isn't addressed to 'klodolph at all, but to the designers of the Linux randomness system call).
You could probably mail me further thoughts you have about my comments on HN, if you wanted to keep talking about it.
We are commenting on a thread about the Linux kernel random developers making exactly the mistake you think I should assume people won't so readily make.
To add to tptacek's excellent comment, it's important to remember that some of the characteristics of /dev/random and /dev/urandom discussed here only apply to Linux; not other operating systems such as Solaris:
https://blogs.oracle.com/darren/entry/solaris_random_number_...
I'm certain OpenBSD differs here as well.
On OpenBSD, /dev/{a,s,u,}random all do the same thing: pour endless nonblocking chacha20 keystream on you.
NSA likes this. Only one random and entropy code to compromise.
As opposed to putting a CSPRNG in userspace, in which case they only need to compromise either of two CSPRNG codebases?
As opposed to have everybody implement his/her own random()/entropy().
...the NSA would love that most of all. The vast majority of programmers lack the knowledge, skill, or both required to correctly implement anything related to cryptography.
"The vast majority of programmers lack the knowledge, skill, or both required to correctly implement anything."
Agreed.
Who's "everybody?" If you mean userspace application/library developers, they don't have a good source of entropy, so they have to get it from the kernel. That means userspace CSPRNGs end up depending on the kernel CSPRNG. Presto, two single points of failure!
I see, so prior to this random() and entropy() addition to Linux nobody could ever write any security related application. Yes, this is true. We should change the pull request to "Enable security for Linux, 2014". God, never too late...
So looks like on older kernels they can keep using the kernel.random.uuid sysctl, and on newer kernels this new syscall ... nice!
The third paragraph in the DESCRIPTION section of the manpage contains a lot of errors:
- "then the /dev/raundom pool will be used" -> should be /dev/urandom
- "Unlike reading from the /dev/urandom" -> author meant /dev/random
- "EGAIN" -> should be EAGAIN
- In fact the entire sentence "Unlike reading from /dev/random, [it either blocks or returns EAGAIN]" should be removed. This blocking/EAGAIN behavior is the same regardless if you read from random or urandom.
> - "Unlike reading from the /dev/urandom" -> author meant /dev/random
Pretty sure he doesn't. He's describing the differences between using the /dev/urandom pool (through getrandom(2)) and actually reading from /dev/urandom.
I see, this one makes sense.
> The /dev/random pool is limited based on the entropy that can be obtained from environmental noise, so if there is insufficient entropy, the requested number of bytes may not be returned. If there is no entropy available at all, getrandom(2) will either return an error with errno set to EAGAIN, or block if the GRND_BLOCK flags bit is set.
Man, that's an ugly interface. So, if there are 8 bits of entropy available and I ask for 16, it will return 8, but if there are none available and I ask for 16 it will block and then return 16.
Also, it's not possible to know how much entropy is available. I'd really like to emphasise that: entropy estimation is nonsense.
Don't block; seriously, don't block. Just spit out the output of a properly-seeded PRF (e.g. AES in CTR mode).
> Don't block; seriously, don't block. Just spit out the output of a properly-seeded PRF (e.g. AES in CTR mode).
/dev/urandom does that and it is the suggested way to get secure random numbers for 99% of the applications.
No, /dev/urandom fails the 'properly-seeded' criterion during early boot.
The real answer is to just change the interface and be done with it, and to hell with backwards-compatibility, but that'll never happen.
The really real answer is to make /dev/random and /dev/urandom the same CSPRNG, seeded as early in boot as possible (and continually refreshed with more entropy, of course), and never blocking thereafter, but that'll never happen either.
"Any userspace program which uses this new functionality must make sure that if it is used in early boot, that it will not cause the boot up scripts or other portions of the system startup to hang indefinitely."
How can the userspace program "make sure that it doesn't hang indefinitely" if it is started early in the boot process and the call blocks?
It can use the expect() system call and timeout / fail after a reasonable period of time if no bits are returned.
expect() is not a system call.
Sorry, I meant select(). Don't know where my head was when I typed that.
Can you select() a system call (the new proposal) instead of an fd (the old interface)?
You don't get a file handle to a syscall. You just load up some registers/stack and call a `SYSENTER`, `SYSCALL` or `INT 0x80` instruction.
select is itself a separate syscall that takes in file handles or something.
Yeah, I know, but it wasn't clear if the parent comment was referring to select() on /dev/random, or the new syscall interface.
There's a flag that determines whether it blocks or just returns an EAGAIN error. A program can use it in non-blocking-ish mode and try again later.
Proper link that lets you see the discussion:
https://lkml.org/lkml/2014/7/17/145
In the synopsis, it says "It should not be used [by] Monte Carlo simulations or for other probabilistic sampling applications." Does anyone know why not?
Probably because it will be slow and deplete the entropy pool. Statistical sampling doesn't require cryptographically-secure random number generation.
"deplete the entropy pool" isn't really a meaningful phrase given the way the Linux /dev/{random,urandom} work. AFAIK /dev/random and /dev/urandom use the same CSPRNG (different instances) which is periodically reseeded with entropy gathered from the system (hw, interrupt timings, etc.), the only difference being that the /dev/random device blocks if an entropy estimator says that there's "too little entropy".
It's not that /dev/random has been "depleted" (which is meaningless given how a CSPRNG works), it's just that some entropy estimation code thinks that it might not be a source of perfect entropy/randomness at a particular time. (This practice is pretty questionable, hence the proposed new syscall.)
This. It'd just be silly to use a slow CSPRNG for that.
With this, entropy pool depletion is not really that much of a concern; "urandom reads" from getrandom(2) will have at least one good seed.
At a guess, those applications routinely require large amounts of mostly random (not CSPRNG random) numbers. This syscall returns data from the kernel's entropy pool, presumably at the speed of /dev/urandom.
The recommendation is probably a way to say "please don't abuse, and please don't make the kernel spin".
Here are some guesses about the reasoning for this admonition:
* Suppose /dev/urandom is not receiving randomness at all, and the Monte Carlo application keeps drawing bits from it like crazy. Then this device is reduced, effectively, to a PRNG. Maybe, as such, it is not a particularly good PRNG; it is not intended to be used that way. It is supposed to take some real entropy bits, and "stretch" them to cover a larger request---but not ridiculously so.
* Simulation applications benefit from repeatability: the same seed fetches exactly the same results. You lose that with urandom.
Monte Carlo simulations need to be repeatable. So you can take a seed from entropy, but then you seed your own deterministic RNG. You log the seed so that you can backtest.
Doesn't appear many people on this thread are considering such use cases, but it appears the authors have.
I stumbled across a possible answer to this question just now in a discussion regarding a proposed interface for a randomness call where the callee would declare what they are using the randomness for - with Monte Carlo as one of the options. I immediately recalled your comment, so, here is your answer:
> I would hope that people would agree there should be a difference between what is needed for a Monte Carlo simulation and other various cryptographic use cases. (I've gotten bug reports from people who insisted on using /dev/urandom for Monte Carlo simulations, and who then complained when it was too slow for their purposes....)
https://www.ietf.org/mail-archive/web/dsfjdssdfsd/current/ms...
> If the GRND_RANDOM flags bit is set, then draw from the /dev/random pool instead of /dev/urandom pool. The /dev/random pool is limited based on the entropy that can be obtained from environmental noise, so if there is insufficient entropy, the requested number of bytes may not be returned. If there is no entropy available at all, getrandom(2) will either return an error with errno set to EAGAIN, or block if the GRND_BLOCK flags bit is set.
> If the GRND_RANDOM flags bit is not set, then the /dev/raundom pool will be used. Unlike reading from the /dev/urandom, if the urandom pool has not been sufficiently initialized, getrandom(2) will either return an error with errno set to EGAIN, or block if the GRND_BLOCK flags bit is set.
Is this complexity necessary? I'm concerned that this will be hard to use properly as there's lots of different behaviours based on the parameters, and different types of failure modes. Security code that is hard to call right or hard to understand tends to get misused, causing security bugs in application code.
Yay! However, a few things that are a bit unfortunate:
- It suggests that I use that entropy to seed a PRNG. No. It should be the PRNG. - Fill the buffer with "up to buflen random bytes"? I asked for n bytes. Please give me actually n bytes. If there's a way to get this wrong, people will get it wrong: the BSD socket API does the same thing and at least it has good reason. However, the wording seems to imply that if you use the "urandom" source, it won't do that? - "It should not be used Monte Carlo simulations or for other probabilistic sampling applications.". Doesn't mention why. Sounds like "run out of entropy" FUD :-(
Its more an example of using the wrong tool for the job. You don't use a sludge hammer to hammer a nail in. Monte Carlo simulations don't need CSPRNG.
Slightly off-topic, but I couldn't help but notice as a programmer. I find the coding style of the patch kind of random. The if statements, some of them have braces and some don't. Without a pattern it seems.
What is this style called?
I only see one place where an if statement has curly braces where it doesn't need them. The rest of them only have curly braces if they're necessary.
Oh right, I see the second if statement at block @@ -1533,6 +1544,26 @@ indeed threw me off. I would value consistency over necessity. The else-if at the same block looks weird and the use of new lines could also help the code to be more legible and maintainable.
Anyway, sorry went off-topic.
Linux Kernel Coding Style - https://www.kernel.org/doc/Documentation/CodingStyle
The rationale for this system call is poor. If you're out of descriptors, so that you cannot open another one, then bail loudly! Do not fall back on some ill-conceived code that uses poor quality pseudo-random bits in a situation where such bits must not be used!
Also, an descriptor to /dev/urandom can be acquired early and retained for the life of the process. When you need bits, you just read from it. This gets around subsequent chroots and descriptor exhaustion.
You missed the 3rd reason for this. Calls to /dev/urandom can not block, so it a script or program requests random bytes before entropy has been set up, it can get back bad random bytes. The new system call getrandoms does block, so early scripts will not get bad entropy data.
The newly proposed system call, as implemented in the mailing list patch, provides the functionality of both random and urandom: the behaviors are distinguished by a bitmask.
Take a look at the code. If the flag GRND_RANDOM is used, then it uses the random device, otherwise urandom.
Independently of this, blocking behavior is requested with GRND_BLOCK. If this is omitted, then the random device bails with errno == EAGAIN if it doesn't have enough entropy, otherwise it blocks.
If GRND_BLOCK is omitted with the urandom method (GRND_RANDOM is omitted), then it will bail with -EAGAIN if the urandom device is not initialized; otherwise it just calls urandom_read. With GRND_BLOCK, it will block for urandom to initialize, if necessary.
The new system call only blocks when given GRND_BLOCK, and it uses urandom unless given the flag GRND_RANDOM. If it is given GRND_RANDOM, but not GRND_BLOCK, and not enough entropy is available, then the system call with bail with errno == EAGAIN. If there is no GRND_RANDOM then it falls back on urandom_read: that is usually non-blocking.
Sure, blocking is a flag. You're ignoring the point, which is that it will never return bad data.
I see; if the point is that /dev/urandom doesn't even block when it is uninitialized, that is valid.
This system call (when used to access urandom) will either block on urandom to be initialized, or else fail loudly with -EAGAIN, which is an improved interface.
How do you know it is not random? http://dilbert.com/dyn/str_strip/000000000/00000000/0000000/...
Does anyone have any idea why Linux has a distinction between /dev/random and /dev/urandom?
Legacy adherence to some defunct interface? Ignorance on the part of kernel devs (seems unlikely)?
I think its because linux got there first and designed these interfaces
To quote Ted Tso (author of the above patch and the original writer of /dev/random in the first place)
The two other interfaces are two character devices /dev/random and * /dev/urandom. /dev/random is suitable for use when very high * quality randomness is desired (for example, for key generation or * one-time pads), as it will only return a maximum of the number of * bits of randomness (as estimated by the random number generator) * contained in the entropy pool. * * The /dev/urandom device does not have this limit, and will return * as many bytes as are requested. As more and more random bytes are * requested without giving time for the entropy pool to recharge, * this will result in random numbers that are merely cryptographically * strong. For many applications, however, this is acceptable.
Awesome. Thanks, Theo!
So, does this mean that security issues are causing the end of the Unix "everything is a file" philosophy?
That hasn't really been true for a very long time, especially in the area of X11.
X11 has always been a bit of a weird beast. And things like /proc are definitely in everything-is-a-file tradition, so it wasn't quite dead. (Maybe just pining for the fjords?)
Here is a problem:
/dev/random and /dev/urandom have security attributes: ownership and permission. The newly proposed syscall makes no security checks whatsoever to protect the entropy resource: it punches a hole through these perms, effectively.
What's the security case for locking down /dev/random?
A security case for protecting a source of random bits is that they are important and that cryptographic applications and middleware can malfunction if these bits are not available. A world-readable /dev/random means that anyone can drain out random bits, making them unavailable to these applications.
Someone somewhere could be setting up permissions in their Linux-based system based on this type of reasoning.
It's a security regression if a character device can be suddenly accessed through a back-channel that ignores permissions.
AFAIK isn't this the kind of thing SELinux can take care of? There needn't be any security regression with this change.
and seccomp
If you don't trust a CSPRNG, then you pretty much might as well turn off all your crypto because you can't trust it.
If you can trust a CSPRNG, then you should be able to trust urandom, as long as it's been properly initialized once. It won't drain out. It doesn't run out of bits. One application reading it doesn't make it unavailable for others. The same is true of the syscall.
/dev/random is a CSPRNG with finite output over a fixed time period, so it is susceptible to denial-of-service attacks.
Which is why secure software doesn't use /dev/random, and prefers instead urandom.
Maybe on Linux, but as I pointed out somewhere else; this doesn't apply to all *NIXes with a /dev/random or /dev/urandom:
Short version is, on Solaris, you should use /dev/random for long-term keys and other high-value keying material. /dev/urandom should be used for everything else such as less demanding or shorter-term cryptographic uses like short-term session keys, paddings, and challenge strings.
I have two concerns with this comment:
1. It's not safe to use a weak RNG for session keys or even padding. Cryptosystems often fail catastrophically when their most obscure parameters are weak --- in fact, more exploitable cryptosystem flaws derive from issues like padding and IVs than keys.
2. FIPS 186-2 is a SHA-based DRBG. It doesn't "run out of entropy". What am I missing? The situation on Solaris w/r/t urandom appears to be identical to that of Linux.
1. The Solaris crypto designers / developers believe their rng is is appropriate for short-term session keys and padding. Read the post I linked before as to why the believe that.
2. /dev/random basically only uses bits from entropy sources while /dev/urandom just uses a few bits from /dev/random as a seed. Read the post I linked again.
Again, I would reiterate, the situation is somewhat different for Solaris. Usage of /dev/random vs. /dev/urandom is actually considered different.
I read this post, both when you linked to it just now, and, prior to that, while I was researching the urandom blog post I wrote a few months ago. I'm being repetitive, but:
* I think the distinction this post is drawing between Solaris random and urandom is functionally very similar to the distinction between Linux random and urandom (I think the distinction on Solaris might be that random has a higher quality rekeying system than urandom).
* I can read a straightforward description of a SHA1 PRNG keyed by kernel entropy, which is what this blog post describes, and unless I'm missing something in the article, I don't see how it is any less cryptographically secure than Solaris /dev/random. It seems to be less forward-secure than /dev/random. That's about it.
Since you appear to have grokked this article better than I did in either of the two cases I read it, could you give a short summary of the argument it presents for avoiding urandom for long-term secrets?
1) You're not completely off here; the primary difference is currently higher quality re-keying for /dev/random, but there are implementation differences.
2) From a cryptography standpoint, the precise implementation of the current framework makes /dev/urandom suitable for short-term session keys and padding. This may not hold true for Linux, but for Solaris it has been reviewed and deemed appropriate usage. Fundamentally, it's about constraints placed on the implementation that currently make /dev/random more "secure".
In conversing with one of the crypto developers for Solaris, they are considering whether to lessen the distinction between /dev/random and /dev/urandom further in the future.
I think the concern is that a process could then exhaust all available entropy, and DOS any other processes depending on /dev/random. Currently, a chroot could protect against that.
Yes, and it is exactly that chroot that is cause for concern and why this system call is being introduced...
Consuming all the entropy. But its not a good argument. The BSDs manage and Linux has syscall controls.
And take a look at those permissions on your machine and see what they show for you:
crw-rw-rw- 1 root root 1, 8 Jul 15 11:27 /dev/random
crw-rw-rw- 1 root root 1, 9 Jul 15 11:27 /dev/urandom
So, if the default permission is read/write by any user, what limitation should this new syscall add?
Oh the joy of NIH.
I didn't understand what you meant by you commentary.
The above excerpt shows they assuming upfront that they got the idea from BSD. As a system call is not an application that may be ported, they need to recode it.
If the point was the name change, I guess this is only to keep consistency within the system, and far from a problem.
They took the sane design from BSD ("here's a single system call that gives you crypto-safe entropy") and effectively made a family of system calls that replicate all the pointless drama of /dev/random and urandom. The NIH criticism is valid.
It seems pointless indeed. On the other hand, I'm almost glad the overengineering only went that far. Actually I'm expecting much worse after reading Theodore's ideas on the IETF list.. we'll see what kind of a library interface they'll come up with, if they make something besides getentropy().
From https://www.ietf.org/mail-archive/web/dsfjdssdfsd/current/ms...
What is a case where a random IV needs a different entropy source than a session key?
And if we're assuming the entropy pool is being compromised (full state read by attacker) from time to time, isn't it foolish to be generating keys on such a machine? Why would new state not be compromised in he same way the previous state was? I understand the system design may want to provide a robust RNG, but further than that seems slightly pointless.
If they are not able to get the exact semantics of the getentropy call, then a namechange is perfectly justified.
if your job is providing security and GOOD security relies on GOOD entropy i fail to see the issue...
Keep reading the thread - Ted explains that decision.
Oh the joy of not reading the thread.
> The getrandom(2) system call is a superset of getentropy(2).
> The reason for the additional flags is that I'm trying to solve more problems than just getentropy()'s raison d'etre.
And you can implement getentropy(2) on top of getrandom(2) if you so desire, which they plan to do in glibc.
http://lists.openwall.net/linux-kernel/2014/07/17/682
Why? So in future they can complain they can't control the entropy sources?