Opened 8 years ago

Closed 8 years ago

Last modified 8 years ago

#646 closed Bug / Defect (notabug)

UDP server with a static key fail to start since update to a version 2.3.9

Reported by: LUTi Owned by: Steffan Karger
Priority: major Milestone: release 2.3.11
Component: Generic / unclassified Version: OpenVPN 2.3.8 (Community Ed)
Severity: Not set (select this one, unless your'e a OpenVPN developer) Keywords:
Cc:

Description

After reverting the src/openvpn/crypto.c patch from 28 Nov 2015:

replace
ASSERT (rand_bytes (output, len));

with
rand_bytes (output, len);

the server starts (almost) normally again, but with a lot of

RAND_bytes() failed

error logs.

RAND_bytes (output, len) is returning 0 value all the time. I am able to connect to the server (2.3.8 and my back-patched 2.3.9 / 2.3.10) from the remote client.

Tested on CentOS 5 and CentOS 6 (EPEL openvpn) with the same result (also with the yesterday's 2.3.10 version). It seemd to worked flawlessly until 2.3.8.

My server configuration is:

local <my.public.ip>
dev tun0
ifconfig 192.168.250.1 192.168.250.2
secret static.key
port 1194
proto udp
comp-lzo
ping 15
ping-restart 45
ping-timer-rem
persist-tun
persist-key

Static key has been generated with:
openvpn --genkey --secret static.key
according to the documentation.

Should we get to this function - prng_bytes () - with a static key at all? Or, is it just about my configuration setting?

Change History (21)

comment:1 Changed 8 years ago by Gert Döring

Owner: set to Steffan Karger
Status: newassigned

RAND_bytes() should never fail... throwing this at syzzer who might have more ideas.

Which version of OpenSSL is this? Does it happen if you build with PolarSSL?

comment:2 Changed 8 years ago by Steffan Karger

Indeed, RAND_bytes() should never fail. prng_bytes() is used by openvpn_encrypt() to generate IVs in CBC mode (which you are using, since you are using static key mode). IVs in CBC mode should be unpredictable, and therefore the random used for those IVs should be good. If rand_bytes() is failing, something is really wrong.

To check that I'm not overlooking something for static key mode, I just tested 2.3.10 in static key mode on my workstation (Ubuntu 15.10, OpenSSL 1.0.2d) - and it works for me.

Answers to the questions by cron2 would be very interesting. What would also be interesting is to know what exactly RAND_bytes() returns. Can you replace rand_bytes() in your crypto_openssl.c with the following, and report back what it logs?

int rand_bytes(uint8_t *output, int len) 
{
  int ret = RAND_bytes (output, len);
  if (unlikely(1 != ret))
    {
      msg(D_CRYPT_ERRORS, "RAND_bytes() failed with %d", ret); 
      return 0;
    }
  return 1;
}

comment:3 Changed 8 years ago by LUTi

Libraries versions (from log file, logged by OpenVPN at startup):

@ CentOS 6: OpenSSL 1.0.1e-fips 11 Feb 2013, LZO 2.03

@ CentOS 5: OpenSSL 0.9.8e-fips-rhel5 01 Jul 2008, LZO 2.06

I will try to build with PolarSSL (don't use it at the moment).

I have already done the test about RAND_bytes() returned value - it is 0.

comment:4 Changed 8 years ago by LUTi

BTW - all of the tests during build are passed successfully.

comment:5 Changed 8 years ago by Steffan Karger

Sorry about the RAND_bytes() return question - I somehow read 'rand_bytes() returns 0', instead of 'RAND_bytes() returns 0' in the OP.

This means that OpenSSL does have a rand method, but it (at that time) is unable to properly use it. Are the openssl error codes / messages more informative? Could you add:

      int err;
      while ((err = ERR_get_error ()))
        {
          msg (D_CRYPT_ERRORS, "OpenSSL error (%d): %s", err,
              ERR_error_string (err, NULL));
        }

inside the if-clause of rand_bytes()?

Edit: also print error code, not just string.

Last edited 8 years ago by Steffan Karger (previous) (diff)

comment:6 Changed 8 years ago by LUTi

About building with PolarSSL:

I've installed mbedtls-1.3.14-1.el6.x86_64 & mbedtls-devel-1.3.14-1.el6.x86_64
and tried to build with:

./configure --with-crypto-library=polarssl

but am getting:
"configure: error: ssl is required but missing"

("/usr/bin/ld: cannot find -lpolarssl" in config.log).

There are only:
/usr/lib64/libmbedtls.so
/usr/lib64/libmbedtls.so.1.3.14
/usr/lib64/libmbedtls.so.9

in my lib dir.

I am going to rebuild with the latest TEST patch as above now, and report.

comment:7 Changed 8 years ago by LUTi

Server start log with OpenSSL error:

Jan 6 11:34:06 test openvpn[20849]: OpenSSL error (604389476): error:24064064:random number generator:SSLEAY_RAND_BYTES:PRNG not seeded
Jan 6 11:34:06 test openvpn[20849]: RAND_bytes() failed with 0

comment:8 Changed 8 years ago by LUTi

Successfuly rebuilt with PolarSSL (creating symlinks libpolarssl.so -> libmbedtls.so and libpolarssl.a -> libmbedtls.a in /usr/lib64) - mbedtls-1.3.14-1.el6.x86_64, mbedtls-devel-1.3.14-1.el6.x86_64, mbedtls-static-1.3.14-1.el6.x86_64 and mbedtls-utils-1.3.14-1.el6.x86_64 EPEL CentOS 6 packages.

Fails as well:

OpenVPN server start log:
Jan 6 12:50:36 test openvpn[5894]: library versions: PolarSSL 1.3.14, LZO 2.03
...
Jan 6 12:50:36 test openvpn[5895]: UDPv4 link local (bound): [AF_INET]192.168.0.10:1194
Jan 6 12:50:36 test openvpn[5895]: UDPv4 link remote: [undef]
Jan 6 12:50:36 test openvpn[5895]: Failed to initialize random generator
Jan 6 12:50:36 test openvpn[5895]: Exiting due to fatal error
Jan 6 12:50:36 test openvpn[5895]: Closing TUN/TAP interface
...

Are you sure the check failing shall not be expected in my case (a static key file)?

Since there is a statement:
"Failing to check the return value occurs if no prng is used (i.e. in static key mode, or when explicitly disabled using --prng none). prng_bytes() is used for generating IVs, session IDs and filenames."
at http://article.gmane.org/gmane.network.openvpn.devel/10636/

comment:9 Changed 8 years ago by Steffan Karger

I will need to look at this more later on, but it looks like openvpn (through openssl/polarssl) can not read from /dev/urandom. Could that be the case?

(This also reminds me I really need to send more of my 'improve debug logging' patches to the list...)

comment:10 Changed 8 years ago by LUTi

I've ran:
strace -xe trace=file,read,write,close openssl rand 10

as root, and it seemd to succeed:
[root@test ~]# strace -xe trace=file,read,write,close openssl rand 10
execve("/usr/bin/openssl", ["openssl", "rand", "10"], 28 vars */) = 0
access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory)
open("/etc/ld.so.cache", O_RDONLY) = 3
close(3) = 0
open("/lib64/libssl.so.6", O_RDONLY) = 3
read(3, "\x7f\x45\x4c\x46\x02\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x03\x00\x3e\x00\x01\x00\x00\x00\xb0\x35\xa1\x04\x3e\x00\x00\x00"..., 832) = 832
close(3) = 0
...
open("/dev/urandom", O_RDONLY|O_NOCTTY|O_NONBLOCK) = 3
read(3, "\xfe\xa7\x2c\x29\x2c\xf6\xe9\xb8\x52\xf6\x2e\x2a\x6b\x0f\xbd\x38\x64\x5d\x70\x90\xad\x5c\xd2\x76\x44\xb5\x99\x37\xee\xc1\x51\x06"..., 48) = 48
close(3) = 0
write(1, "\xae\x59\xee\x4b\x02\xc6\x4b\xf2\xac\x91", 10YKK) = 10
stat("/root/.rnd", {st_mode=S_IFREG|0600, st_size=1024, ...}) = 0
open("/root/.rnd", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 3
chmod("/root/.rnd", 0600) = 0
write(3, "\xf5\x28\x24\xef\x21\x2c\xfa\x45\x7e\x9d\x6f\x90\x96\xa6\xbe\x00\xb8\xde\xa9\xd8\xb6\xcf\xd5\xd6\x35\xa7\x43\xc6\xd2\x2f\x7c\xa1"..., 1024) = 1024
close(3) = 0

There is the following in my config (what I've forgotten to copy yesterday above):
user nobody
group nobody
chroot /etc/openvpn/chroot

Commenting out the last line (chroot) seems to resolve the issue.

So, it looks like OpenVPN (through OpenSSL / PolarSSL) can not access /dev/urandom in chrooted environment.

Creating:
mknod random c 1 8
mknod urandom c 1 9
in /etc/openvpn/chroot/dev

seems to (more or less) resolve the issue.

More or less - there are new kernel logs now:
Jan 6 15:51:13 test kernel: type=1400 audit(1452091873.627:14): avc: denied { read } for pid=3872 comm="openvpn" name="urandom" dev=sda8 ino=3062318 scontext=unconfined_u:system_r:openvpn_t:s0 tcontext=unconfined_u:object_r:openvpn_etc_t:s0 tclass=chr_file
Jan 6 15:51:13 test kernel: type=1400 audit(1452091873.627:15): avc: denied { open } for pid=3872 comm="openvpn" name="urandom" dev=sda8 ino=3062318 scontext=unconfined_u:system_r:openvpn_t:s0 tcontext=unconfined_u:object_r:openvpn_etc_t:s0 tclass=chr_file
Jan 6 15:51:13 test kernel: type=1400 audit(1452091873.627:16): avc: denied { getattr } for pid=3872 comm="openvpn" path="/dev/urandom" dev=sda8 ino=3062318 scontext=unconfined_u:system_r:openvpn_t:s0 tcontext=unconfined_u:object_r:openvpn_etc_t:s0 tclass=chr_file
Jan 6 15:51:13 test kernel: type=1400 audit(1452091873.627:17): avc: denied { ioctl } for pid=3872 comm="openvpn" path="/dev/urandom" dev=sda8 ino=3062318 scontext=unconfined_u:system_r:openvpn_t:s0 tcontext=unconfined_u:object_r:openvpn_etc_t:s0 tclass=chr_file

which seem to be selinux (which I have set to permissive, so not such a big deal for my case) related though.

In any case, it seems that OpenVPN / OpenSSL (PolarSSL) has some issue with the chrooted environment (which doesn't seem to be properly initiated), as it is now.

comment:11 Changed 8 years ago by LUTi

It would probably make sense to include the SSL Error with a RAND_bytes() failed (and, that one with the returned value) log however.

comment:12 in reply to:  11 Changed 8 years ago by Gert Döring

Replying to LUTi:

It would probably make sense to include the SSL Error with a RAND_bytes() failed (and, that one with the returned value) log however.

Definitely :-) @syzzer: on standby for ACK+merge.

Won't make 2.3.10 though.

comment:13 Changed 8 years ago by Gert Döring

Milestone: release 2.3.9release 2.3.11

comment:14 Changed 8 years ago by Steffan Karger

Resolution: notabug
Status: assignedclosed

Logging improvement patches are on the list!

As for this ticket, yes, this definitely was the unavailable /dev/urandom. And this is exactly why we need ASSERT()s at these places in the code. With older version, you were just using no / bad random. Fortunately for you, the OpenVPN CBC packet format is such that you do not immediately loose all your security (due to the not-attacker-controller timestamp + packet id, as described in http://article.gmane.org/gmane.network.openvpn.devel/10636/). Still, CBC needs good random, so make sure to give it to openssl/polarssl :)

Since this was not a bug in OpenVPN, and the patches are on the list, I will close this ticket.

comment:15 Changed 8 years ago by LUTi

I believe that, considering OpenVPN is somehow "responsible" for its own chroot environment, this ticket should still be considered as a bug. The fact is that user does everything according to the documentation (with options supported / available), but OpenVPN fail to start successfully.

I think OpenVPN should take care about providing /dev/urandom (and, /dev/random, if needed) if chroot option is considered as an option in configuration with a static key file.

Or, to include at least some check, and provide the warning / instructions about what to take care about manually. Or, not to support the use of chroot and a static key together?

comment:16 Changed 8 years ago by LUTi

OpenSSL and PolarSSL in my normal environment have a good random available (therfore also all of the test during build process pass). Just not in the OpenVPN's (or, any other) chrooted environment.

Or, should they take care (by themselves) to have /dev/urandom available in the chrooted environment (some other package is establishing / use)?

comment:17 Changed 8 years ago by Gert Döring

Well, the point of a chroot environment is that programs cannot access stuff that is not in there - so they cannot "take care of missing /dev/urandom" either (because they can't create it, at least not without privileges plus lots of system dependent code to know how the device needs to look like - which will differ between Linux, FreeBSD, MacOS, etc.)

What we definitely need to do is provide a proper error message and not just ASSERT() out - patches to that extent have hit the list and are being reviewed.

comment:18 Changed 8 years ago by LUTi

/dev/urandom shall be exactly there!

As above, OpenSSL in case of chroot don't need to access the system /dev/urandom, but seems to need a chrooted one - exactly this is a problem in my case!

My system /dev/urandom is working just fine all of the time, therfore the build process tests have all succeeded (and, the server starts just fine if not in the chroot as well).

If OpenVPN can access and make use of the chrooted environment, I guess it shouldn't be too difficult also to create /dev/urandom, if not there already. I've created the missing device (in linux) simply launching "mknod urandom c 1 9" in the chrooted dev directory (that I had to create manually as well).

This at least prevents my OpenVPN server to fail to start (I don't know the usefulness of such a device for OpenSSL / the quality of the randomness provided with it though).

Last edited 8 years ago by LUTi (previous) (diff)

comment:19 Changed 8 years ago by Steffan Karger

In most cases /dev/urandom should indeed be there. However, that is not the responsibility of openvpn, but the responsibility of the one setting up the chroot. OpenVPN simply can not decide for you and/or the crypto library whether it needs to create it, nor is OpenVPN always run with sufficient privileges to create it.

However, I do agree that this should be documented better. The man page now contains a note about polarssl needing /dev/urandom in the chroot, but fails to mention that openssl does too. I'll cook up a patch to improve the man page.

comment:20 Changed 8 years ago by LUTi

Thank you, syzzer.

Maybe it would be worth to include somewhere the instructions about how exactly to prepare chroot, or even a simple script (to make the life of users easier)?

To summarize - I've needed to create the chroot directory, and within that:
/tmp directory
/dev directory
/dev/urandom device (I've created also /dev/random, just in case...)

The script (to be ran within the directory where chroot shall reside) could be something like (for linux):
mkdir chroot
cd chroot
mkdir tmp
mkdir dev
cd dev
mknod random c 1 8
mknod urandom c 1 9

(if there are some particular privileges required, the script could include that as well).

comment:21 Changed 8 years ago by Steffan Karger

Sure, better instructions are very much welcome. Feel free to write a wiki article (e.g. on https://community.openvpn.net/openvpn/wiki/chroot), or send a patch to update the manpage and/or add a script to contrib/.

Note: See TracTickets for help on using tickets.