unchecked malloc may allow unauthed client to crash Xserver [CVE-2014-8091]

authdes_ezdecode() calls malloc() using a length provided by the
connection handshake sent by a newly connected client in order
to authenticate to the server, so should be treated as untrusted.

It didn't check if malloc() failed before writing to the newly
allocated buffer, so could lead to a server crash if the server
fails to allocate memory (up to UINT16_MAX bytes, since the len
field is a CARD16 in the X protocol).

Reported-by: Ilja Van Sprundel <ivansprundel@ioactive.com>
Signed-off-by: Alan Coopersmith <alan.coopersmith@oracle.com>
Reviewed-by: Peter Hutterer <peter.hutterer@who-t.net>

os/rpcauth.c

@@ -66,6 +66,10 @@ authdes_ezdecode(const char *inmsg, int len)
     SVCXPRT xprt;
 
     temp_inmsg = malloc(len);
+    if (temp_inmsg == NULL) {
+        why = AUTH_FAILED; /* generic error, since there is no AUTH_BADALLOC */
+        return NULL;
+    }
     memmove(temp_inmsg, inmsg, len);
 
     memset((char *) &msg, 0, sizeof(msg));