glx: Add safe_{add,mul,pad} (v3) [CVE-2014-8093 4/6]

These are paranoid about integer overflow, and will return -1 if their
operation would overflow a (signed) integer or if either argument is
negative.

Note that RenderLarge requests are sized with a uint32_t so in principle
this could be sketchy there, but dix limits bigreqs to 128M so you
shouldn't ever notice, and honestly if you're sending more than 2G of
rendering commands you're already doing something very wrong.

v2: Use INT_MAX for consistency with the rest of the server (jcristau)
v3: Reject negative arguments (anholt)

Reviewed-by: Keith Packard <keithp@keithp.com>
Reviewed-by: Julien Cristau <jcristau@debian.org>
Reviewed-by: Michal Srb <msrb@suse.com>
Reviewed-by: Andy Ritger <aritger@nvidia.com>
Signed-off-by: Adam Jackson <ajax@redhat.com>
Signed-off-by: Alan Coopersmith <alan.coopersmith@oracle.com>
(cherry picked from commit 2a5cbc17fc)
Signed-off-by: Julien Cristau <jcristau@debian.org>

glx/glxserver.h

@@ -228,6 +228,47 @@ extern void glxSwapQueryServerStringReply(ClientPtr client,
  * Routines for computing the size of variably-sized rendering commands.
  */
 
+static _X_INLINE int
+safe_add(int a, int b)
+{
+    if (a < 0 || b < 0)
+        return -1;
+
+    if (INT_MAX - a < b)
+        return -1;
+
+    return a + b;
+}
+
+static _X_INLINE int
+safe_mul(int a, int b)
+{
+    if (a < 0 || b < 0)
+        return -1;
+
+    if (a == 0 || b == 0)
+        return 0;
+
+    if (a > INT_MAX / b)
+        return -1;
+
+    return a * b;
+}
+
+static _X_INLINE int
+safe_pad(int a)
+{
+    int ret;
+
+    if (a < 0)
+        return -1;
+
+    if ((ret = safe_add(a, 3)) < 0)
+        return -1;
+
+    return ret & (GLuint)~3;
+}
+
 extern int __glXTypeSize(GLenum enm);
 extern int __glXImageSize(GLenum format, GLenum type,
                           GLenum target, GLsizei w, GLsizei h, GLsizei d,