andreacavalli
/
xserver-multidpi
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity

xserver: fix RGB mask handling 

On FreeBSD 13.0-CURRENT for PowerPC64 big-endian (BE), X was
crashing in some cases. For instance, when twm was started
and the background was clicked to open its menu, X crashed
with a segmentation fault, trying to dereference a null pointer
at CreatePicture().

There were 2 issues with xorg-server handling of RGB masks that
caused the pointer above to be null and thus the crash:
- wrong use of ffs() to get the RGB offsets from the masks
- overflow when shifting a 16-bit integer

This change fixes both issues. They happen when the system is BE
but has a video adapter using a little-endian (LE) ARGB32
framebuffer. In order to display the correct colors, this setup
requires a BE RGBA32 color format to be used by X, by setting
the RGB masks appropriately, that didn't work properly because of
the issues above.
This commit is contained in:
luporl 2020-12-09 11:11:15 -03:00 committed by Povilas Kanapickas
parent 5be3b80b8d
commit 7e142cb2a8
2 changed files with 9 additions and 9 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

6
hw/xfree86/common/xf86Helper.c
View File

@ -729,9 +729,9 @@ xf86SetWeight(ScrnInfoPtr scrp, rgb weight, rgb mask)
scrp->mask.red = mask.red; scrp->mask.red = mask.red;
scrp->mask.green = mask.green; scrp->mask.green = mask.green;
scrp->mask.blue = mask.blue; scrp->mask.blue = mask.blue;
scrp->offset.red = ffs(mask.red); scrp->offset.red = ffs(mask.red) - 1;
scrp->offset.green = ffs(mask.green); scrp->offset.green = ffs(mask.green) - 1;
scrp->offset.blue = ffs(mask.blue); scrp->offset.blue = ffs(mask.blue) - 1;
} }
return TRUE; return TRUE;
} }

12
render/picture.c
View File

@ -523,12 +523,12 @@ PictureMatchVisual(ScreenPtr pScreen, int depth, VisualPtr pVisual)
return format; return format;
} }
else { else {
if (format->direct.redMask << format->direct.red == if ((unsigned long)format->direct.redMask <<
pVisual->redMask && format->direct.red == pVisual->redMask &&
format->direct.greenMask << format->direct.green == (unsigned long)format->direct.greenMask <<
pVisual->greenMask && format->direct.green == pVisual->greenMask &&
format->direct.blueMask << format->direct.blue == (unsigned long)format->direct.blueMask <<
pVisual->blueMask) { format->direct.blue == pVisual->blueMask) {
return format; return format;
} }
} }