andreacavalli/netty5
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity

Improve the allocation algorithm in PoolChunk

Browse Source 
Motivation:

Depth-first search is not always efficient for buddy allocation.

Modification:

Employ a new faster search algorithm with different memoryMap layout.

Result:

With thread-local cache disabled, we see a lot of performance
improvment, especially when the size of the allocation is as small as
the page size, which had the largest search space previously.
This commit is contained in:
Pavan Kumar 2014-06-18 19:57:20 -07:00 committed by Trustin Lee
parent 70f1ce3cdd
commit 6e411fd04d
1 changed files with 253 additions and 202 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

439
buffer/src/main/java/io/netty/buffer/PoolChunk.java
View File

@ -16,25 +16,112 @@
package io.netty.buffer; package io.netty.buffer;
/**
* Description of algorithm for PageRun/PoolSubpage allocation from PoolChunk
*
* Notation: The following terms are important to understand the code
* > page - a page is the smallest unit of memory chunk that can be allocated
* > chunk - a chunk is a collection of pages
* > in this code chunkSize = 2^{maxOrder} * pageSize
*
* To begin we allocate a byte array of size = chunkSize
* Whenever a ByteBuf of given size needs to be created we search for the first position
* in the byte array that has enough empty space to accommodate the requested size and
* return a (long) handle that encodes this offset information, (this memory segment is then
* marked as reserved so it is always used by exactly one ByteBuf and no more)
*
* For simplicity all sizes are normalized according to PoolArena#normalizeCapacity method
* This ensures that when we request for memory segments of size >= pageSize the normalizedCapacity
* equals the next nearest power of 2
*
* To search for the first offset in chunk that has at least requested size available we construct a
* complete balanced binary tree and store it in an array (just like heaps) - memoryMap
*
* The tree looks like this (the size of each node being mentioned in the parenthesis)
*
* depth=0 1 node (chunkSize)
* depth=1 2 nodes (chunkSize/2)
* ..
* ..
* depth=d 2^d nodes (chunkSize/2^d)
* ..
* depth=maxOrder 2^maxOrder nodes (chunkSize/2^{maxOrder} = pageSize)
*
* depth=maxOrder is the last level and the leafs consist of pages
*
* With this tree available searching in chunkArray translates like this:
* To allocate a memory segment of size chunkSize/2^k we search for the first node (from left) at height k
* which is unused
*
* Algorithm:
* ----------
* Encode the tree in memoryMap with the notation
* memoryMap[id] = x => in the subtree rooted at id, the first node that is free to be allocated
* is at depth x (counted from depth=0) i.e., at depths [depth_of_id, x), there is no node that is free
*
* As we allocate & free nodes, we update values stored in memoryMap so that the property is maintained
*
* Initialization -
* In the beginning we construct the memoryMap array by storing the depth of a node at each node
* i.e., memoryMap[id] = depth_of_id
*
* Observations:
* -------------
* 1) memoryMap[id] = depth_of_id => it is free / unallocated
* 2) memoryMap[id] > depth_of_id => at least one of its child nodes is allocated, so we cannot allocate it, but
* some of its children can still be allocated based on their availability
* 3) memoryMap[id] = maxOrder + 1 => the node is fully allocated & thus none of its children can be allocated, it
* is thus marked as unusable
*
* Algorithm: [allocateNode(d) => we want to find the first node (from left) at height h that can be allocated]
* ----------
* 1) start at root (i.e., depth = 0 or id = 1)
* 2) if memoryMap[1] > d => cannot be allocated from this chunk
* 3) if left node value <= h; we can allocate from left subtree so move to left and repeat until found
* 4) else try in right subtree
*
* Algorithm: [allocateRun(size)]
* ----------
* 1) Compute d = log_2(chunkSize/size)
* 2) Return allocateNode(d)
*
* Algorithm: [allocateSubpage(size)]
* ----------
* 1) use allocateNode(maxOrder) to find an empty (i.e., unused) leaf (i.e., page)
* 2) use this handle to construct the PoolSubpage object or if it already exists just call init(normCapacity)
* note that this PoolSubpage object is added to subpagesPool in the PoolArena when we init() it
*
* Note:
* -----
* In the implementation for improving cache coherence,
* we store 2 pieces of information (i.e, 2 byte vals) as a short value in memoryMap
*
* memoryMap[id]= (depth_of_id, x)
* where as per convention defined above
* the second value (i.e, x) indicates that the first node which is free to be allocated is at depth x (from root)
*/
final class PoolChunk<T> { final class PoolChunk<T> {
private static final int ST_UNUSED = 0;
private static final int ST_BRANCH = 1; private static final int BYTE_LENGTH = Byte.SIZE;
private static final int ST_ALLOCATED = 2; private static final int UPPER_BYTE_MASK = - (1 << BYTE_LENGTH);
private static final int ST_ALLOCATED_SUBPAGE = 3;
final PoolArena<T> arena; final PoolArena<T> arena;
final T memory; final T memory;
final boolean unpooled; final boolean unpooled;
private final int[] memoryMap; private final short[] memoryMap;
private final PoolSubpage<T>[] subpages; private final PoolSubpage<T>[] subpages;
/** Used to determine if the requested capacity is equal to or greater than pageSize. */ /** Used to determine if the requested capacity is equal to or greater than pageSize. */
private final int subpageOverflowMask; private final int subpageOverflowMask;
private final int pageSize; private final int pageSize;
private final int pageShifts; private final int pageShifts;
private final int maxOrder;
private final int chunkSize; private final int chunkSize;
private final int log2ChunkSize;
private final int maxSubpageAllocs; private final int maxSubpageAllocs;
/** Used to mark memory as unusable */
private final byte unusable;
private int freeBytes; private int freeBytes;
@ -51,21 +138,26 @@ final class PoolChunk<T> {
this.memory = memory; this.memory = memory;
this.pageSize = pageSize; this.pageSize = pageSize;
this.pageShifts = pageShifts; this.pageShifts = pageShifts;
this.maxOrder = maxOrder;
this.chunkSize = chunkSize; this.chunkSize = chunkSize;
unusable = (byte) (maxOrder + 1);
log2ChunkSize = log2(chunkSize);
subpageOverflowMask = ~(pageSize - 1); subpageOverflowMask = ~(pageSize - 1);
freeBytes = chunkSize; freeBytes = chunkSize;
int chunkSizeInPages = chunkSize >>> pageShifts; assert maxOrder < 30 : "maxOrder should be < 30, but is : " + maxOrder;
maxSubpageAllocs = 1 << maxOrder; maxSubpageAllocs = 1 << maxOrder;
// Generate the memory map. // Generate the memory map.
memoryMap = new int[maxSubpageAllocs << 1]; memoryMap = new short[maxSubpageAllocs << 1];
int memoryMapIndex = 1; int memoryMapIndex = 1;
for (int i = 0; i <= maxOrder; i ++) { for (int d = 0; d <= maxOrder; ++d) { // move down the tree one level at a time
int runSizeInPages = chunkSizeInPages >>> i; short dd = (short) ((d << BYTE_LENGTH) | d);
for (int j = 0; j < chunkSizeInPages; j += runSizeInPages) { int depth = 1 << d;
//noinspection PointlessBitwiseExpression for (int p = 0; p < depth; ++p) {
memoryMap[memoryMapIndex ++] = j << 17 | runSizeInPages << 2 | ST_UNUSED; // in each level traverse left to right and set value to the depth of subtree
memoryMap[memoryMapIndex] = dd;
memoryMapIndex += 1;
} }
} }
@ -82,7 +174,10 @@ final class PoolChunk<T> {
subpageOverflowMask = 0; subpageOverflowMask = 0;
pageSize = 0; pageSize = 0;
pageShifts = 0; pageShifts = 0;
maxOrder = 0;
unusable = (byte) (maxOrder + 1);
chunkSize = size; chunkSize = size;
log2ChunkSize = log2(chunkSize);
maxSubpageAllocs = 0; maxSubpageAllocs = 0;
} }
@ -104,146 +199,124 @@ final class PoolChunk<T> {
} }
long allocate(int normCapacity) { long allocate(int normCapacity) {
int firstVal = memoryMap[1];
if ((normCapacity & subpageOverflowMask) != 0) { // >= pageSize if ((normCapacity & subpageOverflowMask) != 0) { // >= pageSize
return allocateRun(normCapacity, 1, firstVal); return allocateRun(normCapacity);
} else { } else {
return allocateSubpage(normCapacity, 1, firstVal); return allocateSubpage(normCapacity);
} }
} }
private long allocateRun(int normCapacity, int curIdx, int val) { /**
switch (val & 3) { * Update method used by allocate
case ST_UNUSED: * This is triggered only when a successor is allocated and all its predecessors
return allocateRunSimple(normCapacity, curIdx, val); * need to update their state
case ST_BRANCH: * The minimal depth at which subtree rooted at id has some free space
// Try the right node first because it is more likely to be ST_UNUSED. *
// It is because allocateRunSimple() always chooses the left node. * @param id id
final int nextIdxLeft = curIdx << 1; */
final int nextIdxRight = nextIdxLeft ^ 1; private void updateParentsAlloc(int id) {
final int nextValRight = memoryMap[nextIdxRight]; while (id > 1) {
final boolean recurseRight; int parentId = id >>> 1;
byte val1 = value(id);
switch (nextValRight & 3) { byte val2 = value(id ^ 1);
case ST_UNUSED: byte val = val1 < val2 ? val1 : val2;
return allocateRunSimple(normCapacity, nextIdxRight, nextValRight); setValue(parentId, val);
case ST_BRANCH: id = parentId;
recurseRight = true;
break;
default:
recurseRight = false;
}
final int nextValLeft = memoryMap[nextIdxLeft];
final boolean recurseLeft;
switch (nextValLeft & 3) {
case ST_UNUSED:
return allocateRunSimple(normCapacity, nextIdxLeft, nextValLeft);
case ST_BRANCH:
recurseLeft = true;
break;
default:
recurseLeft = false;
}
if (recurseRight) {
long res = branchRun(normCapacity, nextIdxRight);
if (res > 0) {
return res;
} }
} }
if (recurseLeft) { /**
return branchRun(normCapacity, nextIdxLeft); * Update method used by free
* This needs to handle the special case when both children are completely free
* in which case parent be directly allocated on request of size = child-size * 2
*
* @param id id
*/
private void updateParentsFree(int id) {
int logChild = depth(id) + 1;
while (id > 1) {
int parentId = id >>> 1;
byte val1 = value(id);
byte val2 = value(id ^ 1);
logChild -= 1; // in first iteration equals log, subsequently reduce 1 from logChild as we traverse up
if (val1 == logChild && val2 == logChild) {
setValue(parentId, (byte) (logChild - 1));
} else {
byte val = val1 < val2 ? val1 : val2;
setValue(parentId, val);
}
id = parentId;
} }
} }
/**
* Algorithm to allocate an index in memoryMap when we query for a free node
* at depth d
*
* @param d depth
* @return index in memoryMap
*/
private int allocateNode(int d) {
int id = 1;
int initial = - (1 << d); // has last d bits = 0 and rest all = 1
byte val = value(id);
if (val > d) { // unusable
return -1; return -1;
} }
while (val < d || (id & initial) == 0) { // id & initial == 1 << d for all ids at depth d, for < d it is 0
private long branchRun(int normCapacity, int nextIdx) { id = id << 1;
int nextNextIdx = nextIdx << 1; val = value(id);
int nextNextVal = memoryMap[nextNextIdx]; if (val > d) {
long res = allocateRun(normCapacity, nextNextIdx, nextNextVal); id = id ^ 1;
if (res > 0) { val = value(id);
return res;
}
nextNextIdx ^= 1;
nextNextVal = memoryMap[nextNextIdx];
return allocateRun(normCapacity, nextNextIdx, nextNextVal);
}
private long allocateRunSimple(int normCapacity, int curIdx, int val) {
int runLength = runLength(val);
if (normCapacity > runLength) {
return -1;
}
for (;;) {
if (normCapacity == runLength) {
// Found the run that fits.
// Note that capacity has been normalized already, so we don't need to deal with
// the values that are not power of 2.
memoryMap[curIdx] = val & ~3 | ST_ALLOCATED;
freeBytes -= runLength;
return curIdx;
}
int nextIdx = curIdx << 1;
int unusedIdx = nextIdx ^ 1;
memoryMap[curIdx] = val & ~3 | ST_BRANCH;
//noinspection PointlessBitwiseExpression
memoryMap[unusedIdx] = memoryMap[unusedIdx] & ~3 | ST_UNUSED;
runLength >>>= 1;
curIdx = nextIdx;
val = memoryMap[curIdx];
} }
} }
byte value = value(id);
private long allocateSubpage(int normCapacity, int curIdx, int val) { assert value == d && ((id & initial) == 1 << d) : String.format("val = %d, id & initial = %d, d = %d",
switch (val & 3) { value, id & initial, d);
case ST_UNUSED: setValue(id, unusable); // mark as unusable
return allocateSubpageSimple(normCapacity, curIdx, val); updateParentsAlloc(id);
case ST_BRANCH: return id;
// Try the right node first because it is more likely to be ST_UNUSED.
// It is because allocateSubpageSimple() always chooses the left node.
final int nextIdxLeft = curIdx << 1;
final int nextIdxRight = nextIdxLeft ^ 1;
long res = branchSubpage(normCapacity, nextIdxRight);
if (res > 0) {
return res;
} }
return branchSubpage(normCapacity, nextIdxLeft); /**
case ST_ALLOCATED_SUBPAGE: * Allocate a run of pages (>=1)
PoolSubpage<T> subpage = subpages[subpageIdx(curIdx)]; *
int elemSize = subpage.elemSize; * @param normCapacity normalized capacity
if (normCapacity != elemSize) { * @return index in memoryMap
return -1; */
private long allocateRun(int normCapacity) {
int numPages = normCapacity >>> pageShifts;
int d = maxOrder - log2(numPages);
int id = allocateNode(d);
if (id < 0) {
return id;
}
freeBytes -= runLength(id);
return id;
} }
return subpage.allocate(); /**
* Create/ initialize a new PoolSubpage of normCapacity
* Any PoolSubpage created/ initialized here is added to subpage pool in the PoolArena that owns this PoolChunk
*
* @param normCapacity normalized capacity
* @return index in memoryMap
*/
private long allocateSubpage(int normCapacity) {
int d = maxOrder; // subpages are only be allocated from pages i.e., leaves
int id = allocateNode(d);
if (id < 0) {
return id;
} }
freeBytes -= pageSize;
return -1; int subpageIdx = subpageIdx(id);
}
private long allocateSubpageSimple(int normCapacity, int curIdx, int val) {
int runLength = runLength(val);
for (;;) {
if (runLength == pageSize) {
memoryMap[curIdx] = val & ~3 | ST_ALLOCATED_SUBPAGE;
freeBytes -= runLength;
int subpageIdx = subpageIdx(curIdx);
PoolSubpage<T> subpage = subpages[subpageIdx]; PoolSubpage<T> subpage = subpages[subpageIdx];
if (subpage == null) { if (subpage == null) {
subpage = new PoolSubpage<T>(this, curIdx, runOffset(val), pageSize, normCapacity); subpage = new PoolSubpage<T>(this, id, runOffset(id), pageSize, normCapacity);
subpages[subpageIdx] = subpage; subpages[subpageIdx] = subpage;
} else { } else {
subpage.init(normCapacity); subpage.init(normCapacity);
@ -251,86 +324,50 @@ final class PoolChunk<T> {
return subpage.allocate(); return subpage.allocate();
} }
int nextIdx = curIdx << 1; /**
int unusedIdx = nextIdx ^ 1; * Free a subpage or a run of pages
* When a subpage is freed from PoolSubpage, it might be added back to subpage pool of the owning PoolArena
memoryMap[curIdx] = val & ~3 | ST_BRANCH; * If the subpage pool in PoolArena has at least one other PoolSubpage of given elemSize, we can
//noinspection PointlessBitwiseExpression * completely free the owning Page so it is available for subsequent allocations
memoryMap[unusedIdx] = memoryMap[unusedIdx] & ~3 | ST_UNUSED; *
* @param handle handle to free
runLength >>>= 1; */
curIdx = nextIdx;
val = memoryMap[curIdx];
}
}
private long branchSubpage(int normCapacity, int nextIdx) {
int nextVal = memoryMap[nextIdx];
if ((nextVal & 3) != ST_ALLOCATED) {
return allocateSubpage(normCapacity, nextIdx, nextVal);
}
return -1;
}
void free(long handle) { void free(long handle) {
int memoryMapIdx = (int) handle; int memoryMapIdx = (int) handle;
int bitmapIdx = (int) (handle >>> 32); int bitmapIdx = (int) (handle >>> Integer.SIZE);
int val = memoryMap[memoryMapIdx]; if (bitmapIdx != 0) { // free a subpage
int state = val & 3;
if (state == ST_ALLOCATED_SUBPAGE) {
assert bitmapIdx != 0;
PoolSubpage<T> subpage = subpages[subpageIdx(memoryMapIdx)]; PoolSubpage<T> subpage = subpages[subpageIdx(memoryMapIdx)];
assert subpage != null && subpage.doNotDestroy; assert subpage != null && subpage.doNotDestroy;
if (subpage.free(bitmapIdx & 0x3FFFFFFF)) { if (subpage.free(bitmapIdx & 0x3FFFFFFF)) {
return; return;
} }
} else {
assert state == ST_ALLOCATED : "state: " + state;
assert bitmapIdx == 0;
}
freeBytes += runLength(val);
for (;;) {
//noinspection PointlessBitwiseExpression
memoryMap[memoryMapIdx] = val & ~3 | ST_UNUSED;
if (memoryMapIdx == 1) {
assert freeBytes == chunkSize;
return;
}
if ((memoryMap[siblingIdx(memoryMapIdx)] & 3) != ST_UNUSED) {
break;
}
memoryMapIdx = parentIdx(memoryMapIdx);
val = memoryMap[memoryMapIdx];
} }
freeBytes += runLength(memoryMapIdx);
setValue(memoryMapIdx, depth(memoryMapIdx));
updateParentsFree(memoryMapIdx);
} }
void initBuf(PooledByteBuf<T> buf, long handle, int reqCapacity) { void initBuf(PooledByteBuf<T> buf, long handle, int reqCapacity) {
int memoryMapIdx = (int) handle; int memoryMapIdx = (int) handle;
int bitmapIdx = (int) (handle >>> 32); int bitmapIdx = (int) (handle >>> Integer.SIZE);
if (bitmapIdx == 0) { if (bitmapIdx == 0) {
int val = memoryMap[memoryMapIdx]; byte val = value(memoryMapIdx);
assert (val & 3) == ST_ALLOCATED : String.valueOf(val & 3); assert val == unusable : String.valueOf(val);
buf.init(this, handle, runOffset(val), reqCapacity, runLength(val)); buf.init(this, handle, runOffset(memoryMapIdx), reqCapacity, runLength(memoryMapIdx));
} else { } else {
initBufWithSubpage(buf, handle, bitmapIdx, reqCapacity); initBufWithSubpage(buf, handle, bitmapIdx, reqCapacity);
} }
} }
void initBufWithSubpage(PooledByteBuf<T> buf, long handle, int reqCapacity) { void initBufWithSubpage(PooledByteBuf<T> buf, long handle, int reqCapacity) {
initBufWithSubpage(buf, handle, (int) (handle >>> 32), reqCapacity); initBufWithSubpage(buf, handle, (int) (handle >>> Integer.SIZE), reqCapacity);
} }
private void initBufWithSubpage(PooledByteBuf<T> buf, long handle, int bitmapIdx, int reqCapacity) { private void initBufWithSubpage(PooledByteBuf<T> buf, long handle, int bitmapIdx, int reqCapacity) {
assert bitmapIdx != 0; assert bitmapIdx != 0;
int memoryMapIdx = (int) handle; int memoryMapIdx = (int) handle;
int val = memoryMap[memoryMapIdx];
assert (val & 3) == ST_ALLOCATED_SUBPAGE;
PoolSubpage<T> subpage = subpages[subpageIdx(memoryMapIdx)]; PoolSubpage<T> subpage = subpages[subpageIdx(memoryMapIdx)];
assert subpage.doNotDestroy; assert subpage.doNotDestroy;
@ -338,29 +375,43 @@ final class PoolChunk<T> {
buf.init( buf.init(
this, handle, this, handle,
runOffset(val) + (bitmapIdx & 0x3FFFFFFF) * subpage.elemSize, reqCapacity, subpage.elemSize); runOffset(memoryMapIdx) + (bitmapIdx & 0x3FFFFFFF) * subpage.elemSize, reqCapacity, subpage.elemSize);
} }
private static int parentIdx(int memoryMapIdx) { private byte value(int id) {
return memoryMapIdx >>> 1; return (byte) memoryMap[id];
} }
private static int siblingIdx(int memoryMapIdx) { private void setValue(int id, byte val) {
return memoryMapIdx ^ 1; memoryMap[id] = (short) ((memoryMap[id] & UPPER_BYTE_MASK) | val);
} }
private int runLength(int val) { private byte depth(int id) {
return (val >>> 2 & 0x7FFF) << pageShifts; short val = memoryMap[id];
return (byte) (val >>> BYTE_LENGTH);
} }
private int runOffset(int val) { private int log2(int val) {
return val >>> 17 << pageShifts; // compute the (0-based, with lsb = 0) position of highest set bit i.e, log2
return Integer.SIZE - 1 - Integer.numberOfLeadingZeros(val);
}
private int runLength(int id) {
// represents the size in #bytes supported by node 'id' in the tree
return 1 << (log2ChunkSize - depth(id));
}
private int runOffset(int id) {
// represents the 0-based offset in #bytes from start of the byte-array chunk
int shift = id ^ (1 << depth(id));
return shift * runLength(id);
} }
private int subpageIdx(int memoryMapIdx) { private int subpageIdx(int memoryMapIdx) {
return memoryMapIdx - maxSubpageAllocs; return memoryMapIdx ^ maxSubpageAllocs; // remove highest set bit, to get offset
} }
@Override
public String toString() { public String toString() {
StringBuilder buf = new StringBuilder(); StringBuilder buf = new StringBuilder();
buf.append("Chunk("); buf.append("Chunk(");