rocksdb/build_tools
yingsu00 f54d7f5fea Port 3 way SSE4.2 crc32c implementation from Folly
Summary:
**# Summary**

RocksDB uses SSE crc32 intrinsics to calculate the crc32 values but it does it in single way fashion (not pipelined on single CPU core). Intel's whitepaper () published an algorithm that uses 3-way pipelining for the crc32 intrinsics, then use pclmulqdq intrinsic to combine the values. Because pclmulqdq has overhead on its own, this algorithm will show perf gains on buffers larger than 216 bytes, which makes RocksDB a perfect user, since most of the buffers RocksDB call crc32c on is over 4KB. Initial db_bench show tremendous CPU gain.

This change uses the 3-way SSE algorithm by default. The old SSE algorithm is now behind a compiler tag NO_THREEWAY_CRC32C. If user compiles the code with NO_THREEWAY_CRC32C=1 then the old SSE Crc32c algorithm would be used. If the server does not have SSE4.2 at the run time the slow way (Non SSE) will be used.

**# Performance Test Results**
We ran the FillRandom and ReadRandom benchmarks in db_bench. ReadRandom is the point of interest here since it calculates the CRC32 for the in-mem buffers. We did 3 runs for each algorithm.

Before this change the CRC32 value computation takes about 11.5% of total CPU cost, and with the new 3-way algorithm it reduced to around 4.5%. The overall throughput also improved from 25.53MB/s to 27.63MB/s.

1) ReadRandom in db_bench overall metrics

    PER RUN
    Algorithm | run | micros/op | ops/sec |Throughput (MB/s)
    3-way      |  1   | 4.143   | 241387 | 26.7
    3-way      |  2   | 3.775   | 264872 | 29.3
    3-way      | 3    | 4.116   | 242929 | 26.9
    FastCrc32c|1  | 4.037   | 247727 | 27.4
    FastCrc32c|2  | 4.648   | 215166 | 23.8
    FastCrc32c|3  | 4.352   | 229799 | 25.4

     AVG
    Algorithm     |    Average of micros/op |   Average of ops/sec |    Average of Throughput (MB/s)
    3-way           |     4.01                               |      249,729                 |      27.63
    FastCrc32c  |     4.35                              |     230,897                  |      25.53

 2)   Crc32c computation CPU cost (inclusive samples percentage)
    PER RUN
    Implementation | run |  TotalSamples   | Crc32c percentage
    3-way                 |  1    |  4,572,250,000 | 4.37%
    3-way                 |  2    |  3,779,250,000 | 4.62%
    3-way                 |  3    |  4,129,500,000 | 4.48%
    FastCrc32c       |  1    |  4,663,500,000 | 11.24%
    FastCrc32c       |  2    |  4,047,500,000 | 12.34%
    FastCrc32c       |  3    |  4,366,750,000 | 11.68%

 **# Test Plan**
     make -j64 corruption_test && ./corruption_test
      By default it uses 3-way SSE algorithm

     NO_THREEWAY_CRC32C=1 make -j64 corruption_test && ./corruption_test

    make clean && DEBUG_LEVEL=0 make -j64 db_bench
    make clean && DEBUG_LEVEL=0 NO_THREEWAY_CRC32C=1 make -j64 db_bench
Closes https://github.com/facebook/rocksdb/pull/3173

Differential Revision: D6330882

Pulled By: yingsu00

fbshipit-source-id: 8ec3d89719533b63b536a736663ca6f0dd4482e9
2017-12-19 18:26:49 -08:00
..
amalgamate.py New amalgamation target 2015-10-01 08:29:31 +13:00
build_detect_platform Port 3 way SSE4.2 crc32c implementation from Folly 2017-12-19 18:26:49 -08:00
cont_integration.sh Fix /bin/bash shebangs 2017-08-03 15:56:46 -07:00
dependencies_4.8.1.sh Introduce Lua Extension: RocksLuaCompactionFilter 2016-11-16 15:39:12 -08:00
dependencies.sh update dependencies.sh 2017-08-31 15:26:24 -07:00
dockerbuild.sh Fix /bin/bash shebangs 2017-08-03 15:56:46 -07:00
error_filter.py Change RocksDB License 2017-07-15 16:11:23 -07:00
fb_compile_mongo.sh set -e in fb_compile_mongo.sh 2015-06-29 11:43:25 -07:00
fbcode_config4.8.1.sh revert fbcode build behavior 2017-12-07 16:12:52 -08:00
fbcode_config.sh revert fbcode build behavior 2017-12-07 16:12:52 -08:00
format-diff.sh Fix /bin/bash shebangs 2017-08-03 15:56:46 -07:00
gnu_parallel Testing out parallel sandcastle changes 2016-07-27 11:58:57 -07:00
make_package.sh Put lib files into suitable path in RPM package 2017-05-15 14:05:46 -07:00
precommit_checker.py Use gcc-4.9-glibc-2.20-fb python in precommit_checker 2017-02-03 13:39:18 -08:00
regression_build_test.sh Fix /bin/bash shebangs 2017-08-03 15:56:46 -07:00
rocksdb-lego-determinator fix lite build 2017-10-17 08:57:09 -07:00
RocksDBCommonHelper.php Update RocksDBCommonHelper to use escapeshellarg 2017-08-15 06:56:31 -07:00
run_ci_db_test.ps1 Fix crashes, address test issues and adjust windows test script 2017-11-10 10:41:57 -08:00
update_dependencies.sh update dependencies.sh 2017-08-31 15:26:24 -07:00
version.sh Version librocksdb.so 2017-01-24 11:09:10 -08:00