diff options
| author | George Hazan <ghazan@miranda.im> | 2020-01-21 19:03:25 +0300 | 
|---|---|---|
| committer | George Hazan <ghazan@miranda.im> | 2020-01-21 19:03:25 +0300 | 
| commit | 1a2103e23c5b65c9a4e5eb6cc84b23f8395a0ebd (patch) | |
| tree | 24f14318ca117c4ed047af43c4b1e4bc10443365 /libs/libmdbx/src | |
| parent | 999d22f6c3c618cde52097547e53703664d7e665 (diff) | |
libmdbx: update to 0.6
Diffstat (limited to 'libs/libmdbx/src')
56 files changed, 859 insertions, 275 deletions
diff --git a/libs/libmdbx/src/CMakeLists.dist-minimal b/libs/libmdbx/src/CMakeLists.dist-minimal index 48592297a6..fa330a9878 100644 --- a/libs/libmdbx/src/CMakeLists.dist-minimal +++ b/libs/libmdbx/src/CMakeLists.dist-minimal @@ -10,7 +10,7 @@  ##  ## -##  Copyright 2019 Leonid Yuriev <leo@yuriev.ru> +##  Copyright 2020 Leonid Yuriev <leo@yuriev.ru>  ##  and other libmdbx authors: please see AUTHORS file.  ##  All rights reserved.  ## diff --git a/libs/libmdbx/src/CMakeLists.txt b/libs/libmdbx/src/CMakeLists.txt index 3fb33d3f31..e642f31a98 100644 --- a/libs/libmdbx/src/CMakeLists.txt +++ b/libs/libmdbx/src/CMakeLists.txt @@ -1,5 +1,5 @@  ## -##  Copyright 2019 Leonid Yuriev <leo@yuriev.ru> +##  Copyright 2020 Leonid Yuriev <leo@yuriev.ru>  ##  and other libmdbx authors: please see AUTHORS file.  ##  All rights reserved.  ## diff --git a/libs/libmdbx/src/COPYRIGHT b/libs/libmdbx/src/COPYRIGHT index 46e0961046..fc6d9eed24 100644 --- a/libs/libmdbx/src/COPYRIGHT +++ b/libs/libmdbx/src/COPYRIGHT @@ -1,4 +1,4 @@ -Copyright 2015-2019 Leonid Yuriev <leo@yuriev.ru>. +Copyright 2015-2020 Leonid Yuriev <leo@yuriev.ru>.  Copyright 2011-2015 Howard Chu, Symas Corp.  Copyright 2015,2016 Peter-Service R&D LLC.  All rights reserved. diff --git a/libs/libmdbx/src/README.md b/libs/libmdbx/src/README.md index e578f893ed..1b2fb69bdd 100644 --- a/libs/libmdbx/src/README.md +++ b/libs/libmdbx/src/README.md @@ -1,7 +1,7 @@  libmdbx  ======= -_libmdbx_ is an extremely fast, compact, powerful, embedded +_libmdbx_ is an extremely fast, compact, powerful, embedded,  transactional [key-value store](https://en.wikipedia.org/wiki/Key-value_database)  database, with [permissive license](LICENSE).  _MDBX_ has a specific set of properties and capabilities, @@ -11,7 +11,7 @@ focused on creating unique lightweight solutions with extraordinary performance.  2. Provides **extraordinary performance**, minimal overhead through [Memory-Mapping](https://en.wikipedia.org/wiki/Memory-mapped_file) and `Olog(N)` operations costs by virtue of [B+ tree](https://en.wikipedia.org/wiki/B%2B_tree). -3. Requires **no maintenance and no crash recovery** since doesn't use [WAL](https://en.wikipedia.org/wiki/Write-ahead_logging), but that might be a caveat for some workloads. +3. Requires **no maintenance and no crash recovery** since doesn't use [WAL](https://en.wikipedia.org/wiki/Write-ahead_logging), but that might be a caveat for write-intensive workloads.  4. **Compact and friendly for fully embeddeding**. Only 25KLOC of `C11`, 64K x86 binary code,  no internal threads neither processes, but implements a simplified variant of the @@ -127,7 +127,7 @@ transaction journal. No crash recovery needed. No maintenance is required.  - **Page size**: a power of 2, maximum `65536` bytes, default `4096` bytes.  - **Key size**: minimum 0, maximum ≈¼ pagesize (`1300` bytes for default 4K pagesize, `21780` bytes for 64K pagesize).  - **Value size**: minimum 0, maximum `2146435072` (`0x7FF00000`) bytes for maps, ≈¼ pagesize for multimaps (`1348` bytes default 4K pagesize, `21828` bytes for 64K pagesize). -- **Write transaction size**: up to `4194301` (`0x3FFFFD`) pages (16 [GiB](https://en.wikipedia.org/wiki/Gibibyte) for pagesize, 256 [GiB](https://en.wikipedia.org/wiki/Gibibyte) for 64K pagesize). +- **Write transaction size**: up to `4194301` (`0x3FFFFD`) pages (16 [GiB](https://en.wikipedia.org/wiki/Gibibyte) for default 4K pagesize, 256 [GiB](https://en.wikipedia.org/wiki/Gibibyte) for 64K pagesize).  - **Database size**: up to `2147483648` pages (8 [TiB](https://en.wikipedia.org/wiki/Tebibyte) for default 4K pagesize, 128 [TiB](https://en.wikipedia.org/wiki/Tebibyte) for 64K pagesize).  - **Maximum sub-databases**: `32765`. @@ -418,14 +418,14 @@ Performance comparison  All benchmarks were done in 2015 by [IOArena](https://github.com/pmwkaa/ioarena)  and multiple [scripts](https://github.com/pmwkaa/ioarena/tree/HL%2B%2B2015) -runs on Lenovo Carbon-2 laptop, i7-4600U 2.1 GHz, 8 Gb RAM, +runs on Lenovo Carbon-2 laptop, i7-4600U 2.1 GHz (2 physical cores, 4 HyperThreading cores), 8 Gb RAM,  SSD SAMSUNG MZNTD512HAGL-000L1 (DXT23L0Q) 512 Gb.  ## Integral performance  Here showed sum of performance metrics in 3 benchmarks: - - Read/Search on 4 CPU cores machine; + - Read/Search on machine with 4 logical CPU in HyperThreading mode (i.e. actually 2 physical CPU cores);   - Transactions with [CRUD](https://en.wikipedia.org/wiki/CRUD)   operations in sync-write mode (fdatasync is called after each diff --git a/libs/libmdbx/src/appveyor.yml b/libs/libmdbx/src/appveyor.yml index 0cf8f8633f..c1b1085cd7 100644 --- a/libs/libmdbx/src/appveyor.yml +++ b/libs/libmdbx/src/appveyor.yml @@ -1,4 +1,4 @@ -version: 0.5.0.{build} +version: 0.6.0.{build}  environment:    matrix: diff --git a/libs/libmdbx/src/cmake/compiler.cmake b/libs/libmdbx/src/cmake/compiler.cmake index 03b0805bec..7936244f69 100644 --- a/libs/libmdbx/src/cmake/compiler.cmake +++ b/libs/libmdbx/src/cmake/compiler.cmake @@ -1,4 +1,4 @@ -##  Copyright (c) 2012-2019 Leonid Yuriev <leo@yuriev.ru>. +##  Copyright (c) 2012-2020 Leonid Yuriev <leo@yuriev.ru>.  ##  ##  Licensed under the Apache License, Version 2.0 (the "License");  ##  you may not use this file except in compliance with the License. diff --git a/libs/libmdbx/src/cmake/profile.cmake b/libs/libmdbx/src/cmake/profile.cmake index 6507e8dd83..c1c4f78e19 100644 --- a/libs/libmdbx/src/cmake/profile.cmake +++ b/libs/libmdbx/src/cmake/profile.cmake @@ -1,4 +1,4 @@ -##  Copyright (c) 2012-2019 Leonid Yuriev <leo@yuriev.ru>. +##  Copyright (c) 2012-2020 Leonid Yuriev <leo@yuriev.ru>.  ##  ##  Licensed under the Apache License, Version 2.0 (the "License");  ##  you may not use this file except in compliance with the License. diff --git a/libs/libmdbx/src/cmake/utils.cmake b/libs/libmdbx/src/cmake/utils.cmake index c31f53cd3e..22e0d72f57 100644 --- a/libs/libmdbx/src/cmake/utils.cmake +++ b/libs/libmdbx/src/cmake/utils.cmake @@ -1,4 +1,4 @@ -##  Copyright (c) 2012-2019 Leonid Yuriev <leo@yuriev.ru>. +##  Copyright (c) 2012-2020 Leonid Yuriev <leo@yuriev.ru>.  ##  ##  Licensed under the Apache License, Version 2.0 (the "License");  ##  you may not use this file except in compliance with the License. diff --git a/libs/libmdbx/src/example/example-mdbx.c b/libs/libmdbx/src/example/example-mdbx.c index 1d25ef6fc8..0a9e7ed601 100644 --- a/libs/libmdbx/src/example/example-mdbx.c +++ b/libs/libmdbx/src/example/example-mdbx.c @@ -4,7 +4,7 @@   */  /* - * Copyright 2015-2019 Leonid Yuriev <leo@yuriev.ru>. + * Copyright 2015-2020 Leonid Yuriev <leo@yuriev.ru>.   * Copyright 2017 Ilya Shipitsin <chipitsine@gmail.com>.   * Copyright 2012-2015 Howard Chu, Symas Corp.   * All rights reserved. diff --git a/libs/libmdbx/src/example/sample-bdb.txt b/libs/libmdbx/src/example/sample-bdb.txt index 5c89540afa..0f5e190a5e 100644 --- a/libs/libmdbx/src/example/sample-bdb.txt +++ b/libs/libmdbx/src/example/sample-bdb.txt @@ -4,7 +4,7 @@   */  /* - * Copyright 2015-2019 Leonid Yuriev <leo@yuriev.ru>. + * Copyright 2015-2020 Leonid Yuriev <leo@yuriev.ru>.   * Copyright 2012-2015 Howard Chu, Symas Corp.   * Copyright 2015,2016 Peter-Service R&D LLC.   * All rights reserved. diff --git a/libs/libmdbx/src/mdbx.h b/libs/libmdbx/src/mdbx.h index a522501a65..8f2b89c63e 100644 --- a/libs/libmdbx/src/mdbx.h +++ b/libs/libmdbx/src/mdbx.h @@ -460,7 +460,7 @@   *   **** LICENSE AND COPYRUSTING **************************************************   * - * Copyright 2015-2019 Leonid Yuriev <leo@yuriev.ru> + * Copyright 2015-2020 Leonid Yuriev <leo@yuriev.ru>   * and other libmdbx authors: please see AUTHORS file.   * All rights reserved.   * @@ -643,7 +643,7 @@ typedef pthread_t mdbx_tid_t;  /*----------------------------------------------------------------------------*/  #define MDBX_VERSION_MAJOR 0 -#define MDBX_VERSION_MINOR 5 +#define MDBX_VERSION_MINOR 6  #ifndef LIBMDBX_API  #if defined(LIBMDBX_EXPORTS) diff --git a/libs/libmdbx/src/src/CMakeLists.txt b/libs/libmdbx/src/src/CMakeLists.txt index a127eaee88..82c953dabb 100644 --- a/libs/libmdbx/src/src/CMakeLists.txt +++ b/libs/libmdbx/src/src/CMakeLists.txt @@ -1,5 +1,5 @@  ## -##  Copyright 2019 Leonid Yuriev <leo@yuriev.ru> +##  Copyright 2020 Leonid Yuriev <leo@yuriev.ru>  ##  and other libmdbx authors: please see AUTHORS file.  ##  All rights reserved.  ## diff --git a/libs/libmdbx/src/src/alloy.c b/libs/libmdbx/src/src/alloy.c index c40446732b..befdf8be45 100644 --- a/libs/libmdbx/src/src/alloy.c +++ b/libs/libmdbx/src/src/alloy.c @@ -1,5 +1,5 @@  /* - * Copyright 2015-2019 Leonid Yuriev <leo@yuriev.ru> + * Copyright 2015-2020 Leonid Yuriev <leo@yuriev.ru>   * and other libmdbx authors: please see AUTHORS file.   * All rights reserved.   * diff --git a/libs/libmdbx/src/src/elements/config.h b/libs/libmdbx/src/src/elements/config.h new file mode 100644 index 0000000000..fcd035b093 --- /dev/null +++ b/libs/libmdbx/src/src/elements/config.h @@ -0,0 +1,56 @@ +/* This is CMake-template for libmdbx's config.h + ******************************************************************************/ + +/* *INDENT-OFF* */ +/* clang-format off */ + +/* #undef HAVE_VALGRIND_MEMCHECK_H */ +/* #undef HAS_RELAXED_CONSTEXPR */ + +#define LTO_ENABLED +/* #undef MDBX_USE_VALGRIND */ +/* #undef ENABLE_GPROF */ +/* #undef ENABLE_GCOV */ +/* #undef ENABLE_ASAN */ +/* #undef MDBX_FORCE_ASSERTIONS */ + +/* Common */ +#define MDBX_TXN_CHECKOWNER 1 +#define MDBX_TXN_CHECKPID_AUTO +#ifndef MDBX_TXN_CHECKPID_AUTO +#define MDBX_TXN_CHECKPID 0 +#endif +#define MDBX_BUILD_SHARED_LIBRARY 1 +#define MDBX_LOCKING_AUTO +#ifndef MDBX_LOCKING_AUTO +/* #undef MDBX_LOCKING */ +#endif +#define MDBX_TRUST_RTC_AUTO +#ifndef MDBX_TRUST_RTC_AUTO +#define MDBX_TRUST_RTC 0 +#endif + +/* Windows */ +#define MDBX_CONFIG_MANUAL_TLS_CALLBACK 0 +#define MDBX_AVOID_CRT 1 + +/* MacOS */ +#define MDBX_OSX_SPEED_INSTEADOF_DURABILITY 0 + +/* POSIX */ +#define MDBX_DISABLE_GNU_SOURCE 0 +/* #undef MDBX_USE_OFDLOCKS_AUTO */ +#ifndef MDBX_USE_OFDLOCKS_AUTO +#define MDBX_USE_OFDLOCKS 0 +#endif + +/* Build Info */ +#define MDBX_BUILD_TIMESTAMP "2020-01-21T15:55:59Z" +#define MDBX_BUILD_TARGET "x86_64-Windows" +/* #undef MDBX_BUILD_CONFIG */ +#define MDBX_BUILD_COMPILER "MSVC-19.24.28315.0" +#define MDBX_BUILD_FLAGS "LIBMDBX_EXPORTS" +#define MDBX_BUILD_SOURCERY 425d81103531494ef1989854f912bed8cbcb3acfa86c50ce374995c7edf1eb16_v0_6_0_1_g32db811 + +/* *INDENT-ON* */ +/* clang-format on */ diff --git a/libs/libmdbx/src/src/elements/core.c b/libs/libmdbx/src/src/elements/core.c index b5a27ddccf..2cd3445e9b 100644 --- a/libs/libmdbx/src/src/elements/core.c +++ b/libs/libmdbx/src/src/elements/core.c @@ -1,5 +1,5 @@  /* - * Copyright 2015-2019 Leonid Yuriev <leo@yuriev.ru> + * Copyright 2015-2020 Leonid Yuriev <leo@yuriev.ru>   * and other libmdbx authors: please see AUTHORS file.   * All rights reserved.   * @@ -1465,16 +1465,535 @@ static int lcklist_detach_locked(MDBX_env *env) {  }  /*------------------------------------------------------------------------------ - * LY: State of the art quicksort-based sorting, with internal stack and - *     shell-insertion-sort for small chunks (less than half of SORT_THRESHOLD). - */ + * LY: State of the art quicksort-based sorting, with internal stack + *     and bitonic-sort for small chunks. */ -/* LY: Large threshold give some boost due less overhead in the inner qsort - *     loops, but also a penalty in cases reverse-sorted data. - *     So, 42 is magically but reasonable: - *      - 0-3% faster than std::sort (from GNU C++ STL 2018) in most cases. - *      - slower by a few ticks in a few cases for sequences shorter than 21. */ -#define SORT_THRESHOLD 42 +#define SORT_CMP_SWAP(TYPE, CMP, a, b)                                         \ +  do {                                                                         \ +    const TYPE swap_tmp = (a);                                                 \ +    const bool swap_cmp = CMP(swap_tmp, b);                                    \ +    (a) = swap_cmp ? swap_tmp : b;                                             \ +    (b) = swap_cmp ? b : swap_tmp;                                             \ +  } while (0) + +#define SORT_BITONIC_2(TYPE, CMP, begin)                                       \ +  do {                                                                         \ +    SORT_CMP_SWAP(TYPE, CMP, begin[0], begin[1]);                              \ +  } while (0) + +#define SORT_BITONIC_3(TYPE, CMP, begin)                                       \ +  do {                                                                         \ +    SORT_CMP_SWAP(TYPE, CMP, begin[1], begin[2]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[0], begin[2]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[0], begin[1]);                              \ +  } while (0) + +#define SORT_BITONIC_4(TYPE, CMP, begin)                                       \ +  do {                                                                         \ +    SORT_CMP_SWAP(TYPE, CMP, begin[0], begin[1]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[2], begin[3]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[0], begin[2]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[1], begin[3]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[1], begin[2]);                              \ +  } while (0) + +#define SORT_BITONIC_5(TYPE, CMP, begin)                                       \ +  do {                                                                         \ +    SORT_CMP_SWAP(TYPE, CMP, begin[0], begin[1]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[3], begin[4]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[2], begin[4]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[2], begin[3]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[1], begin[4]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[0], begin[3]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[0], begin[2]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[1], begin[3]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[1], begin[2]);                              \ +  } while (0) + +#define SORT_BITONIC_6(TYPE, CMP, begin)                                       \ +  do {                                                                         \ +    SORT_CMP_SWAP(TYPE, CMP, begin[1], begin[2]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[4], begin[5]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[0], begin[2]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[3], begin[5]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[0], begin[1]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[3], begin[4]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[2], begin[5]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[0], begin[3]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[1], begin[4]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[2], begin[4]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[1], begin[3]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[2], begin[3]);                              \ +  } while (0) + +#define SORT_BITONIC_7(TYPE, CMP, begin)                                       \ +  do {                                                                         \ +    SORT_CMP_SWAP(TYPE, CMP, begin[1], begin[2]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[3], begin[4]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[5], begin[6]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[0], begin[2]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[3], begin[5]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[4], begin[6]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[0], begin[1]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[4], begin[5]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[2], begin[6]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[0], begin[4]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[1], begin[5]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[0], begin[3]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[2], begin[5]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[1], begin[3]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[2], begin[4]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[2], begin[3]);                              \ +  } while (0) + +#define SORT_BITONIC_8(TYPE, CMP, begin)                                       \ +  do {                                                                         \ +    SORT_CMP_SWAP(TYPE, CMP, begin[0], begin[1]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[2], begin[3]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[4], begin[5]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[6], begin[7]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[0], begin[2]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[1], begin[3]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[4], begin[6]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[5], begin[7]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[1], begin[2]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[5], begin[6]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[0], begin[4]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[3], begin[7]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[1], begin[5]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[2], begin[6]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[1], begin[4]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[3], begin[6]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[2], begin[4]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[3], begin[5]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[3], begin[4]);                              \ +  } while (0) + +#define SORT_BITONIC_9(TYPE, CMP, begin)                                       \ +  do {                                                                         \ +    SORT_CMP_SWAP(TYPE, CMP, begin[0], begin[1]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[3], begin[4]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[6], begin[7]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[1], begin[2]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[4], begin[5]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[7], begin[8]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[0], begin[1]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[3], begin[4]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[6], begin[7]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[2], begin[5]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[0], begin[3]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[1], begin[4]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[5], begin[8]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[3], begin[6]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[4], begin[7]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[2], begin[5]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[0], begin[3]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[1], begin[4]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[5], begin[7]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[2], begin[6]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[1], begin[3]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[4], begin[6]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[2], begin[4]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[5], begin[6]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[2], begin[3]);                              \ +  } while (0) + +#define SORT_BITONIC_10(TYPE, CMP, begin)                                      \ +  do {                                                                         \ +    SORT_CMP_SWAP(TYPE, CMP, begin[4], begin[9]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[3], begin[8]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[2], begin[7]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[1], begin[6]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[0], begin[5]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[1], begin[4]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[6], begin[9]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[0], begin[3]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[5], begin[8]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[0], begin[2]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[3], begin[6]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[7], begin[9]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[0], begin[1]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[2], begin[4]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[5], begin[7]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[8], begin[9]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[1], begin[2]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[4], begin[6]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[7], begin[8]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[3], begin[5]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[2], begin[5]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[6], begin[8]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[1], begin[3]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[4], begin[7]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[2], begin[3]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[6], begin[7]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[3], begin[4]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[5], begin[6]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[4], begin[5]);                              \ +  } while (0) + +#define SORT_BITONIC_11(TYPE, CMP, begin)                                      \ +  do {                                                                         \ +    SORT_CMP_SWAP(TYPE, CMP, begin[0], begin[1]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[2], begin[3]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[4], begin[5]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[6], begin[7]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[8], begin[9]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[1], begin[3]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[5], begin[7]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[0], begin[2]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[4], begin[6]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[8], begin[10]);                             \ +    SORT_CMP_SWAP(TYPE, CMP, begin[1], begin[2]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[5], begin[6]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[9], begin[10]);                             \ +    SORT_CMP_SWAP(TYPE, CMP, begin[0], begin[4]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[3], begin[7]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[1], begin[5]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[6], begin[10]);                             \ +    SORT_CMP_SWAP(TYPE, CMP, begin[4], begin[8]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[5], begin[9]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[2], begin[6]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[0], begin[4]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[3], begin[8]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[1], begin[5]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[6], begin[10]);                             \ +    SORT_CMP_SWAP(TYPE, CMP, begin[2], begin[3]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[8], begin[9]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[1], begin[4]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[7], begin[10]);                             \ +    SORT_CMP_SWAP(TYPE, CMP, begin[3], begin[5]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[6], begin[8]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[2], begin[4]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[7], begin[9]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[5], begin[6]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[3], begin[4]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[7], begin[8]);                              \ +  } while (0) + +#define SORT_BITONIC_12(TYPE, CMP, begin)                                      \ +  do {                                                                         \ +    SORT_CMP_SWAP(TYPE, CMP, begin[0], begin[1]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[2], begin[3]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[4], begin[5]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[6], begin[7]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[8], begin[9]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[10], begin[11]);                            \ +    SORT_CMP_SWAP(TYPE, CMP, begin[1], begin[3]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[5], begin[7]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[9], begin[11]);                             \ +    SORT_CMP_SWAP(TYPE, CMP, begin[0], begin[2]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[4], begin[6]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[8], begin[10]);                             \ +    SORT_CMP_SWAP(TYPE, CMP, begin[1], begin[2]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[5], begin[6]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[9], begin[10]);                             \ +    SORT_CMP_SWAP(TYPE, CMP, begin[0], begin[4]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[7], begin[11]);                             \ +    SORT_CMP_SWAP(TYPE, CMP, begin[1], begin[5]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[6], begin[10]);                             \ +    SORT_CMP_SWAP(TYPE, CMP, begin[3], begin[7]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[4], begin[8]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[5], begin[9]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[2], begin[6]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[0], begin[4]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[7], begin[11]);                             \ +    SORT_CMP_SWAP(TYPE, CMP, begin[3], begin[8]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[1], begin[5]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[6], begin[10]);                             \ +    SORT_CMP_SWAP(TYPE, CMP, begin[2], begin[3]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[8], begin[9]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[1], begin[4]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[7], begin[10]);                             \ +    SORT_CMP_SWAP(TYPE, CMP, begin[3], begin[5]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[6], begin[8]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[2], begin[4]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[7], begin[9]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[5], begin[6]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[3], begin[4]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[7], begin[8]);                              \ +  } while (0) + +#define SORT_BITONIC_13(TYPE, CMP, begin)                                      \ +  do {                                                                         \ +    SORT_CMP_SWAP(TYPE, CMP, begin[1], begin[7]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[9], begin[11]);                             \ +    SORT_CMP_SWAP(TYPE, CMP, begin[3], begin[4]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[5], begin[8]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[0], begin[12]);                             \ +    SORT_CMP_SWAP(TYPE, CMP, begin[2], begin[6]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[0], begin[1]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[2], begin[3]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[4], begin[6]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[8], begin[11]);                             \ +    SORT_CMP_SWAP(TYPE, CMP, begin[7], begin[12]);                             \ +    SORT_CMP_SWAP(TYPE, CMP, begin[5], begin[9]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[0], begin[2]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[3], begin[7]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[10], begin[11]);                            \ +    SORT_CMP_SWAP(TYPE, CMP, begin[1], begin[4]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[6], begin[12]);                             \ +    SORT_CMP_SWAP(TYPE, CMP, begin[7], begin[8]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[11], begin[12]);                            \ +    SORT_CMP_SWAP(TYPE, CMP, begin[4], begin[9]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[6], begin[10]);                             \ +    SORT_CMP_SWAP(TYPE, CMP, begin[3], begin[4]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[5], begin[6]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[8], begin[9]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[10], begin[11]);                            \ +    SORT_CMP_SWAP(TYPE, CMP, begin[1], begin[7]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[2], begin[6]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[9], begin[11]);                             \ +    SORT_CMP_SWAP(TYPE, CMP, begin[1], begin[3]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[4], begin[7]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[8], begin[10]);                             \ +    SORT_CMP_SWAP(TYPE, CMP, begin[0], begin[5]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[2], begin[5]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[6], begin[8]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[9], begin[10]);                             \ +    SORT_CMP_SWAP(TYPE, CMP, begin[1], begin[2]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[3], begin[5]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[7], begin[8]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[4], begin[6]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[2], begin[3]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[4], begin[5]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[6], begin[7]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[8], begin[9]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[3], begin[4]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[5], begin[6]);                              \ +  } while (0) + +#define SORT_BITONIC_14(TYPE, CMP, begin)                                      \ +  do {                                                                         \ +    SORT_CMP_SWAP(TYPE, CMP, begin[0], begin[1]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[2], begin[3]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[4], begin[5]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[6], begin[7]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[8], begin[9]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[10], begin[11]);                            \ +    SORT_CMP_SWAP(TYPE, CMP, begin[12], begin[13]);                            \ +    SORT_CMP_SWAP(TYPE, CMP, begin[0], begin[2]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[4], begin[6]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[8], begin[10]);                             \ +    SORT_CMP_SWAP(TYPE, CMP, begin[1], begin[3]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[5], begin[7]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[9], begin[11]);                             \ +    SORT_CMP_SWAP(TYPE, CMP, begin[0], begin[4]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[8], begin[12]);                             \ +    SORT_CMP_SWAP(TYPE, CMP, begin[1], begin[5]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[9], begin[13]);                             \ +    SORT_CMP_SWAP(TYPE, CMP, begin[2], begin[6]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[3], begin[7]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[0], begin[8]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[1], begin[9]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[2], begin[10]);                             \ +    SORT_CMP_SWAP(TYPE, CMP, begin[3], begin[11]);                             \ +    SORT_CMP_SWAP(TYPE, CMP, begin[4], begin[12]);                             \ +    SORT_CMP_SWAP(TYPE, CMP, begin[5], begin[13]);                             \ +    SORT_CMP_SWAP(TYPE, CMP, begin[5], begin[10]);                             \ +    SORT_CMP_SWAP(TYPE, CMP, begin[6], begin[9]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[3], begin[12]);                             \ +    SORT_CMP_SWAP(TYPE, CMP, begin[7], begin[11]);                             \ +    SORT_CMP_SWAP(TYPE, CMP, begin[1], begin[2]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[4], begin[8]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[1], begin[4]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[7], begin[13]);                             \ +    SORT_CMP_SWAP(TYPE, CMP, begin[2], begin[8]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[5], begin[6]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[9], begin[10]);                             \ +    SORT_CMP_SWAP(TYPE, CMP, begin[2], begin[4]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[11], begin[13]);                            \ +    SORT_CMP_SWAP(TYPE, CMP, begin[3], begin[8]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[7], begin[12]);                             \ +    SORT_CMP_SWAP(TYPE, CMP, begin[6], begin[8]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[10], begin[12]);                            \ +    SORT_CMP_SWAP(TYPE, CMP, begin[3], begin[5]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[7], begin[9]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[3], begin[4]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[5], begin[6]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[7], begin[8]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[9], begin[10]);                             \ +    SORT_CMP_SWAP(TYPE, CMP, begin[11], begin[12]);                            \ +    SORT_CMP_SWAP(TYPE, CMP, begin[6], begin[7]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[8], begin[9]);                              \ +  } while (0) + +#define SORT_BITONIC_15(TYPE, CMP, begin)                                      \ +  do {                                                                         \ +    SORT_CMP_SWAP(TYPE, CMP, begin[0], begin[1]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[2], begin[3]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[4], begin[5]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[6], begin[7]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[8], begin[9]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[10], begin[11]);                            \ +    SORT_CMP_SWAP(TYPE, CMP, begin[12], begin[13]);                            \ +    SORT_CMP_SWAP(TYPE, CMP, begin[0], begin[2]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[4], begin[6]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[8], begin[10]);                             \ +    SORT_CMP_SWAP(TYPE, CMP, begin[12], begin[14]);                            \ +    SORT_CMP_SWAP(TYPE, CMP, begin[1], begin[3]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[5], begin[7]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[9], begin[11]);                             \ +    SORT_CMP_SWAP(TYPE, CMP, begin[0], begin[4]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[8], begin[12]);                             \ +    SORT_CMP_SWAP(TYPE, CMP, begin[1], begin[5]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[9], begin[13]);                             \ +    SORT_CMP_SWAP(TYPE, CMP, begin[2], begin[6]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[10], begin[14]);                            \ +    SORT_CMP_SWAP(TYPE, CMP, begin[3], begin[7]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[0], begin[8]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[1], begin[9]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[2], begin[10]);                             \ +    SORT_CMP_SWAP(TYPE, CMP, begin[3], begin[11]);                             \ +    SORT_CMP_SWAP(TYPE, CMP, begin[4], begin[12]);                             \ +    SORT_CMP_SWAP(TYPE, CMP, begin[5], begin[13]);                             \ +    SORT_CMP_SWAP(TYPE, CMP, begin[6], begin[14]);                             \ +    SORT_CMP_SWAP(TYPE, CMP, begin[5], begin[10]);                             \ +    SORT_CMP_SWAP(TYPE, CMP, begin[6], begin[9]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[3], begin[12]);                             \ +    SORT_CMP_SWAP(TYPE, CMP, begin[13], begin[14]);                            \ +    SORT_CMP_SWAP(TYPE, CMP, begin[7], begin[11]);                             \ +    SORT_CMP_SWAP(TYPE, CMP, begin[1], begin[2]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[4], begin[8]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[1], begin[4]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[7], begin[13]);                             \ +    SORT_CMP_SWAP(TYPE, CMP, begin[2], begin[8]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[11], begin[14]);                            \ +    SORT_CMP_SWAP(TYPE, CMP, begin[5], begin[6]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[9], begin[10]);                             \ +    SORT_CMP_SWAP(TYPE, CMP, begin[2], begin[4]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[11], begin[13]);                            \ +    SORT_CMP_SWAP(TYPE, CMP, begin[3], begin[8]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[7], begin[12]);                             \ +    SORT_CMP_SWAP(TYPE, CMP, begin[6], begin[8]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[10], begin[12]);                            \ +    SORT_CMP_SWAP(TYPE, CMP, begin[3], begin[5]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[7], begin[9]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[3], begin[4]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[5], begin[6]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[7], begin[8]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[9], begin[10]);                             \ +    SORT_CMP_SWAP(TYPE, CMP, begin[11], begin[12]);                            \ +    SORT_CMP_SWAP(TYPE, CMP, begin[6], begin[7]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[8], begin[9]);                              \ +  } while (0) + +#define SORT_BITONIC_16(TYPE, CMP, begin)                                      \ +  do {                                                                         \ +    SORT_CMP_SWAP(TYPE, CMP, begin[0], begin[1]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[2], begin[3]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[4], begin[5]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[6], begin[7]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[8], begin[9]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[10], begin[11]);                            \ +    SORT_CMP_SWAP(TYPE, CMP, begin[12], begin[13]);                            \ +    SORT_CMP_SWAP(TYPE, CMP, begin[14], begin[15]);                            \ +    SORT_CMP_SWAP(TYPE, CMP, begin[0], begin[2]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[4], begin[6]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[8], begin[10]);                             \ +    SORT_CMP_SWAP(TYPE, CMP, begin[12], begin[14]);                            \ +    SORT_CMP_SWAP(TYPE, CMP, begin[1], begin[3]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[5], begin[7]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[9], begin[11]);                             \ +    SORT_CMP_SWAP(TYPE, CMP, begin[13], begin[15]);                            \ +    SORT_CMP_SWAP(TYPE, CMP, begin[0], begin[4]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[8], begin[12]);                             \ +    SORT_CMP_SWAP(TYPE, CMP, begin[1], begin[5]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[9], begin[13]);                             \ +    SORT_CMP_SWAP(TYPE, CMP, begin[2], begin[6]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[10], begin[14]);                            \ +    SORT_CMP_SWAP(TYPE, CMP, begin[3], begin[7]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[11], begin[15]);                            \ +    SORT_CMP_SWAP(TYPE, CMP, begin[0], begin[8]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[1], begin[9]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[2], begin[10]);                             \ +    SORT_CMP_SWAP(TYPE, CMP, begin[3], begin[11]);                             \ +    SORT_CMP_SWAP(TYPE, CMP, begin[4], begin[12]);                             \ +    SORT_CMP_SWAP(TYPE, CMP, begin[5], begin[13]);                             \ +    SORT_CMP_SWAP(TYPE, CMP, begin[6], begin[14]);                             \ +    SORT_CMP_SWAP(TYPE, CMP, begin[7], begin[15]);                             \ +    SORT_CMP_SWAP(TYPE, CMP, begin[5], begin[10]);                             \ +    SORT_CMP_SWAP(TYPE, CMP, begin[6], begin[9]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[3], begin[12]);                             \ +    SORT_CMP_SWAP(TYPE, CMP, begin[13], begin[14]);                            \ +    SORT_CMP_SWAP(TYPE, CMP, begin[7], begin[11]);                             \ +    SORT_CMP_SWAP(TYPE, CMP, begin[1], begin[2]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[4], begin[8]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[1], begin[4]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[7], begin[13]);                             \ +    SORT_CMP_SWAP(TYPE, CMP, begin[2], begin[8]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[11], begin[14]);                            \ +    SORT_CMP_SWAP(TYPE, CMP, begin[5], begin[6]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[9], begin[10]);                             \ +    SORT_CMP_SWAP(TYPE, CMP, begin[2], begin[4]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[11], begin[13]);                            \ +    SORT_CMP_SWAP(TYPE, CMP, begin[3], begin[8]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[7], begin[12]);                             \ +    SORT_CMP_SWAP(TYPE, CMP, begin[6], begin[8]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[10], begin[12]);                            \ +    SORT_CMP_SWAP(TYPE, CMP, begin[3], begin[5]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[7], begin[9]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[3], begin[4]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[5], begin[6]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[7], begin[8]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[9], begin[10]);                             \ +    SORT_CMP_SWAP(TYPE, CMP, begin[11], begin[12]);                            \ +    SORT_CMP_SWAP(TYPE, CMP, begin[6], begin[7]);                              \ +    SORT_CMP_SWAP(TYPE, CMP, begin[8], begin[9]);                              \ +  } while (0) + +#define SORT_INNER(TYPE, CMP, begin, end, len)                                 \ +  switch (len) {                                                               \ +  default:                                                                     \ +    __unreachable();                                                           \ +  case 0:                                                                      \ +  case 1:                                                                      \ +    break;                                                                     \ +  case 2:                                                                      \ +    SORT_BITONIC_2(TYPE, CMP, begin);                                          \ +    break;                                                                     \ +  case 3:                                                                      \ +    SORT_BITONIC_3(TYPE, CMP, begin);                                          \ +    break;                                                                     \ +  case 4:                                                                      \ +    SORT_BITONIC_4(TYPE, CMP, begin);                                          \ +    break;                                                                     \ +  case 5:                                                                      \ +    SORT_BITONIC_5(TYPE, CMP, begin);                                          \ +    break;                                                                     \ +  case 6:                                                                      \ +    SORT_BITONIC_6(TYPE, CMP, begin);                                          \ +    break;                                                                     \ +  case 7:                                                                      \ +    SORT_BITONIC_7(TYPE, CMP, begin);                                          \ +    break;                                                                     \ +  case 8:                                                                      \ +    SORT_BITONIC_8(TYPE, CMP, begin);                                          \ +    break;                                                                     \ +  case 9:                                                                      \ +    SORT_BITONIC_9(TYPE, CMP, begin);                                          \ +    break;                                                                     \ +  case 10:                                                                     \ +    SORT_BITONIC_10(TYPE, CMP, begin);                                         \ +    break;                                                                     \ +  case 11:                                                                     \ +    SORT_BITONIC_11(TYPE, CMP, begin);                                         \ +    break;                                                                     \ +  case 12:                                                                     \ +    SORT_BITONIC_12(TYPE, CMP, begin);                                         \ +    break;                                                                     \ +  case 13:                                                                     \ +    SORT_BITONIC_13(TYPE, CMP, begin);                                         \ +    break;                                                                     \ +  case 14:                                                                     \ +    SORT_BITONIC_14(TYPE, CMP, begin);                                         \ +    break;                                                                     \ +  case 15:                                                                     \ +    SORT_BITONIC_15(TYPE, CMP, begin);                                         \ +    break;                                                                     \ +  case 16:                                                                     \ +    SORT_BITONIC_16(TYPE, CMP, begin);                                         \ +    break;                                                                     \ +  }  #define SORT_SWAP(TYPE, a, b)                                                  \    do {                                                                         \ @@ -1483,19 +2002,6 @@ static int lcklist_detach_locked(MDBX_env *env) {      (b) = swap_tmp;                                                            \    } while (0) -#define SORT_SHELLPASS(TYPE, CMP, begin, end, gap)                             \ -  for (TYPE *i = begin + gap; i < end; ++i) {                                  \ -    for (TYPE *j = i - (gap); j >= begin && CMP(*i, *j); j -= gap) {           \ -      const TYPE tmp = *i;                                                     \ -      do {                                                                     \ -        j[gap] = *j;                                                           \ -        j -= gap;                                                              \ -      } while (j >= begin && CMP(tmp, *j));                                    \ -      j[gap] = tmp;                                                            \ -      break;                                                                   \ -    }                                                                          \ -  } -  #define SORT_PUSH(low, high)                                                   \    do {                                                                         \      top->lo = (low);                                                           \ @@ -1510,76 +2016,70 @@ static int lcklist_detach_locked(MDBX_env *env) {      high = top->hi;                                                            \    } while (0) -#define SORT_IMPL(NAME, TYPE, CMP)                                             \ +#define SORT_IMPL(NAME, EXPECT_LOW_CARDINALITY_OR_PRESORTED, TYPE, CMP)        \ +                                                                               \ +  static __inline bool NAME##_is_sorted(const TYPE *first, const TYPE *last) { \ +    while (++first <= last)                                                    \ +      if (CMP(first[0], first[-1]))                                            \ +        return false;                                                          \ +    return true;                                                               \ +  }                                                                            \                                                                                 \    typedef struct {                                                             \      TYPE *lo, *hi;                                                             \    } NAME##_stack;                                                              \                                                                                 \    static __hot void NAME(TYPE *const begin, TYPE *const end) {                 \ -    const ptrdiff_t length = end - begin;                                      \ -    if (length < 2)                                                            \ -      return;                                                                  \ +    NAME##_stack stack[sizeof(unsigned) * CHAR_BIT], *top = stack;             \                                                                                 \ -    if (length > SORT_THRESHOLD / 2) {                                         \ -      NAME##_stack stack[sizeof(unsigned) * CHAR_BIT], *top = stack;           \ +    TYPE *hi = end - 1;                                                        \ +    TYPE *lo = begin;                                                          \ +    while (true) {                                                             \ +      const ptrdiff_t len = hi - lo;                                           \ +      if (len < 16) {                                                          \ +        SORT_INNER(TYPE, CMP, lo, hi + 1, len + 1);                            \ +        if (unlikely(top == stack))                                            \ +          break;                                                               \ +        SORT_POP(lo, hi);                                                      \ +        continue;                                                              \ +      }                                                                        \                                                                                 \ -      TYPE *hi = end - 1;                                                      \ -      TYPE *lo = begin;                                                        \ -      while (true) {                                                           \ -        TYPE *mid = lo + ((hi - lo) >> 1);                                     \ -        if (CMP(*mid, *lo))                                                    \ -          SORT_SWAP(TYPE, *mid, *lo);                                          \ -        if (CMP(*hi, *mid)) {                                                  \ -          SORT_SWAP(TYPE, *hi, *mid);                                          \ -          if (CMP(*mid, *lo))                                                  \ -            SORT_SWAP(TYPE, *mid, *lo);                                        \ -        }                                                                      \ +      TYPE *mid = lo + (len >> 1);                                             \ +      SORT_CMP_SWAP(TYPE, CMP, *lo, *mid);                                     \ +      SORT_CMP_SWAP(TYPE, CMP, *mid, *hi);                                     \ +      SORT_CMP_SWAP(TYPE, CMP, *lo, *mid);                                     \                                                                                 \ -        TYPE *right = hi - 1;                                                  \ -        TYPE *left = lo + 1;                                                   \ -        do {                                                                   \ -          while (CMP(*mid, *right))                                            \ -            --right;                                                           \ -          while (CMP(*left, *mid))                                             \ -            ++left;                                                            \ -          if (left < right) {                                                  \ -            SORT_SWAP(TYPE, *left, *right);                                    \ -            if (mid == left)                                                   \ -              mid = right;                                                     \ -            else if (mid == right)                                             \ -              mid = left;                                                      \ -            ++left;                                                            \ -            --right;                                                           \ -          } else if (left == right) {                                          \ -            ++left;                                                            \ -            --right;                                                           \ -            break;                                                             \ +      TYPE *right = hi - 1;                                                    \ +      TYPE *left = lo + 1;                                                     \ +      while (1) {                                                              \ +        while (CMP(*left, *mid))                                               \ +          ++left;                                                              \ +        while (CMP(*mid, *right))                                              \ +          --right;                                                             \ +        if (unlikely(left > right)) {                                          \ +          if (EXPECT_LOW_CARDINALITY_OR_PRESORTED) {                           \ +            if (NAME##_is_sorted(lo, right))                                   \ +              lo = right + 1;                                                  \ +            if (NAME##_is_sorted(left, hi))                                    \ +              hi = left;                                                       \            }                                                                    \ -        } while (left <= right);                                               \ -                                                                               \ -        if (lo + SORT_THRESHOLD > right) {                                     \ -          if (left + SORT_THRESHOLD > hi) {                                    \ -            if (top == stack)                                                  \ -              break;                                                           \ -            else                                                               \ -              SORT_POP(lo, hi);                                                \ -          } else                                                               \ -            lo = left;                                                         \ -        } else if (left + SORT_THRESHOLD > hi)                                 \ -          hi = right;                                                          \ -        else if (right - lo > hi - left) {                                     \ -          SORT_PUSH(lo, right);                                                \ -          lo = left;                                                           \ -        } else {                                                               \ -          SORT_PUSH(left, hi);                                                 \ -          hi = right;                                                          \ +          break;                                                               \          }                                                                      \ +        SORT_SWAP(TYPE, *left, *right);                                        \ +        mid = (mid == left) ? right : (mid == right) ? left : mid;             \ +        ++left;                                                                \ +        --right;                                                               \ +      }                                                                        \ +                                                                               \ +      if (right - lo > hi - left) {                                            \ +        SORT_PUSH(lo, right);                                                  \ +        lo = left;                                                             \ +      } else {                                                                 \ +        SORT_PUSH(left, hi);                                                   \ +        hi = right;                                                            \        }                                                                        \      }                                                                          \                                                                                 \ -    SORT_SHELLPASS(TYPE, CMP, begin, end, 8);                                  \ -    SORT_SHELLPASS(TYPE, CMP, begin, end, 1);                                  \      for (TYPE *scan = begin + 1; scan < end; ++scan)                           \        assert(CMP(scan[-1], scan[0]));                                          \    } @@ -1868,7 +2368,7 @@ static void __hot mdbx_pnl_xmerge(MDBX_PNL dst, const MDBX_PNL src) {    assert(mdbx_pnl_check4assert(dst, MAX_PAGENO + 1));  } -SORT_IMPL(pgno_sort, pgno_t, MDBX_PNL_ORDERED) +SORT_IMPL(pgno_sort, false, pgno_t, MDBX_PNL_ORDERED)  static __hot void mdbx_pnl_sort(MDBX_PNL pnl) {    pgno_sort(MDBX_PNL_BEGIN(pnl), MDBX_PNL_END(pnl));    assert(mdbx_pnl_check(pnl, MAX_PAGENO + 1)); @@ -1978,7 +2478,7 @@ static __always_inline void mdbx_txl_xappend(MDBX_TXL tl, txnid_t id) {  }  #define TXNID_SORT_CMP(first, last) ((first) > (last)) -SORT_IMPL(txnid_sort, txnid_t, TXNID_SORT_CMP) +SORT_IMPL(txnid_sort, false, txnid_t, TXNID_SORT_CMP)  static void mdbx_txl_sort(MDBX_TXL tl) {    txnid_sort(MDBX_PNL_BEGIN(tl), MDBX_PNL_END(tl));  } @@ -1996,7 +2496,7 @@ static int __must_check_result mdbx_txl_append(MDBX_TXL *ptl, txnid_t id) {  /*----------------------------------------------------------------------------*/  #define DP_SORT_CMP(first, last) ((first).pgno < (last).pgno) -SORT_IMPL(dp_sort, MDBX_DP, DP_SORT_CMP) +SORT_IMPL(dp_sort, false, MDBX_DP, DP_SORT_CMP)  static __always_inline MDBX_DPL mdbx_dpl_sort(MDBX_DPL dl) {    assert(dl->length <= MDBX_DPL_TXNFULL);    assert(dl->sorted <= dl->length); @@ -2013,61 +2513,57 @@ static __always_inline MDBX_DPL mdbx_dpl_sort(MDBX_DPL dl) {  SEARCH_IMPL(dp_bsearch, MDBX_DP, pgno_t, DP_SEARCH_CMP)  static unsigned __hot mdbx_dpl_search(MDBX_DPL dl, pgno_t pgno) { -  if (dl->sorted < dl->length) { -    /* unsorted tail case  */ +  if (mdbx_audit_enabled()) { +    for (const MDBX_DP *ptr = dl + dl->sorted; --ptr > dl;) { +      assert(ptr[0].pgno < ptr[1].pgno); +      assert(ptr[0].pgno >= NUM_METAS); +    } +  } + +  switch (dl->length - dl->sorted) { +  default: +    /* sort a whole */ +    dl->sorted = dl->length; +    dp_sort(dl + 1, dl + dl->length + 1); +    __fallthrough; /* fall through */ +  case 0: +    /* whole sorted cases */      if (mdbx_audit_enabled()) { -      for (const MDBX_DP *ptr = dl + dl->sorted; --ptr > dl;) { +      for (const MDBX_DP *ptr = dl + dl->length; --ptr > dl;) {          assert(ptr[0].pgno < ptr[1].pgno);          assert(ptr[0].pgno >= NUM_METAS);        }      } +    return (unsigned)(dp_bsearch(dl + 1, dl->length, pgno) - dl); -    /* try linear search until the threshold */ -    if (dl->length - dl->sorted < SORT_THRESHOLD / 2) { -      unsigned i = dl->length; -      while (i - dl->sorted > 7) { -        if (dl[i].pgno == pgno) -          return i; -        if (dl[i - 1].pgno == pgno) -          return i - 1; -        if (dl[i - 2].pgno == pgno) -          return i - 2; -        if (dl[i - 3].pgno == pgno) -          return i - 3; -        if (dl[i - 4].pgno == pgno) -          return i - 4; -        if (dl[i - 5].pgno == pgno) -          return i - 5; -        if (dl[i - 6].pgno == pgno) -          return i - 6; -        if (dl[i - 7].pgno == pgno) -          return i - 7; -        i -= 8; -      } -      while (i > dl->sorted) { -        if (dl[i].pgno == pgno) -          return i; -        --i; -      } - -      MDBX_DPL it = dp_bsearch(dl + 1, i, pgno); -      return (unsigned)(it - dl); -    } - -    /* sort a whole */ -    dl->sorted = dl->length; -    dp_sort(dl + 1, dl + dl->length + 1); -  } +#define LINEAR_SEARCH_CASE(N)                                                  \ +  case N:                                                                      \ +    if (dl[dl->length - N + 1].pgno == pgno)                                   \ +      return dl->length - N + 1;                                               \ +    __fallthrough -  if (mdbx_audit_enabled()) { -    for (const MDBX_DP *ptr = dl + dl->length; --ptr > dl;) { -      assert(ptr[0].pgno < ptr[1].pgno); -      assert(ptr[0].pgno >= NUM_METAS); -    } +    /* try linear search until the threshold */ +    LINEAR_SEARCH_CASE(16); /* fall through */ +    LINEAR_SEARCH_CASE(15); /* fall through */ +    LINEAR_SEARCH_CASE(14); /* fall through */ +    LINEAR_SEARCH_CASE(13); /* fall through */ +    LINEAR_SEARCH_CASE(12); /* fall through */ +    LINEAR_SEARCH_CASE(11); /* fall through */ +    LINEAR_SEARCH_CASE(10); /* fall through */ +    LINEAR_SEARCH_CASE(9);  /* fall through */ +    LINEAR_SEARCH_CASE(8);  /* fall through */ +    LINEAR_SEARCH_CASE(7);  /* fall through */ +    LINEAR_SEARCH_CASE(6);  /* fall through */ +    LINEAR_SEARCH_CASE(5);  /* fall through */ +    LINEAR_SEARCH_CASE(4);  /* fall through */ +    LINEAR_SEARCH_CASE(3);  /* fall through */ +    LINEAR_SEARCH_CASE(2);  /* fall through */ +  case 1: +    if (dl[dl->length].pgno == pgno) +      return dl->length; +    /* continue bsearch on the sorted part */ +    return (unsigned)(dp_bsearch(dl + 1, dl->sorted, pgno) - dl);    } - -  MDBX_DPL it = dp_bsearch(dl + 1, dl->length, pgno); -  return (unsigned)(it - dl);  }  static __always_inline MDBX_page *mdbx_dpl_find(MDBX_DPL dl, pgno_t pgno) { @@ -2752,8 +3248,7 @@ static __cold __maybe_unused bool mdbx_dirtylist_check(MDBX_txn *txn) {    if (unlikely(loose != txn->tw.loose_count))      return false; -  if (txn->tw.dirtylist->length - txn->tw.dirtylist->sorted < -      SORT_THRESHOLD / 2) { +  if (txn->tw.dirtylist->length - txn->tw.dirtylist->sorted < 16) {      for (unsigned i = 1; i <= MDBX_PNL_SIZE(txn->tw.retired_pages); ++i) {        const MDBX_page *const dp =            mdbx_dpl_find(txn->tw.dirtylist, txn->tw.retired_pages[i]); @@ -10872,7 +11367,7 @@ int mdbx_cursor_put(MDBX_cursor *mc, MDBX_val *key, MDBX_val *data,    } else if ((flags & MDBX_CURRENT) == 0) {      int exact = 0;      MDBX_val d2; -    if (flags & MDBX_APPEND) { +    if ((flags & MDBX_APPEND) != 0 && mc->mc_db->md_entries > 0) {        MDBX_val k2;        rc = mdbx_cursor_last(mc, &k2, &d2);        if (rc == 0) { @@ -14884,19 +15379,21 @@ static int mdbx_dbi_bind(MDBX_txn *txn, const MDBX_dbi dbi, unsigned user_flags,      }    } -  if (!txn->mt_dbxs[dbi].md_cmp || MDBX_DEBUG) { -    if (!keycmp) -      keycmp = mdbx_default_keycmp(user_flags); -    mdbx_tassert(txn, !txn->mt_dbxs[dbi].md_cmp || -                          txn->mt_dbxs[dbi].md_cmp == keycmp); +  if (!keycmp) +    keycmp = txn->mt_dbxs[dbi].md_cmp ? txn->mt_dbxs[dbi].md_cmp +                                      : mdbx_default_keycmp(user_flags); +  if (txn->mt_dbxs[dbi].md_cmp != keycmp) { +    if (txn->mt_dbxs[dbi].md_cmp) +      return MDBX_EINVAL;      txn->mt_dbxs[dbi].md_cmp = keycmp;    } -  if (!txn->mt_dbxs[dbi].md_dcmp || MDBX_DEBUG) { -    if (!datacmp) -      datacmp = mdbx_default_datacmp(user_flags); -    mdbx_tassert(txn, !txn->mt_dbxs[dbi].md_dcmp || -                          txn->mt_dbxs[dbi].md_dcmp == datacmp); +  if (!datacmp) +    datacmp = txn->mt_dbxs[dbi].md_dcmp ? txn->mt_dbxs[dbi].md_dcmp +                                        : mdbx_default_datacmp(user_flags); +  if (txn->mt_dbxs[dbi].md_dcmp != datacmp) { +    if (txn->mt_dbxs[dbi].md_dcmp) +      return MDBX_EINVAL;      txn->mt_dbxs[dbi].md_dcmp = datacmp;    } @@ -14906,13 +15403,14 @@ static int mdbx_dbi_bind(MDBX_txn *txn, const MDBX_dbi dbi, unsigned user_flags,  int mdbx_dbi_open_ex(MDBX_txn *txn, const char *table_name, unsigned user_flags,                       MDBX_dbi *dbi, MDBX_cmp_func *keycmp,                       MDBX_cmp_func *datacmp) { +  if (unlikely(!dbi || (user_flags & ~VALID_FLAGS) != 0)) +    return MDBX_EINVAL; +  *dbi = (MDBX_dbi)-1; +    int rc = check_txn(txn, MDBX_TXN_BLOCKED);    if (unlikely(rc != MDBX_SUCCESS))      return rc; -  if (unlikely(!dbi || (user_flags & ~VALID_FLAGS) != 0)) -    return MDBX_EINVAL; -    switch (user_flags &            (MDBX_INTEGERDUP | MDBX_DUPFIXED | MDBX_DUPSORT | MDBX_REVERSEDUP)) {    default: @@ -14929,8 +15427,10 @@ int mdbx_dbi_open_ex(MDBX_txn *txn, const char *table_name, unsigned user_flags,    /* main table? */    if (!table_name) { -    *dbi = MAIN_DBI; -    return mdbx_dbi_bind(txn, MAIN_DBI, user_flags, keycmp, datacmp); +    rc = mdbx_dbi_bind(txn, MAIN_DBI, user_flags, keycmp, datacmp); +    if (likely(rc == MDBX_SUCCESS)) +      *dbi = MAIN_DBI; +    return rc;    }    if (txn->mt_dbxs[MAIN_DBI].md_cmp == NULL) { diff --git a/libs/libmdbx/src/src/elements/defs.h b/libs/libmdbx/src/src/elements/defs.h index f4265502a9..9e262e2fc2 100644 --- a/libs/libmdbx/src/src/elements/defs.h +++ b/libs/libmdbx/src/src/elements/defs.h @@ -1,5 +1,5 @@  /* - * Copyright 2015-2019 Leonid Yuriev <leo@yuriev.ru> + * Copyright 2015-2020 Leonid Yuriev <leo@yuriev.ru>   * and other libmdbx authors: please see AUTHORS file.   * All rights reserved.   * @@ -154,7 +154,7 @@  #endif /* __fallthrough */  #ifndef __unreachable -#	if __GNUC_PREREQ(4,5) +#	if __GNUC_PREREQ(4,5) || __has_builtin(__builtin_unreachable)  #		define __unreachable() __builtin_unreachable()  #	elif defined(_MSC_VER)  #		define __unreachable() __assume(0) @@ -294,7 +294,7 @@  #endif /* __flatten */  #ifndef likely -#   if (defined(__GNUC__) || defined(__clang__)) && !defined(__COVERITY__) +#   if (defined(__GNUC__) || __has_builtin(__builtin_expect)) && !defined(__COVERITY__)  #       define likely(cond) __builtin_expect(!!(cond), 1)  #   else  #       define likely(x) (x) @@ -302,7 +302,7 @@  #endif /* likely */  #ifndef unlikely -#   if (defined(__GNUC__) || defined(__clang__)) && !defined(__COVERITY__) +#   if (defined(__GNUC__) || __has_builtin(__builtin_expect)) && !defined(__COVERITY__)  #       define unlikely(cond) __builtin_expect(!!(cond), 0)  #   else  #       define unlikely(x) (x) diff --git a/libs/libmdbx/src/src/elements/internals.h b/libs/libmdbx/src/src/elements/internals.h index 7ae594ab9a..80e3500a98 100644 --- a/libs/libmdbx/src/src/elements/internals.h +++ b/libs/libmdbx/src/src/elements/internals.h @@ -1,5 +1,5 @@  /* - * Copyright 2015-2019 Leonid Yuriev <leo@yuriev.ru> + * Copyright 2015-2020 Leonid Yuriev <leo@yuriev.ru>   * and other libmdbx authors: please see AUTHORS file.   * All rights reserved.   * diff --git a/libs/libmdbx/src/src/elements/lck-posix.c b/libs/libmdbx/src/src/elements/lck-posix.c index 84652d7adf..7ef4ca4dee 100644 --- a/libs/libmdbx/src/src/elements/lck-posix.c +++ b/libs/libmdbx/src/src/elements/lck-posix.c @@ -1,5 +1,5 @@  /* - * Copyright 2015-2019 Leonid Yuriev <leo@yuriev.ru> + * Copyright 2015-2020 Leonid Yuriev <leo@yuriev.ru>   * and other libmdbx authors: please see AUTHORS file.   * All rights reserved.   * diff --git a/libs/libmdbx/src/src/elements/lck-windows.c b/libs/libmdbx/src/src/elements/lck-windows.c index 56d77f5ebf..977457de37 100644 --- a/libs/libmdbx/src/src/elements/lck-windows.c +++ b/libs/libmdbx/src/src/elements/lck-windows.c @@ -1,5 +1,5 @@  /* - * Copyright 2015-2019 Leonid Yuriev <leo@yuriev.ru> + * Copyright 2015-2020 Leonid Yuriev <leo@yuriev.ru>   * and other libmdbx authors: please see AUTHORS file.   * All rights reserved.   * diff --git a/libs/libmdbx/src/src/elements/osal.c b/libs/libmdbx/src/src/elements/osal.c index 118e57a191..158413d66a 100644 --- a/libs/libmdbx/src/src/elements/osal.c +++ b/libs/libmdbx/src/src/elements/osal.c @@ -1,7 +1,7 @@  /* https://en.wikipedia.org/wiki/Operating_system_abstraction_layer */  /* - * Copyright 2015-2019 Leonid Yuriev <leo@yuriev.ru> + * Copyright 2015-2020 Leonid Yuriev <leo@yuriev.ru>   * and other libmdbx authors: please see AUTHORS file.   * All rights reserved.   * diff --git a/libs/libmdbx/src/src/elements/osal.h b/libs/libmdbx/src/src/elements/osal.h index c590ee51c9..e7adec72a7 100644 --- a/libs/libmdbx/src/src/elements/osal.h +++ b/libs/libmdbx/src/src/elements/osal.h @@ -1,7 +1,7 @@  /* https://en.wikipedia.org/wiki/Operating_system_abstraction_layer */  /* - * Copyright 2015-2019 Leonid Yuriev <leo@yuriev.ru> + * Copyright 2015-2020 Leonid Yuriev <leo@yuriev.ru>   * and other libmdbx authors: please see AUTHORS file.   * All rights reserved.   * diff --git a/libs/libmdbx/src/src/elements/version.c b/libs/libmdbx/src/src/elements/version.c index 58234230c4..408fde08db 100644 --- a/libs/libmdbx/src/src/elements/version.c +++ b/libs/libmdbx/src/src/elements/version.c @@ -4,7 +4,7 @@  #include "internals.h"  #if MDBX_VERSION_MAJOR != 0 ||                             \ -    MDBX_VERSION_MINOR != 5 +    MDBX_VERSION_MINOR != 6  #error "API version mismatch! Had `git fetch --tags` done?"  #endif @@ -24,11 +24,11 @@ __dll_export  #endif      const mdbx_version_info mdbx_version = {          0, -        5, +        6,          0, -        1879, -        {"2020-01-07T22:30:21+03:00", "f982437718fa106bde7985a976752c08e95599a8", "3ffcc1722de83ce4cd5a37772d3b41e8aea02042", -         "v0.5.0-10-g3ffcc17"}, +        1891, +        {"2020-01-21T18:43:16+03:00", "0ff4c585e913972e7f611a07c979716d3b89ffab", "32db811545ca84f9d554eb5d453d131357f03f81", +         "v0.6.0-1-g32db811"},          sourcery};  __dll_export diff --git a/libs/libmdbx/src/src/man1/mdbx_chk.1 b/libs/libmdbx/src/src/man1/mdbx_chk.1 index f9f603e634..5ae24c9d09 100644 --- a/libs/libmdbx/src/src/man1/mdbx_chk.1 +++ b/libs/libmdbx/src/src/man1/mdbx_chk.1 @@ -1,6 +1,6 @@ -.\" Copyright 2015-2019 Leonid Yuriev <leo@yuriev.ru>. +.\" Copyright 2015-2020 Leonid Yuriev <leo@yuriev.ru>.  .\" Copying restrictions apply.  See COPYRIGHT/LICENSE. -.TH MDBX_CHK 1 "2019-12-05" "MDBX 0.4.x" +.TH MDBX_CHK 1 "2020-01-20" "MDBX 0.6.x"  .SH NAME  mdbx_chk \- MDBX checking tool  .SH SYNOPSIS diff --git a/libs/libmdbx/src/src/man1/mdbx_copy.1 b/libs/libmdbx/src/src/man1/mdbx_copy.1 index 82acdde8fa..ca8f597a22 100644 --- a/libs/libmdbx/src/src/man1/mdbx_copy.1 +++ b/libs/libmdbx/src/src/man1/mdbx_copy.1 @@ -1,8 +1,8 @@ -.\" Copyright 2015-2019 Leonid Yuriev <leo@yuriev.ru>. +.\" Copyright 2015-2020 Leonid Yuriev <leo@yuriev.ru>.  .\" Copyright 2012-2015 Howard Chu, Symas Corp. All Rights Reserved.  .\" Copyright 2015,2016 Peter-Service R&D LLC <http://billing.ru/>.  .\" Copying restrictions apply.  See COPYRIGHT/LICENSE. -.TH MDBX_COPY 1 "2019-12-05" "MDBX 0.4.x" +.TH MDBX_COPY 1 "2020-01-20" "MDBX 0.6.x"  .SH NAME  mdbx_copy \- MDBX environment copy tool  .SH SYNOPSIS diff --git a/libs/libmdbx/src/src/man1/mdbx_dump.1 b/libs/libmdbx/src/src/man1/mdbx_dump.1 index a9481f6bf8..b95260026c 100644 --- a/libs/libmdbx/src/src/man1/mdbx_dump.1 +++ b/libs/libmdbx/src/src/man1/mdbx_dump.1 @@ -1,8 +1,8 @@ -.\" Copyright 2015-2019 Leonid Yuriev <leo@yuriev.ru>. +.\" Copyright 2015-2020 Leonid Yuriev <leo@yuriev.ru>.  .\" Copyright 2014-2015 Howard Chu, Symas Corp. All Rights Reserved.  .\" Copyright 2015,2016 Peter-Service R&D LLC <http://billing.ru/>.  .\" Copying restrictions apply.  See COPYRIGHT/LICENSE. -.TH MDBX_DUMP 1 "2019-12-05" "MDBX 0.4.x" +.TH MDBX_DUMP 1 "2020-01-20" "MDBX 0.6.x"  .SH NAME  mdbx_dump \- MDBX environment export tool  .SH SYNOPSIS diff --git a/libs/libmdbx/src/src/man1/mdbx_load.1 b/libs/libmdbx/src/src/man1/mdbx_load.1 index 927969d5cc..5a13e0ec88 100644 --- a/libs/libmdbx/src/src/man1/mdbx_load.1 +++ b/libs/libmdbx/src/src/man1/mdbx_load.1 @@ -1,8 +1,8 @@ -.\" Copyright 2015-2019 Leonid Yuriev <leo@yuriev.ru>. +.\" Copyright 2015-2020 Leonid Yuriev <leo@yuriev.ru>.  .\" Copyright 2014-2015 Howard Chu, Symas Corp. All Rights Reserved.  .\" Copyright 2015,2016 Peter-Service R&D LLC <http://billing.ru/>.  .\" Copying restrictions apply.  See COPYRIGHT/LICENSE. -.TH MDBX_LOAD 1 "2019-12-05" "MDBX 0.4.x" +.TH MDBX_LOAD 1 "2020-01-20" "MDBX 0.6.x"  .SH NAME  mdbx_load \- MDBX environment import tool  .SH SYNOPSIS diff --git a/libs/libmdbx/src/src/man1/mdbx_stat.1 b/libs/libmdbx/src/src/man1/mdbx_stat.1 index b78df372f6..b312d97a69 100644 --- a/libs/libmdbx/src/src/man1/mdbx_stat.1 +++ b/libs/libmdbx/src/src/man1/mdbx_stat.1 @@ -1,8 +1,8 @@ -.\" Copyright 2015-2019 Leonid Yuriev <leo@yuriev.ru>. +.\" Copyright 2015-2020 Leonid Yuriev <leo@yuriev.ru>.  .\" Copyright 2012-2015 Howard Chu, Symas Corp. All Rights Reserved.  .\" Copyright 2015,2016 Peter-Service R&D LLC <http://billing.ru/>.  .\" Copying restrictions apply.  See COPYRIGHT/LICENSE. -.TH MDBX_STAT 1 "2019-12-05" "MDBX 0.4.x" +.TH MDBX_STAT 1 "2020-01-20" "MDBX 0.6.x"  .SH NAME  mdbx_stat \- MDBX environment status tool  .SH SYNOPSIS diff --git a/libs/libmdbx/src/src/tools/mdbx_chk.c b/libs/libmdbx/src/src/tools/mdbx_chk.c index 61fb74c7e5..e5dd539af5 100644 --- a/libs/libmdbx/src/src/tools/mdbx_chk.c +++ b/libs/libmdbx/src/src/tools/mdbx_chk.c @@ -1,7 +1,7 @@  /* mdbx_chk.c - memory-mapped database check tool */  /* - * Copyright 2015-2019 Leonid Yuriev <leo@yuriev.ru> + * Copyright 2015-2020 Leonid Yuriev <leo@yuriev.ru>   * and other libmdbx authors: please see AUTHORS file.   * All rights reserved.   * diff --git a/libs/libmdbx/src/src/tools/mdbx_copy.c b/libs/libmdbx/src/src/tools/mdbx_copy.c index b3d0ea58b2..8545b38809 100644 --- a/libs/libmdbx/src/src/tools/mdbx_copy.c +++ b/libs/libmdbx/src/src/tools/mdbx_copy.c @@ -1,7 +1,7 @@  /* mdbx_copy.c - memory-mapped database backup tool */  /* - * Copyright 2015-2019 Leonid Yuriev <leo@yuriev.ru> + * Copyright 2015-2020 Leonid Yuriev <leo@yuriev.ru>   * and other libmdbx authors: please see AUTHORS file.   * All rights reserved.   * diff --git a/libs/libmdbx/src/src/tools/mdbx_dump.c b/libs/libmdbx/src/src/tools/mdbx_dump.c index 20f47938af..3afabc0b4a 100644 --- a/libs/libmdbx/src/src/tools/mdbx_dump.c +++ b/libs/libmdbx/src/src/tools/mdbx_dump.c @@ -1,7 +1,7 @@  /* mdbx_dump.c - memory-mapped database dump tool */  /* - * Copyright 2015-2019 Leonid Yuriev <leo@yuriev.ru> + * Copyright 2015-2020 Leonid Yuriev <leo@yuriev.ru>   * and other libmdbx authors: please see AUTHORS file.   * All rights reserved.   * diff --git a/libs/libmdbx/src/src/tools/mdbx_load.c b/libs/libmdbx/src/src/tools/mdbx_load.c index c6b2ee8801..449848e7b8 100644 --- a/libs/libmdbx/src/src/tools/mdbx_load.c +++ b/libs/libmdbx/src/src/tools/mdbx_load.c @@ -1,7 +1,7 @@  /* mdbx_load.c - memory-mapped database load tool */  /* - * Copyright 2015-2019 Leonid Yuriev <leo@yuriev.ru> + * Copyright 2015-2020 Leonid Yuriev <leo@yuriev.ru>   * and other libmdbx authors: please see AUTHORS file.   * All rights reserved.   * @@ -55,7 +55,7 @@ static int version;  static int dbi_flags;  static char *prog; -static int Eof; +static bool Eof;  static MDBX_envinfo envinfo;  static MDBX_val kbuf, dbuf; @@ -85,20 +85,32 @@ static void readhdr(void) {    dbi_flags = 0;    while (fgets(dbuf.iov_base, (int)dbuf.iov_len, stdin) != NULL) {      lineno++; -    if (!strncmp(dbuf.iov_base, "db_pagesize=", STRLENOF("db_pagesize=")) || -        !strncmp(dbuf.iov_base, "duplicates=", STRLENOF("duplicates="))) { -      /* LY: silently ignore information fields. */ + +    if (!strncmp(dbuf.iov_base, "db_pagesize=", STRLENOF("db_pagesize="))) { +      envinfo.mi_dxb_pagesize = +          atoi((char *)dbuf.iov_base + STRLENOF("db_pagesize=")); +      continue; +    } + +    if (!strncmp(dbuf.iov_base, "duplicates=", STRLENOF("duplicates="))) { +      dbi_flags |= MDBX_DUPSORT;        continue; -    } else if (!strncmp(dbuf.iov_base, "VERSION=", STRLENOF("VERSION="))) { +    } + +    if (!strncmp(dbuf.iov_base, "VERSION=", STRLENOF("VERSION="))) {        version = atoi((char *)dbuf.iov_base + STRLENOF("VERSION="));        if (version > 3) {          fprintf(stderr, "%s: line %" PRIiSIZE ": unsupported VERSION %d\n",                  prog, lineno, version);          exit(EXIT_FAILURE);        } -    } else if (!strncmp(dbuf.iov_base, "HEADER=END", STRLENOF("HEADER=END"))) { -      break; -    } else if (!strncmp(dbuf.iov_base, "format=", STRLENOF("format="))) { +      continue; +    } + +    if (!strncmp(dbuf.iov_base, "HEADER=END", STRLENOF("HEADER=END"))) +      return; + +    if (!strncmp(dbuf.iov_base, "format=", STRLENOF("format="))) {        if (!strncmp((char *)dbuf.iov_base + STRLENOF("FORMAT="), "print",                     STRLENOF("print")))          mode |= PRINT; @@ -108,21 +120,30 @@ static void readhdr(void) {                  lineno, (char *)dbuf.iov_base + STRLENOF("FORMAT="));          exit(EXIT_FAILURE);        } -    } else if (!strncmp(dbuf.iov_base, "database=", STRLENOF("database="))) { +      continue; +    } + +    if (!strncmp(dbuf.iov_base, "database=", STRLENOF("database="))) {        ptr = memchr(dbuf.iov_base, '\n', dbuf.iov_len);        if (ptr)          *ptr = '\0';        if (subname)          mdbx_free(subname);        subname = mdbx_strdup((char *)dbuf.iov_base + STRLENOF("database=")); -    } else if (!strncmp(dbuf.iov_base, "type=", STRLENOF("type="))) { +      continue; +    } + +    if (!strncmp(dbuf.iov_base, "type=", STRLENOF("type="))) {        if (strncmp((char *)dbuf.iov_base + STRLENOF("type="), "btree",                    STRLENOF("btree"))) {          fprintf(stderr, "%s: line %" PRIiSIZE ": unsupported type %s\n", prog,                  lineno, (char *)dbuf.iov_base + STRLENOF("type="));          exit(EXIT_FAILURE);        } -    } else if (!strncmp(dbuf.iov_base, "mapaddr=", STRLENOF("mapaddr="))) { +      continue; +    } + +    if (!strncmp(dbuf.iov_base, "mapaddr=", STRLENOF("mapaddr="))) {        int i;        ptr = memchr(dbuf.iov_base, '\n', dbuf.iov_len);        if (ptr) @@ -134,7 +155,10 @@ static void readhdr(void) {                  lineno, (char *)dbuf.iov_base + STRLENOF("mapaddr="));          exit(EXIT_FAILURE);        } -    } else if (!strncmp(dbuf.iov_base, "mapsize=", STRLENOF("mapsize="))) { +      continue; +    } + +    if (!strncmp(dbuf.iov_base, "mapsize=", STRLENOF("mapsize="))) {        int i;        ptr = memchr(dbuf.iov_base, '\n', dbuf.iov_len);        if (ptr) @@ -146,8 +170,10 @@ static void readhdr(void) {                  lineno, (char *)dbuf.iov_base + STRLENOF("mapsize="));          exit(EXIT_FAILURE);        } -    } else if (!strncmp(dbuf.iov_base, -                        "maxreaders=", STRLENOF("maxreaders="))) { +      continue; +    } + +    if (!strncmp(dbuf.iov_base, "maxreaders=", STRLENOF("maxreaders="))) {        int i;        ptr = memchr(dbuf.iov_base, '\n', dbuf.iov_len);        if (ptr) @@ -159,31 +185,33 @@ static void readhdr(void) {                  lineno, (char *)dbuf.iov_base + STRLENOF("maxreaders="));          exit(EXIT_FAILURE);        } -    } else { -      int i; -      for (i = 0; dbflags[i].bit; i++) { -        if (!strncmp(dbuf.iov_base, dbflags[i].name, dbflags[i].len) && -            ((char *)dbuf.iov_base)[dbflags[i].len] == '=') { -          if (((char *)dbuf.iov_base)[dbflags[i].len + 1] == '1') -            dbi_flags |= dbflags[i].bit; -          break; -        } +      continue; +    } + +    int i; +    for (i = 0; dbflags[i].bit; i++) { +      if (!strncmp(dbuf.iov_base, dbflags[i].name, dbflags[i].len) && +          ((char *)dbuf.iov_base)[dbflags[i].len] == '=') { +        if (((char *)dbuf.iov_base)[dbflags[i].len + 1] == '1') +          dbi_flags |= dbflags[i].bit; +        break;        } -      if (!dbflags[i].bit) { -        ptr = memchr(dbuf.iov_base, '=', dbuf.iov_len); -        if (!ptr) { -          fprintf(stderr, "%s: line %" PRIiSIZE ": unexpected format\n", prog, -                  lineno); -          exit(EXIT_FAILURE); -        } else { -          *ptr = '\0'; -          fprintf(stderr, -                  "%s: line %" PRIiSIZE ": unrecognized keyword ignored: %s\n", -                  prog, lineno, (char *)dbuf.iov_base); -        } +    } +    if (!dbflags[i].bit) { +      ptr = memchr(dbuf.iov_base, '=', dbuf.iov_len); +      if (!ptr) { +        fprintf(stderr, "%s: line %" PRIiSIZE ": unexpected format\n", prog, +                lineno); +        exit(EXIT_FAILURE); +      } else { +        *ptr = '\0'; +        fprintf(stderr, +                "%s: line %" PRIiSIZE ": unrecognized keyword ignored: %s\n", +                prog, lineno, (char *)dbuf.iov_base);        }      }    } +  Eof = true;  }  static void badend(void) { @@ -212,14 +240,14 @@ static int readline(MDBX_val *out, MDBX_val *buf) {    if (!(mode & NOHDR)) {      c = fgetc(stdin);      if (c == EOF) { -      Eof = 1; +      Eof = true;        return EOF;      }      if (c != ' ') {        lineno++;        if (fgets(buf->iov_base, (int)buf->iov_len, stdin) == NULL) {        badend: -        Eof = 1; +        Eof = true;          badend();          return EOF;        } @@ -229,7 +257,7 @@ static int readline(MDBX_val *out, MDBX_val *buf) {      }    }    if (fgets(buf->iov_base, (int)buf->iov_len, stdin) == NULL) { -    Eof = 1; +    Eof = true;      return EOF;    }    lineno++; @@ -242,7 +270,7 @@ static int readline(MDBX_val *out, MDBX_val *buf) {    while (c1[len - 1] != '\n') {      buf->iov_base = mdbx_realloc(buf->iov_base, buf->iov_len * 2);      if (!buf->iov_base) { -      Eof = 1; +      Eof = true;        fprintf(stderr, "%s: line %" PRIiSIZE ": out of memory, line too long\n",                prog, lineno);        return EOF; @@ -250,7 +278,7 @@ static int readline(MDBX_val *out, MDBX_val *buf) {      c1 = buf->iov_base;      c1 += l2;      if (fgets((char *)c1, (int)buf->iov_len + 1, stdin) == NULL) { -      Eof = 1; +      Eof = true;        badend();        return EOF;      } @@ -270,7 +298,7 @@ static int readline(MDBX_val *out, MDBX_val *buf) {            *c1++ = '\\';          } else {            if (c2 + 3 > end || !isxdigit(c2[1]) || !isxdigit(c2[2])) { -            Eof = 1; +            Eof = true;              badend();              return EOF;            } @@ -285,13 +313,13 @@ static int readline(MDBX_val *out, MDBX_val *buf) {    } else {      /* odd length not allowed */      if (len & 1) { -      Eof = 1; +      Eof = true;        badend();        return EOF;      }      while (c2 < end) {        if (!isxdigit(*c2) || !isxdigit(c2[1])) { -        Eof = 1; +        Eof = true;          badend();          return EOF;        } @@ -366,7 +394,7 @@ int main(int argc, char *argv[]) {        break;      case 'f':        if (freopen(optarg, "r", stdin) == NULL) { -        fprintf(stderr, "%s: %s: reopen: %s\n", prog, optarg, +        fprintf(stderr, "%s: %s: open: %s\n", prog, optarg,                  mdbx_strerror(errno));          exit(EXIT_FAILURE);        } @@ -433,16 +461,20 @@ int main(int argc, char *argv[]) {    mdbx_env_set_maxdbs(env, 2); -  if (envinfo.mi_maxreaders) -    mdbx_env_set_maxreaders(env, envinfo.mi_maxreaders); +#ifdef MDBX_FIXEDMAP +  if (info.mi_mapaddr) +    envflags |= MDBX_FIXEDMAP; +#endif    if (envinfo.mi_mapsize) { -    if (envinfo.mi_mapsize > SIZE_MAX) { -      fprintf(stderr, "mdbx_env_set_mapsize failed, error %d %s\n", rc, -              mdbx_strerror(MDBX_TOO_LARGE)); -      return EXIT_FAILURE; +    if (envinfo.mi_mapsize > INTPTR_MAX) { +      fprintf(stderr, +              "Database size is too large for current system (mapsize=%" PRIu64 +              " is great than system-limit %zi)\n", +              envinfo.mi_mapsize, INTPTR_MAX); +      goto env_close;      } -    rc = mdbx_env_set_geometry(env, 0, -1, (size_t)envinfo.mi_mapsize, -1, -1, +    rc = mdbx_env_set_geometry(env, 0, 0, (intptr_t)envinfo.mi_mapsize, -1, -1,                                 -1);      if (rc) {        fprintf(stderr, "mdbx_env_set_geometry failed, error %d %s\n", rc, @@ -451,11 +483,6 @@ int main(int argc, char *argv[]) {      }    } -#ifdef MDBX_FIXEDMAP -  if (info.mi_mapaddr) -    envflags |= MDBX_FIXEDMAP; -#endif -    rc = mdbx_env_open(env, envname, envflags, 0664);    if (rc) {      fprintf(stderr, "mdbx_env_open failed, error %d %s\n", rc, @@ -464,7 +491,7 @@ int main(int argc, char *argv[]) {    }    kbuf.iov_len = mdbx_env_get_maxvalsize_ex(env, MDBX_DUPSORT); -  if (kbuf.iov_len >= SIZE_MAX / 4) { +  if (kbuf.iov_len >= INTPTR_MAX / 4) {      fprintf(stderr, "mdbx_env_get_maxkeysize failed, returns %zu\n",              kbuf.iov_len);      goto env_close; @@ -574,6 +601,7 @@ int main(int argc, char *argv[]) {        goto env_close;      }      mdbx_dbi_close(env, dbi); +    subname = NULL;      /* try read next header */      if (!(mode & NOHDR)) diff --git a/libs/libmdbx/src/src/tools/mdbx_stat.c b/libs/libmdbx/src/src/tools/mdbx_stat.c index 461f569afb..0867aedac3 100644 --- a/libs/libmdbx/src/src/tools/mdbx_stat.c +++ b/libs/libmdbx/src/src/tools/mdbx_stat.c @@ -1,7 +1,7 @@  /* mdbx_stat.c - memory-mapped database status tool */  /* - * Copyright 2015-2019 Leonid Yuriev <leo@yuriev.ru> + * Copyright 2015-2020 Leonid Yuriev <leo@yuriev.ru>   * and other libmdbx authors: please see AUTHORS file.   * All rights reserved.   * diff --git a/libs/libmdbx/src/test/append.cc b/libs/libmdbx/src/test/append.cc index a30351dcde..2fc8e429da 100644 --- a/libs/libmdbx/src/test/append.cc +++ b/libs/libmdbx/src/test/append.cc @@ -1,5 +1,5 @@  /* - * Copyright 2017-2019 Leonid Yuriev <leo@yuriev.ru> + * Copyright 2017-2020 Leonid Yuriev <leo@yuriev.ru>   * and other libmdbx authors: please see AUTHORS file.   * All rights reserved.   * diff --git a/libs/libmdbx/src/test/base.h b/libs/libmdbx/src/test/base.h index 05dfd1c4e3..01cf171455 100644 --- a/libs/libmdbx/src/test/base.h +++ b/libs/libmdbx/src/test/base.h @@ -1,5 +1,5 @@  /* - * Copyright 2017-2019 Leonid Yuriev <leo@yuriev.ru> + * Copyright 2017-2020 Leonid Yuriev <leo@yuriev.ru>   * and other libmdbx authors: please see AUTHORS file.   * All rights reserved.   * diff --git a/libs/libmdbx/src/test/cases.cc b/libs/libmdbx/src/test/cases.cc index b01169004b..98255f52ca 100644 --- a/libs/libmdbx/src/test/cases.cc +++ b/libs/libmdbx/src/test/cases.cc @@ -1,5 +1,5 @@  /* - * Copyright 2017-2019 Leonid Yuriev <leo@yuriev.ru> + * Copyright 2017-2020 Leonid Yuriev <leo@yuriev.ru>   * and other libmdbx authors: please see AUTHORS file.   * All rights reserved.   * diff --git a/libs/libmdbx/src/test/chrono.cc b/libs/libmdbx/src/test/chrono.cc index 315d379088..43cd2a370c 100644 --- a/libs/libmdbx/src/test/chrono.cc +++ b/libs/libmdbx/src/test/chrono.cc @@ -1,5 +1,5 @@  /* - * Copyright 2017-2019 Leonid Yuriev <leo@yuriev.ru> + * Copyright 2017-2020 Leonid Yuriev <leo@yuriev.ru>   * and other libmdbx authors: please see AUTHORS file.   * All rights reserved.   * diff --git a/libs/libmdbx/src/test/chrono.h b/libs/libmdbx/src/test/chrono.h index 07cdef66cf..f3d8debe74 100644 --- a/libs/libmdbx/src/test/chrono.h +++ b/libs/libmdbx/src/test/chrono.h @@ -1,5 +1,5 @@  /* - * Copyright 2017-2019 Leonid Yuriev <leo@yuriev.ru> + * Copyright 2017-2020 Leonid Yuriev <leo@yuriev.ru>   * and other libmdbx authors: please see AUTHORS file.   * All rights reserved.   * diff --git a/libs/libmdbx/src/test/config.cc b/libs/libmdbx/src/test/config.cc index eedfd7b2d4..f7f960c579 100644 --- a/libs/libmdbx/src/test/config.cc +++ b/libs/libmdbx/src/test/config.cc @@ -1,5 +1,5 @@  /* - * Copyright 2017-2019 Leonid Yuriev <leo@yuriev.ru> + * Copyright 2017-2020 Leonid Yuriev <leo@yuriev.ru>   * and other libmdbx authors: please see AUTHORS file.   * All rights reserved.   * diff --git a/libs/libmdbx/src/test/config.h b/libs/libmdbx/src/test/config.h index 9e083f3f22..7efd09dd54 100644 --- a/libs/libmdbx/src/test/config.h +++ b/libs/libmdbx/src/test/config.h @@ -1,5 +1,5 @@  /* - * Copyright 2017-2019 Leonid Yuriev <leo@yuriev.ru> + * Copyright 2017-2020 Leonid Yuriev <leo@yuriev.ru>   * and other libmdbx authors: please see AUTHORS file.   * All rights reserved.   * diff --git a/libs/libmdbx/src/test/dead.cc b/libs/libmdbx/src/test/dead.cc index 8f83bbeb2e..97f326a2f4 100644 --- a/libs/libmdbx/src/test/dead.cc +++ b/libs/libmdbx/src/test/dead.cc @@ -1,5 +1,5 @@  /* - * Copyright 2017-2019 Leonid Yuriev <leo@yuriev.ru> + * Copyright 2017-2020 Leonid Yuriev <leo@yuriev.ru>   * and other libmdbx authors: please see AUTHORS file.   * All rights reserved.   * diff --git a/libs/libmdbx/src/test/hill.cc b/libs/libmdbx/src/test/hill.cc index 9d9890951a..37f748c9f7 100644 --- a/libs/libmdbx/src/test/hill.cc +++ b/libs/libmdbx/src/test/hill.cc @@ -1,5 +1,5 @@  /* - * Copyright 2017-2019 Leonid Yuriev <leo@yuriev.ru> + * Copyright 2017-2020 Leonid Yuriev <leo@yuriev.ru>   * and other libmdbx authors: please see AUTHORS file.   * All rights reserved.   * diff --git a/libs/libmdbx/src/test/jitter.cc b/libs/libmdbx/src/test/jitter.cc index c7b8bccb51..d62d71d651 100644 --- a/libs/libmdbx/src/test/jitter.cc +++ b/libs/libmdbx/src/test/jitter.cc @@ -1,5 +1,5 @@  /* - * Copyright 2017-2019 Leonid Yuriev <leo@yuriev.ru> + * Copyright 2017-2020 Leonid Yuriev <leo@yuriev.ru>   * and other libmdbx authors: please see AUTHORS file.   * All rights reserved.   * diff --git a/libs/libmdbx/src/test/keygen.cc b/libs/libmdbx/src/test/keygen.cc index c2098e6ef9..374537dd6e 100644 --- a/libs/libmdbx/src/test/keygen.cc +++ b/libs/libmdbx/src/test/keygen.cc @@ -1,5 +1,5 @@  /* - * Copyright 2017-2019 Leonid Yuriev <leo@yuriev.ru> + * Copyright 2017-2020 Leonid Yuriev <leo@yuriev.ru>   * and other libmdbx authors: please see AUTHORS file.   * All rights reserved.   * diff --git a/libs/libmdbx/src/test/keygen.h b/libs/libmdbx/src/test/keygen.h index d25b88349f..b5674f1ca6 100644 --- a/libs/libmdbx/src/test/keygen.h +++ b/libs/libmdbx/src/test/keygen.h @@ -1,5 +1,5 @@  /* - * Copyright 2017-2019 Leonid Yuriev <leo@yuriev.ru> + * Copyright 2017-2020 Leonid Yuriev <leo@yuriev.ru>   * and other libmdbx authors: please see AUTHORS file.   * All rights reserved.   * diff --git a/libs/libmdbx/src/test/log.cc b/libs/libmdbx/src/test/log.cc index 2b22e28785..f568e774fc 100644 --- a/libs/libmdbx/src/test/log.cc +++ b/libs/libmdbx/src/test/log.cc @@ -1,5 +1,5 @@  /* - * Copyright 2017-2019 Leonid Yuriev <leo@yuriev.ru> + * Copyright 2017-2020 Leonid Yuriev <leo@yuriev.ru>   * and other libmdbx authors: please see AUTHORS file.   * All rights reserved.   * diff --git a/libs/libmdbx/src/test/log.h b/libs/libmdbx/src/test/log.h index bb8f997b36..bb24893779 100644 --- a/libs/libmdbx/src/test/log.h +++ b/libs/libmdbx/src/test/log.h @@ -1,5 +1,5 @@  /* - * Copyright 2017-2019 Leonid Yuriev <leo@yuriev.ru> + * Copyright 2017-2020 Leonid Yuriev <leo@yuriev.ru>   * and other libmdbx authors: please see AUTHORS file.   * All rights reserved.   * diff --git a/libs/libmdbx/src/test/main.cc b/libs/libmdbx/src/test/main.cc index c00837fa72..656d7c52f9 100644 --- a/libs/libmdbx/src/test/main.cc +++ b/libs/libmdbx/src/test/main.cc @@ -1,5 +1,5 @@  /* - * Copyright 2017-2019 Leonid Yuriev <leo@yuriev.ru> + * Copyright 2017-2020 Leonid Yuriev <leo@yuriev.ru>   * and other libmdbx authors: please see AUTHORS file.   * All rights reserved.   * diff --git a/libs/libmdbx/src/test/nested.cc b/libs/libmdbx/src/test/nested.cc index edb3242fab..beebd88071 100644 --- a/libs/libmdbx/src/test/nested.cc +++ b/libs/libmdbx/src/test/nested.cc @@ -1,5 +1,5 @@  /* - * Copyright 2017-2019 Leonid Yuriev <leo@yuriev.ru> + * Copyright 2017-2020 Leonid Yuriev <leo@yuriev.ru>   * and other libmdbx authors: please see AUTHORS file.   * All rights reserved.   * diff --git a/libs/libmdbx/src/test/osal-unix.cc b/libs/libmdbx/src/test/osal-unix.cc index 4b8694c708..edc3e91a48 100644 --- a/libs/libmdbx/src/test/osal-unix.cc +++ b/libs/libmdbx/src/test/osal-unix.cc @@ -1,5 +1,5 @@  /* - * Copyright 2017-2019 Leonid Yuriev <leo@yuriev.ru> + * Copyright 2017-2020 Leonid Yuriev <leo@yuriev.ru>   * and other libmdbx authors: please see AUTHORS file.   * All rights reserved.   * diff --git a/libs/libmdbx/src/test/osal.h b/libs/libmdbx/src/test/osal.h index 6d0e1c4eb6..e47450b931 100644 --- a/libs/libmdbx/src/test/osal.h +++ b/libs/libmdbx/src/test/osal.h @@ -1,5 +1,5 @@  /* - * Copyright 2017-2019 Leonid Yuriev <leo@yuriev.ru> + * Copyright 2017-2020 Leonid Yuriev <leo@yuriev.ru>   * and other libmdbx authors: please see AUTHORS file.   * All rights reserved.   * diff --git a/libs/libmdbx/src/test/pcrf/pcrf_test.c b/libs/libmdbx/src/test/pcrf/pcrf_test.c index 206bd21d4e..b16e79f8ee 100644 --- a/libs/libmdbx/src/test/pcrf/pcrf_test.c +++ b/libs/libmdbx/src/test/pcrf/pcrf_test.c @@ -1,5 +1,5 @@  /* - * Copyright 2016-2019 Leonid Yuriev <leo@yuriev.ru>. + * Copyright 2016-2020 Leonid Yuriev <leo@yuriev.ru>.   * Copyright 2015 Vladimir Romanov   * <https://www.linkedin.com/in/vladimirromanov>, Yota Lab.   * diff --git a/libs/libmdbx/src/test/test.cc b/libs/libmdbx/src/test/test.cc index ff1676274b..9da2cf2fff 100644 --- a/libs/libmdbx/src/test/test.cc +++ b/libs/libmdbx/src/test/test.cc @@ -1,5 +1,5 @@  /* - * Copyright 2017-2019 Leonid Yuriev <leo@yuriev.ru> + * Copyright 2017-2020 Leonid Yuriev <leo@yuriev.ru>   * and other libmdbx authors: please see AUTHORS file.   * All rights reserved.   * diff --git a/libs/libmdbx/src/test/test.h b/libs/libmdbx/src/test/test.h index 630059ac3f..8837cb598b 100644 --- a/libs/libmdbx/src/test/test.h +++ b/libs/libmdbx/src/test/test.h @@ -1,5 +1,5 @@  /* - * Copyright 2017-2019 Leonid Yuriev <leo@yuriev.ru> + * Copyright 2017-2020 Leonid Yuriev <leo@yuriev.ru>   * and other libmdbx authors: please see AUTHORS file.   * All rights reserved.   * diff --git a/libs/libmdbx/src/test/ttl.cc b/libs/libmdbx/src/test/ttl.cc index 782a8b4d56..92e99b82f6 100644 --- a/libs/libmdbx/src/test/ttl.cc +++ b/libs/libmdbx/src/test/ttl.cc @@ -1,5 +1,5 @@  /* - * Copyright 2017-2019 Leonid Yuriev <leo@yuriev.ru> + * Copyright 2017-2020 Leonid Yuriev <leo@yuriev.ru>   * and other libmdbx authors: please see AUTHORS file.   * All rights reserved.   * diff --git a/libs/libmdbx/src/test/utils.cc b/libs/libmdbx/src/test/utils.cc index 311cf544ef..596e66e3f5 100644 --- a/libs/libmdbx/src/test/utils.cc +++ b/libs/libmdbx/src/test/utils.cc @@ -1,5 +1,5 @@  /* - * Copyright 2017-2019 Leonid Yuriev <leo@yuriev.ru> + * Copyright 2017-2020 Leonid Yuriev <leo@yuriev.ru>   * and other libmdbx authors: please see AUTHORS file.   * All rights reserved.   * diff --git a/libs/libmdbx/src/test/utils.h b/libs/libmdbx/src/test/utils.h index 2a5a54de6d..a5061c1444 100644 --- a/libs/libmdbx/src/test/utils.h +++ b/libs/libmdbx/src/test/utils.h @@ -1,5 +1,5 @@  /* - * Copyright 2017-2019 Leonid Yuriev <leo@yuriev.ru> + * Copyright 2017-2020 Leonid Yuriev <leo@yuriev.ru>   * and other libmdbx authors: please see AUTHORS file.   * All rights reserved.   *  | 
