d0s3t3ch/wownero
1
0
Fork 0
mirror of https://git.wownero.com/wownero/wownero.git synced 2024-08-15 01:03:23 +00:00
Code Issues Projects Releases Wiki Activity
wownero/src/blockchain_db/lmdb/db_lmdb.h

456 lines
16 KiB
C
Raw Normal View History

Update 2018 copyright
2018-01-07 05:05:16 +00:00
// Copyright (c) 2014-2018, The Monero Project
Initial commit of lmdb BlockchainDB impl
2014-10-21 20:33:43 +00:00
// All rights reserved.
Replace tabs with two spaces for consistency with rest of codebase Remove trailing whitespace in same files.
2015-12-14 04:54:39 +00:00
//
Initial commit of lmdb BlockchainDB impl
2014-10-21 20:33:43 +00:00
// Redistribution and use in source and binary forms, with or without modification, are
// permitted provided that the following conditions are met:
Replace tabs with two spaces for consistency with rest of codebase Remove trailing whitespace in same files.
2015-12-14 04:54:39 +00:00
//
Initial commit of lmdb BlockchainDB impl
2014-10-21 20:33:43 +00:00
// 1. Redistributions of source code must retain the above copyright notice, this list of
// conditions and the following disclaimer.
Replace tabs with two spaces for consistency with rest of codebase Remove trailing whitespace in same files.
2015-12-14 04:54:39 +00:00
//
Initial commit of lmdb BlockchainDB impl
2014-10-21 20:33:43 +00:00
// 2. Redistributions in binary form must reproduce the above copyright notice, this list
// of conditions and the following disclaimer in the documentation and/or other
// materials provided with the distribution.
Replace tabs with two spaces for consistency with rest of codebase Remove trailing whitespace in same files.
2015-12-14 04:54:39 +00:00
//
Initial commit of lmdb BlockchainDB impl
2014-10-21 20:33:43 +00:00
// 3. Neither the name of the copyright holder nor the names of its contributors may be
// used to endorse or promote products derived from this software without specific
// prior written permission.
Replace tabs with two spaces for consistency with rest of codebase Remove trailing whitespace in same files.
2015-12-14 04:54:39 +00:00
//
Initial commit of lmdb BlockchainDB impl
2014-10-21 20:33:43 +00:00
// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY
// EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
// MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL
// THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
// SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
// PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
// INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,
// STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF
// THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
Steps toward multiple dbs available -- working There will need to be some more refactoring for these changes to be considered complete/correct, but for now it's working. new daemon cli argument "--db-type", works for LMDB and BerkeleyDB. A good deal of refactoring is also present in this commit, namely Blockchain no longer instantiates BlockchainDB, but rather is passed a pointer to an already-instantiated BlockchainDB on init().
2015-03-25 15:41:30 +00:00
#pragma once
Initial commit of lmdb BlockchainDB impl
2014-10-21 20:33:43 +00:00
LMDB should now dynamically resize the mapsize Some filesystems (*cough* NTFS *cough*) aren't good with sparse files, so this makes LMDB dynamically resize its mapsize as needed. Note: the check interval is currently every 10 blocks (for testing) and will probably need to change to 1000 or something. Default mapsize set to 1GiB. Blockchain conversion tools using batching will probably segfault, I'll fix that in the next commit.
2015-05-17 02:05:54 +00:00
#include <atomic>
Moved BlockchainDB into its own src/ subfolder Ostensibly janitorial work, but should be more relevant later down the line. Things that depend on core cryptonote things (i.e. cryptonote_core) don't necessarily depend on BlockchainDB and thus have no need to have BlockchainDB baked in with them.
2015-03-06 20:20:45 +00:00
#include "blockchain_db/blockchain_db.h"
move includes around to lessen overall load
2017-11-25 22:25:05 +00:00
#include "cryptonote_basic/blobdatatype.h" // for type blobdata
blockchain_db: add functions for adding/removing/getting rct commitments
2016-06-04 13:18:37 +00:00
#include "ringct/rctTypes.h"
read txn/cursor stuff Could wrap more later.
2016-02-18 12:09:57 +00:00
#include <boost/thread/tss.hpp>
Initial commit of lmdb BlockchainDB impl
2014-10-21 20:33:43 +00:00
#include <lmdb.h>
** CHANGES ARE EXPERIMENTAL (FOR TESTING ONLY) Bockchain: 1. Optim: Multi-thread long-hash computation when encountering groups of blocks. 2. Optim: Cache verified txs and return result from cache instead of re-checking whenever possible. 3. Optim: Preload output-keys when encoutering groups of blocks. Sort by amount and global-index before bulk querying database and multi-thread when possible. 4. Optim: Disable double spend check on block verification, double spend is already detected when trying to add blocks. 5. Optim: Multi-thread signature computation whenever possible. 6. Patch: Disable locking (recursive mutex) on called functions from check_tx_inputs which causes slowdowns (only seems to happen on ubuntu/VMs??? Reason: TBD) 7. Optim: Removed looped full-tx hash computation when retrieving transactions from pool (???). 8. Optim: Cache difficulty/timestamps (735 blocks) for next-difficulty calculations so that only 2 db reads per new block is needed when a new block arrives (instead of 1470 reads). Berkeley-DB: 1. Fix: 32-bit data errors causing wrong output global indices and failure to send blocks to peers (etc). 2. Fix: Unable to pop blocks on reorganize due to transaction errors. 3. Patch: Large number of transaction aborts when running multi-threaded bulk queries. 4. Patch: Insufficient locks error when running full sync. 5. Patch: Incorrect db stats when returning from an immediate exit from "pop block" operation. 6. Optim: Add bulk queries to get output global indices. 7. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 8. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 9. Optim: Added thread-safe buffers used when multi-threading bulk queries. 10. Optim: Added support for nosync/write_nosync options for improved performance (*see --db-sync-mode option for details) 11. Mod: Added checkpoint thread and auto-remove-logs option. 12. *Now usable on 32-bit systems like RPI2. LMDB: 1. Optim: Added custom comparison for 256-bit key tables (minor speed-up, TBD: get actual effect) 2. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 3. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 4. Optim: Added support for sync/writemap options for improved performance (*see --db-sync-mode option for details) 5. Mod: Auto resize to +1GB instead of multiplier x1.5 ETC: 1. Minor optimizations for slow-hash for ARM (RPI2). Incomplete. 2. Fix: 32-bit saturation bug when computing next difficulty on large blocks. [PENDING ISSUES] 1. Berkely db has a very slow "pop-block" operation. This is very noticeable on the RPI2 as it sometimes takes > 10 MINUTES to pop a block during reorganization. This does not happen very often however, most reorgs seem to take a few seconds but it possibly depends on the number of outputs present. TBD. 2. Berkeley db, possible bug "unable to allocate memory". TBD. [NEW OPTIONS] (*Currently all enabled for testing purposes) 1. --fast-block-sync arg=[0:1] (default: 1) a. 0 = Compute long hash per block (may take a while depending on CPU) b. 1 = Skip long-hash and verify blocks based on embedded known good block hashes (faster, minimal CPU dependence) 2. --db-sync-mode arg=[[safe|fast|fastest]:[sync|async]:[nblocks_per_sync]] (default: fastest:async:1000) a. safe = fdatasync/fsync (or equivalent) per stored block. Very slow, but safest option to protect against power-out/crash conditions. b. fast/fastest = Enables asynchronous fdatasync/fsync (or equivalent). Useful for battery operated devices or STABLE systems with UPS and/or systems with battery backed write cache/solid state cache. Fast - Write meta-data but defer data flush. Fastest - Defer meta-data and data flush. Sync - Flush data after nblocks_per_sync and wait. Async - Flush data after nblocks_per_sync but do not wait for the operation to finish. 3. --prep-blocks-threads arg=[n] (default: 4 or system max threads, whichever is lower) Max number of threads to use when computing long-hash in groups. 4. --show-time-stats arg=[0:1] (default: 1) Show benchmark related time stats. 5. --db-auto-remove-logs arg=[0:1] (default: 1) For berkeley-db only. Auto remove logs if enabled. **Note: lmdb and berkeley-db have changes to the tables and are not compatible with official git head version. At the moment, you need a full resync to use this optimized version. [PERFORMANCE COMPARISON] **Some figures are approximations only. Using a baseline machine of an i7-2600K+SSD+(with full pow computation): 1. The optimized lmdb/blockhain core can process blocks up to 585K for ~1.25 hours + download time, so it usually takes 2.5 hours to sync the full chain. 2. The current head with memory can process blocks up to 585K for ~4.2 hours + download time, so it usually takes 5.5 hours to sync the full chain. 3. The current head with lmdb can process blocks up to 585K for ~32 hours + download time and usually takes 36 hours to sync the full chain. Averate procesing times (with full pow computation): lmdb-optimized: 1. tx_ave = 2.5 ms / tx 2. block_ave = 5.87 ms / block memory-official-repo: 1. tx_ave = 8.85 ms / tx 2. block_ave = 19.68 ms / block lmdb-official-repo (0f4a036437fd41a5498ee5e74e2422ea6177aa3e) 1. tx_ave = 47.8 ms / tx 2. block_ave = 64.2 ms / block **Note: The following data denotes processing times only (does not include p2p download time) lmdb-optimized processing times (with full pow computation): 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.25 hours processing time (--db-sync-mode=fastest:async:1000). 2. Laptop, Dual-core / 4-threads U4200 (3Mb) - 4.90 hours processing time (--db-sync-mode=fastest:async:1000). 3. Embedded, Quad-core / 4-threads Z3735F (2x1Mb) - 12.0 hours processing time (--db-sync-mode=fastest:async:1000). lmdb-optimized processing times (with per-block-checkpoint) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 10 minutes processing time (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with full pow computation) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.8 hours processing time (--db-sync-mode=fastest:async:1000). 2. RPI2. Improved from estimated 3 months(???) into 2.5 days (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with per-block-checkpoint) 1. RPI2. 12-15 hours (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000).
2015-07-10 20:09:32 +00:00
#define ENABLE_AUTO_RESIZE
Initial commit of lmdb BlockchainDB impl
2014-10-21 20:33:43 +00:00
namespace cryptonote
{
read txn/cursor stuff Could wrap more later.
2016-02-18 12:09:57 +00:00
typedef struct mdb_txn_cursors
Use cursors in write txns Saves a bit of seek overhead. LMDB frees them automatically in txn_(commit|abort) so they need no cleanup.
2016-01-07 06:33:22 +00:00
{
MDB_cursor *m_txc_blocks;
MDB_cursor *m_txc_block_heights;
Consolidated block info
2016-03-03 04:03:04 +00:00
MDB_cursor *m_txc_block_info;
Use cursors in write txns Saves a bit of seek overhead. LMDB frees them automatically in txn_(commit|abort) so they need no cleanup.
2016-01-07 06:33:22 +00:00
MDB_cursor *m_txc_output_txs;
MDB_cursor *m_txc_output_amounts;
MDB_cursor *m_txc_txs;
db_lmdb: save pruned and prunable tx data separately This bumps DB version to 2, migration code will run for v1 DBs
2017-10-01 10:24:33 +00:00
MDB_cursor *m_txc_txs_pruned;
MDB_cursor *m_txc_txs_prunable;
MDB_cursor *m_txc_txs_prunable_hash;
Schema update: tx_indices
2016-03-04 17:38:15 +00:00
MDB_cursor *m_txc_tx_indices;
Use cursors in write txns Saves a bit of seek overhead. LMDB frees them automatically in txn_(commit|abort) so they need no cleanup.
2016-01-07 06:33:22 +00:00
MDB_cursor *m_txc_tx_outputs;
MDB_cursor *m_txc_spent_keys;
Move txpool to the database Integration could go further (ie, return_tx_to_pool calls should not be needed anymore, possibly other things). poolstate.bin is now obsolete.
2017-05-14 13:06:55 +00:00
MDB_cursor *m_txc_txpool_meta;
MDB_cursor *m_txc_txpool_blob;
read txn/cursor stuff Could wrap more later.
2016-02-18 12:09:57 +00:00
MDB_cursor *m_txc_hf_versions;
} mdb_txn_cursors;
#define m_cur_blocks m_cursors->m_txc_blocks
#define m_cur_block_heights m_cursors->m_txc_block_heights
Consolidated block info
2016-03-03 04:03:04 +00:00
#define m_cur_block_info m_cursors->m_txc_block_info
read txn/cursor stuff Could wrap more later.
2016-02-18 12:09:57 +00:00
#define m_cur_output_txs m_cursors->m_txc_output_txs
#define m_cur_output_amounts m_cursors->m_txc_output_amounts
#define m_cur_txs m_cursors->m_txc_txs
db_lmdb: save pruned and prunable tx data separately This bumps DB version to 2, migration code will run for v1 DBs
2017-10-01 10:24:33 +00:00
#define m_cur_txs_pruned m_cursors->m_txc_txs_pruned
#define m_cur_txs_prunable m_cursors->m_txc_txs_prunable
#define m_cur_txs_prunable_hash m_cursors->m_txc_txs_prunable_hash
Schema update: tx_indices
2016-03-04 17:38:15 +00:00
#define m_cur_tx_indices m_cursors->m_txc_tx_indices
read txn/cursor stuff Could wrap more later.
2016-02-18 12:09:57 +00:00
#define m_cur_tx_outputs m_cursors->m_txc_tx_outputs
#define m_cur_spent_keys m_cursors->m_txc_spent_keys
Move txpool to the database Integration could go further (ie, return_tx_to_pool calls should not be needed anymore, possibly other things). poolstate.bin is now obsolete.
2017-05-14 13:06:55 +00:00
#define m_cur_txpool_meta m_cursors->m_txc_txpool_meta
#define m_cur_txpool_blob m_cursors->m_txc_txpool_blob
read txn/cursor stuff Could wrap more later.
2016-02-18 12:09:57 +00:00
#define m_cur_hf_versions m_cursors->m_txc_hf_versions
typedef struct mdb_rflags
{
bool m_rf_txn;
bool m_rf_blocks;
bool m_rf_block_heights;
Consolidated block info
2016-03-03 04:03:04 +00:00
bool m_rf_block_info;
read txn/cursor stuff Could wrap more later.
2016-02-18 12:09:57 +00:00
bool m_rf_output_txs;
bool m_rf_output_amounts;
bool m_rf_txs;
db_lmdb: save pruned and prunable tx data separately This bumps DB version to 2, migration code will run for v1 DBs
2017-10-01 10:24:33 +00:00
bool m_rf_txs_pruned;
bool m_rf_txs_prunable;
bool m_rf_txs_prunable_hash;
Schema update: tx_indices
2016-03-04 17:38:15 +00:00
bool m_rf_tx_indices;
read txn/cursor stuff Could wrap more later.
2016-02-18 12:09:57 +00:00
bool m_rf_tx_outputs;
bool m_rf_spent_keys;
Move txpool to the database Integration could go further (ie, return_tx_to_pool calls should not be needed anymore, possibly other things). poolstate.bin is now obsolete.
2017-05-14 13:06:55 +00:00
bool m_rf_txpool_meta;
bool m_rf_txpool_blob;
read txn/cursor stuff Could wrap more later.
2016-02-18 12:09:57 +00:00
bool m_rf_hf_versions;
} mdb_rflags;
typedef struct mdb_threadinfo
{
MDB_txn *m_ti_rtxn; // per-thread read txn
mdb_txn_cursors m_ti_rcursors; // per-thread read cursors
mdb_rflags m_ti_rflags; // per-thread read state
~mdb_threadinfo();
} mdb_threadinfo;
Use cursors in write txns Saves a bit of seek overhead. LMDB frees them automatically in txn_(commit|abort) so they need no cleanup.
2016-01-07 06:33:22 +00:00
Try to not pollute cryptonote namespace
2015-03-14 21:24:51 +00:00
struct mdb_txn_safe
Parts of LMDB impl of BlockchainDB done and working The rest should just be tedious copypasta and modification.
2014-10-23 19:37:10 +00:00
{
Add destructor for readtxns Only if we created the readtxn. Was missing cleanups from exceptions before.
2016-03-16 10:24:48 +00:00
mdb_txn_safe(const bool check=true);
Move mdb_txn_safe implementation to cpp file
2015-05-16 00:42:47 +00:00
~mdb_txn_safe();
Parts of LMDB impl of BlockchainDB done and working The rest should just be tedious copypasta and modification.
2014-10-23 19:37:10 +00:00
Move mdb_txn_safe implementation to cpp file
2015-05-16 00:42:47 +00:00
void commit(std::string message = "");
Parts of LMDB impl of BlockchainDB done and working The rest should just be tedious copypasta and modification.
2014-10-23 19:37:10 +00:00
BlockchainLMDB: Add support for batch transactions
2015-02-11 23:55:53 +00:00
// This should only be needed for batch transaction which must be ensured to
// be aborted before mdb_env_close, not after. So we can't rely on
Try to not pollute cryptonote namespace
2015-03-14 21:24:51 +00:00
// BlockchainLMDB destructor to call mdb_txn_safe destructor, as that's too late
BlockchainLMDB: Add support for batch transactions
2015-02-11 23:55:53 +00:00
// to properly abort, since mdb_env_close would have been called earlier.
Move mdb_txn_safe implementation to cpp file
2015-05-16 00:42:47 +00:00
void abort();
Don't create readtxn until after txn_safe gate check
2018-01-31 18:18:01 +00:00
void uncheck();
BlockchainLMDB: Add support for batch transactions
2015-02-11 23:55:53 +00:00
Parts of LMDB impl of BlockchainDB done and working The rest should just be tedious copypasta and modification.
2014-10-23 19:37:10 +00:00
operator MDB_txn*()
{
return m_txn;
}
operator MDB_txn**()
{
return &m_txn;
}
blockchain_db: add a few const
2015-05-27 18:03:46 +00:00
uint64_t num_active_tx() const;
LMDB should now dynamically resize the mapsize Some filesystems (*cough* NTFS *cough*) aren't good with sparse files, so this makes LMDB dynamically resize its mapsize as needed. Note: the check interval is currently every 10 blocks (for testing) and will probably need to change to 1000 or something. Default mapsize set to 1GiB. Blockchain conversion tools using batching will probably segfault, I'll fix that in the next commit.
2015-05-17 02:05:54 +00:00
static void prevent_new_txns();
static void wait_no_active_txns();
static void allow_new_txns();
Add destructor for readtxns Only if we created the readtxn. Was missing cleanups from exceptions before.
2016-03-16 10:24:48 +00:00
mdb_threadinfo* m_tinfo;
Parts of LMDB impl of BlockchainDB done and working The rest should just be tedious copypasta and modification.
2014-10-23 19:37:10 +00:00
MDB_txn* m_txn;
BlockchainLMDB: Add support for batch transactions
2015-02-11 23:55:53 +00:00
bool m_batch_txn = false;
Add destructor for readtxns Only if we created the readtxn. Was missing cleanups from exceptions before.
2016-03-16 10:24:48 +00:00
bool m_check;
LMDB should now dynamically resize the mapsize Some filesystems (*cough* NTFS *cough*) aren't good with sparse files, so this makes LMDB dynamically resize its mapsize as needed. Note: the check interval is currently every 10 blocks (for testing) and will probably need to change to 1000 or something. Default mapsize set to 1GiB. Blockchain conversion tools using batching will probably segfault, I'll fix that in the next commit.
2015-05-17 02:05:54 +00:00
static std::atomic<uint64_t> num_active_txns;
// could use a mutex here, but this should be sufficient.
static std::atomic_flag creation_gate;
Parts of LMDB impl of BlockchainDB done and working The rest should just be tedious copypasta and modification.
2014-10-23 19:37:10 +00:00
};
LMDB should now dynamically resize the mapsize Some filesystems (*cough* NTFS *cough*) aren't good with sparse files, so this makes LMDB dynamically resize its mapsize as needed. Note: the check interval is currently every 10 blocks (for testing) and will probably need to change to 1000 or something. Default mapsize set to 1GiB. Blockchain conversion tools using batching will probably segfault, I'll fix that in the next commit.
2015-05-17 02:05:54 +00:00
// If m_batch_active is set, a batch transaction exists beyond this class, such
// as a batch import with verification enabled, or possibly (later) a batch
// network sync.
//
// For some of the lookup methods, such as get_block_timestamp(), tx_exists(),
// and get_tx(), when m_batch_active is set, the lookup uses the batch
// transaction. This isn't only because the transaction is available, but it's
// necessary so that lookups include the database updates only present in the
// current batch write.
//
// A regular network sync without batch writes is expected to open a new read
// transaction, as those lookups are part of the validation done prior to the
// write for block and tx data, so no write transaction is open at the time.
Initial commit of lmdb BlockchainDB impl
2014-10-21 20:33:43 +00:00
class BlockchainLMDB : public BlockchainDB
{
public:
db_lmdb: enable batch transactions by default
2018-06-14 02:15:53 +00:00
BlockchainLMDB(bool batch_transactions=true);
Initial commit of lmdb BlockchainDB impl
2014-10-21 20:33:43 +00:00
~BlockchainLMDB();
Add support for database open with flags Add support to: - BlockchainDB, BlockchainLMDB - blockchain_import utility to open LMDB database with one or more LMDB flags. Sample use: $ blockchain_import --database lmdb#nosync $ blockchain_import --database lmdb#nosync,nometasync
2015-02-12 00:02:20 +00:00
virtual void open(const std::string& filename, const int mdb_flags=0);
Initial commit of lmdb BlockchainDB impl
2014-10-21 20:33:43 +00:00
virtual void close();
virtual void sync();
Toggle SAFE syncmode on and off automatically If monerod is started with default sync mode, set it to SAFE after synchronization completes. Set it back to FAST if synchronization restarts (e.g. because another peer has a longer blockchain). If monerod is started with an explicit sync mode, none of this automation takes effect.
2017-08-19 18:36:51 +00:00
virtual void safesyncmode(const bool onoff);
Initial commit of lmdb BlockchainDB impl
2014-10-21 20:33:43 +00:00
virtual void reset();
blockchain_db: add consts where appropriate
2014-12-06 21:37:22 +00:00
virtual std::vector<std::string> get_filenames() const;
Initial commit of lmdb BlockchainDB impl
2014-10-21 20:33:43 +00:00
add --regtest and --fixed-difficulty for regression testing on_generateblocks RPC call combines functionality from the on_getblocktemplate and on_submitblock RPC calls to allow rapid block creation. Difficulty is set permanently to 1 for regtest. Makes use of FAKECHAIN network type, but takes hard fork heights from mainchain Default reserve_size in generate_blocks RPC call is now 1. If it is 0, the following error occurs 'Failed to calculate offset for'. Queries hard fork heights info of other network types
2018-06-14 19:11:49 +00:00
virtual bool remove_data_file(const std::string& folder) const;
BlockchainDB implementations have names now In order to make things more general, BlockchainDB now has get_db_name() which should return a string with the "name" of that type of db. This "name" will be the subfolder name that holds that db type's files within the monero folder. Small bugfix: blockchain_converter was not correctly appending this in the prior hard-coded-string implementation of the subfolder data directory concept.
2015-03-14 01:39:27 +00:00
virtual std::string get_db_name() const;
Initial commit of lmdb BlockchainDB impl
2014-10-21 20:33:43 +00:00
virtual bool lock();
virtual void unlock();
core: faster find_blockchain_supplement Since this queries block heights for blocks that may or may not exist, queries for non existing blocks would throw an exception, and that would slow down the loop a lot. 7 seconds to go through a 30 hash list. Fix this by adding an optional return block height to block_exists and using this instead. Actual errors will still throw an exception. This also cuts down on log exception spam.
2016-08-30 15:39:33 +00:00
virtual bool block_exists(const crypto::hash& h, uint64_t *height = NULL) const;
Initial commit of lmdb BlockchainDB impl
2014-10-21 20:33:43 +00:00
blockchain_db: add consts where appropriate
2014-12-06 21:37:22 +00:00
virtual uint64_t get_block_height(const crypto::hash& h) const;
Initial commit of lmdb BlockchainDB impl
2014-10-21 20:33:43 +00:00
blockchain_db: add consts where appropriate
2014-12-06 21:37:22 +00:00
virtual block_header get_block_header(const crypto::hash& h) const;
Initial commit of lmdb BlockchainDB impl
2014-10-21 20:33:43 +00:00
blockchain_db: add "raw" blobdata getters for block and transaction This speeds up operations such as serving blocks to syncing peers
2017-01-15 16:05:55 +00:00
virtual cryptonote::blobdata get_block_blob(const crypto::hash& h) const;
virtual cryptonote::blobdata get_block_blob_from_height(const uint64_t& height) const;
Initial commit of lmdb BlockchainDB impl
2014-10-21 20:33:43 +00:00
db: store cumulative rct output distribution in the db for speed This gets rid of the temporary precalc cache. Also make the RPC able to send data back in binary or JSON, since there can be a lot of data This bumps the LMDB database format to v3, with migration.
2018-05-22 13:46:30 +00:00
virtual std::vector<uint64_t> get_block_cumulative_rct_outputs(const std::vector<uint64_t> &heights) const;
blockchain_db: add consts where appropriate
2014-12-06 21:37:22 +00:00
virtual uint64_t get_block_timestamp(const uint64_t& height) const;
Initial commit of lmdb BlockchainDB impl
2014-10-21 20:33:43 +00:00
blockchain_db: add consts where appropriate
2014-12-06 21:37:22 +00:00
virtual uint64_t get_top_block_timestamp() const;
Initial commit of lmdb BlockchainDB impl
2014-10-21 20:33:43 +00:00
blockchain_db: add consts where appropriate
2014-12-06 21:37:22 +00:00
virtual size_t get_block_size(const uint64_t& height) const;
Initial commit of lmdb BlockchainDB impl
2014-10-21 20:33:43 +00:00
blockchain_db: add consts where appropriate
2014-12-06 21:37:22 +00:00
virtual difficulty_type get_block_cumulative_difficulty(const uint64_t& height) const;
Initial commit of lmdb BlockchainDB impl
2014-10-21 20:33:43 +00:00
blockchain_db: add consts where appropriate
2014-12-06 21:37:22 +00:00
virtual difficulty_type get_block_difficulty(const uint64_t& height) const;
Initial commit of lmdb BlockchainDB impl
2014-10-21 20:33:43 +00:00
blockchain_db: add consts where appropriate
2014-12-06 21:37:22 +00:00
virtual uint64_t get_block_already_generated_coins(const uint64_t& height) const;
Initial commit of lmdb BlockchainDB impl
2014-10-21 20:33:43 +00:00
blockchain_db: add consts where appropriate
2014-12-06 21:37:22 +00:00
virtual crypto::hash get_block_hash_from_height(const uint64_t& height) const;
Initial commit of lmdb BlockchainDB impl
2014-10-21 20:33:43 +00:00
blockchain_db: add consts where appropriate
2014-12-06 21:37:22 +00:00
virtual std::vector<block> get_blocks_range(const uint64_t& h1, const uint64_t& h2) const;
Initial commit of lmdb BlockchainDB impl
2014-10-21 20:33:43 +00:00
blockchain_db: add consts where appropriate
2014-12-06 21:37:22 +00:00
virtual std::vector<crypto::hash> get_hashes_range(const uint64_t& h1, const uint64_t& h2) const;
Initial commit of lmdb BlockchainDB impl
2014-10-21 20:33:43 +00:00
blockchain_db: add consts where appropriate
2014-12-06 21:37:22 +00:00
virtual crypto::hash top_block_hash() const;
Initial commit of lmdb BlockchainDB impl
2014-10-21 20:33:43 +00:00
blockchain_db: add consts where appropriate
2014-12-06 21:37:22 +00:00
virtual block get_top_block() const;
Initial commit of lmdb BlockchainDB impl
2014-10-21 20:33:43 +00:00
blockchain_db: add consts where appropriate
2014-12-06 21:37:22 +00:00
virtual uint64_t height() const;
Initial commit of lmdb BlockchainDB impl
2014-10-21 20:33:43 +00:00
blockchain_db: add consts where appropriate
2014-12-06 21:37:22 +00:00
virtual bool tx_exists(const crypto::hash& h) const;
Schema update: tx_indices - improve further with less indirection
2016-03-04 18:59:20 +00:00
virtual bool tx_exists(const crypto::hash& h, uint64_t& tx_index) const;
Initial commit of lmdb BlockchainDB impl
2014-10-21 20:33:43 +00:00
blockchain_db: add consts where appropriate
2014-12-06 21:37:22 +00:00
virtual uint64_t get_tx_unlock_time(const crypto::hash& h) const;
Initial commit of lmdb BlockchainDB impl
2014-10-21 20:33:43 +00:00
blockchain_db: add "raw" blobdata getters for block and transaction This speeds up operations such as serving blocks to syncing peers
2017-01-15 16:05:55 +00:00
virtual bool get_tx_blob(const crypto::hash& h, cryptonote::blobdata &tx) const;
db_lmdb: save pruned and prunable tx data separately This bumps DB version to 2, migration code will run for v1 DBs
2017-10-01 10:24:33 +00:00
virtual bool get_pruned_tx_blob(const crypto::hash& h, cryptonote::blobdata &tx) const;
virtual bool get_prunable_tx_hash(const crypto::hash& tx_hash, crypto::hash &prunable_hash) const;
blockchain: allow marking "tx not found" without an exception This is a normal occurence in many cases, and there is no need to spam the log with those when it is.
2016-12-23 16:38:28 +00:00
blockchain_db: add consts where appropriate
2014-12-06 21:37:22 +00:00
virtual uint64_t get_tx_count() const;
Initial commit of lmdb BlockchainDB impl
2014-10-21 20:33:43 +00:00
blockchain_db: add consts where appropriate
2014-12-06 21:37:22 +00:00
virtual std::vector<transaction> get_tx_list(const std::vector<crypto::hash>& hlist) const;
Initial commit of lmdb BlockchainDB impl
2014-10-21 20:33:43 +00:00
blockchain_db: add consts where appropriate
2014-12-06 21:37:22 +00:00
virtual uint64_t get_tx_block_height(const crypto::hash& h) const;
Initial commit of lmdb BlockchainDB impl
2014-10-21 20:33:43 +00:00
blockchain_db: add consts where appropriate
2014-12-06 21:37:22 +00:00
virtual uint64_t get_num_outputs(const uint64_t& amount) const;
Initial commit of lmdb BlockchainDB impl
2014-10-21 20:33:43 +00:00
** CHANGES ARE EXPERIMENTAL (FOR TESTING ONLY) Bockchain: 1. Optim: Multi-thread long-hash computation when encountering groups of blocks. 2. Optim: Cache verified txs and return result from cache instead of re-checking whenever possible. 3. Optim: Preload output-keys when encoutering groups of blocks. Sort by amount and global-index before bulk querying database and multi-thread when possible. 4. Optim: Disable double spend check on block verification, double spend is already detected when trying to add blocks. 5. Optim: Multi-thread signature computation whenever possible. 6. Patch: Disable locking (recursive mutex) on called functions from check_tx_inputs which causes slowdowns (only seems to happen on ubuntu/VMs??? Reason: TBD) 7. Optim: Removed looped full-tx hash computation when retrieving transactions from pool (???). 8. Optim: Cache difficulty/timestamps (735 blocks) for next-difficulty calculations so that only 2 db reads per new block is needed when a new block arrives (instead of 1470 reads). Berkeley-DB: 1. Fix: 32-bit data errors causing wrong output global indices and failure to send blocks to peers (etc). 2. Fix: Unable to pop blocks on reorganize due to transaction errors. 3. Patch: Large number of transaction aborts when running multi-threaded bulk queries. 4. Patch: Insufficient locks error when running full sync. 5. Patch: Incorrect db stats when returning from an immediate exit from "pop block" operation. 6. Optim: Add bulk queries to get output global indices. 7. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 8. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 9. Optim: Added thread-safe buffers used when multi-threading bulk queries. 10. Optim: Added support for nosync/write_nosync options for improved performance (*see --db-sync-mode option for details) 11. Mod: Added checkpoint thread and auto-remove-logs option. 12. *Now usable on 32-bit systems like RPI2. LMDB: 1. Optim: Added custom comparison for 256-bit key tables (minor speed-up, TBD: get actual effect) 2. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 3. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 4. Optim: Added support for sync/writemap options for improved performance (*see --db-sync-mode option for details) 5. Mod: Auto resize to +1GB instead of multiplier x1.5 ETC: 1. Minor optimizations for slow-hash for ARM (RPI2). Incomplete. 2. Fix: 32-bit saturation bug when computing next difficulty on large blocks. [PENDING ISSUES] 1. Berkely db has a very slow "pop-block" operation. This is very noticeable on the RPI2 as it sometimes takes > 10 MINUTES to pop a block during reorganization. This does not happen very often however, most reorgs seem to take a few seconds but it possibly depends on the number of outputs present. TBD. 2. Berkeley db, possible bug "unable to allocate memory". TBD. [NEW OPTIONS] (*Currently all enabled for testing purposes) 1. --fast-block-sync arg=[0:1] (default: 1) a. 0 = Compute long hash per block (may take a while depending on CPU) b. 1 = Skip long-hash and verify blocks based on embedded known good block hashes (faster, minimal CPU dependence) 2. --db-sync-mode arg=[[safe|fast|fastest]:[sync|async]:[nblocks_per_sync]] (default: fastest:async:1000) a. safe = fdatasync/fsync (or equivalent) per stored block. Very slow, but safest option to protect against power-out/crash conditions. b. fast/fastest = Enables asynchronous fdatasync/fsync (or equivalent). Useful for battery operated devices or STABLE systems with UPS and/or systems with battery backed write cache/solid state cache. Fast - Write meta-data but defer data flush. Fastest - Defer meta-data and data flush. Sync - Flush data after nblocks_per_sync and wait. Async - Flush data after nblocks_per_sync but do not wait for the operation to finish. 3. --prep-blocks-threads arg=[n] (default: 4 or system max threads, whichever is lower) Max number of threads to use when computing long-hash in groups. 4. --show-time-stats arg=[0:1] (default: 1) Show benchmark related time stats. 5. --db-auto-remove-logs arg=[0:1] (default: 1) For berkeley-db only. Auto remove logs if enabled. **Note: lmdb and berkeley-db have changes to the tables and are not compatible with official git head version. At the moment, you need a full resync to use this optimized version. [PERFORMANCE COMPARISON] **Some figures are approximations only. Using a baseline machine of an i7-2600K+SSD+(with full pow computation): 1. The optimized lmdb/blockhain core can process blocks up to 585K for ~1.25 hours + download time, so it usually takes 2.5 hours to sync the full chain. 2. The current head with memory can process blocks up to 585K for ~4.2 hours + download time, so it usually takes 5.5 hours to sync the full chain. 3. The current head with lmdb can process blocks up to 585K for ~32 hours + download time and usually takes 36 hours to sync the full chain. Averate procesing times (with full pow computation): lmdb-optimized: 1. tx_ave = 2.5 ms / tx 2. block_ave = 5.87 ms / block memory-official-repo: 1. tx_ave = 8.85 ms / tx 2. block_ave = 19.68 ms / block lmdb-official-repo (0f4a036437fd41a5498ee5e74e2422ea6177aa3e) 1. tx_ave = 47.8 ms / tx 2. block_ave = 64.2 ms / block **Note: The following data denotes processing times only (does not include p2p download time) lmdb-optimized processing times (with full pow computation): 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.25 hours processing time (--db-sync-mode=fastest:async:1000). 2. Laptop, Dual-core / 4-threads U4200 (3Mb) - 4.90 hours processing time (--db-sync-mode=fastest:async:1000). 3. Embedded, Quad-core / 4-threads Z3735F (2x1Mb) - 12.0 hours processing time (--db-sync-mode=fastest:async:1000). lmdb-optimized processing times (with per-block-checkpoint) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 10 minutes processing time (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with full pow computation) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.8 hours processing time (--db-sync-mode=fastest:async:1000). 2. RPI2. Improved from estimated 3 months(???) into 2.5 days (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with per-block-checkpoint) 1. RPI2. 12-15 hours (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000).
2015-07-10 20:09:32 +00:00
virtual output_data_t get_output_key(const uint64_t& amount, const uint64_t& index);
blockchain_db: do not throw on expected partial results getting keys When scanning for outputs used in a set of incoming blocks, we expect that some of the inputs in their transactions will not be found in the blockchain, as they could be in previous blocks in that set. Those outputs will be scanned there at a later point. In this case, we add a flag to control wehther an output not being found is expected or not.
2017-02-13 19:05:30 +00:00
virtual void get_output_key(const uint64_t &amount, const std::vector<uint64_t> &offsets, std::vector<output_data_t> &outputs, bool allow_partial = false);
Initial commit of lmdb BlockchainDB impl
2014-10-21 20:33:43 +00:00
Fixes a bug with getting output metadata from BlockchainDB Thanks to moneromooo-monero for spotting the bug.
2014-12-14 20:20:41 +00:00
virtual tx_out_index get_output_tx_and_index_from_global(const uint64_t& index) const;
** CHANGES ARE EXPERIMENTAL (FOR TESTING ONLY) Bockchain: 1. Optim: Multi-thread long-hash computation when encountering groups of blocks. 2. Optim: Cache verified txs and return result from cache instead of re-checking whenever possible. 3. Optim: Preload output-keys when encoutering groups of blocks. Sort by amount and global-index before bulk querying database and multi-thread when possible. 4. Optim: Disable double spend check on block verification, double spend is already detected when trying to add blocks. 5. Optim: Multi-thread signature computation whenever possible. 6. Patch: Disable locking (recursive mutex) on called functions from check_tx_inputs which causes slowdowns (only seems to happen on ubuntu/VMs??? Reason: TBD) 7. Optim: Removed looped full-tx hash computation when retrieving transactions from pool (???). 8. Optim: Cache difficulty/timestamps (735 blocks) for next-difficulty calculations so that only 2 db reads per new block is needed when a new block arrives (instead of 1470 reads). Berkeley-DB: 1. Fix: 32-bit data errors causing wrong output global indices and failure to send blocks to peers (etc). 2. Fix: Unable to pop blocks on reorganize due to transaction errors. 3. Patch: Large number of transaction aborts when running multi-threaded bulk queries. 4. Patch: Insufficient locks error when running full sync. 5. Patch: Incorrect db stats when returning from an immediate exit from "pop block" operation. 6. Optim: Add bulk queries to get output global indices. 7. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 8. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 9. Optim: Added thread-safe buffers used when multi-threading bulk queries. 10. Optim: Added support for nosync/write_nosync options for improved performance (*see --db-sync-mode option for details) 11. Mod: Added checkpoint thread and auto-remove-logs option. 12. *Now usable on 32-bit systems like RPI2. LMDB: 1. Optim: Added custom comparison for 256-bit key tables (minor speed-up, TBD: get actual effect) 2. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 3. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 4. Optim: Added support for sync/writemap options for improved performance (*see --db-sync-mode option for details) 5. Mod: Auto resize to +1GB instead of multiplier x1.5 ETC: 1. Minor optimizations for slow-hash for ARM (RPI2). Incomplete. 2. Fix: 32-bit saturation bug when computing next difficulty on large blocks. [PENDING ISSUES] 1. Berkely db has a very slow "pop-block" operation. This is very noticeable on the RPI2 as it sometimes takes > 10 MINUTES to pop a block during reorganization. This does not happen very often however, most reorgs seem to take a few seconds but it possibly depends on the number of outputs present. TBD. 2. Berkeley db, possible bug "unable to allocate memory". TBD. [NEW OPTIONS] (*Currently all enabled for testing purposes) 1. --fast-block-sync arg=[0:1] (default: 1) a. 0 = Compute long hash per block (may take a while depending on CPU) b. 1 = Skip long-hash and verify blocks based on embedded known good block hashes (faster, minimal CPU dependence) 2. --db-sync-mode arg=[[safe|fast|fastest]:[sync|async]:[nblocks_per_sync]] (default: fastest:async:1000) a. safe = fdatasync/fsync (or equivalent) per stored block. Very slow, but safest option to protect against power-out/crash conditions. b. fast/fastest = Enables asynchronous fdatasync/fsync (or equivalent). Useful for battery operated devices or STABLE systems with UPS and/or systems with battery backed write cache/solid state cache. Fast - Write meta-data but defer data flush. Fastest - Defer meta-data and data flush. Sync - Flush data after nblocks_per_sync and wait. Async - Flush data after nblocks_per_sync but do not wait for the operation to finish. 3. --prep-blocks-threads arg=[n] (default: 4 or system max threads, whichever is lower) Max number of threads to use when computing long-hash in groups. 4. --show-time-stats arg=[0:1] (default: 1) Show benchmark related time stats. 5. --db-auto-remove-logs arg=[0:1] (default: 1) For berkeley-db only. Auto remove logs if enabled. **Note: lmdb and berkeley-db have changes to the tables and are not compatible with official git head version. At the moment, you need a full resync to use this optimized version. [PERFORMANCE COMPARISON] **Some figures are approximations only. Using a baseline machine of an i7-2600K+SSD+(with full pow computation): 1. The optimized lmdb/blockhain core can process blocks up to 585K for ~1.25 hours + download time, so it usually takes 2.5 hours to sync the full chain. 2. The current head with memory can process blocks up to 585K for ~4.2 hours + download time, so it usually takes 5.5 hours to sync the full chain. 3. The current head with lmdb can process blocks up to 585K for ~32 hours + download time and usually takes 36 hours to sync the full chain. Averate procesing times (with full pow computation): lmdb-optimized: 1. tx_ave = 2.5 ms / tx 2. block_ave = 5.87 ms / block memory-official-repo: 1. tx_ave = 8.85 ms / tx 2. block_ave = 19.68 ms / block lmdb-official-repo (0f4a036437fd41a5498ee5e74e2422ea6177aa3e) 1. tx_ave = 47.8 ms / tx 2. block_ave = 64.2 ms / block **Note: The following data denotes processing times only (does not include p2p download time) lmdb-optimized processing times (with full pow computation): 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.25 hours processing time (--db-sync-mode=fastest:async:1000). 2. Laptop, Dual-core / 4-threads U4200 (3Mb) - 4.90 hours processing time (--db-sync-mode=fastest:async:1000). 3. Embedded, Quad-core / 4-threads Z3735F (2x1Mb) - 12.0 hours processing time (--db-sync-mode=fastest:async:1000). lmdb-optimized processing times (with per-block-checkpoint) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 10 minutes processing time (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with full pow computation) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.8 hours processing time (--db-sync-mode=fastest:async:1000). 2. RPI2. Improved from estimated 3 months(???) into 2.5 days (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with per-block-checkpoint) 1. RPI2. 12-15 hours (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000).
2015-07-10 20:09:32 +00:00
virtual void get_output_tx_and_index_from_global(const std::vector<uint64_t> &global_indices,
Replace tabs with two spaces for consistency with rest of codebase Remove trailing whitespace in same files.
2015-12-14 04:54:39 +00:00
std::vector<tx_out_index> &tx_out_indices) const;
Fixes a bug with getting output metadata from BlockchainDB Thanks to moneromooo-monero for spotting the bug.
2014-12-14 20:20:41 +00:00
new unlocked parameter to output_histogram This constrains the number of instances of any amount to the unlocked ones (as defined by the default unlock time setting: outputs with non default unlock time are not considered, so may be counted as unlocked even if they are not actually unlocked).
2016-08-01 21:16:00 +00:00
virtual tx_out_index get_output_tx_and_index(const uint64_t& amount, const uint64_t& index) const;
virtual void get_output_tx_and_index(const uint64_t& amount, const std::vector<uint64_t> &offsets, std::vector<tx_out_index> &indices) const;
Initial commit of lmdb BlockchainDB impl
2014-10-21 20:33:43 +00:00
Cleanup and clarify Try to rationalize the variable names, document usage.
2016-04-04 01:10:58 +00:00
virtual std::vector<uint64_t> get_tx_amount_output_indices(const uint64_t tx_id) const;
Initial commit of lmdb BlockchainDB impl
2014-10-21 20:33:43 +00:00
blockchain_db: add consts where appropriate
2014-12-06 21:37:22 +00:00
virtual bool has_key_image(const crypto::key_image& img) const;
Initial commit of lmdb BlockchainDB impl
2014-10-21 20:33:43 +00:00
Move txpool to the database Integration could go further (ie, return_tx_to_pool calls should not be needed anymore, possibly other things). poolstate.bin is now obsolete.
2017-05-14 13:06:55 +00:00
virtual void add_txpool_tx(const transaction &tx, const txpool_tx_meta_t& meta);
virtual void update_txpool_tx(const crypto::hash &txid, const txpool_tx_meta_t& meta);
Protect node privacy by proper filtering in restricted-mode RPC answers This patch allows to filter out sensitive information for queries that rely on the pool state, when running in restricted mode. This filtering is only applied to data sent back to RPC queries. Results of inline commands typed locally in the daemon are not affected. In practice, when running with `--restricted-rpc`: * get_transaction_pool will list relayed transactions with the fields "last relayed time" and "received time" set to zero. * get_transaction_pool will not list transaction that have do_not_relay set to true, and will not list key images that are used only for such transactions * get_transaction_pool_hashes.bin will not list such transaction * get_transaction_pool_stats will not count such transactions in any of the aggregated values that are computed The implementation does not make filtering the default, so developers should be mindful of this if they add new RPC functionality. Fixes #2590.
2017-11-08 12:06:41 +00:00
virtual uint64_t get_txpool_tx_count(bool include_unrelayed_txes = true) const;
Move txpool to the database Integration could go further (ie, return_tx_to_pool calls should not be needed anymore, possibly other things). poolstate.bin is now obsolete.
2017-05-14 13:06:55 +00:00
virtual bool txpool_has_tx(const crypto::hash &txid) const;
virtual void remove_txpool_tx(const crypto::hash& txid);
Fix exceptions not finding txpool txes when relaying
2017-12-14 17:09:30 +00:00
virtual bool get_txpool_tx_meta(const crypto::hash& txid, txpool_tx_meta_t &meta) const;
blockchain_db: add a txpool tx getter which returns existence Avoids exception spam for the "nope, not found" case
2017-06-11 14:10:18 +00:00
virtual bool get_txpool_tx_blob(const crypto::hash& txid, cryptonote::blobdata &bd) const;
Move txpool to the database Integration could go further (ie, return_tx_to_pool calls should not be needed anymore, possibly other things). poolstate.bin is now obsolete.
2017-05-14 13:06:55 +00:00
virtual cryptonote::blobdata get_txpool_tx_blob(const crypto::hash& txid) const;
Protect node privacy by proper filtering in restricted-mode RPC answers This patch allows to filter out sensitive information for queries that rely on the pool state, when running in restricted mode. This filtering is only applied to data sent back to RPC queries. Results of inline commands typed locally in the daemon are not affected. In practice, when running with `--restricted-rpc`: * get_transaction_pool will list relayed transactions with the fields "last relayed time" and "received time" set to zero. * get_transaction_pool will not list transaction that have do_not_relay set to true, and will not list key images that are used only for such transactions * get_transaction_pool_hashes.bin will not list such transaction * get_transaction_pool_stats will not count such transactions in any of the aggregated values that are computed The implementation does not make filtering the default, so developers should be mindful of this if they add new RPC functionality. Fixes #2590.
2017-11-08 12:06:41 +00:00
virtual bool for_all_txpool_txes(std::function<bool(const crypto::hash&, const txpool_tx_meta_t&, const cryptonote::blobdata*)> f, bool include_blob = false, bool include_unrelayed_txes = true) const;
Move txpool to the database Integration could go further (ie, return_tx_to_pool calls should not be needed anymore, possibly other things). poolstate.bin is now obsolete.
2017-05-14 13:06:55 +00:00
Add functions to iterate through blocks, txes, outputs, key images
2015-10-25 10:45:25 +00:00
virtual bool for_all_key_images(std::function<bool(const crypto::key_image&)>) const;
Don't copy blockchain for coinbase_tx_sum Changed Blockchain::for_all_blocks() to for_blocks_range() Operate on blockchain in-place instead of building a copy first.
2017-06-01 12:29:51 +00:00
virtual bool for_blocks_range(const uint64_t& h1, const uint64_t& h2, std::function<bool(uint64_t, const crypto::hash&, const cryptonote::block&)>) const;
db_lmdb: save pruned and prunable tx data separately This bumps DB version to 2, migration code will run for v1 DBs
2017-10-01 10:24:33 +00:00
virtual bool for_all_transactions(std::function<bool(const crypto::hash&, const cryptonote::transaction&)>, bool pruned) const;
add RPC to get a histogram of outputs of a given amount
2018-02-19 11:15:15 +00:00
virtual bool for_all_outputs(std::function<bool(uint64_t amount, const crypto::hash &tx_hash, uint64_t height, size_t tx_idx)> f) const;
virtual bool for_all_outputs(uint64_t amount, const std::function<bool(uint64_t height)> &f) const;
Add functions to iterate through blocks, txes, outputs, key images
2015-10-25 10:45:25 +00:00
Parts of LMDB impl of BlockchainDB done and working The rest should just be tedious copypasta and modification.
2014-10-23 19:37:10 +00:00
virtual uint64_t add_block( const block& blk
, const size_t& block_size
, const difficulty_type& cumulative_difficulty
, const uint64_t& coins_generated
, const std::vector<transaction>& txs
);
BlockchainLMDB: Add support for batch transactions
2015-02-11 23:55:53 +00:00
virtual void set_batch_transactions(bool batch_transactions);
Use actual batch size for resize estimates And optimize import startup: Remember start_height position during initial count_blocks pass to avoid having to reread entire file again to arrive at start_height
2017-09-16 23:12:42 +00:00
virtual bool batch_start(uint64_t batch_num_blocks=0, uint64_t batch_bytes=0);
BlockchainLMDB: Add support for batch transactions
2015-02-11 23:55:53 +00:00
virtual void batch_commit();
virtual void batch_stop();
virtual void batch_abort();
read txn/cursor stuff Could wrap more later.
2016-02-18 12:09:57 +00:00
virtual void block_txn_start(bool readonly);
BlockchainDB/LMDB/BDB: Extract DB txn functions for block add/remove
2016-02-08 16:32:19 +00:00
virtual void block_txn_stop();
virtual void block_txn_abort();
Hack for read/write txn mixup save the thread ID of the writer thread so we don't try to use the writetxn from reader threads
2016-03-14 18:26:15 +00:00
virtual bool block_rtxn_start(MDB_txn **mtxn, mdb_txn_cursors **mcur) const;
read txn/cursor stuff Could wrap more later.
2016-02-18 12:09:57 +00:00
virtual void block_rtxn_stop() const;
BlockchainDB/LMDB/BDB: Extract DB txn functions for block add/remove
2016-02-08 16:32:19 +00:00
Minor bugfixes, redundancy removal Minor bugfixes in block removal Storing outputs outside their transactions is largely unnecessary, and thus has been removed.
2014-10-30 22:33:35 +00:00
virtual void pop_block(block& blk, std::vector<transaction>& txs);
** CHANGES ARE EXPERIMENTAL (FOR TESTING ONLY) Bockchain: 1. Optim: Multi-thread long-hash computation when encountering groups of blocks. 2. Optim: Cache verified txs and return result from cache instead of re-checking whenever possible. 3. Optim: Preload output-keys when encoutering groups of blocks. Sort by amount and global-index before bulk querying database and multi-thread when possible. 4. Optim: Disable double spend check on block verification, double spend is already detected when trying to add blocks. 5. Optim: Multi-thread signature computation whenever possible. 6. Patch: Disable locking (recursive mutex) on called functions from check_tx_inputs which causes slowdowns (only seems to happen on ubuntu/VMs??? Reason: TBD) 7. Optim: Removed looped full-tx hash computation when retrieving transactions from pool (???). 8. Optim: Cache difficulty/timestamps (735 blocks) for next-difficulty calculations so that only 2 db reads per new block is needed when a new block arrives (instead of 1470 reads). Berkeley-DB: 1. Fix: 32-bit data errors causing wrong output global indices and failure to send blocks to peers (etc). 2. Fix: Unable to pop blocks on reorganize due to transaction errors. 3. Patch: Large number of transaction aborts when running multi-threaded bulk queries. 4. Patch: Insufficient locks error when running full sync. 5. Patch: Incorrect db stats when returning from an immediate exit from "pop block" operation. 6. Optim: Add bulk queries to get output global indices. 7. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 8. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 9. Optim: Added thread-safe buffers used when multi-threading bulk queries. 10. Optim: Added support for nosync/write_nosync options for improved performance (*see --db-sync-mode option for details) 11. Mod: Added checkpoint thread and auto-remove-logs option. 12. *Now usable on 32-bit systems like RPI2. LMDB: 1. Optim: Added custom comparison for 256-bit key tables (minor speed-up, TBD: get actual effect) 2. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 3. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 4. Optim: Added support for sync/writemap options for improved performance (*see --db-sync-mode option for details) 5. Mod: Auto resize to +1GB instead of multiplier x1.5 ETC: 1. Minor optimizations for slow-hash for ARM (RPI2). Incomplete. 2. Fix: 32-bit saturation bug when computing next difficulty on large blocks. [PENDING ISSUES] 1. Berkely db has a very slow "pop-block" operation. This is very noticeable on the RPI2 as it sometimes takes > 10 MINUTES to pop a block during reorganization. This does not happen very often however, most reorgs seem to take a few seconds but it possibly depends on the number of outputs present. TBD. 2. Berkeley db, possible bug "unable to allocate memory". TBD. [NEW OPTIONS] (*Currently all enabled for testing purposes) 1. --fast-block-sync arg=[0:1] (default: 1) a. 0 = Compute long hash per block (may take a while depending on CPU) b. 1 = Skip long-hash and verify blocks based on embedded known good block hashes (faster, minimal CPU dependence) 2. --db-sync-mode arg=[[safe|fast|fastest]:[sync|async]:[nblocks_per_sync]] (default: fastest:async:1000) a. safe = fdatasync/fsync (or equivalent) per stored block. Very slow, but safest option to protect against power-out/crash conditions. b. fast/fastest = Enables asynchronous fdatasync/fsync (or equivalent). Useful for battery operated devices or STABLE systems with UPS and/or systems with battery backed write cache/solid state cache. Fast - Write meta-data but defer data flush. Fastest - Defer meta-data and data flush. Sync - Flush data after nblocks_per_sync and wait. Async - Flush data after nblocks_per_sync but do not wait for the operation to finish. 3. --prep-blocks-threads arg=[n] (default: 4 or system max threads, whichever is lower) Max number of threads to use when computing long-hash in groups. 4. --show-time-stats arg=[0:1] (default: 1) Show benchmark related time stats. 5. --db-auto-remove-logs arg=[0:1] (default: 1) For berkeley-db only. Auto remove logs if enabled. **Note: lmdb and berkeley-db have changes to the tables and are not compatible with official git head version. At the moment, you need a full resync to use this optimized version. [PERFORMANCE COMPARISON] **Some figures are approximations only. Using a baseline machine of an i7-2600K+SSD+(with full pow computation): 1. The optimized lmdb/blockhain core can process blocks up to 585K for ~1.25 hours + download time, so it usually takes 2.5 hours to sync the full chain. 2. The current head with memory can process blocks up to 585K for ~4.2 hours + download time, so it usually takes 5.5 hours to sync the full chain. 3. The current head with lmdb can process blocks up to 585K for ~32 hours + download time and usually takes 36 hours to sync the full chain. Averate procesing times (with full pow computation): lmdb-optimized: 1. tx_ave = 2.5 ms / tx 2. block_ave = 5.87 ms / block memory-official-repo: 1. tx_ave = 8.85 ms / tx 2. block_ave = 19.68 ms / block lmdb-official-repo (0f4a036437fd41a5498ee5e74e2422ea6177aa3e) 1. tx_ave = 47.8 ms / tx 2. block_ave = 64.2 ms / block **Note: The following data denotes processing times only (does not include p2p download time) lmdb-optimized processing times (with full pow computation): 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.25 hours processing time (--db-sync-mode=fastest:async:1000). 2. Laptop, Dual-core / 4-threads U4200 (3Mb) - 4.90 hours processing time (--db-sync-mode=fastest:async:1000). 3. Embedded, Quad-core / 4-threads Z3735F (2x1Mb) - 12.0 hours processing time (--db-sync-mode=fastest:async:1000). lmdb-optimized processing times (with per-block-checkpoint) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 10 minutes processing time (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with full pow computation) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.8 hours processing time (--db-sync-mode=fastest:async:1000). 2. RPI2. Improved from estimated 3 months(???) into 2.5 days (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with per-block-checkpoint) 1. RPI2. 12-15 hours (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000).
2015-07-10 20:09:32 +00:00
virtual bool can_thread_bulk_indices() const { return true; }
New RPC and daemon command to get output histogram This is a list of existing output amounts along with the number of outputs of that amount in the blockchain. The daemon command takes: - no parameters: all outputs with at least 3 instances - one parameter: all outputs with at least that many instances - two parameters: all outputs within that many instances The default starts at 3 to avoid massive spamming of all dust outputs in the blockchain, and is the current minimum mixin requirement. An optional vector of amounts may be passed, to request histogram only for those outputs.
2016-03-26 14:30:23 +00:00
/**
* @brief return a histogram of outputs on the blockchain
*
* @param amounts optional set of amounts to lookup
new unlocked parameter to output_histogram This constrains the number of instances of any amount to the unlocked ones (as defined by the default unlock time setting: outputs with non default unlock time are not considered, so may be counted as unlocked even if they are not actually unlocked).
2016-08-01 21:16:00 +00:00
* @param unlocked whether to restrict count to unlocked outputs
wallet: select part of the fake outs from recent outputs 25% of the outputs are selected from the last 5 days (if possible), in order to avoid the common case of sending recently received outputs again. 25% and 5 days are subject to review later, since it's just a wallet level change.
2016-09-17 14:45:51 +00:00
* @param recent_cutoff timestamp to determine which outputs are recent
speedup get_output_histogram for all amounts when min_count > 0 This skips the vast majority of "dust" output amounts with just one instance on the chain. Clocks in at 0.15% of the original time on testnet.
2018-03-22 17:51:58 +00:00
* @param min_count return only amounts with at least that many instances
New RPC and daemon command to get output histogram This is a list of existing output amounts along with the number of outputs of that amount in the blockchain. The daemon command takes: - no parameters: all outputs with at least 3 instances - one parameter: all outputs with at least that many instances - two parameters: all outputs within that many instances The default starts at 3 to avoid massive spamming of all dust outputs in the blockchain, and is the current minimum mixin requirement. An optional vector of amounts may be passed, to request histogram only for those outputs.
2016-03-26 14:30:23 +00:00
*
* @return a set of amount/instances
*/
speedup get_output_histogram for all amounts when min_count > 0 This skips the vast majority of "dust" output amounts with just one instance on the chain. Clocks in at 0.15% of the original time on testnet.
2018-03-22 17:51:58 +00:00
std::map<uint64_t, std::tuple<uint64_t, uint64_t, uint64_t>> get_output_histogram(const std::vector<uint64_t> &amounts, bool unlocked, uint64_t recent_cutoff, uint64_t min_count) const;
New RPC and daemon command to get output histogram This is a list of existing output amounts along with the number of outputs of that amount in the blockchain. The daemon command takes: - no parameters: all outputs with at least 3 instances - one parameter: all outputs with at least that many instances - two parameters: all outputs within that many instances The default starts at 3 to avoid massive spamming of all dust outputs in the blockchain, and is the current minimum mixin requirement. An optional vector of amounts may be passed, to request histogram only for those outputs.
2016-03-26 14:30:23 +00:00
speed up get_output_distribution (and precalc common case)
2018-05-16 21:41:50 +00:00
bool get_output_distribution(uint64_t amount, uint64_t from_height, uint64_t to_height, std::vector<uint64_t> &distribution, uint64_t &base) const;
Initial commit of lmdb BlockchainDB impl
2014-10-21 20:33:43 +00:00
private:
Check and resize if needed at batch transaction start This currently only affects blockchain_import and blockchain_converter. When the number of blocks expected for the batch transaction is provided, make an estimate of the DB space needed. If not enough free space remains, resize the DB. The estimate is made based on: - the average size of the last 500 blocks, or if larger, a min. block size of 4k - a factor for the expanded size a block occupies in the DB across the sub-dbs/tables - a safety factor (1.7) to allow for a "reasonable" average block size increase over the batch Increase the DB size by whichever is greater: the estimated size needed or a minimum increase size, currently 128 MB. The conservative factors in the estimate help in testing that the resize occurs when needed, and without gratuitous size increases. For common use, the safety factor and minimum increase size could reasonably be increased. For testing, setting DEFAULT_MAPSIZE (blockchain_db/lmdb/db_lmdb.h) to 1 << 27 (128 MB) and recompiling will ensure DB resizes take place sooner and more frequently.
2015-07-12 05:46:16 +00:00
void do_resize(uint64_t size_increase=0);
LMDB should now dynamically resize the mapsize Some filesystems (*cough* NTFS *cough*) aren't good with sparse files, so this makes LMDB dynamically resize its mapsize as needed. Note: the check interval is currently every 10 blocks (for testing) and will probably need to change to 1000 or something. Default mapsize set to 1GiB. Blockchain conversion tools using batching will probably segfault, I'll fix that in the next commit.
2015-05-17 02:05:54 +00:00
Check and resize if needed at batch transaction start This currently only affects blockchain_import and blockchain_converter. When the number of blocks expected for the batch transaction is provided, make an estimate of the DB space needed. If not enough free space remains, resize the DB. The estimate is made based on: - the average size of the last 500 blocks, or if larger, a min. block size of 4k - a factor for the expanded size a block occupies in the DB across the sub-dbs/tables - a safety factor (1.7) to allow for a "reasonable" average block size increase over the batch Increase the DB size by whichever is greater: the estimated size needed or a minimum increase size, currently 128 MB. The conservative factors in the estimate help in testing that the resize occurs when needed, and without gratuitous size increases. For common use, the safety factor and minimum increase size could reasonably be increased. For testing, setting DEFAULT_MAPSIZE (blockchain_db/lmdb/db_lmdb.h) to 1 << 27 (128 MB) and recompiling will ensure DB resizes take place sooner and more frequently.
2015-07-12 05:46:16 +00:00
bool need_resize(uint64_t threshold_size=0) const;
Use actual batch size for resize estimates And optimize import startup: Remember start_height position during initial count_blocks pass to avoid having to reread entire file again to arrive at start_height
2017-09-16 23:12:42 +00:00
void check_and_resize_for_batch(uint64_t batch_num_blocks, uint64_t batch_bytes);
uint64_t get_estimated_batch_size(uint64_t batch_num_blocks, uint64_t batch_bytes) const;
LMDB should now dynamically resize the mapsize Some filesystems (*cough* NTFS *cough*) aren't good with sparse files, so this makes LMDB dynamically resize its mapsize as needed. Note: the check interval is currently every 10 blocks (for testing) and will probably need to change to 1000 or something. Default mapsize set to 1GiB. Blockchain conversion tools using batching will probably segfault, I'll fix that in the next commit.
2015-05-17 02:05:54 +00:00
Initial commit of lmdb BlockchainDB impl
2014-10-21 20:33:43 +00:00
virtual void add_block( const block& blk
, const size_t& block_size
, const difficulty_type& cumulative_difficulty
, const uint64_t& coins_generated
db: store cumulative rct output distribution in the db for speed This gets rid of the temporary precalc cache. Also make the RPC able to send data back in binary or JSON, since there can be a lot of data This bumps the LMDB database format to v3, with migration.
2018-05-22 13:46:30 +00:00
, uint64_t num_rct_outs
Improve block and tx processing efficiency by less repeat hashing BlockchainLMDB::add_block() BlockchainLMDB::add_transaction_data() BlockchainDB::add_transaction()
2015-02-11 23:55:53 +00:00
, const crypto::hash& block_hash
Initial commit of lmdb BlockchainDB impl
2014-10-21 20:33:43 +00:00
);
Parts of LMDB impl of BlockchainDB done and working The rest should just be tedious copypasta and modification.
2014-10-23 19:37:10 +00:00
virtual void remove_block();
Initial commit of lmdb BlockchainDB impl
2014-10-21 20:33:43 +00:00
db_lmdb: save pruned and prunable tx data separately This bumps DB version to 2, migration code will run for v1 DBs
2017-10-01 10:24:33 +00:00
virtual uint64_t add_transaction_data(const crypto::hash& blk_hash, const transaction& tx, const crypto::hash& tx_hash, const crypto::hash& tx_prunable_hash);
Initial commit of lmdb BlockchainDB impl
2014-10-21 20:33:43 +00:00
Complete implementation of transaction removal Complete method BlockchainLMDB::remove_output() - use output index as the key for: m_output_indices, m_output_txs, m_output_keys - call new method BlockchainLMDB::remove_amount_output_index() Add method to remove amount output index. - BlockchainLMDB::remove_amount_output_index() - for m_output_amounts This also fixes the segfault when blockchain reorganization is attempted.
2015-01-12 02:04:04 +00:00
virtual void remove_transaction_data(const crypto::hash& tx_hash, const transaction& tx);
Initial commit of lmdb BlockchainDB impl
2014-10-21 20:33:43 +00:00
Cleanup and clarify Try to rationalize the variable names, document usage.
2016-04-04 01:10:58 +00:00
virtual uint64_t add_output(const crypto::hash& tx_hash,
Update schema for "tx_outputs" to use array containing amount output indices This speeds up wallet refresh by directly retrieving a tx's amount output indices. It removes the indirection and walking the amount output duplicate list for every amount in each requested tx. "tx_outputs" is used by: Amount output indices are needed for wallet refresh. Global output indices are needed for removing a tx. Both amount output indices and global output indices are now stored in an array of 64-bit unsigned ints: tx_outputs[<tx_hash>] -> [ <a1_oi, a1_gi, a2_oi, a2_gi, ...> ] Previously it was: tx_outputs[<tx_hash>] -> duplicate list of <a1_gi, a2_gi, a3_gi, ...> The amount output list had to be walked for every amount in order to find each amount's output index, by comparing the amount's global output index with each one in the duplicate list until a match was found. See also d045dfa7ce0bf131681193c97560da26f9f37900
2016-01-31 13:10:14 +00:00
const tx_out& tx_output,
const uint64_t& local_index,
move the rct commitments to the output_amounts database Since these are needed at the same time as the output pubkeys, this is a whole lot faster, and takes less space. Only outputs of 0 amount store the commitment. When reading other outputs, a fake commitment is regenerated on the fly. This avoids having to rewrite the database to add space for fake commitments for existing outputs. This code relies on two things: - LMDB must support fixed size records per key, rather than per database (ie, all records on key 0 are the same size, all records for non 0 keys are same size, but records from key 0 and non 0 keys do have different sizes). - the commitment must be directly after the rest of the data in outkey and output_data_t.
2016-06-29 18:55:49 +00:00
const uint64_t unlock_time,
const rct::key *commitment
Update schema for "tx_outputs" to use array containing amount output indices This speeds up wallet refresh by directly retrieving a tx's amount output indices. It removes the indirection and walking the amount output duplicate list for every amount in each requested tx. "tx_outputs" is used by: Amount output indices are needed for wallet refresh. Global output indices are needed for removing a tx. Both amount output indices and global output indices are now stored in an array of 64-bit unsigned ints: tx_outputs[<tx_hash>] -> [ <a1_oi, a1_gi, a2_oi, a2_gi, ...> ] Previously it was: tx_outputs[<tx_hash>] -> duplicate list of <a1_gi, a2_gi, a3_gi, ...> The amount output list had to be walked for every amount in order to find each amount's output index, by comparing the amount's global output index with each one in the duplicate list until a match was found. See also d045dfa7ce0bf131681193c97560da26f9f37900
2016-01-31 13:10:14 +00:00
);
Initial commit of lmdb BlockchainDB impl
2014-10-21 20:33:43 +00:00
Cleanup and clarify Try to rationalize the variable names, document usage.
2016-04-04 01:10:58 +00:00
virtual void add_tx_amount_output_indices(const uint64_t tx_id,
const std::vector<uint64_t>& amount_output_indices
Update schema for "tx_outputs" to use array containing amount output indices This speeds up wallet refresh by directly retrieving a tx's amount output indices. It removes the indirection and walking the amount output duplicate list for every amount in each requested tx. "tx_outputs" is used by: Amount output indices are needed for wallet refresh. Global output indices are needed for removing a tx. Both amount output indices and global output indices are now stored in an array of 64-bit unsigned ints: tx_outputs[<tx_hash>] -> [ <a1_oi, a1_gi, a2_oi, a2_gi, ...> ] Previously it was: tx_outputs[<tx_hash>] -> duplicate list of <a1_gi, a2_gi, a3_gi, ...> The amount output list had to be walked for every amount in order to find each amount's output index, by comparing the amount's global output index with each one in the duplicate list until a match was found. See also d045dfa7ce0bf131681193c97560da26f9f37900
2016-01-31 13:10:14 +00:00
);
Initial commit of lmdb BlockchainDB impl
2014-10-21 20:33:43 +00:00
Cleanup and clarify Try to rationalize the variable names, document usage.
2016-04-04 01:10:58 +00:00
void remove_tx_outputs(const uint64_t tx_id, const transaction& tx);
Minor bugfixes, redundancy removal Minor bugfixes in block removal Storing outputs outside their transactions is largely unnecessary, and thus has been removed.
2014-10-30 22:33:35 +00:00
Cleanup drop obsolete remove_output() fix get_output_key(global), fix crash in blockchain_dump
2016-04-04 16:28:31 +00:00
void remove_output(const uint64_t amount, const uint64_t& out_index);
Minor bugfixes, redundancy removal Minor bugfixes in block removal Storing outputs outside their transactions is largely unnecessary, and thus has been removed.
2014-10-30 22:33:35 +00:00
Initial commit of lmdb BlockchainDB impl
2014-10-21 20:33:43 +00:00
virtual void add_spent_key(const crypto::key_image& k_image);
virtual void remove_spent_key(const crypto::key_image& k_image);
Can't cache num_txs or num_outputs either Same reason as 3ff54bdd7a8b5e08e4e8ac17b7fff23ad3a82312
2017-02-18 18:46:26 +00:00
uint64_t num_outputs() const;
hardfork: most state now saved to the DB There will be a delay on first load of an existing blockchain as it gets reparsed for this state data.
2015-09-20 17:41:38 +00:00
// Hard fork
virtual void set_hard_fork_version(uint64_t height, uint8_t version);
virtual uint8_t get_hard_fork_version(uint64_t height) const;
Fix hf when import with verify off Delete the hf tables, so the next open will rescan and regenerate
2016-01-15 14:00:58 +00:00
virtual void check_hard_fork_info();
blockchain_import: Add --drop-hard-fork command
2016-02-05 01:15:37 +00:00
virtual void drop_hard_fork_info();
hardfork: most state now saved to the DB There will be a delay on first load of an existing blockchain as it gets reparsed for this state data.
2015-09-20 17:41:38 +00:00
All LMDB BlockchainDB implemented, not tested All of the functionality for the LMDB implementation of BlockchainDB is implemented, but only what is in tests/unit_tests/BlockchainDB.cpp has been tested. This is basically add a block, see if you can get the block and a tx from the block. More tests should be added at some point.
2014-10-28 00:45:33 +00:00
/**
* @brief convert a tx output to a blob for storage
*
* @param output the output to convert
*
* @return the resultant blob
*/
blockchain_db: add a few const
2015-05-27 18:03:46 +00:00
blobdata output_to_blob(const tx_out& output) const;
All LMDB BlockchainDB implemented, not tested All of the functionality for the LMDB implementation of BlockchainDB is implemented, but only what is in tests/unit_tests/BlockchainDB.cpp has been tested. This is basically add a block, see if you can get the block and a tx from the block. More tests should be added at some point.
2014-10-28 00:45:33 +00:00
/**
* @brief convert a tx output blob to a tx output
*
* @param blob the blob to convert
*
* @return the resultant tx output
*/
blockchain_db: add consts where appropriate
2014-12-06 21:37:22 +00:00
tx_out output_from_blob(const blobdata& blob) const;
All LMDB BlockchainDB implemented, not tested All of the functionality for the LMDB implementation of BlockchainDB is implemented, but only what is in tests/unit_tests/BlockchainDB.cpp has been tested. This is basically add a block, see if you can get the block and a tx from the block. More tests should be added at some point.
2014-10-28 00:45:33 +00:00
blockchain_db: add consts where appropriate
2014-12-06 21:37:22 +00:00
void check_open() const;
Parts of LMDB impl of BlockchainDB done and working The rest should just be tedious copypasta and modification.
2014-10-23 19:37:10 +00:00
BlockchainDB: skip fixup check if read-only database
2015-12-26 22:27:35 +00:00
virtual bool is_read_only() const;
rpc: add blockchain disk size to getinfo This should help new nodes predict how much disk space will be needed for a full sync
2018-06-17 21:07:15 +00:00
virtual uint64_t get_database_size() const;
blockchain_db: fixup missing key images in early DB version Early DB versions did not store key images for inputs if the transaction spending them had no outputs (ie, all fee). This is not correct, as this would allow these outputs to be double spent. This was fixed in 533acc30eda7792c802ea8b6417917fa99b8bc2b a few months ago, but databases having synced blocks 2021612 and 685498 with a faulty version will be missing those key images in the spent keys database. This code checks for this, and adds those key images if they are missing.
2015-12-06 20:48:17 +00:00
// fix up anything that may be wrong due to past bugs
virtual void fixup();
Migration Migrate from DB version 0 to version 1 on startup
2016-04-06 17:05:20 +00:00
// migrate from older DB version to current
void migrate(const uint32_t oldversion);
// migrate from DB version 0 to 1
void migrate_0_1();
db_lmdb: save pruned and prunable tx data separately This bumps DB version to 2, migration code will run for v1 DBs
2017-10-01 10:24:33 +00:00
// migrate from DB version 1 to 2
void migrate_1_2();
db: store cumulative rct output distribution in the db for speed This gets rid of the temporary precalc cache. Also make the RPC able to send data back in binary or JSON, since there can be a lot of data This bumps the LMDB database format to v3, with migration.
2018-05-22 13:46:30 +00:00
// migrate from DB version 2 to 3
void migrate_2_3();
Fix blockchain_import wedge on exception in cleanup_handle_incoming_blocks
2017-08-29 14:43:32 +00:00
void cleanup_batch();
private:
Initial commit of lmdb BlockchainDB impl
2014-10-21 20:33:43 +00:00
MDB_env* m_env;
MDB_dbi m_blocks;
All LMDB BlockchainDB implemented, not tested All of the functionality for the LMDB implementation of BlockchainDB is implemented, but only what is in tests/unit_tests/BlockchainDB.cpp has been tested. This is basically add a block, see if you can get the block and a tx from the block. More tests should be added at some point.
2014-10-28 00:45:33 +00:00
MDB_dbi m_block_heights;
Consolidated block info
2016-03-03 04:03:04 +00:00
MDB_dbi m_block_info;
Initial commit of lmdb BlockchainDB impl
2014-10-21 20:33:43 +00:00
MDB_dbi m_txs;
db_lmdb: save pruned and prunable tx data separately This bumps DB version to 2, migration code will run for v1 DBs
2017-10-01 10:24:33 +00:00
MDB_dbi m_txs_pruned;
MDB_dbi m_txs_prunable;
MDB_dbi m_txs_prunable_hash;
Schema update: tx_indices
2016-03-04 17:38:15 +00:00
MDB_dbi m_tx_indices;
Adding block data to LMDB BlockchainDB coded Still needs testing (and need to write a few more unit tests), but everything should be there. Lots of unfortunate duplication, but...well, I can't see a way around it using LMDB. A couple of other minor changes in this commit, only slightly relevant.
2014-10-23 23:47:36 +00:00
MDB_dbi m_tx_outputs;
Initial commit of lmdb BlockchainDB impl
2014-10-21 20:33:43 +00:00
Adding block data to LMDB BlockchainDB coded Still needs testing (and need to write a few more unit tests), but everything should be there. Lots of unfortunate duplication, but...well, I can't see a way around it using LMDB. A couple of other minor changes in this commit, only slightly relevant.
2014-10-23 23:47:36 +00:00
MDB_dbi m_output_txs;
MDB_dbi m_output_amounts;
MDB_dbi m_spent_keys;
Parts of LMDB impl of BlockchainDB done and working The rest should just be tedious copypasta and modification.
2014-10-23 19:37:10 +00:00
Move txpool to the database Integration could go further (ie, return_tx_to_pool calls should not be needed anymore, possibly other things). poolstate.bin is now obsolete.
2017-05-14 13:06:55 +00:00
MDB_dbi m_txpool_meta;
MDB_dbi m_txpool_blob;
hardfork: most state now saved to the DB There will be a delay on first load of an existing blockchain as it gets reparsed for this state data.
2015-09-20 17:41:38 +00:00
MDB_dbi m_hf_starting_heights;
MDB_dbi m_hf_versions;
db_lmdb: add versioning, to detect incompatible format changes
2015-10-26 15:52:07 +00:00
MDB_dbi m_properties;
Keep a running blocksize count Used in batch size estimation, avoids rereading already processed blocks during import
2016-02-14 20:27:43 +00:00
mutable uint64_t m_cum_size; // used in batch size estimation
Add a few read txns to streamline Slight perf gain, but mainly to reduce spam at loglevel 3
2017-02-18 17:27:33 +00:00
mutable unsigned int m_cum_count;
Parts of LMDB impl of BlockchainDB done and working The rest should just be tedious copypasta and modification.
2014-10-23 19:37:10 +00:00
std::string m_folder;
Try to not pollute cryptonote namespace
2015-03-14 21:24:51 +00:00
mdb_txn_safe* m_write_txn; // may point to either a short-lived txn or a batch txn
LMDB should now dynamically resize the mapsize Some filesystems (*cough* NTFS *cough*) aren't good with sparse files, so this makes LMDB dynamically resize its mapsize as needed. Note: the check interval is currently every 10 blocks (for testing) and will probably need to change to 1000 or something. Default mapsize set to 1GiB. Blockchain conversion tools using batching will probably segfault, I'll fix that in the next commit.
2015-05-17 02:05:54 +00:00
mdb_txn_safe* m_write_batch_txn; // persist batch txn outside of BlockchainLMDB
Hack for read/write txn mixup save the thread ID of the writer thread so we don't try to use the writetxn from reader threads
2016-03-14 18:26:15 +00:00
boost::thread::id m_writer;
BlockchainLMDB: Add support for batch transactions
2015-02-11 23:55:53 +00:00
bool m_batch_transactions; // support for batch transactions
bool m_batch_active; // whether batch transaction is in progress
LMDB should now dynamically resize the mapsize Some filesystems (*cough* NTFS *cough*) aren't good with sparse files, so this makes LMDB dynamically resize its mapsize as needed. Note: the check interval is currently every 10 blocks (for testing) and will probably need to change to 1000 or something. Default mapsize set to 1GiB. Blockchain conversion tools using batching will probably segfault, I'll fix that in the next commit.
2015-05-17 02:05:54 +00:00
read txn/cursor stuff Could wrap more later.
2016-02-18 12:09:57 +00:00
mdb_txn_cursors m_wcursors;
mutable boost::thread_specific_ptr<mdb_threadinfo> m_tinfo;
Use cursors in write txns Saves a bit of seek overhead. LMDB frees them automatically in txn_(commit|abort) so they need no cleanup.
2016-01-07 06:33:22 +00:00
** CHANGES ARE EXPERIMENTAL (FOR TESTING ONLY) Bockchain: 1. Optim: Multi-thread long-hash computation when encountering groups of blocks. 2. Optim: Cache verified txs and return result from cache instead of re-checking whenever possible. 3. Optim: Preload output-keys when encoutering groups of blocks. Sort by amount and global-index before bulk querying database and multi-thread when possible. 4. Optim: Disable double spend check on block verification, double spend is already detected when trying to add blocks. 5. Optim: Multi-thread signature computation whenever possible. 6. Patch: Disable locking (recursive mutex) on called functions from check_tx_inputs which causes slowdowns (only seems to happen on ubuntu/VMs??? Reason: TBD) 7. Optim: Removed looped full-tx hash computation when retrieving transactions from pool (???). 8. Optim: Cache difficulty/timestamps (735 blocks) for next-difficulty calculations so that only 2 db reads per new block is needed when a new block arrives (instead of 1470 reads). Berkeley-DB: 1. Fix: 32-bit data errors causing wrong output global indices and failure to send blocks to peers (etc). 2. Fix: Unable to pop blocks on reorganize due to transaction errors. 3. Patch: Large number of transaction aborts when running multi-threaded bulk queries. 4. Patch: Insufficient locks error when running full sync. 5. Patch: Incorrect db stats when returning from an immediate exit from "pop block" operation. 6. Optim: Add bulk queries to get output global indices. 7. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 8. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 9. Optim: Added thread-safe buffers used when multi-threading bulk queries. 10. Optim: Added support for nosync/write_nosync options for improved performance (*see --db-sync-mode option for details) 11. Mod: Added checkpoint thread and auto-remove-logs option. 12. *Now usable on 32-bit systems like RPI2. LMDB: 1. Optim: Added custom comparison for 256-bit key tables (minor speed-up, TBD: get actual effect) 2. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 3. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 4. Optim: Added support for sync/writemap options for improved performance (*see --db-sync-mode option for details) 5. Mod: Auto resize to +1GB instead of multiplier x1.5 ETC: 1. Minor optimizations for slow-hash for ARM (RPI2). Incomplete. 2. Fix: 32-bit saturation bug when computing next difficulty on large blocks. [PENDING ISSUES] 1. Berkely db has a very slow "pop-block" operation. This is very noticeable on the RPI2 as it sometimes takes > 10 MINUTES to pop a block during reorganization. This does not happen very often however, most reorgs seem to take a few seconds but it possibly depends on the number of outputs present. TBD. 2. Berkeley db, possible bug "unable to allocate memory". TBD. [NEW OPTIONS] (*Currently all enabled for testing purposes) 1. --fast-block-sync arg=[0:1] (default: 1) a. 0 = Compute long hash per block (may take a while depending on CPU) b. 1 = Skip long-hash and verify blocks based on embedded known good block hashes (faster, minimal CPU dependence) 2. --db-sync-mode arg=[[safe|fast|fastest]:[sync|async]:[nblocks_per_sync]] (default: fastest:async:1000) a. safe = fdatasync/fsync (or equivalent) per stored block. Very slow, but safest option to protect against power-out/crash conditions. b. fast/fastest = Enables asynchronous fdatasync/fsync (or equivalent). Useful for battery operated devices or STABLE systems with UPS and/or systems with battery backed write cache/solid state cache. Fast - Write meta-data but defer data flush. Fastest - Defer meta-data and data flush. Sync - Flush data after nblocks_per_sync and wait. Async - Flush data after nblocks_per_sync but do not wait for the operation to finish. 3. --prep-blocks-threads arg=[n] (default: 4 or system max threads, whichever is lower) Max number of threads to use when computing long-hash in groups. 4. --show-time-stats arg=[0:1] (default: 1) Show benchmark related time stats. 5. --db-auto-remove-logs arg=[0:1] (default: 1) For berkeley-db only. Auto remove logs if enabled. **Note: lmdb and berkeley-db have changes to the tables and are not compatible with official git head version. At the moment, you need a full resync to use this optimized version. [PERFORMANCE COMPARISON] **Some figures are approximations only. Using a baseline machine of an i7-2600K+SSD+(with full pow computation): 1. The optimized lmdb/blockhain core can process blocks up to 585K for ~1.25 hours + download time, so it usually takes 2.5 hours to sync the full chain. 2. The current head with memory can process blocks up to 585K for ~4.2 hours + download time, so it usually takes 5.5 hours to sync the full chain. 3. The current head with lmdb can process blocks up to 585K for ~32 hours + download time and usually takes 36 hours to sync the full chain. Averate procesing times (with full pow computation): lmdb-optimized: 1. tx_ave = 2.5 ms / tx 2. block_ave = 5.87 ms / block memory-official-repo: 1. tx_ave = 8.85 ms / tx 2. block_ave = 19.68 ms / block lmdb-official-repo (0f4a036437fd41a5498ee5e74e2422ea6177aa3e) 1. tx_ave = 47.8 ms / tx 2. block_ave = 64.2 ms / block **Note: The following data denotes processing times only (does not include p2p download time) lmdb-optimized processing times (with full pow computation): 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.25 hours processing time (--db-sync-mode=fastest:async:1000). 2. Laptop, Dual-core / 4-threads U4200 (3Mb) - 4.90 hours processing time (--db-sync-mode=fastest:async:1000). 3. Embedded, Quad-core / 4-threads Z3735F (2x1Mb) - 12.0 hours processing time (--db-sync-mode=fastest:async:1000). lmdb-optimized processing times (with per-block-checkpoint) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 10 minutes processing time (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with full pow computation) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.8 hours processing time (--db-sync-mode=fastest:async:1000). 2. RPI2. Improved from estimated 3 months(???) into 2.5 days (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with per-block-checkpoint) 1. RPI2. 12-15 hours (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000).
2015-07-10 20:09:32 +00:00
#if defined(__arm__)
// force a value so it can compile with 32-bit ARM
constexpr static uint64_t DEFAULT_MAPSIZE = 1LL << 31;
#else
#if defined(ENABLE_AUTO_RESIZE)
constexpr static uint64_t DEFAULT_MAPSIZE = 1LL << 30;
#else
constexpr static uint64_t DEFAULT_MAPSIZE = 1LL << 33;
#endif
#endif
db_lmdb: resize blockchain database when 90% filled instead of a random ratio from 60% to 90%.
2018-05-25 10:59:59 +00:00
constexpr static float RESIZE_PERCENT = 0.9f;
Initial commit of lmdb BlockchainDB impl
2014-10-21 20:33:43 +00:00
};
} // namespace cryptonote
Reference in a new issue Copy permalink