Commit graph

62 commits

Author SHA1 Message Date
Howard Chu
d2d8a41c47
Use actual batch size for resize estimates
And optimize import startup:

Remember start_height position during initial count_blocks pass
to avoid having to reread entire file again to arrive at start_height
2017-09-17 22:08:40 +01:00
moneromooo-monero
a95e460c71
move db specific options to BlockchainDB
Avoids common depending on blockchain_db, which can cause
link errors.
2017-08-29 11:43:34 +01:00
Howard Chu
9a859844f4
Toggle SAFE syncmode on and off automatically
If monerod is started with default sync mode, set it to SAFE after
synchronization completes. Set it back to FAST if synchronization
restarts (e.g. because another peer has a longer blockchain).

If monerod is started with an explicit sync mode, none of this
automation takes effect.
2017-08-20 16:30:28 +01:00
Howard Chu
4c7f8ac04f
DB cleanup
Hide LMDB-specific stuff behind blockchain_db.h. Nobody besides blockchain_db.cpp
should ever be including DB-specific headers any more.
2017-08-19 18:11:38 +01:00
moneromooo-monero
235df7f484
blockchain_db: add a txpool tx getter which returns existence
Avoids exception spam for the "nope, not found" case
2017-06-11 15:36:48 +01:00
Howard Chu
d17c0fc2d0
Don't copy blockchain for coinbase_tx_sum
Changed Blockchain::for_all_blocks() to for_blocks_range()
Operate on blockchain in-place instead of building a copy first.
2017-06-01 14:14:24 +01:00
moneromooo-monero
b52abd1370
Move txpool to the database
Integration could go further (ie, return_tx_to_pool calls should
not be needed anymore, possibly other things).

poolstate.bin is now obsolete.
2017-05-25 22:23:37 +01:00
kenshi84
7d07c64fe5
fix dependency: put HardFork back to cryptonote_basic, made some BlockchainDB functions virtual again to avoid missing symbols error 2017-03-10 11:22:39 +09:00
moneromooo-monero
12adb4a3f3
core: move hardfork back to cryptonote_core
should fix a cross dependency betewen cryptonote_basic and
blockchain_db
2017-02-25 16:41:35 +00:00
Riccardo Spagni
c3599fa7b9
update copyright year, fix occasional lack of newline at line end 2017-02-21 19:38:18 +02:00
Riccardo Spagni
49efd3add9
Merge pull request #1727
0288310e blockchain_db: add "raw" blobdata getters for block and transaction (moneromooo-monero)
2017-02-21 11:27:15 +02:00
moneromooo-monero
0288310e3b
blockchain_db: add "raw" blobdata getters for block and transaction
This speeds up operations such as serving blocks to syncing peers
2017-02-13 21:11:37 +00:00
moneromooo-monero
cca95c1c7a
blockchain_db: do not throw on expected partial results getting keys
When scanning for outputs used in a set of incoming blocks,
we expect that some of the inputs in their transactions will
not be found in the blockchain, as they could be in previous
blocks in that set. Those outputs will be scanned there at
a later point. In this case, we add a flag to control wehther
an output not being found is expected or not.
2017-02-13 19:05:30 +00:00
kenshi84
8027ce0c75 extract some basic code from libcryptonote_core into libcryptonote_basic 2017-02-08 22:45:15 +09:00
Riccardo Spagni
65e33b1bc5
Merge pull request #1506
3ff54bdd Check for correct thread before ending batch transaction (Howard Chu)
eaf8470b Must wait for previous batch to finish before starting new one (Howard Chu)
c903c554 Don't cache block height, always get from DB (Howard Chu)
eb1fb601 Tweak default db-sync-mode to fast:async:1 (Howard Chu)
0693cff9 Use batch transactions when syncing (Howard Chu)
2017-01-15 14:43:12 -05:00
Howard Chu
0693cff925
Use batch transactions when syncing
Faster throughput while avoiding corruption. I.e., makes
running with --db-sync-mode safe more tolerable.
2017-01-14 22:43:06 +00:00
moneromooo-monero
0478ac6848
blockchain: allow marking "tx not found" without an exception
This is a normal occurence in many cases, and there is no need
to spam the log with those when it is.
2017-01-07 20:52:17 +00:00
moneromooo-monero
88faec75fe
wallet: select part of the fake outs from recent outputs
25% of the outputs are selected from the last 5 days (if possible),
in order to avoid the common case of sending recently received
outputs again. 25% and 5 days are subject to review later, since
it's just a wallet level change.
2016-10-15 18:17:16 +01:00
moneromooo-monero
6cf8ca2a7f
core: faster find_blockchain_supplement
Since this queries block heights for blocks that may or may not
exist, queries for non existing blocks would throw an exception,
and that would slow down the loop a lot. 7 seconds to go through
a 30 hash list.

Fix this by adding an optional return block height to block_exists
and using this instead. Actual errors will still throw an
exception.

This also cuts down on log exception spam.
2016-08-31 10:03:32 +01:00
moneromooo-monero
59a66e209a
move the rct commitments to the output_amounts database
Since these are needed at the same time as the output pubkeys,
this is a whole lot faster, and takes less space. Only outputs
of 0 amount store the commitment. When reading other outputs,
a fake commitment is regenerated on the fly. This avoids having
to rewrite the database to add space for fake commitments for
existing outputs.

This code relies on two things:

- LMDB must support fixed size records per key, rather than
per database (ie, all records on key 0 are the same size, all
records for non 0 keys are same size, but records from key 0
and non 0 keys do have different sizes).

- the commitment must be directly after the rest of the data
in outkey and output_data_t.
2016-08-28 21:29:02 +01:00
moneromooo-monero
eb56d0f994
blockchain_db: add functions for adding/removing/getting rct commitments 2016-08-28 21:28:11 +01:00
moneromooo-monero
1593553e03
new unlocked parameter to output_histogram
This constrains the number of instances of any amount
to the unlocked ones (as defined by the default unlock time
setting: outputs with non default unlock time are not
considered, so may be counted as unlocked even if they are
not actually unlocked).
2016-08-01 22:16:00 +01:00
moneromooo-monero
d7b681cd65
remove hf_starting_height db
It's not really needed, it used to be an optimization for when
that code was not using the db and needed to recalculate things
fast on startup.
2016-07-13 21:38:34 +01:00
Howard Chu
2b0fa05f0d Another take on migration
Delete old indices and recreate them, rather than updating them
Maybe not quite as slow as before.
2016-04-24 17:46:50 +01:00
Howard Chu
d7ea7d9a23 Merge branch 'performance' into master 2016-04-05 21:13:16 +01:00
Howard Chu
372acee723 Cleanup
drop obsolete remove_output()
fix get_output_key(global), fix crash in blockchain_dump
2016-04-05 21:05:24 +01:00
Howard Chu
591e421875 Cleanup and clarify
Try to rationalize the variable names, document usage.
2016-04-05 20:57:45 +01:00
warptangent
9aadedb1d0 Schema update: tx_indices - consolidate the tx subdbs from 5 to 3 2016-04-05 20:54:06 +01:00
warptangent
a2f518aa01 Schema update: tx_indices - yet less indirection 2016-04-05 20:54:06 +01:00
warptangent
8d12a8df2c Schema update: tx_indices - improve further with less indirection 2016-04-05 20:54:06 +01:00
warptangent
132c666f67 Update schema for "tx_outputs" to use array containing amount output indices
This speeds up wallet refresh by directly retrieving a tx's amount output indices.

It removes the indirection and walking the amount output duplicate list
for every amount in each requested tx.

"tx_outputs" is used by:
Amount output indices are needed for wallet refresh.
Global output indices are needed for removing a tx.

Both amount output indices and global output indices are now stored in
an array of 64-bit unsigned ints:

tx_outputs[<tx_hash>] -> [ <a1_oi, a1_gi, a2_oi, a2_gi, ...> ]

Previously it was:
tx_outputs[<tx_hash>] -> duplicate list of <a1_gi, a2_gi, a3_gi, ...>

The amount output list had to be walked for every amount in order to
find each amount's output index, by comparing the amount's global output
index with each one in the duplicate list until a match was found.

See also d045dfa7ce
2016-04-05 20:30:50 +01:00
moneromooo-monero
600a3cf0c0
New RPC and daemon command to get output histogram
This is a list of existing output amounts along with the number
of outputs of that amount in the blockchain.

The daemon command takes:
- no parameters: all outputs with at least 3 instances
- one parameter: all outputs with at least that many instances
- two parameters: all outputs within that many instances

The default starts at 3 to avoid massive spamming of all dust
outputs in the blockchain, and is the current minimum mixin
requirement.

An optional vector of amounts may be passed, to request
histogram only for those outputs.
2016-03-26 21:10:43 +00:00
Riccardo Spagni
0d30b65785 Merge pull request #749
bfd4a28 Update BlockchainDB documentation (Thomas Winget)
797357e Change Doxyfile, Blockchain not blockchain_storage (Thomas Winget)
c835215 remove defunct code from cryptonote::core (Thomas Winget)
50dba6d cryptonote::core doxygen documentation (Thomas Winget)
8ac329d doxygen documentation for difficulty functions (Thomas Winget)
540a76c Move checkpoint functions into checkpoints class (Thomas Winget)
1b0c98e doxygen documentation for checkpoints.{h,cpp} (Thomas Winget)
89c24ac Remove unnecessary or defunct code (Thomas Winget)
ab0ed14 doxygen include private and static members (Thomas Winget)
3a48449 Updated documentation for blockchain.* (Thomas Winget)
2016-03-25 09:12:25 +02:00
Riccardo Spagni
01e0a69c17 Revert "Merge pull request #749"
This reverts commit 7fa63a82a1, reversing
changes made to cb6be986c3.
2016-03-25 08:42:42 +02:00
Thomas Winget
bfd4a28c41
Update BlockchainDB documentation
BlockchainDB is now Doxygen-compliant and its documentation is
up-to-date with recent changes.
2016-03-24 03:40:04 -04:00
moneromooo-monero
f7301c3563
Revert "Print stack trace upon exceptions"
Ain't nobody got time for link/cmake skullduggery.

This reverts commit fff238ec94.
2016-03-21 10:12:23 +00:00
moneromooo-monero
fff238ec94
Print stack trace upon exceptions
Useful for debugging users' logs
2016-03-19 21:48:36 +00:00
Howard Chu
8cc7a36f0b read txn/cursor stuff
Could wrap more later.
2016-02-23 20:47:15 +00:00
Riccardo Spagni
63e2d4195b
Merge pull request #657
1995923 BlockchainLMDB: Deal with DB exceptions at block level with particularity (warptangent)
c16cc20 BlockchainLMDB: Add sanity check for inconsistent state (warptangent)
9118d0a BlockchainLMDB: Call destructor on allocated txn if setup fails (warptangent)
f5581c3 BlockchainLMDB: Replace remaining txn pointer NULLs with nullptr (warptangent)
2016-02-13 19:54:47 +02:00
warptangent
1995923559
BlockchainLMDB: Deal with DB exceptions at block level with particularity
Add another DB error exception type to distinguish failed txn setup from
general use of txn.

This keeps the error handling flow the same as before the block-level
txn setup changes that moved control up a layer to BlockchainDB.
2016-02-13 05:12:39 -08:00
moneromooo-monero
8649b9f1ef
blockchain_db: pass hard fork object as a simple pointer 2016-02-08 20:56:31 +00:00
warptangent
3800875406
Make HardFork object available to BlockchainDB and derived DB implementations
This will later allow the HardFork object's DB update functions to be
called when the DB transaction that persists across block add/remove is
open.
2016-02-08 09:28:17 -08:00
warptangent
f3a6000094
BlockchainDB/LMDB/BDB: Extract DB txn functions for block add/remove 2016-02-08 09:28:14 -08:00
warptangent
c657e772c4
blockchain_import: Add --drop-hard-fork command 2016-02-08 08:50:47 -08:00
Howard Chu
30f92f5630 Fix hf when import with verify off
Delete the hf tables, so the next open will rescan and regenerate
2016-01-15 17:26:19 +00:00
Riccardo Spagni
810a11267c
fixed copyrights with bad year references 2015-12-31 08:37:27 +02:00
warptangent
ee9d71e9f9
BlockchainDB: skip fixup check if read-only database 2015-12-26 14:30:20 -08:00
warptangent
725acc7f17
Replace tabs with two spaces for consistency with rest of codebase
Remove trailing whitespace in same files.
2015-12-15 06:22:06 -08:00
moneromooo-monero
a98e976f9e
blockchain_db: fixup missing key images in early DB version
Early DB versions did not store key images for inputs if the
transaction spending them had no outputs (ie, all fee). This
is not correct, as this would allow these outputs to be double
spent. This was fixed in 533acc30ed
a few months ago, but databases having synced blocks 2021612 and
685498 with a faulty version will be missing those key images
in the spent keys database. This code checks for this, and adds
those key images if they are missing.
2015-12-06 21:55:05 +00:00
moneromooo-monero
a3c5ca077c
blockchain_db: make the indexing base a BlockchainDB virtual function 2015-12-05 18:41:29 +00:00