Commit graph

131 commits

Author SHA1 Message Date
warptangent
7c013f66e9 Add batch warning for further review 2016-04-05 20:54:06 +01:00
warptangent
ae0854a431 Schema update: tx_indices 2016-04-05 20:54:06 +01:00
Howard Chu
8d252a4214 Consolidated block info 2016-04-05 20:53:59 +01:00
warptangent
132c666f67 Update schema for "tx_outputs" to use array containing amount output indices
This speeds up wallet refresh by directly retrieving a tx's amount output indices.

It removes the indirection and walking the amount output duplicate list
for every amount in each requested tx.

"tx_outputs" is used by:
Amount output indices are needed for wallet refresh.
Global output indices are needed for removing a tx.

Both amount output indices and global output indices are now stored in
an array of 64-bit unsigned ints:

tx_outputs[<tx_hash>] -> [ <a1_oi, a1_gi, a2_oi, a2_gi, ...> ]

Previously it was:
tx_outputs[<tx_hash>] -> duplicate list of <a1_gi, a2_gi, a3_gi, ...>

The amount output list had to be walked for every amount in order to
find each amount's output index, by comparing the amount's global output
index with each one in the duplicate list until a match was found.

See also d045dfa7ce
2016-04-05 20:30:50 +01:00
Riccardo Spagni
32dc08d19a
Merge pull request #700
bdec7cb BlockchainLMDB: Use DB error helper consistently (warptangent)
c5932eb BlockchainLMDB: Add DB error to exception (warptangent)
a49c355 Blockchain: Omit verbose time stats messages by default (warptangent)
2016-03-05 23:10:38 +02:00
Riccardo Spagni
c3af15702f
Merge pull request #698
ee7a8b8 Get rid of lmdb_cur (Howard Chu)
2016-03-05 23:06:26 +02:00
warptangent
bdec7cbfb3
BlockchainLMDB: Use DB error helper consistently 2016-03-03 21:27:13 -08:00
warptangent
c5932eb5ea BlockchainLMDB: Add DB error to exception 2016-03-03 20:15:37 -08:00
Howard Chu
ee7a8b87f9 Get rid of lmdb_cur
We don't need it now with per-txn cursors.
2016-03-03 19:06:27 +00:00
Howard Chu
f5affbef26 More for df239428c0
Let ARMv7 work again
2016-03-03 00:13:13 +00:00
warptangent
2c823c1589
Make partial revert consistent
If user-defined comparator is used, subdb shouldn't be opened with
MDB_INTEGERKEY.

TODO: Again, this will be added back with future schema updates.
2016-03-01 02:47:16 -08:00
warptangent
df239428c0
Revert part of 7db89ed2ee
For now, so existing databases work.

TODO: add these back with future schema updates.
2016-02-29 18:03:41 -08:00
warptangent
c9c4060dc6
BlockchainLMDB: Add height and db error to exception 2016-02-29 17:52:54 -08:00
Riccardo Spagni
d60bf4ee36
Merge pull request #686
7db89ed ARMv7: fix unaligned accesses (Howard Chu)
5a07cef Wrap some more actions in a larger read txn (Howard Chu)
8cc7a36 read txn/cursor stuff (Howard Chu)
86a7f2b core: check whether an update is needed straight away (moneromooo-monero)
ea5fa5e core: print "update needed" hard fork notifications in red (moneromooo-monero)
2016-02-29 17:29:26 -05:00
Howard Chu
7db89ed2ee ARMv7: fix unaligned accesses
And cleanup some key comparators
2016-02-25 13:57:00 +00:00
Howard Chu
8cc7a36f0b read txn/cursor stuff
Could wrap more later.
2016-02-23 20:47:15 +00:00
moneromooo-monero
0eb4d054f6
db_bdb: add missing include for std::condition_variable 2016-02-20 15:38:43 +00:00
warptangent
57e75fa33f
BlockchainBDB: Check if hard fork subdbs need reset
See f7e337e625 for LMDB equivalent.
2016-02-18 06:38:56 -08:00
warptangent
47f6cf8bea
BlockchainBDB: Support blockchain_import --drop-hard-fork command
See c657e772c4 for LMDB equivalent.
2016-02-18 06:37:45 -08:00
Howard Chu
02abe3590d Use MDB_PREV_MULTIPLE
in get_global_output_indices
2016-02-17 16:14:50 +00:00
Howard Chu
7a4755d3a4 Fixup after lmdb master resync 2016-02-17 16:14:38 +00:00
Howard Chu
3b13a7473e Shutup about VERSION 0 2016-02-17 05:06:19 +00:00
Howard Chu
1537477c9f Use cursor in get_output_key 2016-02-17 04:05:29 +00:00
Howard Chu
f2faf8cdd9 Use MDB_APPEND mode where possible
When keys are contiguous and monotonically increasing, this gets
denser page utilization (doesn't leave padding in page splits).
Can't be used for keys that are inserted in random order (e.g. hashes)
In total this only saves around 1.5% of space compared to original
DB code. The previous patch accounted for 0.8% savings on its own;
the blocks tables just aren't that big.
2016-02-17 04:05:29 +00:00
Howard Chu
090b548c3b Use cursors in write txns
Saves a bit of seek overhead. LMDB frees them automatically
in txn_(commit|abort) so they need no cleanup.
2016-02-17 04:05:29 +00:00
Howard Chu
ed08d2152e Keep a running blocksize count
Used in batch size estimation, avoids rereading already processed
blocks during import
2016-02-17 04:05:28 +00:00
Howard Chu
0fc93345ea Win32 import batchsize tweaks
Reduce frequency of resizes: bump minimum increase from 128MB to 512MB
Use a bigger safety margin at small batch sizes
2016-02-17 04:05:28 +00:00
Riccardo Spagni
63e2d4195b
Merge pull request #657
1995923 BlockchainLMDB: Deal with DB exceptions at block level with particularity (warptangent)
c16cc20 BlockchainLMDB: Add sanity check for inconsistent state (warptangent)
9118d0a BlockchainLMDB: Call destructor on allocated txn if setup fails (warptangent)
f5581c3 BlockchainLMDB: Replace remaining txn pointer NULLs with nullptr (warptangent)
2016-02-13 19:54:47 +02:00
warptangent
1995923559
BlockchainLMDB: Deal with DB exceptions at block level with particularity
Add another DB error exception type to distinguish failed txn setup from
general use of txn.

This keeps the error handling flow the same as before the block-level
txn setup changes that moved control up a layer to BlockchainDB.
2016-02-13 05:12:39 -08:00
warptangent
c16cc204d3
BlockchainLMDB: Add sanity check for inconsistent state
This hasn't been known to occur in block-level txn abort, but throw
exception if it does.
2016-02-13 04:17:37 -08:00
warptangent
9118d0a44c
BlockchainLMDB: Call destructor on allocated txn if setup fails 2016-02-13 04:17:37 -08:00
warptangent
f5581c3536
BlockchainLMDB: Replace remaining txn pointer NULLs with nullptr
For consistency.
2016-02-13 04:17:36 -08:00
moneromooo-monero
8649b9f1ef
blockchain_db: pass hard fork object as a simple pointer 2016-02-08 20:56:31 +00:00
warptangent
e02577f594
Move HardFork DB update to BlockchainDB::add_block()
Ensures the database is consistent.

Also simplifes blockchain_import in that verify mode off has less to
work around.
2016-02-08 09:28:19 -08:00
warptangent
3800875406
Make HardFork object available to BlockchainDB and derived DB implementations
This will later allow the HardFork object's DB update functions to be
called when the DB transaction that persists across block add/remove is
open.
2016-02-08 09:28:17 -08:00
warptangent
fd46c96dce
BlockchainDB/LMDB: Refactor block-scope DB txn handling for add block
Move block-scope txn start and stop from BlockchainLMDB to BlockchainDB.
2016-02-08 09:28:15 -08:00
warptangent
f3a6000094
BlockchainDB/LMDB/BDB: Extract DB txn functions for block add/remove 2016-02-08 09:28:14 -08:00
warptangent
f47d5b0fe3
BlockchainLMDB: Allow two HardFork functions to update DB during block add
Note that this doesn't yet cause them to be called during block add.
2016-02-08 09:28:12 -08:00
warptangent
15ee0bef4b
BlockchainLMDB: extract txn macros used during block add/remove 2016-02-08 09:28:10 -08:00
warptangent
f7e337e625
fake_core: Check if hard fork subdbs need reset at start 2016-02-08 08:50:48 -08:00
warptangent
c657e772c4
blockchain_import: Add --drop-hard-fork command 2016-02-08 08:50:47 -08:00
moneromooo-monero
3a0f4d8a57
berkeleydb: fix delete/free mismatch
Despite being C++, the stats object is allocated by the underlying
C layer using malloc(3).
2016-02-03 08:49:11 +00:00
warptangent
b8cc6d46bd
BlockchainBDB: Continue when tx has no outputs
See 24b66243c2
2016-01-28 21:15:46 -08:00
warptangent
b8707466e2
BlockchainBDB: When removing, find amount output index fast by starting at end
This improves blockchain reorganization time by allowing one of the more
expensive DB lookups when popping a block to not have to seek through a
long dup list in the "output_amounts" db. This is most noticeable for
HDDs.

See ffcf6bdb95
2016-01-28 21:15:44 -08:00
warptangent
1a5c3fa729
BlockchainBDB: Remove tx outputs in reverse order
Data should be removed in the reverse order it was added.

This matches the order of removal in
blockchain_storage::pop_transaction_from_global_index.

See f11def012f
2016-01-28 21:15:42 -08:00
me0wmix
28f95eb001 OpenBSD support for Monero. 2016-01-21 11:18:26 -07:00
warptangent
24b66243c2
db_lmdb: Continue when tx has no outputs 2016-01-19 16:55:31 -08:00
Howard Chu
30f92f5630 Fix hf when import with verify off
Delete the hf tables, so the next open will rescan and regenerate
2016-01-15 17:26:19 +00:00
Riccardo Spagni
de03926850
updated copyright year 2015-12-31 08:39:56 +02:00
Riccardo Spagni
810a11267c
fixed copyrights with bad year references 2015-12-31 08:37:27 +02:00