Commit graph

298 commits

Author SHA1 Message Date
flozilla
f9fad18608
blockchain_db: sanity check on tx/hash vector sizes
It could trip on a corrupt/crafted file if the user has disabled
input verification.
2017-11-15 10:02:23 +00:00
Riccardo Spagni
5de492be40
Merge pull request #2629
383ff4f6 remove "using namespace std" from headers (moneromooo-monero)
2017-11-14 21:29:49 +02:00
moneromooo-monero
383ff4f689
remove "using namespace std" from headers
It's nasty, and actually breaks on Solaris, where if.h fails to
build due to:

  struct map *if_memmap;
2017-11-14 16:56:10 +00:00
Riccardo Spagni
0d0ce8c21e
Merge pull request #2742
7c7d3672 Increase LMDB maxreaders if large number of threads in use (Howard Chu)
6738753b Use max_concurrency as-is (Howard Chu)
2017-11-14 15:30:03 +02:00
Riccardo Spagni
fd0740e5e7
Merge pull request #2615
10013e94 Protect node privacy by proper filtering in restricted-mode RPC answers (binaryFate)
2017-11-14 14:53:10 +02:00
binaryFate
10013e9434 Protect node privacy by proper filtering in restricted-mode RPC answers
This patch allows to filter out sensitive information for queries that rely on the pool state, when running in restricted mode.
This filtering is only applied to data sent back to RPC queries. Results of inline commands typed locally in the daemon are not affected.
In practice, when running with `--restricted-rpc`:
* get_transaction_pool will list relayed transactions with the fields "last relayed time" and "received time" set to zero.
* get_transaction_pool will not list transaction that have do_not_relay set to true, and will not list key images that are used only for such transactions
* get_transaction_pool_hashes.bin will not list such transaction
* get_transaction_pool_stats will not count such transactions in any of the aggregated values that are computed

The implementation does not make filtering the default, so developers should be mindful of this if they add new RPC functionality.
Fixes #2590.
2017-11-08 13:07:07 +01:00
moneromooo-monero
ccf53a566c
track double spending in the txpool
Transactions in the txpool are marked when another transaction
is seen double spending one or more of its inputs.
This is then exposed wherever appropriate.

Note that being marked with this "double spend seen" flag does
NOT mean this transaction IS a double spend and will never be
mined: it just means that the network has seen at least another
transaction spending at least one of the same inputs, so care
should be taken to wait for a few confirmations before acting
upon that transaction (ie, mostly of use for merchants wanting
to accept unconfirmed transactions).
2017-11-06 00:05:44 +00:00
Howard Chu
7c7d36725a
Increase LMDB maxreaders if large number of threads in use 2017-11-02 16:33:48 +00:00
ston1th
f946bb7be6 fixed an LMDB issue on OpenBSD (#2699) which leads to a crash in monerod 2017-10-26 17:16:06 +02:00
Riccardo Spagni
b293ad9a1a
Merge pull request #2640
22b51e06 db_lmdb: include chain height when failing to find an output key (moneromooo-monero)
5db433b3 blockchain: avoid exceptions in output verification (moneromooo-monero)
2017-10-15 18:59:45 +02:00
moneromooo-monero
22b51e0672
db_lmdb: include chain height when failing to find an output key
To help debugging logs.
2017-10-12 13:28:28 +01:00
moneromooo-monero
bc1b93333e
frob level 1 logs a bit for consistency
Level 1 logs map to INFO, so setting log level to 1 should
show these. Demote some stuff to DEBUG to avoid spam, though.
2017-10-07 20:03:42 +01:00
Riccardo Spagni
e5a2cdbfad
Merge pull request #2457
d2d8a41c Use actual batch size for resize estimates (Howard Chu)
2017-09-25 16:57:07 +02:00
moneromooo-monero
9aeae7a224
db_lmdb: fix use of uninitialized key in for_blocks_range 2017-09-19 14:11:24 +01:00
Howard Chu
d2d8a41c47
Use actual batch size for resize estimates
And optimize import startup:

Remember start_height position during initial count_blocks pass
to avoid having to reread entire file again to arrive at start_height
2017-09-17 22:08:40 +01:00
Riccardo Spagni
1e57e48342
Merge pull request #2372
c867357a cryptonote_protocol: error handling on cleanup_handle_incoming_blocks (moneromooo-monero)
ce901fcb Fix blockchain_import wedge on exception in cleanup_handle_incoming_blocks (moneromooo-monero)
84fa015e core: guard against exceptions in handle_incoming_{block,tx} (moneromooo-monero)
2017-09-02 11:30:57 +02:00
moneromooo-monero
ce901fcb31
Fix blockchain_import wedge on exception in cleanup_handle_incoming_blocks 2017-08-29 15:43:32 +01:00
moneromooo-monero
a95e460c71
move db specific options to BlockchainDB
Avoids common depending on blockchain_db, which can cause
link errors.
2017-08-29 11:43:34 +01:00
Riccardo Spagni
36b978f50e
Merge pull request #2314
c22d22e2 Cleanup test impact of adding safesyncmode() method (Howard Chu)
9a859844 Toggle SAFE syncmode on and off automatically (Howard Chu)
2017-08-26 23:43:22 +02:00
Howard Chu
9a859844f4
Toggle SAFE syncmode on and off automatically
If monerod is started with default sync mode, set it to SAFE after
synchronization completes. Set it back to FAST if synchronization
restarts (e.g. because another peer has a longer blockchain).

If monerod is started with an explicit sync mode, none of this
automation takes effect.
2017-08-20 16:30:28 +01:00
Howard Chu
80344740bd
More DB support cleanup
Hide DB types from db_types.h - no reason to recompile dependencies
when DB types change.

Also remove lingering in-memory DB references, they've been
obsolete since 9e82b694da
2017-08-20 13:57:36 +01:00
Howard Chu
4c7f8ac04f
DB cleanup
Hide LMDB-specific stuff behind blockchain_db.h. Nobody besides blockchain_db.cpp
should ever be including DB-specific headers any more.
2017-08-19 18:11:38 +01:00
moneromooo-monero
c6e200a8ab
core: add --db-salvage command line flag
Use to load the database when the primary meta page is corrupted
2017-08-12 19:04:44 +01:00
moneromooo-monero
214fd81e93
some include cleanup 2017-07-31 16:36:52 +01:00
moneromooo-monero
235df7f484
blockchain_db: add a txpool tx getter which returns existence
Avoids exception spam for the "nope, not found" case
2017-06-11 15:36:48 +01:00
Howard Chu
d17c0fc2d0
Don't copy blockchain for coinbase_tx_sum
Changed Blockchain::for_all_blocks() to for_blocks_range()
Operate on blockchain in-place instead of building a copy first.
2017-06-01 14:14:24 +01:00
Riccardo Spagni
a4c13ea092
Merge pull request #2039
4b932ff3 changed crypto to cncrypto so it generated libcncrypto (Gentian)
2017-05-30 21:24:11 +02:00
moneromooo-monero
b52abd1370
Move txpool to the database
Integration could go further (ie, return_tx_to_pool calls should
not be needed anymore, possibly other things).

poolstate.bin is now obsolete.
2017-05-25 22:23:37 +01:00
Gentian
4b932ff314 changed crypto to cncrypto so it generated libcncrypto
fix a cmakelist
2017-05-23 07:45:40 -04:00
Howard Chu
d3aaf74080
Partial fix #1876, check thread in block_txn_start()
Don't allow use of existing batch txn if it's from the wrong thread
2017-03-19 12:27:03 +00:00
kenshi84
7d07c64fe5
fix dependency: put HardFork back to cryptonote_basic, made some BlockchainDB functions virtual again to avoid missing symbols error 2017-03-10 11:22:39 +09:00
Howard Chu
eb62dcc871
Remove redundant num_txs() method
Cleanup of bf1348b7e2
2017-03-03 16:52:45 +00:00
moneromooo-monero
12adb4a3f3
core: move hardfork back to cryptonote_core
should fix a cross dependency betewen cryptonote_basic and
blockchain_db
2017-02-25 16:41:35 +00:00
Riccardo Spagni
c3599fa7b9
update copyright year, fix occasional lack of newline at line end 2017-02-21 19:38:18 +02:00
Riccardo Spagni
49efd3add9
Merge pull request #1727
0288310e blockchain_db: add "raw" blobdata getters for block and transaction (moneromooo-monero)
2017-02-21 11:27:15 +02:00
Riccardo Spagni
59d96eedd8
Merge pull request #1724
cca95c1c blockchain_db: do not throw on expected partial results getting keys (moneromooo-monero)
2017-02-21 11:23:03 +02:00
Howard Chu
2e9136767e
Handle map resizes from other processes 2017-02-18 22:22:24 +00:00
Howard Chu
bf1348b7e2
Can't cache num_txs or num_outputs either
Same reason as 3ff54bdd7a
2017-02-18 21:01:24 +00:00
Howard Chu
dc53e9eef2
Add a few read txns to streamline
Slight perf gain, but mainly to reduce spam at loglevel 3
2017-02-18 21:01:24 +00:00
moneromooo-monero
0288310e3b
blockchain_db: add "raw" blobdata getters for block and transaction
This speeds up operations such as serving blocks to syncing peers
2017-02-13 21:11:37 +00:00
moneromooo-monero
cca95c1c7a
blockchain_db: do not throw on expected partial results getting keys
When scanning for outputs used in a set of incoming blocks,
we expect that some of the inputs in their transactions will
not be found in the blockchain, as they could be in previous
blocks in that set. Those outputs will be scanned there at
a later point. In this case, we add a flag to control wehther
an output not being found is expected or not.
2017-02-13 19:05:30 +00:00
moneromooo-monero
f2986ccfc1
db_lmdb: fix bad height saved in tx data
The recent change to not keep separate track of the blockchain
height caused the reported height to jump early in the lmdb
transaction (when the block data is added to the blocks table),
rather than at the end, after everything succeeded. Since the
block data is added before the transaction data, this caused
the transaction data to be saved with a height one more than
its expected value.

Fix this by saving the block data last. This should have no
side effects.
2017-02-11 10:16:18 +00:00
kenshi84
8027ce0c75 extract some basic code from libcryptonote_core into libcryptonote_basic 2017-02-08 22:45:15 +09:00
Miguel Herranz
95f3e193f0 Update and use blockchain data files defines 2017-01-28 15:55:14 +01:00
Howard Chu
843769f88f
Tweak some msg loglevels 2017-01-16 20:42:59 +00:00
Howard Chu
990e08f090
Fix PR#1506, off by one in chain height 2017-01-16 10:00:06 +00:00
moneromooo-monero
5833d66f65
Change logging to easylogging++
This replaces the epee and data_loggers logging systems with
a single one, and also adds filename:line and explicit severity
levels. Categories may be defined, and logging severity set
by category (or set of categories). epee style 0-4 log level
maps to a sensible severity configuration. Log files now also
rotate when reaching 100 MB.

To select which logs to output, use the MONERO_LOGS environment
variable, with a comma separated list of categories (globs are
supported), with their requested severity level after a colon.
If a log matches more than one such setting, the last one in
the configuration string applies. A few examples:

This one is (mostly) silent, only outputting fatal errors:

MONERO_LOGS=*:FATAL

This one is very verbose:

MONERO_LOGS=*:TRACE

This one is totally silent (logwise):

MONERO_LOGS=""

This one outputs all errors and warnings, except for the
"verify" category, which prints just fatal errors (the verify
category is used for logs about incoming transactions and
blocks, and it is expected that some/many will fail to verify,
hence we don't want the spam):

MONERO_LOGS=*:WARNING,verify:FATAL

Log levels are, in decreasing order of priority:
FATAL, ERROR, WARNING, INFO, DEBUG, TRACE

Subcategories may be added using prefixes and globs. This
example will output net.p2p logs at the TRACE level, but all
other net* logs only at INFO:

MONERO_LOGS=*:ERROR,net*:INFO,net.p2p:TRACE

Logs which are intended for the user (which Monero was using
a lot through epee, but really isn't a nice way to go things)
should use the "global" category. There are a few helper macros
for using this category, eg: MGINFO("this shows up by default")
or MGINFO_RED("this is red"), to try to keep a similar look
and feel for now.

Existing epee log macros still exist, and map to the new log
levels, but since they're used as a "user facing" UI element
as much as a logging system, they often don't map well to log
severities (ie, a log level 0 log may be an error, or may be
something we want the user to see, such as an important info).
In those cases, I tried to use the new macros. In other cases,
I left the existing macros in. When modifying logs, it is
probably best to switch to the new macros with explicit levels.

The --log-level options and set_log commands now also accept
category settings, in addition to the epee style log levels.
2017-01-16 00:25:46 +00:00
Riccardo Spagni
65e33b1bc5
Merge pull request #1506
3ff54bdd Check for correct thread before ending batch transaction (Howard Chu)
eaf8470b Must wait for previous batch to finish before starting new one (Howard Chu)
c903c554 Don't cache block height, always get from DB (Howard Chu)
eb1fb601 Tweak default db-sync-mode to fast:async:1 (Howard Chu)
0693cff9 Use batch transactions when syncing (Howard Chu)
2017-01-15 14:43:12 -05:00
Howard Chu
3ff54bdd7a
Check for correct thread before ending batch transaction 2017-01-15 15:51:57 +00:00
Howard Chu
c903c5541e
Don't cache block height, always get from DB 2017-01-14 22:43:06 +00:00
Howard Chu
0693cff925
Use batch transactions when syncing
Faster throughput while avoiding corruption. I.e., makes
running with --db-sync-mode safe more tolerable.
2017-01-14 22:43:06 +00:00
Riccardo Spagni
006bb0e957
Merge pull request #1538
aff28178 Remove db-auto-remove-logs (Miguel Herranz)
1229c685 Remove berkeley from db_type initialization (Miguel Herranz)
e3090558 Show available types for db-type command (Miguel Herranz)
046ab33d Remove berkeley from blockchain_db_types (Miguel Herranz)
2017-01-13 14:33:59 -05:00
Miguel Herranz
046ab33d1b Remove berkeley from blockchain_db_types 2017-01-10 11:17:48 +01:00
moneromooo-monero
0478ac6848
blockchain: allow marking "tx not found" without an exception
This is a normal occurence in many cases, and there is no need
to spam the log with those when it is.
2017-01-07 20:52:17 +00:00
moneromooo-monero
0e18f46522
db_lmdb: add info in an error message when we can't get an output
Will be useful to debug
2016-12-01 14:31:40 +00:00
moneromooo-monero
c96f9b0255
db_lmdb: guard against going out of sync on unexpected db results
m_num_outputs keeps track of the number of outputs, which should
be the same as the size of both the output_txs and output_amounts
databases. If one goes out of sync, we need to throw to abort
whatever it is we were doing.
2016-12-01 14:29:35 +00:00
moneromooo-monero
bef51e677e
db_lmdb: minor pedantic tweaks
Add consts in a few places where it makes sense, avoid unnecessary
memory reallocation where we know the full size needed at the outset,
simplify and avoid memory copy.
2016-12-01 14:28:09 +00:00
moneromooo-monero
3465c4ebc7
db_lmdb: set same packing format for output_data_t and pre_rct_output_data_t
For safety, though it seems to have been the case already.

Also add a comment about the necessary layout identity.
2016-12-01 14:26:18 +00:00
moneromooo-monero
88faec75fe
wallet: select part of the fake outs from recent outputs
25% of the outputs are selected from the last 5 days (if possible),
in order to avoid the common case of sending recently received
outputs again. 25% and 5 days are subject to review later, since
it's just a wallet level change.
2016-10-15 18:17:16 +01:00
Riccardo Spagni
4cb1348548
Merge pull request #1139
01ec195 Update CMakeLists.txt (codehalo)
446ebbc Update CMakeLists.txt (codehalo)
bd773e7 Update CMakeLists.txt (codehalo)
3627cea Cleanup. Dropped "bit" from bitmonero. (Randi Joseph)
2a51396 Dropped "bit" from bitmonero. (Randi Joseph)
78b13d6 Cleanup. Dropped "bit" from bitmonero. (Randi Joseph)
1e6aedb Cleanup. Dropped "bit" from bitmonero. (Randi Joseph)
9e54616 Dropped "bit" from bitmonero. (Randi Joseph)
2016-10-04 12:11:03 +02:00
Riccardo Spagni
995dde4f95
Merge pull request #1128
d2e11f3 db_lmdb: do not error if dropping a non existent hard fork table (moneromooo-monero)
2016-09-29 22:16:15 +02:00
Randi Joseph
9e54616924 Dropped "bit" from bitmonero. 2016-09-26 17:22:30 -04:00
moneromooo-monero
d2e11f374a
db_lmdb: do not error if dropping a non existent hard fork table 2016-09-24 16:15:50 +01:00
Myagui
12427dd8d3 Fix minor typo in "need resize" message.
Message observed while synchronizing a node from scratch.
"LMDB memory map needs resized"
Proposing a change to:
"LMDB memory map needs to be resized"
2016-09-21 20:40:07 +01:00
redfish
e1c7af35d4 cmake: transitive deps and remove deprecated LINK_*
Keep the immediate direct deps at the library that depends on them,
declare deps as PUBLIC so that targets that link against that library
get the library's deps as transitive deps.

Break dep cycle between blockchain_db <-> crytonote_core.
No code refactoring, just hide cycle from cmake so that
it doesn't complain (cycles are allowed only between
static libs, not shared libs).

This is in preparation for supproting BUILD_SHARED_LIBS cmake
built-in option for building internal libs as shared.
2016-09-18 02:56:26 -04:00
moneromooo-monero
6cf8ca2a7f
core: faster find_blockchain_supplement
Since this queries block heights for blocks that may or may not
exist, queries for non existing blocks would throw an exception,
and that would slow down the loop a lot. 7 seconds to go through
a 30 hash list.

Fix this by adding an optional return block height to block_exists
and using this instead. Actual errors will still throw an
exception.

This also cuts down on log exception spam.
2016-08-31 10:03:32 +01:00
moneromooo-monero
1cdba42a77
db_lmdb: fix reorg with "halfway rct" coinbase outputs 2016-08-29 22:44:17 +01:00
moneromooo-monero
c3b3260ae5
New "Halfway RingCT" outputs for coinbase transactions
When RingCT is enabled, outputs from coinbase transactions
are created as a single output, and stored as RingCT output,
with a fake mask. Their amount is not hidden on the blockchain
itself, but they are then able to be used as fake inputs in
a RingCT ring. Since the output amounts are hidden, their
"dustiness" is not an obstacle anymore to mixing, and this
makes the coinbase transactions a lot smaller, as well as
helping the TXO set to grow more slowly.

Also add a new "Null" type of rct signature, which decreases
the size required when no signatures are to be stored, as
in a coinbase tx.
2016-08-28 21:30:26 +01:00
moneromooo-monero
59a66e209a
move the rct commitments to the output_amounts database
Since these are needed at the same time as the output pubkeys,
this is a whole lot faster, and takes less space. Only outputs
of 0 amount store the commitment. When reading other outputs,
a fake commitment is regenerated on the fly. This avoids having
to rewrite the database to add space for fake commitments for
existing outputs.

This code relies on two things:

- LMDB must support fixed size records per key, rather than
per database (ie, all records on key 0 are the same size, all
records for non 0 keys are same size, but records from key 0
and non 0 keys do have different sizes).

- the commitment must be directly after the rest of the data
in outkey and output_data_t.
2016-08-28 21:29:02 +01:00
moneromooo-monero
dc4aad7eb5
add rct to the protocol
It is not yet constrained to a fork, so don't use on the real network
or you'll be orphaned or rejected.
2016-08-28 21:28:37 +01:00
moneromooo-monero
211d1db762
db_lmdb: update reset for recent db changes
- we need to drop the new m_tx_indices database
- we reset the version to current version

This fixes the core tests failing to initialize.
2016-08-28 21:28:35 +01:00
moneromooo-monero
eb56d0f994
blockchain_db: add functions for adding/removing/getting rct commitments 2016-08-28 21:28:11 +01:00
moneromooo-monero
f88029e72a
db_lmdb: do not try to modify the database in read only mode 2016-08-26 19:33:25 +01:00
Riccardo Spagni
0faf572db8
Merge pull request #948
11dc091 Fake outs set is now decided by the wallet (moneromooo-monero)
1593553 new unlocked parameter to output_histogram (moneromooo-monero)
2016-08-11 22:43:14 +02:00
moneromooo-monero
240864f529
db_lmdb: errors dropping hf starting height db on reset are not fatal
This db is now dropped unconditionally, so may or may not be there
in the first place.
2016-08-10 10:50:17 +01:00
moneromooo-monero
1593553e03
new unlocked parameter to output_histogram
This constrains the number of instances of any amount
to the unlocked ones (as defined by the default unlock time
setting: outputs with non default unlock time are not
considered, so may be counted as unlocked even if they are
not actually unlocked).
2016-08-01 22:16:00 +01:00
moneromooo-monero
121165f222
db_lmdb: add some missing api call checks 2016-07-26 22:39:51 +01:00
moneromooo-monero
d7b681cd65
remove hf_starting_height db
It's not really needed, it used to be an optimization for when
that code was not using the db and needed to recalculate things
fast on startup.
2016-07-13 21:38:34 +01:00
Riccardo Spagni
30dce45643 Merge branch 'performance' of https://github.com/LMDB/bitmonero 2016-04-29 21:41:43 +02:00
moneromooo-monero
513a658c87
add a --max-concurrency flag
It sets the max number of threads to use for a parallel job.
This is different that the number of total threads, since monero
binaries typically start a lot of them.
2016-04-28 20:33:59 +01:00
Howard Chu
2b0fa05f0d Another take on migration
Delete old indices and recreate them, rather than updating them
Maybe not quite as slow as before.
2016-04-24 17:46:50 +01:00
Howard Chu
c14f9efd52 Migration
Migrate from DB version 0 to version 1 on startup
2016-04-08 03:11:05 +01:00
Howard Chu
d7ea7d9a23 Merge branch 'performance' into master 2016-04-05 21:13:16 +01:00
Howard Chu
372acee723 Cleanup
drop obsolete remove_output()
fix get_output_key(global), fix crash in blockchain_dump
2016-04-05 21:05:24 +01:00
Howard Chu
591e421875 Cleanup and clarify
Try to rationalize the variable names, document usage.
2016-04-05 20:57:45 +01:00
Howard Chu
b2f1c58805 Use cursors in some remove functions
Helps when they're called repeatedly in one txn
2016-04-05 20:57:31 +01:00
Howard Chu
118dd69dd5 Use DUPFIXED for block_info and output_txs
Saves another ~150MB or so on the full blockchain
2016-04-05 20:55:16 +01:00
Howard Chu
6225716f3c More outputs consolidation
Also bumped DB VERSION to 1
Another significant speedup and space savings:
Get rid of global_output_indices, remove indirection from output to keys

This is the change warptangent described on irc but never got to finish.
2016-04-05 20:55:12 +01:00
Howard Chu
7c5abdc3a3 Use DUPFIXED for output_keys
Saves another 90MB on 200000 block import.
Had to bring back compare_uint64 for this, but it's safe since
this table is always 64-bit aligned.
2016-04-05 20:54:42 +01:00
Howard Chu
8e9d8e3364 Use DUPFIXED for tx_indices
Small space savings, no measurable speedup
2016-04-05 20:54:39 +01:00
Howard Chu
a12f9365f8 Use DUPFIXED for block_heights
Only a small savings...
2016-04-05 20:54:07 +01:00
Howard Chu
38c2277d6f Use DUPFIXED for spent_keys 2016-04-05 20:54:07 +01:00
warptangent
bf769c32ba Add back changes from revert.
m_tx_outputs doesn't need to be changed, as it's no longer dup list.
2016-04-05 20:54:07 +01:00
warptangent
46b991b362 Use MDB_APPEND mode with two tx subdbs
This is possible on those using a tx index as a key.
2016-04-05 20:54:07 +01:00
warptangent
9aadedb1d0 Schema update: tx_indices - consolidate the tx subdbs from 5 to 3 2016-04-05 20:54:06 +01:00
warptangent
a2f518aa01 Schema update: tx_indices - yet less indirection 2016-04-05 20:54:06 +01:00
warptangent
8d12a8df2c Schema update: tx_indices - improve further with less indirection 2016-04-05 20:54:06 +01:00
warptangent
7c013f66e9 Add batch warning for further review 2016-04-05 20:54:06 +01:00
warptangent
ae0854a431 Schema update: tx_indices 2016-04-05 20:54:06 +01:00
Howard Chu
8d252a4214 Consolidated block info 2016-04-05 20:53:59 +01:00