Commit graph

55 commits

Author SHA1 Message Date
NoodleDoodleNoodleDoodleNoodleDoodleNoo
e5d2680094 ** CHANGES ARE EXPERIMENTAL (FOR TESTING ONLY)
Bockchain:
1. Optim: Multi-thread long-hash computation when encountering groups of blocks.
2. Optim: Cache verified txs and return result from cache instead of re-checking whenever possible.
3. Optim: Preload output-keys when encoutering groups of blocks. Sort by amount and global-index before bulk querying database and multi-thread when possible.
4. Optim: Disable double spend check on block verification, double spend is already detected when trying to add blocks.
5. Optim: Multi-thread signature computation whenever possible.
6. Patch: Disable locking (recursive mutex) on called functions from check_tx_inputs which causes slowdowns (only seems to happen on ubuntu/VMs??? Reason: TBD)
7. Optim: Removed looped full-tx hash computation when retrieving transactions from pool (???).
8. Optim: Cache difficulty/timestamps (735 blocks) for next-difficulty calculations so that only 2 db reads per new block is needed when a new block arrives (instead of 1470 reads).

Berkeley-DB:
1. Fix: 32-bit data errors causing wrong output global indices and failure to send blocks to peers (etc).
2. Fix: Unable to pop blocks on reorganize due to transaction errors.
3. Patch: Large number of transaction aborts when running multi-threaded bulk queries.
4. Patch: Insufficient locks error when running full sync.
5. Patch: Incorrect db stats when returning from an immediate exit from "pop block" operation.
6. Optim: Add bulk queries to get output global indices.
7. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3)
8. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key
9. Optim: Added thread-safe buffers used when multi-threading bulk queries.
10. Optim: Added support for nosync/write_nosync options for improved performance (*see --db-sync-mode option for details)
11. Mod: Added checkpoint thread and auto-remove-logs option.
12. *Now usable on 32-bit systems like RPI2.

LMDB:
1. Optim: Added custom comparison for 256-bit key tables (minor speed-up, TBD: get actual effect)
2. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3)
3. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key
4. Optim: Added support for sync/writemap options for improved performance (*see --db-sync-mode option for details)
5. Mod: Auto resize to +1GB instead of multiplier x1.5

ETC:
1. Minor optimizations for slow-hash for ARM (RPI2). Incomplete.
2. Fix: 32-bit saturation bug when computing next difficulty on large blocks.

[PENDING ISSUES]
1. Berkely db has a very slow "pop-block" operation. This is very noticeable on the RPI2 as it sometimes takes > 10 MINUTES to pop a block during reorganization.
   This does not happen very often however, most reorgs seem to take a few seconds but it possibly depends on the number of outputs present. TBD.
2. Berkeley db, possible bug "unable to allocate memory". TBD.

[NEW OPTIONS] (*Currently all enabled for testing purposes)
1. --fast-block-sync arg=[0:1] (default: 1)
	a. 0 = Compute long hash per block (may take a while depending on CPU)
	b. 1 = Skip long-hash and verify blocks based on embedded known good block hashes (faster, minimal CPU dependence)
2. --db-sync-mode arg=[[safe|fast|fastest]:[sync|async]:[nblocks_per_sync]] (default: fastest:async:1000)
	a. safe = fdatasync/fsync (or equivalent) per stored block. Very slow, but safest option to protect against power-out/crash conditions.
	b. fast/fastest = Enables asynchronous fdatasync/fsync (or equivalent). Useful for battery operated devices or STABLE systems with UPS and/or systems with battery backed write cache/solid state cache.
	Fast    - Write meta-data but defer data flush.
	Fastest - Defer meta-data and data flush.
	Sync    - Flush data after nblocks_per_sync and wait.
	Async   - Flush data after nblocks_per_sync but do not wait for the operation to finish.
3. --prep-blocks-threads arg=[n] (default: 4 or system max threads, whichever is lower)
        Max number of threads to use when computing long-hash in groups.
4. --show-time-stats arg=[0:1] (default: 1)
	Show benchmark related time stats.
5. --db-auto-remove-logs arg=[0:1] (default: 1)
	For berkeley-db only. Auto remove logs if enabled.

**Note: lmdb and berkeley-db have changes to the tables and are not compatible with official git head version.
	At the moment, you need a full resync to use this optimized version.

[PERFORMANCE COMPARISON]
**Some figures are approximations only.
Using a baseline machine of an i7-2600K+SSD+(with full pow computation):
1. The optimized lmdb/blockhain core can process blocks up to 585K for ~1.25 hours + download time, so it usually takes 2.5 hours to sync the full chain.
2. The current head with memory can process blocks up to 585K for ~4.2 hours + download time, so it usually takes 5.5 hours to sync the full chain.
3. The current head with lmdb can process blocks up to 585K for ~32 hours + download time and usually takes 36 hours to sync the full chain.

Averate procesing times (with full pow computation):
lmdb-optimized:
1. tx_ave = 2.5 ms / tx
2. block_ave = 5.87 ms / block
memory-official-repo:
1. tx_ave = 8.85 ms / tx
2. block_ave = 19.68 ms / block
lmdb-official-repo (0f4a036437)
1. tx_ave = 47.8 ms / tx
2. block_ave = 64.2 ms / block

**Note: The following data denotes processing times only (does not include p2p download time)
lmdb-optimized processing times (with full pow computation):
1. Desktop,  Quad-core / 8-threads 2600k  (8Mb) - 1.25 hours processing time (--db-sync-mode=fastest:async:1000).
2. Laptop,   Dual-core / 4-threads U4200  (3Mb) - 4.90 hours processing time (--db-sync-mode=fastest:async:1000).
3. Embedded, Quad-core / 4-threads Z3735F (2x1Mb) - 12.0 hours processing time (--db-sync-mode=fastest:async:1000).

lmdb-optimized processing times (with per-block-checkpoint)
1. Desktop,  Quad-core / 8-threads 2600k  (8Mb) - 10 minutes processing time (--db-sync-mode=fastest:async:1000).

berkeley-db optimized processing times (with full pow computation)
1. Desktop, Quad-core / 8-threads 2600k  (8Mb) - 1.8 hours processing time (--db-sync-mode=fastest:async:1000).
2. RPI2. Improved from estimated 3 months(???) into 2.5 days (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000).

berkeley-db optimized processing times (with per-block-checkpoint)
1. RPI2. 12-15 hours (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000).
2015-07-15 23:20:16 -07:00
Javier Smooth
1b19ee040f disable non-working optimized slow_hash on 32 bit intel 2015-04-13 16:38:33 -07:00
Riccardo Spagni
b2b612794d
no newline at end of file 2015-04-05 17:29:54 +02:00
Riccardo Spagni
3ece1584f0
make fallback compatible with newer slow-hash 2015-01-28 11:32:52 +02:00
Riccardo Spagni
9b3673a54c
added portable slow-hash, updated makefile targets, fixed readme 2015-01-27 09:43:15 +02:00
Riccardo Spagni
e70bc3d369
fixed sse2 ifdef 2015-01-27 09:15:14 +02:00
Riccardo Spagni
2b3b70a261
remove aes flag, detect no sse2 2015-01-27 09:05:47 +02:00
Riccardo Spagni
f4b69d553a
year updated in license 2015-01-02 18:52:46 +02:00
Ben Boeckel
f53f04724c cmake: handle private vs. public headers 2014-10-23 16:42:34 -04:00
Ben Boeckel
55ca7d3b34 cmake: refactor common code with libraries 2014-10-23 16:42:34 -04:00
Ben Boeckel
89cff7bddc cmake: put each library into its own directory
This cleans up the CMake code and shows patterns more easily (to be
refactored in the next commit).
2014-10-23 16:42:34 -04:00
David G. Andersen
28dfb4b178 capitalize Keccak everywhere 2014-10-06 17:44:23 -04:00
David G. Andersen
ce2ad65599 typo fix - cryptonight, not cryptonote 2014-10-06 17:42:58 -04:00
David G. Andersen
e4fb913b2d Fix comment start for two functions 2014-10-06 10:15:21 -04:00
David G. Andersen
46f26ffe88 another typo fix 2014-10-05 20:32:49 -04:00
David G. Andersen
ac6bc4859f fix typo 2014-10-05 14:14:59 -04:00
David G. Andersen
d744dd1be5 More documentation 2014-10-05 13:47:13 -04:00
David G. Andersen
4d493f6d4f initial doxygen commenting of the CryptoNight proof-of-work code 2014-10-05 12:33:39 -04:00
Oran Juice
4f693d715c
Merge with origin/master 2014-09-28 15:52:05 +05:30
Oran Juice
6be139b511
Moved mnemonics code to src/mnemonics 2014-09-23 17:04:04 +05:30
Zachary Michaels
aba3497fdd More preprocessor fixes for slow-hash 2014-09-15 22:33:10 +02:00
Zachary Michaels
cf91545734 Correct includes for mingw in slow-hash
This needs testing
2014-09-15 22:33:10 +02:00
Zachary Michaels
2b76e9d8a3 Change Windows include to windows
mingw is case sensitive
2014-09-15 22:33:10 +02:00
fluffypony
ccb2ab2b7b mmap on FreeBSD doesn't have MAP_HUGETLB 2014-09-10 14:01:39 +02:00
fluffypony
110f110181 Include sys/time.h on BSD 2014-09-10 13:58:43 +02:00
fluffypony
a8d043b6dd replace ftime with gettimeofday on FreeBSD because lcompat is stupid 2014-09-10 13:55:39 +02:00
fluffypony
266c639f4f more FreeBSD fixes 2014-09-09 14:49:13 +02:00
fluffypony
e1d31e0a8b malloc.h reference fixed for FreeBSD 2014-09-09 14:07:57 +02:00
fluffypony
6afbdd9754 FreeBSD alloca.h reference fixed 2014-09-09 12:37:19 +02:00
rfree2monero
2ef0aee81d Fix tree-hash cnt n^2. Asserts, comment. Squash2 2014-09-06 21:56:24 +02:00
fluffypony
f8025cada9 fixed OS X malloc.h reference 2014-09-02 12:08:27 +02:00
fluffypony
6fc995fe5d License updated to BSD 3-clause 2014-07-23 15:03:52 +02:00
Zachary Michaels
849c52c7ed Remove unused array in cn_slow_hash 2014-07-01 14:57:25 -04:00
NoodleDoodleNoodleDoodleNoodleDoodleNoo
e12ccdf905 Update slow-hash.c
1. Fix for Mac OSX compilation errors.
2014-06-19 23:49:25 -07:00
NoodleDoodleNoodleDoodleNoodleDoodleNoo
156312f64a Update slow-hash.c
1. Added multiplication support in 32-bit mode
2014-06-16 21:13:05 -07:00
NoodleDoodleNoodleDoodleNoodleDoodleNoo
72643c47da Update slow-hash.c
1. Added huge pages support and optimized scratchpad twiddling. (credits to dga).
2. Added aes-ni key expansion support.
3. Minor speedup to scratchpad initialization/finalization.
2014-06-16 03:35:09 -07:00
Thomas Winget
537b911af1 stupid signed arithmetic. 2014-06-12 19:05:52 -04:00
Thomas Winget
8bc032ed09 more pointer-based derp 2014-06-12 19:05:51 -04:00
Thomas Winget
72c3f36ca4 fixed some pointer- and loop-based derps 2014-06-12 19:05:51 -04:00
Thomas Winget
d22e458c6c builds, but doesn't link. other than that, electrum-style recovery implemented (but not tested\!) 2014-06-12 19:05:51 -04:00
Thomas Winget
8661f9a970 missing '.at' on two map calls. built on some machines, very strange... 2014-06-12 19:05:51 -04:00
Thomas Winget
e89949c766 revert 'dirty hack' change to crypto.h. need a better way for sure. 2014-06-12 19:05:51 -04:00
tom
cd63132502 compile issues, resolve later. commiting because switching machines 2014-06-12 19:05:51 -04:00
Thomas Winget
4f382b3830 most functions prototyped/modified for wallet recovery 2014-06-12 19:05:51 -04:00
mydesktop
3a3a817678 0.8.8update 2014-05-25 13:06:40 -04:00
NoodleDoodleNoodleDoodleNoodleDoodleNoo
44f61c3965 Update slow-hash.c
1. Added AES-NI support for modern processors.
2014-05-21 09:29:34 -07:00
NoodleDoodleNoodleDoodleNoodleDoodleNoo
ab7b25da5e Create aesb.c 2014-05-17 16:25:58 -07:00
NoodleDoodleNoodleDoodleNoodleDoodleNoo
49d55d3c30 Update slow-hash.c
1. Various optimizations for faster hashing performance.
2014-05-17 16:25:00 -07:00
NoodleDoodleNoodleDoodleNoodleDoodleNoo
e4f7b92ff9 Update oaes_lib.c
1. Moved structs oaes_ctx and oaes_key into oeas_lib header.
2014-05-17 16:22:07 -07:00
NoodleDoodleNoodleDoodleNoodleDoodleNoo
98e0326596 Update oaes_lib.h
1. Moved structs oaes_ctx and oaes_key into oeas_lib header.
2014-05-17 16:20:47 -07:00