2019-03-05 21:05:34 +00:00
|
|
|
// Copyright (c) 2014-2019, The Monero Project
|
2015-02-16 12:20:55 +00:00
|
|
|
//
|
|
|
|
// All rights reserved.
|
|
|
|
//
|
|
|
|
// Redistribution and use in source and binary forms, with or without modification, are
|
|
|
|
// permitted provided that the following conditions are met:
|
|
|
|
//
|
|
|
|
// 1. Redistributions of source code must retain the above copyright notice, this list of
|
|
|
|
// conditions and the following disclaimer.
|
|
|
|
//
|
|
|
|
// 2. Redistributions in binary form must reproduce the above copyright notice, this list
|
|
|
|
// of conditions and the following disclaimer in the documentation and/or other
|
|
|
|
// materials provided with the distribution.
|
|
|
|
//
|
|
|
|
// 3. Neither the name of the copyright holder nor the names of its contributors may be
|
|
|
|
// used to endorse or promote products derived from this software without specific
|
|
|
|
// prior written permission.
|
|
|
|
//
|
|
|
|
// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY
|
|
|
|
// EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
|
|
|
|
// MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL
|
|
|
|
// THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
|
|
|
|
// SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
|
|
|
|
// PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
|
|
|
|
// INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,
|
|
|
|
// STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF
|
|
|
|
// THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
|
|
|
|
|
2015-05-08 20:46:26 +00:00
|
|
|
#include "bootstrap_file.h"
|
2015-10-16 18:45:35 +00:00
|
|
|
#include "blocksdat_file.h"
|
2015-02-16 12:20:55 +00:00
|
|
|
#include "common/command_line.h"
|
2016-05-26 20:48:18 +00:00
|
|
|
#include "cryptonote_core/tx_pool.h"
|
2017-10-28 15:06:43 +00:00
|
|
|
#include "cryptonote_core/cryptonote_core.h"
|
2016-03-05 14:33:04 +00:00
|
|
|
#include "blockchain_db/blockchain_db.h"
|
2015-02-16 12:20:55 +00:00
|
|
|
#include "version.h"
|
|
|
|
|
Change logging to easylogging++
This replaces the epee and data_loggers logging systems with
a single one, and also adds filename:line and explicit severity
levels. Categories may be defined, and logging severity set
by category (or set of categories). epee style 0-4 log level
maps to a sensible severity configuration. Log files now also
rotate when reaching 100 MB.
To select which logs to output, use the MONERO_LOGS environment
variable, with a comma separated list of categories (globs are
supported), with their requested severity level after a colon.
If a log matches more than one such setting, the last one in
the configuration string applies. A few examples:
This one is (mostly) silent, only outputting fatal errors:
MONERO_LOGS=*:FATAL
This one is very verbose:
MONERO_LOGS=*:TRACE
This one is totally silent (logwise):
MONERO_LOGS=""
This one outputs all errors and warnings, except for the
"verify" category, which prints just fatal errors (the verify
category is used for logs about incoming transactions and
blocks, and it is expected that some/many will fail to verify,
hence we don't want the spam):
MONERO_LOGS=*:WARNING,verify:FATAL
Log levels are, in decreasing order of priority:
FATAL, ERROR, WARNING, INFO, DEBUG, TRACE
Subcategories may be added using prefixes and globs. This
example will output net.p2p logs at the TRACE level, but all
other net* logs only at INFO:
MONERO_LOGS=*:ERROR,net*:INFO,net.p2p:TRACE
Logs which are intended for the user (which Monero was using
a lot through epee, but really isn't a nice way to go things)
should use the "global" category. There are a few helper macros
for using this category, eg: MGINFO("this shows up by default")
or MGINFO_RED("this is red"), to try to keep a similar look
and feel for now.
Existing epee log macros still exist, and map to the new log
levels, but since they're used as a "user facing" UI element
as much as a logging system, they often don't map well to log
severities (ie, a log level 0 log may be an error, or may be
something we want the user to see, such as an important info).
In those cases, I tried to use the new macros. In other cases,
I left the existing macros in. When modifying logs, it is
probably best to switch to the new macros with explicit levels.
The --log-level options and set_log commands now also accept
category settings, in addition to the epee style log levels.
2017-01-01 16:34:23 +00:00
|
|
|
#undef MONERO_DEFAULT_LOG_CATEGORY
|
|
|
|
#define MONERO_DEFAULT_LOG_CATEGORY "bcutil"
|
|
|
|
|
2015-02-16 12:20:55 +00:00
|
|
|
namespace po = boost::program_options;
|
Change logging to easylogging++
This replaces the epee and data_loggers logging systems with
a single one, and also adds filename:line and explicit severity
levels. Categories may be defined, and logging severity set
by category (or set of categories). epee style 0-4 log level
maps to a sensible severity configuration. Log files now also
rotate when reaching 100 MB.
To select which logs to output, use the MONERO_LOGS environment
variable, with a comma separated list of categories (globs are
supported), with their requested severity level after a colon.
If a log matches more than one such setting, the last one in
the configuration string applies. A few examples:
This one is (mostly) silent, only outputting fatal errors:
MONERO_LOGS=*:FATAL
This one is very verbose:
MONERO_LOGS=*:TRACE
This one is totally silent (logwise):
MONERO_LOGS=""
This one outputs all errors and warnings, except for the
"verify" category, which prints just fatal errors (the verify
category is used for logs about incoming transactions and
blocks, and it is expected that some/many will fail to verify,
hence we don't want the spam):
MONERO_LOGS=*:WARNING,verify:FATAL
Log levels are, in decreasing order of priority:
FATAL, ERROR, WARNING, INFO, DEBUG, TRACE
Subcategories may be added using prefixes and globs. This
example will output net.p2p logs at the TRACE level, but all
other net* logs only at INFO:
MONERO_LOGS=*:ERROR,net*:INFO,net.p2p:TRACE
Logs which are intended for the user (which Monero was using
a lot through epee, but really isn't a nice way to go things)
should use the "global" category. There are a few helper macros
for using this category, eg: MGINFO("this shows up by default")
or MGINFO_RED("this is red"), to try to keep a similar look
and feel for now.
Existing epee log macros still exist, and map to the new log
levels, but since they're used as a "user facing" UI element
as much as a logging system, they often don't map well to log
severities (ie, a log level 0 log may be an error, or may be
something we want the user to see, such as an important info).
In those cases, I tried to use the new macros. In other cases,
I left the existing macros in. When modifying logs, it is
probably best to switch to the new macros with explicit levels.
The --log-level options and set_log commands now also accept
category settings, in addition to the epee style log levels.
2017-01-01 16:34:23 +00:00
|
|
|
using namespace epee;
|
2015-02-16 12:20:55 +00:00
|
|
|
|
|
|
|
int main(int argc, char* argv[])
|
|
|
|
{
|
2017-03-05 19:24:47 +00:00
|
|
|
TRY_ENTRY();
|
|
|
|
|
2017-02-12 19:36:52 +00:00
|
|
|
epee::string_tools::set_module_name_and_folder(argv[0]);
|
|
|
|
|
2015-02-16 12:20:55 +00:00
|
|
|
uint32_t log_level = 0;
|
2015-05-16 00:14:36 +00:00
|
|
|
uint64_t block_stop = 0;
|
2015-10-16 18:45:35 +00:00
|
|
|
bool blocks_dat = false;
|
2015-02-16 12:20:55 +00:00
|
|
|
|
2017-10-03 09:12:57 +00:00
|
|
|
tools::on_startup();
|
2015-11-21 10:13:10 +00:00
|
|
|
|
2015-08-14 19:45:12 +00:00
|
|
|
boost::filesystem::path output_file_path;
|
2015-02-16 12:20:55 +00:00
|
|
|
|
|
|
|
po::options_description desc_cmd_only("Command line options");
|
|
|
|
po::options_description desc_cmd_sett("Command line options and settings options");
|
2015-08-14 19:45:12 +00:00
|
|
|
const command_line::arg_descriptor<std::string> arg_output_file = {"output-file", "Specify output file", "", true};
|
2017-03-05 19:45:00 +00:00
|
|
|
const command_line::arg_descriptor<std::string> arg_log_level = {"log-level", "0-4 or categories", ""};
|
2015-05-16 00:14:36 +00:00
|
|
|
const command_line::arg_descriptor<uint64_t> arg_block_stop = {"block-stop", "Stop at block number", block_stop};
|
2015-10-16 18:45:35 +00:00
|
|
|
const command_line::arg_descriptor<bool> arg_blocks_dat = {"blocksdat", "Output in blocks.dat format", blocks_dat};
|
2015-02-16 12:20:55 +00:00
|
|
|
|
|
|
|
|
2018-01-21 15:29:55 +00:00
|
|
|
command_line::add_arg(desc_cmd_sett, cryptonote::arg_data_dir);
|
2015-08-14 19:45:12 +00:00
|
|
|
command_line::add_arg(desc_cmd_sett, arg_output_file);
|
2017-10-28 15:06:43 +00:00
|
|
|
command_line::add_arg(desc_cmd_sett, cryptonote::arg_testnet_on);
|
2018-02-16 11:04:04 +00:00
|
|
|
command_line::add_arg(desc_cmd_sett, cryptonote::arg_stagenet_on);
|
2015-05-16 00:14:36 +00:00
|
|
|
command_line::add_arg(desc_cmd_sett, arg_log_level);
|
|
|
|
command_line::add_arg(desc_cmd_sett, arg_block_stop);
|
2015-10-16 18:45:35 +00:00
|
|
|
command_line::add_arg(desc_cmd_sett, arg_blocks_dat);
|
2015-02-16 12:20:55 +00:00
|
|
|
|
|
|
|
command_line::add_arg(desc_cmd_only, command_line::arg_help);
|
|
|
|
|
|
|
|
po::options_description desc_options("Allowed options");
|
|
|
|
desc_options.add(desc_cmd_only).add(desc_cmd_sett);
|
|
|
|
|
|
|
|
po::variables_map vm;
|
|
|
|
bool r = command_line::handle_error_helper(desc_options, [&]()
|
|
|
|
{
|
|
|
|
po::store(po::parse_command_line(argc, argv, desc_options), vm);
|
|
|
|
po::notify(vm);
|
|
|
|
return true;
|
|
|
|
});
|
|
|
|
if (! r)
|
|
|
|
return 1;
|
|
|
|
|
|
|
|
if (command_line::get_arg(vm, command_line::arg_help))
|
|
|
|
{
|
2015-12-31 21:12:13 +00:00
|
|
|
std::cout << "Monero '" << MONERO_RELEASE_NAME << "' (v" << MONERO_VERSION_FULL << ")" << ENDL << ENDL;
|
2015-02-16 12:20:55 +00:00
|
|
|
std::cout << desc_options << std::endl;
|
|
|
|
return 1;
|
|
|
|
}
|
|
|
|
|
2017-03-05 19:45:00 +00:00
|
|
|
mlog_configure(mlog_get_default_log_path("monero-blockchain-export.log"), true);
|
2017-10-06 09:55:36 +00:00
|
|
|
if (!command_line::is_arg_defaulted(vm, arg_log_level))
|
2017-03-05 19:45:00 +00:00
|
|
|
mlog_set_log(command_line::get_arg(vm, arg_log_level).c_str());
|
|
|
|
else
|
|
|
|
mlog_set_log(std::string(std::to_string(log_level) + ",bcutil:INFO").c_str());
|
2015-05-16 00:14:36 +00:00
|
|
|
block_stop = command_line::get_arg(vm, arg_block_stop);
|
2015-02-16 12:20:55 +00:00
|
|
|
|
|
|
|
LOG_PRINT_L0("Starting...");
|
|
|
|
|
2017-10-28 15:06:43 +00:00
|
|
|
bool opt_testnet = command_line::get_arg(vm, cryptonote::arg_testnet_on);
|
2018-02-16 11:04:04 +00:00
|
|
|
bool opt_stagenet = command_line::get_arg(vm, cryptonote::arg_stagenet_on);
|
|
|
|
if (opt_testnet && opt_stagenet)
|
|
|
|
{
|
|
|
|
std::cerr << "Can't specify more than one of --testnet and --stagenet" << std::endl;
|
|
|
|
return 1;
|
|
|
|
}
|
2015-10-16 18:45:35 +00:00
|
|
|
bool opt_blocks_dat = command_line::get_arg(vm, arg_blocks_dat);
|
2015-02-16 12:20:55 +00:00
|
|
|
|
|
|
|
std::string m_config_folder;
|
|
|
|
|
2018-01-21 15:29:55 +00:00
|
|
|
m_config_folder = command_line::get_arg(vm, cryptonote::arg_data_dir);
|
2015-08-14 19:45:12 +00:00
|
|
|
|
|
|
|
if (command_line::has_arg(vm, arg_output_file))
|
|
|
|
output_file_path = boost::filesystem::path(command_line::get_arg(vm, arg_output_file));
|
|
|
|
else
|
|
|
|
output_file_path = boost::filesystem::path(m_config_folder) / "export" / BLOCKCHAIN_RAW;
|
|
|
|
LOG_PRINT_L0("Export output file: " << output_file_path.string());
|
2015-02-16 12:20:55 +00:00
|
|
|
|
|
|
|
// If we wanted to use the memory pool, we would set up a fake_core.
|
|
|
|
|
2015-03-16 18:57:26 +00:00
|
|
|
// Use Blockchain instead of lower-level BlockchainDB for two reasons:
|
|
|
|
// 1. Blockchain has the init() method for easy setup
|
|
|
|
// 2. exporter needs to use get_current_blockchain_height(), get_block_id_by_height(), get_block_by_hash()
|
|
|
|
//
|
|
|
|
// cannot match blockchain_storage setup above with just one line,
|
|
|
|
// e.g.
|
|
|
|
// Blockchain* core_storage = new Blockchain(NULL);
|
|
|
|
// because unlike blockchain_storage constructor, which takes a pointer to
|
|
|
|
// tx_memory_pool, Blockchain's constructor takes tx_memory_pool object.
|
|
|
|
LOG_PRINT_L0("Initializing source blockchain (BlockchainDB)");
|
|
|
|
Blockchain* core_storage = NULL;
|
|
|
|
tx_memory_pool m_mempool(*core_storage);
|
|
|
|
core_storage = new Blockchain(m_mempool);
|
2015-05-01 07:46:23 +00:00
|
|
|
|
2019-06-26 14:39:41 +00:00
|
|
|
BlockchainDB* db = new_db();
|
2017-08-19 14:27:13 +00:00
|
|
|
if (db == NULL)
|
2016-03-05 14:37:47 +00:00
|
|
|
{
|
2019-06-26 14:39:41 +00:00
|
|
|
LOG_ERROR("Failed to initialize a database");
|
|
|
|
throw std::runtime_error("Failed to initialize a database");
|
2016-03-05 14:37:47 +00:00
|
|
|
}
|
2019-06-26 14:39:41 +00:00
|
|
|
LOG_PRINT_L0("database: LMDB");
|
2016-03-05 14:37:47 +00:00
|
|
|
|
2015-05-01 07:46:23 +00:00
|
|
|
boost::filesystem::path folder(m_config_folder);
|
|
|
|
folder /= db->get_db_name();
|
|
|
|
const std::string filename = folder.string();
|
2015-05-16 08:29:02 +00:00
|
|
|
|
|
|
|
LOG_PRINT_L0("Loading blockchain from folder " << filename << " ...");
|
2015-05-01 07:46:23 +00:00
|
|
|
try
|
|
|
|
{
|
2017-08-19 14:27:13 +00:00
|
|
|
db->open(filename, DBF_RDONLY);
|
2015-05-01 07:46:23 +00:00
|
|
|
}
|
|
|
|
catch (const std::exception& e)
|
|
|
|
{
|
|
|
|
LOG_PRINT_L0("Error opening database: " << e.what());
|
2015-10-27 16:15:41 +00:00
|
|
|
return 1;
|
2015-05-01 07:46:23 +00:00
|
|
|
}
|
2018-02-16 11:04:04 +00:00
|
|
|
r = core_storage->init(db, opt_testnet ? cryptonote::TESTNET : opt_stagenet ? cryptonote::STAGENET : cryptonote::MAINNET);
|
2015-02-16 12:20:55 +00:00
|
|
|
|
2019-05-24 14:11:56 +00:00
|
|
|
if (core_storage->get_blockchain_pruning_seed() && !opt_blocks_dat)
|
Pruning
The blockchain prunes seven eighths of prunable tx data.
This saves about two thirds of the blockchain size, while
keeping the node useful as a sync source for an eighth
of the blockchain.
No other data is currently pruned.
There are three ways to prune a blockchain:
- run monerod with --prune-blockchain
- run "prune_blockchain" in the monerod console
- run the monero-blockchain-prune utility
The first two will prune in place. Due to how LMDB works, this
will not reduce the blockchain size on disk. Instead, it will
mark parts of the file as free, so that future data will use
that free space, causing the file to not grow until free space
grows scarce.
The third way will create a second database, a pruned copy of
the original one. Since this is a new file, this one will be
smaller than the original one.
Once the database is pruned, it will stay pruned as it syncs.
That is, there is no need to use --prune-blockchain again, etc.
2018-04-29 22:30:51 +00:00
|
|
|
{
|
|
|
|
LOG_PRINT_L0("Blockchain is pruned, cannot export");
|
|
|
|
return 1;
|
|
|
|
}
|
|
|
|
|
Fix compiler warnings with Clang 6.0.0.
monero/src/cryptonote_protocol/block_queue.cpp:208:44: error:
suggest braces around initialization of subobject [-Werror,-Wmissing-braces]
static const boost::uuids::uuid uuid0 = {0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0};
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
{ }
monero/src/wallet/wallet_rpc_server.cpp:1895:43: error:
lambda capture 'wal' is not used [-Werror,-Wunused-lambda-capture]
tools::signal_handler::install([&wrpc, &wal](int) {
^
monero/src/cryptonote_protocol/cryptonote_protocol_handler.inl:1616:40: error:
lambda capture 'arg' is not used [-Werror,-Wunused-lambda-capture]
m_p2p->for_each_connection([this, &arg, &fluffy_arg, &exclude_context, &fullConnections...
^
monero/src/cryptonote_protocol/cryptonote_protocol_handler.inl:1616:46: error:
lambda capture 'fluffy_arg' is not used [-Werror,-Wunused-lambda-capture]
m_p2p->for_each_connection([this, &arg, &fluffy_arg, &exclude_context, &fullConnections...
^
monero/src/blockchain_utilities/blockchain_export.cpp:181:3: error:
bool literal returned from 'main' [-Werror,-Wmain]
CHECK_AND_ASSERT_MES(r, false, "Failed to initialize source blockchain storage");
^ ~~~~~
monero/contrib/epee/include/misc_log_ex.h:180:97: note:
expanded from macro 'CHECK_AND_ASSERT_MES'
...fail_ret_val, message) do{if(!(expr)) {LOG_ERROR(message); return fail_ret_val;};}while(0)
^ ~~~~~~~~~~~~
monero/src/blockchain_utilities/blockchain_export.cpp:195:3: error:
bool literal returned from 'main' [-Werror,-Wmain]
CHECK_AND_ASSERT_MES(r, false, "Failed to export blockchain raw data");
^ ~~~~~
monero/contrib/epee/include/misc_log_ex.h:180:97: note:
expanded from macro 'CHECK_AND_ASSERT_MES'
...fail_ret_val, message) do{if(!(expr)) {LOG_ERROR(message); return fail_ret_val;};}while(0)
^ ~~~~~~~~~~~~
2017-10-07 16:21:04 +00:00
|
|
|
CHECK_AND_ASSERT_MES(r, 1, "Failed to initialize source blockchain storage");
|
2015-02-16 12:20:55 +00:00
|
|
|
LOG_PRINT_L0("Source blockchain storage initialized OK");
|
|
|
|
LOG_PRINT_L0("Exporting blockchain raw data...");
|
|
|
|
|
2015-10-16 18:45:35 +00:00
|
|
|
if (opt_blocks_dat)
|
|
|
|
{
|
|
|
|
BlocksdatFile blocksdat;
|
|
|
|
r = blocksdat.store_blockchain_raw(core_storage, NULL, output_file_path, block_stop);
|
|
|
|
}
|
|
|
|
else
|
|
|
|
{
|
|
|
|
BootstrapFile bootstrap;
|
|
|
|
r = bootstrap.store_blockchain_raw(core_storage, NULL, output_file_path, block_stop);
|
|
|
|
}
|
Fix compiler warnings with Clang 6.0.0.
monero/src/cryptonote_protocol/block_queue.cpp:208:44: error:
suggest braces around initialization of subobject [-Werror,-Wmissing-braces]
static const boost::uuids::uuid uuid0 = {0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0};
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
{ }
monero/src/wallet/wallet_rpc_server.cpp:1895:43: error:
lambda capture 'wal' is not used [-Werror,-Wunused-lambda-capture]
tools::signal_handler::install([&wrpc, &wal](int) {
^
monero/src/cryptonote_protocol/cryptonote_protocol_handler.inl:1616:40: error:
lambda capture 'arg' is not used [-Werror,-Wunused-lambda-capture]
m_p2p->for_each_connection([this, &arg, &fluffy_arg, &exclude_context, &fullConnections...
^
monero/src/cryptonote_protocol/cryptonote_protocol_handler.inl:1616:46: error:
lambda capture 'fluffy_arg' is not used [-Werror,-Wunused-lambda-capture]
m_p2p->for_each_connection([this, &arg, &fluffy_arg, &exclude_context, &fullConnections...
^
monero/src/blockchain_utilities/blockchain_export.cpp:181:3: error:
bool literal returned from 'main' [-Werror,-Wmain]
CHECK_AND_ASSERT_MES(r, false, "Failed to initialize source blockchain storage");
^ ~~~~~
monero/contrib/epee/include/misc_log_ex.h:180:97: note:
expanded from macro 'CHECK_AND_ASSERT_MES'
...fail_ret_val, message) do{if(!(expr)) {LOG_ERROR(message); return fail_ret_val;};}while(0)
^ ~~~~~~~~~~~~
monero/src/blockchain_utilities/blockchain_export.cpp:195:3: error:
bool literal returned from 'main' [-Werror,-Wmain]
CHECK_AND_ASSERT_MES(r, false, "Failed to export blockchain raw data");
^ ~~~~~
monero/contrib/epee/include/misc_log_ex.h:180:97: note:
expanded from macro 'CHECK_AND_ASSERT_MES'
...fail_ret_val, message) do{if(!(expr)) {LOG_ERROR(message); return fail_ret_val;};}while(0)
^ ~~~~~~~~~~~~
2017-10-07 16:21:04 +00:00
|
|
|
CHECK_AND_ASSERT_MES(r, 1, "Failed to export blockchain raw data");
|
2015-02-16 12:20:55 +00:00
|
|
|
LOG_PRINT_L0("Blockchain raw data exported OK");
|
2017-03-05 19:24:47 +00:00
|
|
|
return 0;
|
|
|
|
|
|
|
|
CATCH_ENTRY("Export error", 1);
|
2015-02-16 12:20:55 +00:00
|
|
|
}
|