ReScrap/ScrapHacks
ReScrap/ScrapHacks
2
1
Fork 1
Code Issues 5 Pull requests Projects 3 Releases Wiki Activity

Add web-based .packed explorer, updated parser and ghidra untility script

Browse source
This commit is contained in:
Daniel S. 2023-05-07 21:29:21 +02:00
parent 8e0df74541
commit 58407ecc9f
35 changed files with 3897 additions and 353 deletions
Download patch file Download diff file Expand all files Collapse all files

24
scrapper_web/.gitignore vendored Normal file
View file

@ -0,0 +1,24 @@
# Logs
logs
*.log
npm-debug.log*
yarn-debug.log*
yarn-error.log*
pnpm-debug.log*
lerna-debug.log*
node_modules
dist
dist-ssr
*.local
# Editor directories and files
.vscode/*
!.vscode/extensions.json
.idea
.DS_Store
*.suo
*.ntvs*
*.njsproj
*.sln
*.sw?

3
scrapper_web/.vscode/extensions.json vendored Normal file
View file

@ -0,0 +1,3 @@
{
"recommendations": ["svelte.svelte-vscode"]
}

47
scrapper_web/README.md Normal file
View file

@ -0,0 +1,47 @@
# Svelte + Vite
This template should help get you started developing with Svelte in Vite.
## Recommended IDE Setup
[VS Code](https://code.visualstudio.com/) + [Svelte](https://marketplace.visualstudio.com/items?itemName=svelte.svelte-vscode).
## Need an official Svelte framework?
Check out [SvelteKit](https://github.com/sveltejs/kit#readme), which is also powered by Vite. Deploy anywhere with its serverless-first approach and adapt to various platforms, with out of the box support for TypeScript, SCSS, and Less, and easily-added support for mdsvex, GraphQL, PostCSS, Tailwind CSS, and more.
## Technical considerations
**Why use this over SvelteKit?**
- It brings its own routing solution which might not be preferable for some users.
- It is first and foremost a framework that just happens to use Vite under the hood, not a Vite app.
This template contains as little as possible to get started with Vite + Svelte, while taking into account the developer experience with regards to HMR and intellisense. It demonstrates capabilities on par with the other `create-vite` templates and is a good starting point for beginners dipping their toes into a Vite + Svelte project.
Should you later need the extended capabilities and extensibility provided by SvelteKit, the template has been structured similarly to SvelteKit so that it is easy to migrate.
**Why `global.d.ts` instead of `compilerOptions.types` inside `jsconfig.json` or `tsconfig.json`?**
Setting `compilerOptions.types` shuts out all other types not explicitly listed in the configuration. Using triple-slash references keeps the default TypeScript setting of accepting type information from the entire workspace, while also adding `svelte` and `vite/client` type information.
**Why include `.vscode/extensions.json`?**
Other templates indirectly recommend extensions via the README, but this file allows VS Code to prompt the user to install the recommended extension upon opening the project.
**Why enable `checkJs` in the JS template?**
It is likely that most cases of changing variable types in runtime are likely to be accidental, rather than deliberate. This provides advanced typechecking out of the box. Should you like to take advantage of the dynamically-typed nature of JavaScript, it is trivial to change the configuration.
**Why is HMR not preserving my local component state?**
HMR state preservation comes with a number of gotchas! It has been disabled by default in both `svelte-hmr` and `@sveltejs/vite-plugin-svelte` due to its often surprising behavior. You can read the details [here](https://github.com/rixo/svelte-hmr#svelte-hmr).
If you have state that's important to retain within a component, consider creating an external store which would not be replaced by HMR.
```js
// store.js
// An extremely simple external store
import { writable } from 'svelte/store'
export default writable(0)
```

13
scrapper_web/index.html Normal file
View file

@ -0,0 +1,13 @@
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8" />
<link rel="icon" type="image/svg+xml" href="/vite.svg" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>Vite + Svelte</title>
</head>
<body>
<div id="app"></div>
<script type="module" src="/src/main.js"></script>
</body>
</html>

33
scrapper_web/jsconfig.json Normal file
View file

@ -0,0 +1,33 @@
{
"compilerOptions": {
"moduleResolution": "Node",
"target": "ESNext",
"module": "ESNext",
/**
* svelte-preprocess cannot figure out whether you have
* a value or a type, so tell TypeScript to enforce using
* `import type` instead of `import` for Types.
*/
"importsNotUsedAsValues": "error",
"isolatedModules": true,
"resolveJsonModule": true,
/**
* To have warnings / errors of the Svelte compiler at the
* correct position, enable source maps by default.
*/
"sourceMap": true,
"esModuleInterop": true,
"skipLibCheck": true,
"forceConsistentCasingInFileNames": true,
/**
* Typecheck JS in `.svelte` and `.js` files by default.
* Disable this if you'd like to use dynamic types.
*/
"checkJs": false
},
/**
* Use global.d.ts instead of compilerOptions.types
* to avoid limiting type declarations.
*/
"include": ["src/**/*.d.ts", "src/**/*.js", "src/**/*.svelte"]
}

26
scrapper_web/package.json Normal file
View file

@ -0,0 +1,26 @@
{
"name": "scrapper_web",
"private": true,
"version": "0.0.0",
"type": "module",
"scripts": {
"dev": "vite",
"build": "wasm-pack build ./scrapper -t web && vite build",
"preview": "vite preview"
},
"devDependencies": {
"@sveltejs/vite-plugin-svelte": "^2.0.2",
"@tailwindcss/forms": "^0.5.3",
"autoprefixer": "^10.4.13",
"cssnano": "^5.1.14",
"cssnano-preset-advanced": "^5.3.9",
"daisyui": "^2.50.0",
"filedrop-svelte": "^0.1.2",
"postcss": "^8.4.21",
"svelte": "^3.55.1",
"svelte-preprocess": "^5.0.1",
"tailwindcss": "^3.2.4",
"vite": "^4.1.0",
"vite-plugin-wasm-pack": "^0.1.12"
}
}

1777
scrapper_web/pnpm-lock.yaml generated Normal file
View file

File diff suppressed because it is too large Load diff

11
scrapper_web/postcss.config.cjs Normal file
View file

@ -0,0 +1,11 @@
let cssnano_plugin = {};
if (process.env.NODE_ENV === "production") {
cssnano_plugin = { cssnano: { preset: "advanced" } };
}
module.exports = {
plugins: {
tailwindcss: {},
autoprefixer: {},
...cssnano_plugin,
},
};

1
scrapper_web/public/vite.svg Normal file
View file

@ -0,0 +1 @@
<svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" class="iconify iconify--logos" width="31.88" height="32" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 257"><defs><linearGradient id="IconifyId1813088fe1fbc01fb466" x1="-.828%" x2="57.636%" y1="7.652%" y2="78.411%"><stop offset="0%" stop-color="#41D1FF"></stop><stop offset="100%" stop-color="#BD34FE"></stop></linearGradient><linearGradient id="IconifyId1813088fe1fbc01fb467" x1="43.376%" x2="50.316%" y1="2.242%" y2="89.03%"><stop offset="0%" stop-color="#FFEA83"></stop><stop offset="8.333%" stop-color="#FFDD35"></stop><stop offset="100%" stop-color="#FFA800"></stop></linearGradient></defs><path fill="url(#IconifyId1813088fe1fbc01fb466)" d="M255.153 37.938L134.897 252.976c-2.483 4.44-8.862 4.466-11.382.048L.875 37.958c-2.746-4.814 1.371-10.646 6.827-9.67l120.385 21.517a6.537 6.537 0 0 0 2.322-.004l117.867-21.483c5.438-.991 9.574 4.796 6.877 9.62Z"></path><path fill="url(#IconifyId1813088fe1fbc01fb467)" d="M185.432.063L96.44 17.501a3.268 3.268 0 0 0-2.634 3.014l-5.474 92.456a3.268 3.268 0 0 0 3.997 3.378l24.777-5.718c2.318-.535 4.413 1.507 3.936 3.838l-7.361 36.047c-.495 2.426 1.782 4.5 4.151 3.78l15.304-4.649c2.372-.72 4.652 1.36 4.15 3.788l-11.698 56.621c-.732 3.542 3.979 5.473 5.943 2.437l1.313-2.028l72.516-144.72c1.215-2.423-.88-5.186-3.54-4.672l-25.505 4.922c-2.396.462-4.435-1.77-3.759-4.114l16.646-57.705c.677-2.35-1.37-4.583-3.769-4.113Z"></path></svg>
Side by side

After

Width:  |  Height:  |  Size: 1.5 KiB

14
scrapper_web/scrapper/.gitignore vendored Normal file
View file

@ -0,0 +1,14 @@
# Generated by Cargo
# will have compiled files and executables
debug/
target/
# Remove Cargo.lock from gitignore if creating an executable, leave it for libraries
# More information here https://doc.rust-lang.org/cargo/guide/cargo-toml-vs-cargo-lock.html
Cargo.lock
# These are backup files generated by rustfmt
**/*.rs.bk
# MSVC Windows builds of rustc generate these, which store debugging information
*.pdb

31
scrapper_web/scrapper/Cargo.toml Normal file
View file

@ -0,0 +1,31 @@
[package]
name = "scrapper"
version = "0.1.0"
authors = []
edition = "2021"
[lib]
crate-type = ["cdylib", "rlib"]
[profile.release]
lto = true
# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html
[dependencies]
aes = "0.8.2"
anyhow = "1.0.69"
binrw = "0.11.1"
cbc = "0.1.2"
console_error_panic_hook = "0.1.7"
derivative = "2.2.0"
js-sys = "0.3.61"
pelite = "0.10.0"
serde = { version = "1.0.152", features = ["derive"] }
serde-wasm-bindgen = "0.4.5"
wasm-bindgen = "0.2.83"
wasm-bindgen-file-reader = "1.0.0"
web-sys = { version = "0.3.61", features = ["File", "BlobPropertyBag", "Blob", "Url"] }
[package.metadata.wasm-pack.profile.release]
wasm-opt = ["-O4"]

23
scrapper_web/scrapper/README.md Normal file
View file

@ -0,0 +1,23 @@
# scrapper
## Usage
[rsw-rs doc](https://github.com/lencx/rsw-rs)
```bash
# install rsw
cargo install rsw
# --- help ---
# rsw help
rsw -h
# new help
rsw new -h
# --- usage ---
# dev
rsw watch
# production
rsw build
```

155
scrapper_web/scrapper/src/lib.rs Normal file
View file

@ -0,0 +1,155 @@
use binrw::{binread, BinReaderExt};
use serde::Serialize;
use std::collections::BTreeMap;
use std::io::{Read, Seek, SeekFrom};
use wasm_bindgen::prelude::*;
use wasm_bindgen_file_reader::WebSysFile;
use web_sys::{Blob, File};
type JsResult<T> = Result<T,JsValue>;
#[binread]
#[derive(Serialize, Debug)]
struct ScrapFile {
#[br(temp)]
name_len: u32,
#[br(count = name_len)]
#[br(map = |s: Vec<u8>| String::from_utf8_lossy(&s).to_string())]
path: String,
size: u32,
offset: u32,
}
#[binread]
#[br(magic = b"BFPK", little)]
#[derive(Serialize, Debug)]
struct PackedHeader {
version: u32,
#[br(temp)]
num_files: u32,
#[br(count= num_files)]
files: Vec<ScrapFile>,
}
#[derive(Serialize, Debug)]
#[serde(tag = "type", rename_all = "snake_case")]
enum DirectoryTree {
File {
size: u32,
offset: u32,
file_index: u8,
},
Directory {
entries: BTreeMap<String, DirectoryTree>,
},
}
#[wasm_bindgen(inspectable)]
pub struct MultiPack {
files: Vec<(String,WebSysFile)>,
tree: DirectoryTree,
}
fn blob_url(buffer: &[u8]) -> JsResult<String> {
let uint8arr =
js_sys::Uint8Array::new(&unsafe { js_sys::Uint8Array::view(buffer) }.into());
let array = js_sys::Array::new();
array.push(&uint8arr.buffer());
let blob = Blob::new_with_u8_array_sequence_and_options(
&array,
web_sys::BlobPropertyBag::new().type_("application/octet-stream"),
)
.unwrap();
web_sys::Url::create_object_url_with_blob(&blob)
}
#[wasm_bindgen]
impl MultiPack {
#[wasm_bindgen(constructor)]
pub fn parse(files: Vec<File>) -> Self {
let mut tree = DirectoryTree::default();
let mut web_files = vec![];
for (file_index, file) in files.into_iter().enumerate() {
let file_name = file.name();
let mut fh = WebSysFile::new(file);
let header = fh.read_le::<PackedHeader>().unwrap();
tree.merge(&header.files, file_index.try_into().unwrap());
web_files.push((file_name,fh));
}
Self {
tree,
files: web_files,
}
}
#[wasm_bindgen]
pub fn tree(&self) -> JsValue {
serde_wasm_bindgen::to_value(&self.tree).unwrap()
}
#[wasm_bindgen]
pub fn download(
&mut self,
file_index: u8,
offset: u32,
size: u32,
) -> Result<JsValue, JsValue> {
let Some((_,file)) = self.files.get_mut(file_index as usize) else {
return Err("File not found".into());
};
let mut buffer = vec![0u8; size as usize];
file.seek(SeekFrom::Start(offset as u64))
.map_err(|e| format!("Failed to seek file: {e}"))?;
file.read(&mut buffer)
.map_err(|e| format!("Failed to read from file: {e}"))?;
Ok(blob_url(&buffer)?.into())
}
}
impl Default for DirectoryTree {
fn default() -> Self {
Self::Directory {
entries: Default::default(),
}
}
}
impl DirectoryTree {
fn add_child(&mut self, name: &str, node: Self) -> &mut Self {
match self {
Self::File { .. } => panic!("Can't add child to file!"),
Self::Directory {
entries
} => entries.entry(name.to_owned()).or_insert(node),
}
}
fn merge(&mut self, files: &[ScrapFile], file_index: u8) {
for file in files {
let mut folder = &mut *self;
let path: Vec<_> = file.path.split('/').collect();
if let Some((filename, path)) = path.as_slice().split_last() {
for part in path {
let DirectoryTree::Directory { entries } = folder else {
unreachable!();
};
folder = entries.entry(part.to_string()).or_default();
}
folder.add_child(
filename,
DirectoryTree::File {
size: file.size,
offset: file.offset,
file_index,
},
);
}
}
}
}
#[wasm_bindgen(start)]
pub fn main() -> Result<(), JsValue> {
console_error_panic_hook::set_once();
Ok(())
}

13
scrapper_web/src/App.svelte Normal file
View file

@ -0,0 +1,13 @@
<script>
import Explorer from "./lib/Explorer.svelte";
</script>
<main>
<div>
<h1>Scrapland .packed explorer</h1>
<Explorer />
</div>
</main>
<style>
</style>

109
scrapper_web/src/app.pcss Normal file
View file

@ -0,0 +1,109 @@
:root {
font-family: Inter, system-ui, Avenir, Helvetica, Arial, sans-serif;
line-height: 1.5;
font-weight: 400;
color-scheme: light dark;
color: rgba(255, 255, 255, 0.87);
background-color: #242424;
font-synthesis: none;
text-rendering: optimizeLegibility;
-webkit-font-smoothing: antialiased;
-moz-osx-font-smoothing: grayscale;
-webkit-text-size-adjust: 100%;
}
a {
font-weight: 500;
color: #646cff;
text-decoration: inherit;
}
a:hover {
color: #535bf2;
}
body {
margin: 0;
display: flex;
place-items: center;
min-width: 320px;
min-height: 100vh;
}
h1 {
font-size: 3.2em;
line-height: 1.1;
}
.card {
padding: 2em;
}
#app {
max-width: 1280px;
margin: 0 auto;
padding: 2rem;
text-align: center;
}
li {
text-align: left;
}
button {
border-radius: 8px;
border: 1px solid transparent;
padding: 0.6em 1.2em;
font-size: 1em;
font-weight: 500;
font-family: inherit;
background-color: #1a1a1a;
cursor: pointer;
transition: border-color 0.25s;
}
button:hover {
border-color: #646cff;
}
button:focus,
button:focus-visible {
outline: 4px auto -webkit-focus-ring-color;
}
@media (prefers-color-scheme: light) {
:root {
color: #213547;
background-color: #ffffff;
}
a:hover {
color: #747bff;
}
button {
background-color: #f9f9f9;
}
}
.lds-dual-ring {
display: inline-block;
width: 80px;
height: 80px;
}
.lds-dual-ring:after {
content: " ";
display: block;
width: 64px;
height: 64px;
margin: 8px;
border-radius: 50%;
border: 6px solid #fff;
border-color: #fff transparent #fff transparent;
animation: lds-dual-ring 1.2s linear infinite;
}
@keyframes lds-dual-ring {
0% {
transform: rotate(0deg);
}
100% {
transform: rotate(360deg);
}
}

52
scrapper_web/src/lib/Explorer.svelte Normal file
View file

@ -0,0 +1,52 @@
<script>
import { onMount } from "svelte";
import TreeView from "./TreeView.svelte";
import ScrapWorker from "../scrapper.worker?worker";
let worker;
let tree;
let busy;
busy = false;
onMount(async () => {
worker = new ScrapWorker();
worker.onmessage = (msg) => {
console.log({ msg });
if (msg.data) {
if (msg.data.parse) {
tree = msg.data.parse;
busy = false;
}
if (msg.data.download) {
let [file_name, url] = msg.data.download;
let dl = document.createElement("a");
dl.href = url;
dl.download = file_name;
dl.click();
}
}
};
});
let files;
function process() {
console.log({ files });
busy = true;
worker.postMessage({ parse: files });
}
</script>
<div class:lds-dual-ring={busy}>
<input
type="file"
multiple
accept=".packed"
class="file-input file-input-bordered w-full max-w-xs"
disabled={busy}
bind:files
on:change={process}
/>
</div>
{#if tree}
{#each [...tree.entries] as [name, child]}
<TreeView scrap={worker} label={name} tree={child} />
{/each}
{/if}

56
scrapper_web/src/lib/TreeView.svelte Normal file
View file

@ -0,0 +1,56 @@
<script>
export let tree;
export let scrap;
export let label=undefined;
let expanded = false;
function toggleExpansion() {
expanded = !expanded;
};
function download() {
console.log({label,tree});
scrap.postMessage({download:{label,...tree}});
console.log(tree);
}
</script>
<ul>
<li>
{#if tree.type == "directory" && tree.entries}
<span on:click={toggleExpansion} on:keydown={toggleExpansion}>
{#if expanded}
<span class="arrow">[-]</span>
{:else}
<span class="arrow">[+]</span>
{/if}
{label}
</span>
{#if tree.entries && expanded}
{#each [...tree.entries] as [name, child]}
<svelte:self {scrap} label={name} tree={child} />
{/each}
{/if}
{:else}
<span>
<span class="no-arrow" />
<a href="#download" title="{tree.size} bytes" on:click={download}>{label}</a>
</span>
{/if}
</li>
</ul>
<style>
ul {
margin: 0;
list-style: none;
padding-left: 1.2rem;
user-select: none;
}
.no-arrow {
padding-left: 1rem;
}
.arrow {
cursor: pointer;
display: inline-block;
}
</style>

6
scrapper_web/src/main.js Normal file
View file

@ -0,0 +1,6 @@
import './app.pcss'
import App from './App.svelte'
export default new App({
target: document.getElementById('app'),
});

28
scrapper_web/src/scrapper.worker.js Normal file
View file

@ -0,0 +1,28 @@
import wasm, { MultiPack } from "scrapper";
async function initialize() {
await wasm();
let pack;
let handlers = {
parse(data) {
pack = new MultiPack(data);
return pack.tree();
},
download(data) {
if (pack) {
let { label, file_index, offset, size } = data;
return [label, pack.download(file_index, offset, size)];
}
},
};
self.onmessage = (event) => {
for (var [name, func] of Object.entries(handlers)) {
let data = event.data[name];
if (data) {
postMessage(Object.fromEntries([[name, func(data)]]));
}
}
};
}
initialize();

2
scrapper_web/src/vite-env.d.ts vendored Normal file
View file

@ -0,0 +1,2 @@
/// <reference types="svelte" />
/// <reference types="vite/client" />

6
scrapper_web/svelte.config.js Normal file
View file

@ -0,0 +1,6 @@
import { vitePreprocess } from '@sveltejs/vite-plugin-svelte'
export default {
// Consult https://svelte.dev/docs#compile-time-svelte-preprocess
// for more information about preprocessors
preprocess: vitePreprocess(),
}

36
scrapper_web/tailwind.config.cjs Normal file
View file

@ -0,0 +1,36 @@
module.exports = {
content: ["./src/**/*.{svelte,js,ts}"],
plugins: [require("@tailwindcss/forms"),require("daisyui")],
theme: {
container: {
center: true,
},
},
daisyui: {
styled: true,
themes: true,
base: true,
utils: true,
logs: true,
rtl: false,
prefix: "",
darkTheme: "scraptool",
themes: [
{
scraptool: {
primary: "#F28C18",
secondary: "#b45309",
accent: "#22d3ee",
neutral: "#1B1D1D",
"base-100": "#212121",
info: "#2463EB",
success: "#16A249",
warning: "#DB7706",
error: "#DC2828",
// "--rounded-box": "0.4rem",
// "--rounded-btn": "0.2rem"
},
},
],
},
};

10
scrapper_web/vite.config.js Normal file
View file

@ -0,0 +1,10 @@
import { defineConfig } from 'vite'
import { svelte } from '@sveltejs/vite-plugin-svelte'
import wasmPack from 'vite-plugin-wasm-pack';
import preprocess from 'svelte-preprocess';
export default defineConfig({
plugins: [wasmPack("./scrapper/"),svelte({
preprocess: preprocess({ postcss: true })
})]
});