Lots of changes, expand to read

- Add notes folder with MDBook documentation (the NOTES.md file was getting kind of large) - Add rz_analyze.py, does the same a r2_analyze.py just with Rizin instead of radare2 so the project can be loaded in Cutter (*and* it's faster) - Add Scrap.rzdb, Rizin database for the Scrap.exe executable - Add Scrapper_rs, Rust version of .packed extractor and repacker - replace helplib.txt with helplib.md - add Py_Docs folder which contains generated documentation for the binary python modules built into Scrap.exe
2021-01-20 23:53:14 +01:00 · 2021-01-20 23:53:14 +01:00 · 7afdfb5869
commit 7afdfb5869
parent 43c01e81d2
50 changed files with 483086 additions and 1709 deletions
--- a/notes/.gitignore
+++ b/notes/.gitignore
@ -0,0 +1,9 @@
+
+# Created by https://www.toptal.com/developers/gitignore/api/mdbook
+# Edit at https://www.toptal.com/developers/gitignore?templates=mdbook
+
+### MdBook ###
+book
+
+
+# End of https://www.toptal.com/developers/gitignore/api/mdbook
--- a/notes/book.toml
+++ b/notes/book.toml
@ -0,0 +1,25 @@
+[book]
+authors = ["Daniel Seiller"]
+language = "en"
+multilingual = false
+src = "src"
+title = "Scrapland Reverse Engineering Notes"
+
+[output.html]
+preferred-dark-theme = "ayu"
+mathjax-support = true
+
+# [preprocessor.graphviz]
+# command = "mdbook-graphviz"
+
+[preprocessor.svgbob]
+text_width = 8.0
+text_height = 16.0
+class = "bob"
+font_family = "arial"
+font_size = 14.0
+stroke_width = 2.0
+# there's using css-variables from theme:
+stroke_color = "var(--fg)" # see default theme / variables.css
+background_color = "transparent" # also useful `var(--bg)`
+# all properties are optional.
--- a/notes/src/Chunked.md
+++ b/notes/src/Chunked.md
@ -0,0 +1,140 @@
+# Chunked Formats
+
+# General Block format
+
+```cpp
+struct Block {
+    unsigned char block_id[4],
+    uint32_t size,
+    unsigned char data[size],
+}
+
+template<typename T>
+struct Block {
+    unsigned char block_id[4],
+    uint32_t size,
+    T data,
+}
+```
+
+# Block IDs
+
+ File ID | Chunk IDs                                                                      
+ ------- | ------------------------------------------------------------------------------ 
+ AMC     | AMC, CMSH, QUAD                                                                
+ CM3     | ANI, CM3, EVA, NAE, NAM, SCN                                                   
+ DUM     | DUM, INI                                                                       
+ EMI     | EMI, LFVF, MAP, MAT, TRI                                                       
+ SM3     | ANI, CAM, INI, LFVF, LUZ, MAP, MAT, MD3D, NAE, NAM, PORT, SCN, SM3, SPR3, SUEL 
+
+Read types:
+
+- `i`: 4-byte unsigned integer
+- `s`: 4-byte signed integer
+- `p`: length prefixed string
+- `f`: 4-byte float
+- `3f`: array of 3 4-byte floats
+- `3i`: array of 3 4-byte unsigned integers
+
+ Chunk ID | Description                 | Reads                    
+ -------- | --------------------------- | ------------------------ 
+ AMC      | Collision Data              |                          
+ ANI      | Animation data?             |                          
+ CAM      | Camera info?                |                          
+ CMSH     | Collision Mesh Data         |                          
+ DUM      | Dummy (map object) data     |                          
+ INI      | INI-Configuration data      |                          
+ LFVF     | FVF Vertex Data             |                          
+ LUZ      | Lighting information        |                          
+ MAP      | UV Map?                     |                          
+ MAT      | Material information        |                          
+ NAE      | Animation Data?             |                          
+ NAM      | Animation Data?             |                          
+ PORT     | Map portals?                | i==1, i, i, 4, 4         
+ QUAD     | Mesh data?                  |                          
+ SCN      | Scene data?                 |                          
+ SUEL     | Ground plane?               | 0x18, 0xc, 4, 4, 4, 0x18 
+ TRI      | Triangle strip definitions? |                          
+ MD3D     | 3D Model definition?        |                          
+
+# Format of Specific chunks
+
+## INI
+
+Configuration Data
+
+```cpp
+struct INI {
+    uint32_t num_sections;
+    struct {
+        uint32_t num_lines;
+        struct {
+            uint32_t num_chars;
+            char line[num_chars]
+        } lines[num_lines];
+    } sections[num_sections];
+};
+```
+
+
+## LFVF
+
+DirectX Flexible Vertex Format Data
+
+```cpp
+struct Vertex { // fields according to flags
+    float position[3]; // D3DFVF_XYZ | D3DFVF_XYZRHW | D3DFVF_XYZB*
+    float rhw; // D3DFVF_XYZRHW
+    float weights[3]; // D3DFVF_XYZB*
+    float normal[3]; // D3DFVF_NORMAL
+    float point_size; // D3DFVF_PSIZE
+    uint32_t diffuse; // D3DFVF_DIFFUSE, RGBA
+    uint32_t specular; //D3DFVF_SPECULAR, RGBA
+    float tex_coords[D3DFVF_TEXTUREFORMAT][D3DFVF_TEX]; // D3DFVF_TEX* and D3DFVF_TEXTUREFORMAT*
+};
+
+struct LFVF {
+    uint32_t unk;
+    uint32_t num_entries;
+    struct {
+        uint32_t FVF; // FVF vertex data configuration
+        uint32_t vert_size; //?,
+        uint32_t num_verts;
+        Vertex vertices[num_vers];
+    } entry[num_entries];
+};
+```
+
+## DUM
+
+Map object data
+
+```cpp
+struct DUM {
+    uint32_t unk_1;
+    uint32_t num_dummies;
+    uint32_t unk_2;
+    struct {
+        uint32_t name_length;
+        char name[name_length];
+        float position[3];
+        float rotation[3];
+        uint32_t has_ini;
+        if (has_ini) {
+            Block<INI> ini;
+        };
+        uint32_t unk_1; // has_next?
+    } sections[num_sections];
+};
+```
+
+## MAP
+
+```cpp
+struct MAP {
+    uint32_t version;
+    uint32_t tex_name_len;
+    char tex_name[tex_name_len];
+    // TODO: rest
+}
+```
--- a/notes/src/Entities.md
+++ b/notes/src/Entities.md
@ -0,0 +1 @@
+# Entities
--- a/notes/src/File.md
+++ b/notes/src/File.md
@ -0,0 +1,18 @@
+# File Formats
+
+
+ File Extension | Description              | Chunked 
+ -------------- | ------------------------ | ------- 
+ .packed        | Game Data Archive        | n       
+ .cm3           | Animation file           | y       
+ .sm3           | 3d model file            | y       
+ .dum           | Dummy (map object) file  | y       
+ .pth           | AI Path                  | n       
+ .emi           | Emission maps/Materials? | y       
+ .amc           | Collision Data           | y       
+ .ini           | Configuration            | n       
+ .txa           | Texture Animation Config | n       
+
+- [Chunked](Chunked.md)
+- [Packed](Packed.md)
+- [AI Pathfinding Graph](Nodegraph.md)
--- a/notes/src/File_Formats.md
+++ b/notes/src/File_Formats.md
@ -0,0 +1 @@
+# File Formats
--- a/notes/src/Netplay.md
+++ b/notes/src/Netplay.md
@ -0,0 +1,41 @@
+# Netplay
+
+Game Info Packet
+
+```
+Server 'B':FZ (0/10) Ver 1.0 at 192.168.99.1:28086
+[0-3] header/ID?
+[4-5] port (16-bit)
+[6-7] max_players (16-bit)
+[8-9] curr_player (16-bit)
+[10-x] server name (char*)
+
+           0  1  2  3  4  5  6  7  8  9  A  B  C  D  E  F  0123456789ABCDEF
+0019fdc0  ba ce 00 01 b6 6d 0a 00 00 00 42 00 30 fe 19 00  .....m....B.0...
+0019fdd0  ff ff ff ff 27 2b b3 9b c7 3e bb 00 9c af 29 00  ....'+...>....).
+0019fde0  db 69 00 00 00 00 00 00 00 00 44 65 61 74 68 4d  .i........DeathM
+0019fdf0  61 74 63 68 00 00 00 00 ff ff 46 5a 00 4a 91 f0  atch......FZ.J..
+0019fe00  92 8b 57 4e 7f 00 00 00 10 21 fe 38 0d ae 00 00  ..WN.....!.8....
+0019fe10  f0 ce f3 36 a0 e8 0b 77 a0 e8                    ...6...w..
+```
+
+
+Player Join Packet
+
+```
+[0-3] header/ID?
+[6-x] Player name
+
+           0  1  2  3  4  5  6  7  8  9  A  B  C  D  E  F  0123456789ABCDEF
+09c9dfe8  7f 47 00 00 00 0e 55 6e 6e 61 6d 65 64 20 50 6c  .G....Unnamed Pl
+09c9dff8  61 79 65 72 06 53 42 6f 73 73 31 b9 00 07 50 5f  ayer.SBoss1...P_
+09c9e008  42 65 74 74 79 06 4d 42 4f 53 53 31 06 4d 42 4f  Betty.MBOSS1.MBO
+09c9e018  53 53 31 00 00 10 30 2c 31 35 2c 30 2c 30 2c 31  SS1...0,15,0,0,1
+09c9e028  35 2c 31 35 2c 31 02 00 00 00                    5,15,1....
+```
+
+ Message                                  | Description                                                       
+ ---------------------------------------- | ----------------------------------------------------------------- 
+ `5c68625c32383230395c73637261706c616e64` | "Scrapland Server" announcement broadcast (`\hb\28209\scrapland`) 
+ `7f01000007`                             | Retrieve Game info                                                
+ `48423d35323932322c3235363a323830383600` | Connection Information (`HB=52922,256:28086`)                     
--- a/notes/src/Nodegraph.md
+++ b/notes/src/Nodegraph.md
@ -0,0 +1,26 @@
+# AI Nodegraph
+
+Used for pathfinding using the A*-Algorithm.
+
+```cpp
+// n= number of dimension (3 or 2)
+
+template<size_t n>
+struct Node {
+    float pos[n];
+};
+
+template<size_t n>
+struct Edge {
+    uint32_t num_edge_nodes;
+    Node<n> nodes[];
+};
+
+template<size_t n>
+struct Graph {
+    uint32_t num_nodes;
+    Node<n> nodes[];
+    uint32_t num_edges;
+    Edge<n> edges[];
+};
+```
--- a/notes/src/Overview.md
+++ b/notes/src/Overview.md
@ -0,0 +1 @@
+# Overview
--- a/notes/src/Packed.md
+++ b/notes/src/Packed.md
@ -0,0 +1,15 @@
+# Packed
+
+```cpp
+struct Header {
+    unsigned char magic[4]; // always BFPK
+    uint32_t version;
+    uint32_t number_of_files;
+    struct File {
+        uint32_t path_length;
+        char path[]; // latin1 encoding
+        uint32_t data_size;
+        uint32_t data_offset; // offset includes header size so it can be used directly in a seek() call
+    } files[];
+};
+```
--- a/notes/src/Python.md
+++ b/notes/src/Python.md
--- a/notes/src/Python_API.md
+++ b/notes/src/Python_API.md
@ -0,0 +1 @@
+# Python API
--- a/notes/src/SUMMARY.md
+++ b/notes/src/SUMMARY.md
@ -0,0 +1,13 @@
+# Summary
+
+- [Overview](./Overview.md)
+- [ScrapEngine](./ScrapEngine.md)
+    - [World](./World.md)
+    - [Entities](./Entities.md)
+    - [Netplay](./Netplay.md)
+    - [Python API](./Python_API.md)
+    - [File Formats](./File_Formats.md)
+      - [Chunked Formats](./Chunked.md)
+      - [Packed](./Packed.md)
+      - [AI Nodegraph](./Nodegraph.md)
+- [ScrapHacks](./ScrapHacks.md)
--- a/notes/src/ScrapEngine.md
+++ b/notes/src/ScrapEngine.md
@ -0,0 +1,54 @@
+# ScrapEngine
+
+- Ingame Scripting Language: Python 1.5.2
+- Interesting memory locations and functions are noted in `config.yml`
+
+## Launch options
+
+- `-console`: open external console window on start
+- `-wideWindow`: start game in widescreen mode
+- `-dedicated`: start in multiplayer dedicated server mode (needs to be used with `-server`)
+- `-server`: start in multiplayer server mode
+- `-build`: Rebuild `Data.packed` (needs a `filelist.2Bpack`)
+
+## Ingame-Console
+
+(Ctrl+\^ or right click on window title bar and select "switch console") (Handler @ `0x402190`)
+
+* `<Code>`: Evaluate Python code
+* `:<Var>`: Get Game Engine Variable
+* `:<Var> <Val>`: Set Game Engine Variable
+* `?`: Show all Game Engine Variables
+* `?<String>`: Show all Game Engine Variables matching `<String>`
+* `/<command>`: Run Command defined in `QuickConsole.py`
+  * `import quickconsole;quickconsole.%s()`
+* `/<command> <arg>,<arg>`: Run function in `QuickConsole.py` with argument(s)
+  * `import quickconsole;quickconsole.%s(%s)`
+
+## External Console
+
+(Scene graph debugging?) (Handler @ `0x5f9520`)
+
+* `listar luces` List lights in scene
+* `listar` list models in scene
+* `arbol <model_name>` show details for model 
+* `mem` (doesn't do anything?)
+* `ver uniones` 
+* Easter Eggs:
+  * `imbecil`
+  * `idiota`
+  * `capullo`
+
+
+## Other interesting Memory Addresses
+
+- `0x852914`: D3D8-Device pointer
+- `0x7FCC00`: number of opened `.packed` files
+- `0x84cb64`: pointer to console command handler
+- `0x7fac84`: pointer to C++ callback list structure
+- `0x80b2cc`: pointer to ActionClassList (???)
+- `0x807a20`: pointer to SScorer (ingame GUI/Menu/Text system) structure (???)
+- `0x80a398`: pointer to SoundSystem (???)
+- `0x8b18f0`: pointer to Models Data (can be dumped using scene graph debugging console)
+- `0x8b18f4`: pointer to Scenes Data (can be dumped using scene graph debugging console)
+- `0x8b18f8`: pointer to active Models Data (can be dumped using scene graph debugging console)
--- a/notes/src/ScrapHacks.md
+++ b/notes/src/ScrapHacks.md
@ -0,0 +1 @@
+# ScrapHacks
--- a/notes/src/World.md
+++ b/notes/src/World.md
@ -0,0 +1,80 @@
+# World
+
+## Game World/State Pointer @ `0x7fe944`
+
+Points to World struct
+
+ Offset | Type                     | Description                            
+ ------ | ------------------------ | -------------------------------------- 
+ 0x0000 | `void**`                 | Virtual Method Table                   
+ 0x0004 | `uint32_t`               | Slots in Entity Hashtable              
+ 0x0008 | `void**`                 | Pointer to Entity Hashtable            
+ 0x00B0 | `??`                     | Pointer to Ground Object (?)           
+ 0x0288 | `pyEntity*`              | UsrEntity[0]                           
+ 0x028C | `pyEntity*`              | UsrEntity[1]                           
+ 0x0290 | `pyEntity*`              | UsrEntity[2]                           
+ 0x0294 | `pyEntity*`              | UsrEntity[3]                           
+ 0x0298 | `uint32_t`               | Slots in Model Hashtable               
+ 0x029C | `void**`                 | Pointer to Model Hashtable             
+ 0x02B8 | `uint32_t`               | Slots in Entity lists Hashtable        
+ 0x02BC | `void**`                 | Pointer to Entity list Hashtable       
+ 0x0330 | `float[3]`               | Time (why 3 times?)                    
+ 0x1C6C | `float`                  | Alarm level                            
+ 0x1C68 | `float`                  | Alarm Grow Level                       
+ 0x2158 | `float`                  | Used in `World_Init`                   
+ 0x2170 | `???`                    | Used in `World_Init`                   
+ 0x2180 | `float`                  | Used in `World_Init`                   
+ 0x2188 | `void*`                  | Used in `World_Init`                   
+ 0x218C | `void*`                  | Used in `World_Init`                   
+ 0x2190 | `float`                  | Used in `World_Init`                   
+ 0x2198 | `void*`                  | Used in `World_Init`                   
+ 0x219C | `void*`                  | Used in `World_Init`                   
+ 0x21A0 | `void**`                 | Used in `World_Init` (VTable pointer?) 
+ 0x21B4 | `void**`                 | Used in `World_Init` (VTable pointer?) 
+ 0x21C8 | `???`                    | Used in `World_Init`                   
+ 0x2204 | `uint32_t` or `uint16_t` | Used in `World_Init`                   
+ 0x2230 | `float`                  | Used in `World_Init`                   
+ 0x2238 | `???`                    | Used in `World_Init`                   
+ 0x2254 | `float`                  | Used in `World_Init`                   
+ 
+## cPyEntity structure
+
+ Offset | Type     | Description          
+ ------ | -------- | -------------------- 
+ 0x0000 | `void**` | Virtual Method Table 
+ 0x0004 | `char*`  | Name                 
+ 0x0008 | `void*`  | ???                  
+
+
+## Entity Hash Table
+
+Hash-function used: [PJW](https://en.wikipedia.org/wiki/PJW_hash_function) (Same parameters as the example implementation)
+
+Entry format:
+
+```cpp
+struct HT_Entry {
+  void* data;
+  const char* key;
+  HT_Entry* next;
+}
+```
+
+Data format:
+
+ Offset | Type          | Description              
+ ------ | ------------- | ------------------------ 
+ 0x0    | `void**`      | Virtual Method Table (?) 
+ 0x4    | `const char*` | name as string           
+ 0x14   | `void*`       | pointer to self (why?)   
+ 0x28   | `float[3]`    | Position in Game World   
+
+## EntityList Hash Table
+
+Attributes:
+- `Near`
+- `First`
+- `Num`
+- `OnDeath`
+- `OnDamage`
+- ...