cobertos
f6d0427a45
* made parallel generic (not tied to TaskTarget) * pulled common higher-order/frontend operations into io.ts * split timelinize specific functionality into own file * Tests made to pass and match previous facebook export snapshots _exactly_
25 lines
1.1 KiB
Markdown
25 lines
1.1 KiB
Markdown
# base-data-manager
|
|
|
|
A Typescript project for parsing through many types of data exports to tabular formats
|
|
|
|
** This is heavily WIP, and mostly just a toy for myself **
|
|
|
|
### Installation
|
|
|
|
* Install `jq`
|
|
* Install sqlite `csv.so` extension (Hardcoded to `/home/cobertos/sqlite-files/` currently)
|
|
* Install `node` + `pnpm i`
|
|
* See `main.ts` for current example usage
|
|
|
|
|
|
### Proposed Architecture
|
|
|
|
The architecture runs in 2 steps.
|
|
|
|
The first step is unopinionated in it's output format. It's meant to take the source data exactly as-is and output it as csv. All source data should pass through, but will be normalized in csv
|
|
|
|
**TODO: It's not completely unopinionated, there is some normalization for names of columns I think we want to apply? Or maybe we apply that later...**
|
|
|
|
An optional second step combines everything into a single SQLite database. From here we normalize many different types of data across multiple exports into a single opinionated output. For example, message threads/channels should all have the same table format, or end up in the same table
|
|
|
|
**TODO: No idea if the second part should be a part of this project... but it currently is**
|