# base-data-manager

A Typescript project for parsing through many types of data exports to tabular formats

** This is heavily WIP, and mostly just a toy for myself **

### Installation

* Install `jq`
* Install sqlite `csv.so` extension (Hardcoded to `/home/cobertos/sqlite-files/` currently)
* Install `node` + `pnpm i`
* See `main.ts` for current example usage


### Proposed Architecture

The architecture runs in 2 steps.

The first step is unopinionated in it's output format. It's meant to take the source data exactly as-is and output it as csv. All source data should pass through, but will be normalized in csv

**TODO: It's not completely unopinionated, there is some normalization for names of columns I think we want to apply? Or maybe we apply that later...**

An optional second step combines everything into a single SQLite database. From here we normalize many different types of data across multiple exports into a single opinionated output. For example, message threads/channels should all have the same table format, or end up in the same table

**TODO: No idea if the second part should be a part of this project... but it currently is**