cobertos/base-data-manager
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
base-data-manager/README.md
cobertos f6d0427a45 Converted TaskTargetPipelineHelper to more functional style, added aggregate() functionality to bring together multiple exports (no tests, but works) 
* made parallel generic (not tied to TaskTarget)
* pulled common higher-order/frontend operations into io.ts
* split timelinize specific functionality into own file
* Tests made to pass and match previous facebook export snapshots _exactly_
2026-02-26 00:14:10 -05:00

1.1 KiB
Raw Permalink Blame History

base-data-manager

A Typescript project for parsing through many types of data exports to tabular formats

** This is heavily WIP, and mostly just a toy for myself **

Installation

  • Install jq
  • Install sqlite csv.so extension (Hardcoded to /home/cobertos/sqlite-files/ currently)
  • Install node + pnpm i
  • See main.ts for current example usage

Proposed Architecture

The architecture runs in 2 steps.

The first step is unopinionated in it's output format. It's meant to take the source data exactly as-is and output it as csv. All source data should pass through, but will be normalized in csv

TODO: It's not completely unopinionated, there is some normalization for names of columns I think we want to apply? Or maybe we apply that later...

An optional second step combines everything into a single SQLite database. From here we normalize many different types of data across multiple exports into a single opinionated output. For example, message threads/channels should all have the same table format, or end up in the same table

TODO: No idea if the second part should be a part of this project... but it currently is