Chomp through data exports (Facebook, Google, wtf ever). Very WIP, do not use

Find a file

cobertos e7bb41d8ab Claude: Output summary as csv with csv-stringify		2026-03-04 13:54:22 +00:00
data-export	Claude: Add fitbit export, though tests are broken due to snapshot oddities	2026-03-03 07:39:50 +00:00
test	Use custom snapshots instead of nodejs's to handle carriage returns and better diffing/flow of running tests	2026-03-04 06:00:01 -05:00
util	Scrub now handles CSVs (in Typescript), scrub now has tests (for both TS and jq scrubbing), scrub has many more cases	2026-03-02 01:54:20 -05:00
.gitignore	Added 3 more scrubbed fixtures for new exports, scrub added boolean and numeric key scrubbing	2026-02-27 03:39:42 -05:00
main.ts	Rewrote timelinize.ts to work, and added new features for it. Added aggregateColumns for aggregated header, added metaIdValue to track which aggregate has metadata for another TaskTarget, added each() to allow a method of using cmd() with .id and other properties, added execPaths to make the initial definition of TaskTarget array to be a little more succinct	2026-02-26 16:32:33 -05:00
package.json	Use custom snapshots instead of nodejs's to handle carriage returns and better diffing/flow of running tests	2026-03-04 06:00:01 -05:00
pnpm-lock.yaml	Use custom snapshots instead of nodejs's to handle carriage returns and better diffing/flow of running tests	2026-03-04 06:00:01 -05:00
README.md	Converted TaskTargetPipelineHelper to more functional style, added aggregate() functionality to bring together multiple exports (no tests, but works)	2026-02-26 00:14:10 -05:00
summary.ts	Claude: Output summary as csv with csv-stringify	2026-03-04 13:54:22 +00:00
timelinize.ts	Rewrote timelinize.ts to work, and added new features for it. Added aggregateColumns for aggregated header, added metaIdValue to track which aggregate has metadata for another TaskTarget, added each() to allow a method of using cmd() with .id and other properties, added execPaths to make the initial definition of TaskTarget array to be a little more succinct	2026-02-26 16:32:33 -05:00
tsconfig.json	Initial commit	2025-11-24 22:42:27 -05:00

README.md

base-data-manager

A Typescript project for parsing through many types of data exports to tabular formats

** This is heavily WIP, and mostly just a toy for myself **

Installation

Install jq
Install sqlite csv.so extension (Hardcoded to /home/cobertos/sqlite-files/ currently)
Install node + pnpm i
See main.ts for current example usage

Proposed Architecture

The architecture runs in 2 steps.

The first step is unopinionated in it's output format. It's meant to take the source data exactly as-is and output it as csv. All source data should pass through, but will be normalized in csv

TODO: It's not completely unopinionated, there is some normalization for names of columns I think we want to apply? Or maybe we apply that later...

An optional second step combines everything into a single SQLite database. From here we normalize many different types of data across multiple exports into a single opinionated output. For example, message threads/channels should all have the same table format, or end up in the same table

TODO: No idea if the second part should be a part of this project... but it currently is