- iCloud logo was just a rasterized SVG, derp. Now it's a true SVG. Much smaller.
- Contact lists and vcards are pretty slow to import due to downloading profile pictures, so move those last
The sorting can help imports go faster if we put DB-heavy sources first, when the database is still small.
The data source names were also standardized to use snake_case like most other word-IDs in the app.
* Add WhatsApp importer
A first pass at importing WhatsApp chat exports.
Some open questions:
- Do we want to import context messages ("you deleted this message")?
- In WhatsApp its possible to have groups with the same participants but a different group name. Is it possible to tag a conversation with a "group name" in Timelinize? If not, this may end up with different conversations being interleaved.
- Is it safe to assume the current location for timezone analysis on import? WhatsApp exports use timezoneless timestamps, which (I've confirmed manually) are just "what the time would have been where you are now" (for me, messages sent in summer are in BST, and in winter are GMT)
Annoying quirks of the export format we should find good ways to communicate to users:
- Any caption text sent with an attachment isn't exported by WhatsApp. (The text is lost and unavailable to Timelinize — I've opened a bug with Meta, for all the good that'll do)
- If there are silent members of a group chat, their presence isn't recorded in the data WhatsApp exports
Todo:
- I _think_ it's safe to assume there's only ever one attachment per message, this would change & simplify the way I parse attachment lines. I'll keep exploring my own exports to identify if this is reasonable.
* Include polls & locations in tests
Polls are currently ignored, but I'll move them to being imported as a message, or as some special datatype, after discussion.
* Add text formatting examples, and show they're not processed
* Fix lint issues
* WhatsApp: Add Retrieval keys to messages
The key on the message isn't perfect, as it'll change if the person exporting their chat history has changed the name of one of the participants between exports (this would mean that participant's name would be different between exports, and their retrieval key would be different).
This seems as close as we can get without exported IDs though.
(I can't find a good way to test that the retrieval key is set properly)
* WhatsApp: Polls, Locations, Metadata
- Correctly parses attachments (even those which have been omitted, as not being available on the device that performed the export)
- Parses Polls (only in English, for now), including adding metadata for the Poll
- Extracts location metadata (Foursquare ID for named locations, or Lat/Long)
- Adds more test data to demonstrate other kinds of messages included in exports
* WhatsApp: Handle other locales
- 🤦♂️ The timestamp format changes based on the locale of the device performing the export — which makes accurate extraction of dates impossible between DD/MM/YYYY and MM/DD/YYYY dates. This parser will assume DD/MM/YYYY date if the last set of digits is 4 long. Perhaps we need an import option for "I'm using American dates"?
- Swaps the Poll scraping structure to allow for the localised words used when the exporting phone is set to other locales (eg. OPCIÓN instead of OPTION)
- Added a chat line test fixture to illustrate this (though normally the entire file would only ever be in a single locale)
* WhatsApp: Correct Poll Structure & fix parsing
I had incorrect POLL lines in the test fixtures; this commit fixes them, and the importer so it can read them properly.
* Use snake case for datasource name
Co-authored-by: Matt Holt <mholt@users.noreply.github.com>
* WhatsApp: Be cautious with matching
Be slightly less confident with matching `_chat.txt` files as WhatsApp exports!
* WhatsApp: Fix lint errors
Fix magic number linting errors
* WhatsApp: swap metadata namespaces
Switch to using "Pin" instead of "Location" to more accurately describe what's being tagged with the metadata.
---------
Co-authored-by: Matt Holt <mholt@users.noreply.github.com>
* [WiP] Firefox data source
Work in progress.
Implements a new Firefox datasource capable of reading its
places.sqlite database to import the browser history (page visits).
The implementation currently has a number of issues:
* Firefox (and Firefox based) browser keeps an exclusive lock on the
places.sqlite database, and we can't dump or backup it while the
browser is open, at least on Linux. To work around that,
we copy the database to a temporary directory and import from it.
This generally works, but isn't safe, as there's a risk of database
corruption when doing the hot copy. Potential alternatives:
* Ask the user to close the browser while the import happens, which
isn't convenient/possible if this is happening regularly in the background.
* Ignore and retry, as it'll eventually succeed, in the rare case the
temporary db copy is corrupted and unreadable
* Something else, no expert here.
* You need to point Timelinize to the places.sqlite file directly. Pointing
it to the Firefox profile directory doesn't seem to work, as it
doesn't seem to scan recursively or list all the directory files and
pass them to Recognize. I'm probably missing something obvious here.
* Missing tests (will be added)
* Linter fixes
* Adapt it to the new API
* Send the full path to process
* Simplify import process
* Add datasource description
* Use the URL as the item content
* Add basic tests
* Give the test some more time
* Do not return an error if context was cancelled
Also change the checkbox dropdown to a more interactive tomselect (type-to-search dropdown with chips) with pictures.
This makes it so data sources can be added to a timeline dynamically.
In the future, data sources can be implemented externally and push data to the timeline, so these need to not be rigidly hard-coded into the app and assumed to never change.
This essentially adds all their info (name, title, description, image, etc) into each timeline DB.
