1
0
Fork 0
Commit graph

202 commits

Author SHA1 Message Date
Matthew Holt
1587a11bfb
Fix dropped messages in logs (fixes job-related UIs) 2025-01-15 12:03:03 -07:00
Matthew Holt
4e89fca643
Fix relationship de-duping; speed up imports a bit more 2025-01-10 15:31:20 -07:00
Matthew Holt
29e2bc8fef
Fix iphone/imessage: Update attribute_id in DB if inserting item piecewise
iMessage db may send a reaction graph for a message before sending the message itself to the pipeline, thus an empty item with only an original ID gets inserted, and later the full message item comes in, but I had neglected to add attribute_id to updateOverrides.
2025-01-09 18:09:58 -07:00
Matthew Holt
bb9151628f
WIP: new entity page; delete almost all DB indexes
Imports are now 4-5x faster and queries are still just about as fast. New indexes should only be created after proving their usefulness.
2025-01-07 13:42:05 -07:00
Matthew Holt
5ec783ecb8
Coupla fixes; also file picker now remembers last path 2025-01-02 20:33:50 -07:00
Matthew Holt
5816c571c6
WIP settings: save is starting to function 2024-12-31 20:58:07 -07:00
Matthew Holt
4bd08bd91c
WIP settings endpoints; use attr as alternate display name in messages 2024-12-31 10:09:20 -07:00
Matthew Holt
3d11d65b8d
WIP settings page; #map mobility; WIP interactive imports
Settings page is started; non-functional, but location picker works.

Moving maps between container elements is improved by moving to nearest to mouse pointer, rather than just most center to the viewport. It also emits an event when the map is moved, allowing us to change/reset map configurations for certain displays.

More progress on interactive imports. More thought is needed before continuing.

Upgraded Mapbox libraries.
2024-12-26 11:51:47 -07:00
Matthew Holt
ce297389b0
Thumbnail job streaming; WIP: interactive imports 2024-12-19 06:51:06 -07:00
Matthew Holt
294e2a72a9
Reconnect after disconnection; improve checkpointing 2024-12-17 14:27:50 -07:00
Matthew Holt
e34667bcce
googlelocation: Fix checkpoints 2024-12-16 20:21:16 -07:00
Matthew Holt
d7b1d73796
Auto-resume jobs on start; improve checkpoint performance 2024-12-16 16:23:26 -07:00
Matthew Holt
e319e1f60d
Checkpoints for Google Location; minor jobs fix 2024-12-16 10:57:34 -07:00
Matthew Holt
a4d8bc923d
Data source checkpoints; refine import concurrency
And related improvements and fixes
2024-12-15 22:40:58 -07:00
Matthew Holt
cbaa39b1b9
Implement proper ANALYZE maintenance 2024-12-13 20:55:22 -07:00
Matthew Holt
024e9a4622
Implement job start/restart 2024-12-13 14:30:13 -07:00
Matthew Holt
fcaa238634
Implement pause/unpause 2024-12-13 13:02:06 -07:00
Matthew Holt
22628833a7
Refactor obfuscation mode and some processing logic 2024-12-13 07:19:27 -07:00
Matthew Holt
0063bbe396
Some fixes for import streaming 2024-12-12 10:37:51 -07:00
Matthew Holt
786f516696
Refine import stream 2024-12-12 10:18:28 -07:00
Matthew Holt
742a543977
WIP live stream of imported items 2024-12-12 08:26:37 -07:00
Matthew Holt
5844c5755b
Fix most (all?) lint warnings 2024-12-11 18:59:24 -07:00
Matthew Holt
27e314e347
Few more minor jobs fixes, tunings 2024-12-11 15:41:13 -07:00
Matthew Holt
b4149dc649
More live job stats/rendering improvements 2024-12-11 14:04:07 -07:00
Matthew Holt
aa12d85c22
Fix job cancellation; wire up more of job UI 2024-12-10 23:13:14 -07:00
Matthew Holt
9ce1efa117
WIP live view of active jobs 2024-12-09 21:55:44 -07:00
Matthew Holt
063501c0f9
Fix stuck thumbnail loaders 2024-12-08 08:03:20 -07:00
Matthew Holt
13131aba65
Run ANALYZE after imports and at startup; add NMEA icon 2024-12-08 05:29:17 -07:00
Matthew Holt
37461545be
Fix fs.SkipDir usage; and minor bug in NMEA
fs.SkipDir documentation is a bit unclear: does it skip the remainder of files in the directory when returned from walking a file, or does it no-op on files and only skip going INTO dirs when on a dir?

I thought it was the latter, and thus, we didn't need to check whether the current DirEntry was a directory before returning (most commonly, when we are trying to skip hidden files/folders). But nope, it's the former -- SkipDir will skip the rest of the entries in the directory, which is NOT what we want. We just want to avoid going INTO a hidden directory in our case.

So unfortunately we now have to check IsDir() before returning.

Also fixed a slight bug with NMEA processing.
2024-12-07 21:16:36 -07:00
Matthew Holt
447b421a67
Start debugging slow conversation queries 2024-12-07 14:57:11 -07:00
Matthew Holt
53ca6063ab
Several fixes, performance improvements 2024-12-07 12:36:42 -07:00
Matt Holt
746e5d6b5c
Refactored import flow, new import UI, thumbnails stored in timeline, etc. (close #3) (#43)
* Schema revisions for new import flow and thumbnails

* WIP settings

* WIP quick schema fix

* gallery: Image search using ML embeddings

Still very rough around the edges, but basically works.

'uv' gets auto-installed, but currently requires restarting Timelinize before it can be used.

Lots of tunings and optimizations are needed. There is much room for improvement.

Still migrating from imports -> jobs, so that part of the code and schema is still a mess.

* Implement search for similar items

* Finish import/planning rewrite; it compiles and tests pass

* Fix some bugs, probably introduce other bugs

* WIP new import planning page

* Fix Google Photos and Twitter recognition

* Finish most of import page UI; start button still WIP

* WIP: Start Import button

* Fixes to jobs, thumbnail job, import job, etc.

* Implement proper checkpointing support; jobs fixes
2024-12-06 11:03:29 -07:00
Matthew Holt
332bf09ac6
Fix new lint errors 2024-09-11 12:39:27 -06:00
Matthew Holt
ab4c3bfe04
Allow bookmark items to be empty (with metadata)
Bookmark items may later be filled in by a snapshot.
2024-09-06 11:51:03 -06:00
Sergio Rubio
ef287c1bb9
GitHub data source (#48)
* GitHub stars data source

Data source that imports GitHub starred repositories in JSON format.

Each starred repo is imported individually, the starred repo
metadata comes from the GitHub API.

The item timestamp is set to the starred date, so they appear in the
timeline the day the repo was starred.

A small json file is saved in the timeline repository data directory
with the metadata retrieved from the GitHub API, which looks like:

```
{
 "id": 841044067,
 "name": "timelinize",
 "html_url": "https://github.com/timelinize/timelinize",
 "description": "Store your data from all your accounts and devices in a single cohesive timeline on your own computer",
 "created_at": "2024-08-11T13:27:39Z",
 "updated_at": "2024-09-03T07:17:29Z",
 "pushed_at": "2024-09-02T15:31:59Z",
 "stargazers_count": 504,
 "language": "Go",
 "full_name": "timelinize/timelinize",
 "topics": null,
 "is_template": false,
 "Topics": "archival,data-archiving,data-import,timeline",
 "private": false,
 "starred_at": "2024-08-12T17:55:48Z"
}
```

The data source currently expects the JSON file to be named like:

- ghstars.json
- ghstars-<ISO date>.json
- ghstars-<UNIX timestamp>.json

* Linter fixes

* Remove optional options

* Add the URL label to the bookmark class

* Change the data source name to GitHub

* Rename data source directory also

* Rename datasource main file

* Store GitHub starred repo URL only

* rename symbols

* Add basic tests

* moar tests

* Linter fix

* You can read on closed channels

* Add bookmark svg for the frontend

* Update package docs

* 💄 docs

* Update datasources/github/github.go

Co-authored-by: Matt Holt <mholt@users.noreply.github.com>

* Update datasources/github/github.go

Co-authored-by: Matt Holt <mholt@users.noreply.github.com>

* Update datasources/github/github.go

Co-authored-by: Matt Holt <mholt@users.noreply.github.com>

* Remove content from item

---------

Co-authored-by: Matt Holt <mholt@users.noreply.github.com>
2024-09-06 11:22:07 -06:00
Matthew Holt
750a78b3b5
Allow storing markdown in the DB 2024-09-04 09:46:35 -06:00
Sergio Rubio
304f8638c5
Test for couldBeMarkdown (#46)
* Test for couldBeMarkdown

couldBeMarkdown is greedy, and makes the processing[^1] step set the content
type of HTML and plain text files as text/markdown.

This has the side effect of excluding text file from in database
storage, since that only happens for text/plain files[^2]

[^1]: 8f0f491ad5/timeline/processing.go (L1479)
[^2]: https://github.com/timelinize/timelinize/pull/45#issuecomment-2325018564

* Add HTML test

* gofmt
2024-09-04 09:21:49 -06:00
Matthew Holt
32ce6e029b processor: Fix couldBeMarkdown detection 2024-09-04 08:50:24 -06:00
Sergio Rubio
fe114eed47
processor: Bump max size of text stored in DB to 50KiB (#45) 2024-09-03 21:35:03 -06:00
Matthew Holt
e3680c219b
Fix more lint errors; couple of minor regressions 2024-08-30 07:19:42 -06:00
Sergio Rubio
ac29bbd26b
chore: linter fixes (#39)
* chore: linter fix, remove unused argument

* min is a built-in function (predeclared identifier)
2024-08-30 06:31:12 -06:00
Matthew Holt
55b503ae4d
More lint fixes 2024-08-29 20:39:56 -06:00
Matthew Holt
3066ddbeb9
Major linting overhaul
I've addressed most of the "fast" linters errors locally in my editor.

Some linters are broken or buggy.
2024-08-29 16:43:52 -06:00
Matthew Holt
21d5a2ed8e
chore: Fix some lint errors (add package comments) 2024-08-28 16:05:43 -06:00
Sergio Rubio
9d9bed5bf0
Add new Document and Note classes (#37) 2024-08-28 13:53:53 -06:00
Matthew Holt
d8a5b6cf61
Improve classification docs 2024-08-28 11:41:59 -06:00
Matthew Holt
8f0f491ad5
Better support for text, markdown, and HTML items
- UI support for upcoming note and document classifications
- Display HTML, text, and markdown items in timeline
- Detect Markdown content in processor
- Tweak logo URL in readme
- Make import modal stay even when clicked outside
- Upgrade luxon to 3.5.0
- Add markdown parsing and DOM sanitizing libs
2024-08-28 08:17:54 -06:00
Matthew Holt
fe87b7f20c timeline: Improve retrieval key docs 2024-08-23 10:10:13 -06:00
Sergio Rubio
12300753a9
geojson: property parsing fixes (#35)
* Accept geojson files without a time property: do not return an error
  if well-know time properties aren't found, which prevents the entire
  feature from being ingested.
* Random properties aren't stored sometimes: do not modify the map while
  iterating it, as that can have unpredictable effects according to the spec
  (https://go.dev/ref/spec#For_range).
2024-08-23 07:53:20 -06:00
Matthew Holt
bc5c07d706
geojson: Accept freeform feature properties as metadata
Certain property names related to time, altitude, heading, etc., are extracted to the best of our ability.

Updates #33

Also a minor fix for a bug in spa.js when creating a new timeline with no owner info (other than name).
2024-08-22 16:55:53 -06:00