AutomergeStorageService

The primary on-disk storage backend. An actor conforming to FileStorageServiceProtocol that reads and writes tags and time records as binary Automerge CRDT documents.

Source: Shared/Sources/MinutaShared/Services/AutomergeStorageService.swift.

Responsibilities

• Persist tags to a single tags.automerge document.
• Persist records into per-month documents under records/YYYY/MM.automerge.
• Cache loaded Document objects in memory, invalidated by the on-disk stamp (mtime and size).
• Store and load image attachments alongside their record’s month directory.
• Keep each record in exactly one month document: move it (and its image files) when its start time changes month.
• Expose a read-only probe (hasExternalChanges()) for external writers.
• Report record ids found in more than one month document (recordIdsInMultipleMonths()).
• Keep working when a file is unreadable: never overwrite it, serve the readable files, write the affected month to a recovery document, and report the file (unreadableFiles()). See Unreadable files and the recovery document.

It implements the same FileStorageServiceProtocol as the legacy JSON backend, so callers (AppState, the CLI, services) are agnostic to the format. See File storage service.

On-disk layout

<storage>/
  tags.automerge                  # one Automerge doc, keyed by tag UUID
  records/
    2026/
      04.automerge                # one Automerge doc per month, keyed by record UUID
      05.automerge
      04.recovery.automerge       # only while (or just after) 04.automerge is unreadable
      04/
        <filename>                # image files of April's records

Each Document is a CRDT map at .ROOT. Tag keys are tag UUID strings; record keys are record UUID strings. The wrappers AutomergeTag and AutomergeTimeRecord translate between the Swift models and the Automerge object graph.

For the format rationale and CRDT semantics, see Automerge storage architecture.

mtime-based cache

Loaded documents are cached, but never trusted blindly. Each load compares the file’s current stamp against the one observed when the document was last read. A stamp (FileStamp) is the file’s modification date and its size:

private func loadTagsDocument() throws -> Document {
    let url = tagsFileURL
    let currentStamp = fileStamp(url)

    if let cached = tagsDocument, tagsDocumentStamp == currentStamp {
        return cached
    }
    // ... otherwise re-read from disk and record the new stamp
}

Size is part of the stamp because mtime alone misses one case: a sync service that fills in a zero-byte placeholder and then sets the file’s original modification date. With the same mtime the unreadable file would look unchanged and never be read again; the changed size makes it a different stamp, so the file is retried.

This means a write by another process (the CLI, or a sync service dropping a new file) is picked up on the next load without an explicit cache flush. The stamp is recorded for a failed read too, and the failed read is not cached: every loadTags() of an unreadable tags file reads the file again and throws again, while hasExternalChanges() sees an unchanged file.

Record documents use the same scheme, keyed by "YYYY-MM", with one difference: a month is validated by a stamp pair (MonthStamp), the stamps of MM.automerge and of MM.recovery.automerge (nil where the file is absent). workingDocument(year:month:) serves the cached document only while both match what was seen when the document was built; a change of either file rebuilds it. The pair matters while the month document is unreadable: writers then share the recovery file, and a cached writer that compared only the month document’s stamp would overwrite another process’s recovery write.

Record documents are held in a cache capped at maxCachedDocuments (12 month documents); evictOldDocumentsIfNeeded() drops the oldest months (lowest "YYYY-MM" keys) when the cap is reached. Eviction removes only the Document: the month’s state (monthStates, holding the stamp pair, the write target and any read failure) and its index entries stay, so an evicted month is still a file the service has already seen and change detection is unaffected.

Record index

recordMonthIndex: [UUID: Set<String>] maps each record id to the month documents ("YYYY-MM") that hold it, with monthRecordIds as its reverse. A month’s entries are replaced whenever its document is loaded from disk or saved, and dropped when the file is gone or a write to it failed. The index covers the working documents of exactly the months that have an entry in monthStates, so it survives cache eviction.

Normally an id maps to one month. More than one means a duplicate, left by an older build or by a sync merge.

refreshIndexLocked() brings the index in line with disk: it stats every month that has a month file or a recovery file, reloads those whose stamp pair differs from the last seen one or that were never seen, and forgets months whose files are gone. It is stat-only when nothing changed. It runs inside the write lock, so no other process can move a record between the refresh and the write that relies on it. An unreadable month does not fail the refresh: it is indexed by its working document (the recovery document, or nothing when both files are unreadable), so one corrupt file does not block edits to records in other months. The refresh also persists every pending consolidation of a recovery document, see Consolidation.

Writes

Mutating operations (saveTags, mutateTags, saveRecord, updateRecord, deleteRecord) take the cross-process write lock before touching disk. The image methods (saveImage, deleteImage) write files directly and take no lock:

try StorageFileLock.withExclusiveLock(at: baseURL) {
    // read-mutate-write under flock
}

The write path is read-mutate-write: load the current document from disk under the lock, apply the change, commit with a timestamp, and atomically overwrite the file (.write(to:options: .atomic)). For a month, the file written is the one its state names as the write target: normally MM.automerge, the recovery file while MM.automerge is unreadable, and nothing (the write throws) when both are unreadable. The combination of the flock and stamp invalidation — not an in-process merge — is what keeps two writers from clobbering each other. saveTags and mutateTags explicitly clear the cached tags document first so a stale merged cache can’t reintroduce keys another writer deleted. Record writes (saveRecord, updateRecord, deleteRecord, and the consolidation in refreshIndexLocked) load each month they write with workingDocument(year:month:fresh: true), which ignores the cached document whenever a file exists. A matching stamp is not proof enough for a write: on a file system with one-second mtimes (exFAT, SMB, some sync folders) another writer can replace the file within the same tick with one of the same size, and a write from the cache would erase that change. Reads keep using the stamp. The id-to-months index is still refreshed by stamp, so a record another process moved to another month within the same tick and size is not found there (a duplicate minuta doctor reports).

Tag saves diff against the on-disk key set and delete keys that are no longer present, so removals propagate as Automerge deletes rather than orphaned entries.

saveRecord and updateRecord

Both write the record into the month document of its startTime and then remove its id from every other month the index lists, under one lock. They differ in whether the index is refreshed first:

updateRecord is refused when the record cannot be found while a month document is unreadable. After the refresh, if no month’s working document holds the id and at least one month document (MM.automerge) is unreadable, it throws AutomergeStorageError.recordMayBeInUnreadableMonth(months:) with the sorted list of those months and writes nothing. The record, if it is stored at all, is inside an unreadable file, and the caller’s value is a stale copy of it; writing that copy to a recovery document would overwrite the stored record’s newer fields once the file reads again. A record that was created or saved during the outage is in the recovery document, so the index knows it and it is edited there as usual. saveRecord has no such check, which is why a record that is not in storage (the undo of a delete, an import) goes through it; see TimeTrackingService.restoreRecord(_:) in Time tracking.

So editing a record’s start time into another month moves it: updateRecord writes it to the new month document, moves the image files named in record.images from the old month folder to the new one, and deletes the id from the old month document. The order is write new, move images, delete old; an interruption leaves the old copy in place as a duplicate that the next save cleans up, instead of losing the record. A failed file move or a month that cannot be cleaned is logged, not thrown: the caller’s change is already persisted.

If the write to the target month fails, the in-memory document is discarded so the next load re-reads disk rather than serving unsaved state.

deleteRecord

deleteRecord refreshes the index under the lock, then removes the id from the month of the argument’s startTime. If the id is not there (the caller’s value is stale, or another process moved the record), it is removed from the month(s) the index says it is in.

If the month document is unreadable and its working document does not hold the id, deleteRecord throws AutomergeStorageError.recordInUnreadableMonth and writes nothing: the record, if it is stored at all, is inside the unreadable file, where the deletion cannot be applied. A record that was created or saved during the outage is in the recovery document and is deleted from there.

Image files are deleted only after the document write succeeded, and only the files named in the argument’s images, resolved against the month the id was found in. Files the stored record references but the argument omits stay on disk, so passing a copy with images = [] deletes the record and keeps its files; that is how the app’s undo window works (see Deletion toast). A file that cannot be removed is logged and left as an orphan.

Duplicates

Duplicates that already exist on disk are not collapsed on read, because nothing says which copy is newer: loadRecords / loadAllRecords return every copy. They are resolved by writes:

• saving a record (updateRecord, or saveRecord if the index already knows the other month) keeps the copy in the month of its start time and removes the others;
• deleting removes only the copy in the month of the argument’s startTime when that month holds the id; a copy in another month is left alone.

recordIdsInMultipleMonths() is the read-only diagnostic: it returns each duplicated id with the sorted list of months holding it. Unreadable months are compared by their working documents, so the check still runs when a file is corrupt. minuta doctor reports it as the duplicate records check, see the CLI reference.

Unreadable files and the recovery document

The storage folder is shared with a sync service, so a truncated, half-downloaded or zero-byte file is a normal event. The rule: a file that cannot be read is never overwritten, renamed or deleted by this service (ConflictResolutionService follows the same rule). Everything else keeps working around it. The types live in Shared/Sources/MinutaShared/Services/AutomergeStorageRecovery.swift (MonthRecovery, MonthState, MonthStamp, FileStamp, UnreadableStorageFile, AutomergeStorageError).

A file is unreadable in one of two ways:

Zero-byte files take the first path. Document(Data()) does not throw, it returns a new empty document, so a zero-byte file (a sync placeholder, a truncated write) used to read as an empty month and was overwritten by the next save. AutomergeDocument.load(from:) now rejects empty data; the reported reason is Deserialization failed: the file is empty. This applies to tags.automerge as well.

Reads

loadRecords, loadAllRecords, loadRunningRecords and recordIdsInMultipleMonths() do not throw because of an unreadable month. They return the records of every readable month plus, for an unreadable one, the records in its recovery document; the records inside the unreadable file are not returned. A caller that must tell “no records” from “records not shown” asks unreadableFiles() after loading. Reads never write to disk.

Month states

MonthRecovery.resolve(primary:recovery:) reads both files of a month and decides what reads and writes work on (the working document) and where writes go. “Month document” is records/YYYY/MM.automerge, “recovery document” is records/YYYY/MM.recovery.automerge.

One more case: both files parse but the merge itself throws. The month document is then read again from disk (the in-memory one may hold part of the merge) and served alone, and the recovery file is reported with the reason Merge failed: ... and left on disk.

The result is stored per month as a MonthState: the stamp pair, the write target (primary, recovery or none), needsConsolidation, the failure of either file and the time of the read. States live in monthStates, outside the document cache, so they survive eviction.

A recovery file is created only to hold records: while the write target is the recovery file and it does not exist yet, a write whose document is empty (for example a cleanup that found nothing to remove) creates nothing.

Merge and the conflict fix-up

When both files are readable, MonthRecovery.merge(recovery:into:) runs primary.merge(other: recovery) and then repairs conflicting keys.

The repair exists because of how the two documents relate. A record saved during the outage is created in the recovery document as a new map object, unrelated to the object the month document holds under the same record id. Automerge does not merge two independently created objects field by field: it keeps both as conflicting values of the key and serves one of them whole, chosen by its own ordering and not by time. Without the repair, a timer that was running before the outage and stopped during it could come back running after the merge.

So for every record key of the recovery document that is left with more than one value in the merged document (getAll(obj:key:).count > 1), the record is written again from the recovery document as a fresh object, which supersedes both. The recovery document’s version of a record wins.

The merge is idempotent. Merging the same recovery document a second time brings no new changes, the keys stay single-valued, and nothing is rewritten, so an edit made to the month document after the first merge is kept.

Consolidation

The merge happens in memory, on load (a read may trigger it, and reads never write). It is persisted, and the recovery file removed, by:

• the first write to that month (saveRecordDocument with the month document as the write target);
• refreshIndexLocked(), which runs at the start of updateRecord and deleteRecord and consolidates every month that is pending, not only the one being written.

Both run under the storage write lock. saveRecord (a new record) consolidates only the month it writes to.

The order is: write the month document, then remove the recovery file, and only if the recovery file’s stamp is still the one that was merged. A recovery file that changed in between (a sync service delivered another device’s recovery writes) stays on disk; its stamp no longer matches, so the next load merges it. A crash between the write and the removal is harmless: the next load merges the same recovery document again, which changes nothing, and the next write removes the file. If the removal fails, it is logged and retried by the next write to the month.

What is deliberately not done

• No automatic quarantine. The service never renames, moves or deletes an unreadable file, and the app has no button for it. If the file is still syncing, it becomes readable on its own. If it is lost for good, the user moves or deletes it by hand; the month then falls into the “absent / readable” row above and the recovery document takes its place on the next write.
• No queued deletions or edits. Deleting a record that lives inside an unreadable month document is refused (recordInUnreadableMonth), and so is editing it (recordMayBeInUnreadableMonth). Neither is remembered and applied later; the caller repeats it once the file reads again.
• No recovery file for tags. Tags are matched by name. A tag created in a parallel tags document during an outage could have the same name as a tag inside the unreadable file under a different id, and after a merge both would exist: same-name duplicates, with records split between them. So while tags.automerge is unreadable, tags are read-only at best: loadTags() throws tagsFileUnreadable, and saveTags / mutateTags throw the same error before writing anything, because every tag write loads the document first. The file is never replaced by an empty document. Records are unaffected: they load, and they can be saved with a tagId the caller already knows.

Known limits

The refusals above cover the common cases, not every one:

• A record hidden in an unreadable recovery file. updateRecord is refused only while a month document is unreadable. When the month document reads and only its recovery file is unreadable, a stale update of a record that lives only in that recovery file is accepted and written to the month document.
• saveRecord with an id that is already stored in an unreadable month. saveRecord does not check: the record is written to the month of its start time, and the copy inside the unreadable file shows up as a duplicate once the file reads again. See Duplicates.
• deleteRecord with a stale start time. The refusal looks at the month of the argument’s startTime when the index does not know the id. If that month is readable while the record sits inside another, unreadable month document, nothing is removed and no error is thrown.

unreadableFiles()

public func unreadableFiles() -> [UnreadableStorageFile]

Lists the storage files that could not be read, sorted by month with the tags file last. HybridStorageService.unreadableFiles() forwards to it (as an async method); FileStorageServiceProtocol is unchanged, so callers need the concrete service.

The method does no disk I/O. It reports what the loads of this instance have observed so far: a file appears after a load that visited it (loadAllRecords visits every month, loadTags the tags file) and disappears after a load that read it successfully. It is empty right after invalidateCache(), so call it after loading, on the same instance.

Typed errors

AutomergeStorageError (LocalizedError) is thrown when a file that would have to be overwritten cannot be read, or when a record cannot be found because a month document is unreadable. The file named in the error is left untouched. Messages, with example values:

How the two front ends surface all this: Storage warnings in the app, the stderr warning and doctor rows in the CLI reference. The view across processes and devices: Cross-process safety.

External change detection

hasExternalChanges() is a read-only probe used by the app’s ExternalChangeMonitor. It compares on-disk stamps (mtime and size) against the last-observed ones and returns true when:

• the stamp of tags.automerge differs from the one last seen, or the file appeared or vanished;
• for any month seen so far (whether its document is still cached or was evicted), the stamp pair differs: either the month file or the recovery file changed, appeared or vanished;
• a month has a month file or a recovery file on disk and the service has never loaded it;
• a file that could not be opened (a transient failure, tags or month) was last tried 60 or more seconds ago.

An unreadable file counts as seen. While its mtime and size are unchanged the probe returns false, so a corrupt file does not make the app reload every 5 seconds; when either changes (the sync finished) the probe fires, the app reloads and the file is parsed again.

It performs only stat-level I/O and mutates no state, so it is cheap to call on a short interval.

invalidateCache() drops all cached documents, month states (stamps and recorded failures), the tags stamp and failure, and the record index; callers reload through the normal load methods afterward.

Related

• File storage service — the protocol this actor implements.
• Automerge storage architecture — on-disk format and CRDT design.
• Cross-process safety — flock, stamp-based invalidation, and the app/CLI contract.
• Conflict resolution — merging sync-service conflict copies.
• External change monitor — the app-side poller that drives reloads.