Cross-Process Safety

The macOS/iOS app and the minuta CLI operate on the same .automerge files. This page explains the mechanics that keep concurrent writers from losing data and keep each process’s view fresh.

Concurrency model

  • Same host, multiple writers (e.g., app running while you run a CLI command in Terminal): protected by an advisory file lock.
  • Multiple hosts (e.g., laptop + desktop both open while Dropbox syncs): protected by Automerge CRDT merge semantics. File-lock state does not cross hosts; Dropbox/iCloud copy content, not kernel locks.

The storage layer treats these as different problems. The file lock ensures single-host atomicity. The CRDT format ensures cross-host merging is safe eventually.

Single-host write lock

Filename: <storage>/.minuta.writelock.nosync

  • Created on first write. The .nosync suffix keeps iCloud Drive from syncing it (Apple convention).
  • Any process that writes takes flock(LOCK_EX) around its critical section and releases it before returning.
  • Both the app (AutomergeStorageService) and the CLI (same shared code) take the same lock. ConflictResolutionService takes it too, see Conflict resolution under the lock.
  • The lock is not re-entrant. withExclusiveLock opens a new file descriptor on every call and blocks in flock(LOCK_EX); a nested call from code that already holds the lock waits on itself forever. Helpers that run inside the critical section (refreshIndexLocked, saveRecordDocument, mergeConflictFiles) therefore expect the caller to hold the lock and never take it themselves. The closure is synchronous, so nothing inside it can await.

Implementation: Shared/Sources/MinutaShared/Services/StorageFileLock.swift — a thin wrapper around open(2) + flock(2).

Save path under the lock

1. Acquire flock(LOCK_EX) on .minuta.writelock.nosync
2. Invalidate the relevant in-memory cache slice (tags or month doc)
3. Reload the doc from disk (fresh bytes under lock)
4. Apply the mutation to the doc
5. commitWith(timestamp:) + save().write(atomically:)
6. Record new stamp (mtime, size) in cache
7. Release flock

Tags (saveTags, mutateTags) clear the cached tags document; record writes (saveRecord, updateRecord, deleteRecord) read each month they write from disk (workingDocument(fresh: true)) instead of comparing stamps. A stamp (mtime and size) can match a changed file: with one-second mtimes (exFAT, SMB, some sync folders) a write by another process in the same tick that leaves the size unchanged would otherwise be overwritten by the cached document. updateRecord and deleteRecord additionally refresh the id-to-months index under the lock (re-reading every month file whose stamp changed), so a record that another process moved to a different month is found where it is now. Details: AutomergeStorageService.

Step 2 is essential when the caller of mutateTags comes in holding a stale cached doc — without invalidation, a subsequent diff-based write (e.g., “delete all keys the caller didn’t include”) would clobber keys another writer added.

Why the in-save merge was removed

An earlier design did inMemory.merge(with: onDisk) inside the save path. That triggered an upstream Automerge Rust panic (PatchLogMismatch) when merging documents with overlapping histories. Since the flock guarantees exclusive disk access for the read-mutate-write window, the merge was unnecessary for correctness on a single host. Cross-host merge still happens — via mtime-triggered reloads — just on the read path.

mtime-based cache invalidation

AutomergeStorageService remembers the stamp of each file when it last loaded it. A stamp (FileStamp) is the modification date and the size:

  • tagsDocumentStamp: FileStamp?
  • monthStates: [String: MonthState] keyed by YYYY-MM; each state holds a stamp pair (MonthStamp): the stamp of the month file MM.automerge and of its recovery file MM.recovery.automerge, nil where the file is absent

On each load, it stats the file (for a month, both files) and compares. If a stamp differs from last-seen, the cache is dropped and the doc re-read from disk. This is how the app picks up CLI writes and sync-service drops without an explicit refresh. Writes do not rely on it (see the save path above): a same-size change within the same mtime tick is invisible to the stamp until the next change of the file.

Size is compared because mtime alone misses a sync service that fills in a zero-byte placeholder and then sets the file’s original modification date: with the same mtime the file would look unchanged and never be read again.

The stamp is recorded for a file that could not be read as well, so an unreadable file is “seen” like any other and is read again only when it changes. See Unreadable files.

External change polling (app side)

The app also actively polls for external writes rather than waiting for the next user action. Without this, the UI could show stale data for minutes while the CLI or Dropbox quietly updates the folder.

Two pieces:

AutomergeStorageService.hasExternalChanges()

A read-only probe. Stats tags.automerge and, for every month seen so far, the month file and its recovery file; also scans for months that appeared since last time. Returns true if:

  • any stamp (mtime or size) differs from the cached value (for a month: either half of the stamp pair)
  • a month file or a recovery file exists for a month the service has never seen
  • a cached file vanished from disk
  • a file that could not be opened (permissions, I/O) was last tried 60 or more seconds ago

An unreadable file whose mtime and size have not changed is not a change: without that, one corrupt file would make the app invalidate and reload on every 5-second tick.

No I/O beyond stat. Safe to call many times per minute. The last-seen stamps are kept for months evicted from the 12-document cache, so eviction does not make a known file look new.

ExternalChangeMonitor (app only)

Minuta/Sources/Services/ExternalChangeMonitor.swift. A @MainActor timer that calls the probe on a cadence:

Scene phaseCadence
.active5 seconds
.inactive60 seconds
.background60 seconds

When the phase transitions to .active, an immediate probe fires — that’s the “on window focus” trigger. Overlapping ticks are coalesced via an inFlight flag.

On a positive result, the monitor calls AppState.reloadFromExternalChanges():

  1. await storage.invalidateCache()
  2. await resolveConflictsIfNeeded() — handles (conflicted copy) files from Dropbox/iCloud
  3. await loadData() — reloads tags, today’s records, running timers, and bumps historyRevision so History reloads its date range too (a change to a record of an earlier day changes nothing in today’s records)

HybridStorageService.invalidateCache() is async and returns only once both underlying stores are invalidated. It used to start an unawaited Task and return at once, and callers reload right after: an invalidation landing after the reload would wipe the stamps that reload had just recorded.

Is invalidation safe?

Yes. All user writes go through await trackingService.updateRecord(...) (or equivalent), which doesn’t return until AutomergeStorageService has written to disk under the flock. There is no “dirty in-memory state” that invalidation could destroy — the cache is read-through, not write-back. At worst, invalidation forces the next load to re-read bytes we just wrote.

The CLI does not run a monitor: it’s a one-shot process, exits after each command, and rebuilds its cache on the next invocation.

Unreadable files

A month file or the tags file can be unreadable for a while (truncated or half-downloaded by a sync service, zero bytes, no permission). The single-process mechanics, the month-state table and the merge are described in AutomergeStorageService. In short: the unreadable file is never overwritten, renamed or deleted; new records for that month go to a parallel recovery file records/YYYY/MM.recovery.automerge; a record that may be inside the unreadable file cannot be edited or deleted until it reads again (recordMayBeInUnreadableMonth, recordInUnreadableMonth); when the month file reads again the recovery document is merged into it and the recovery file removed. This section is the view across processes and devices.

Two processes writing the same recovery file

While MM.automerge is unreadable, the app and the CLI both write that month to MM.recovery.automerge. They are ordinary writers: each takes the flock, and inside it loads the month’s working document, which compares the stamp pair. A process holding a cached recovery document sees that the recovery file’s stamp moved (the other process wrote it), re-reads it, applies its change and writes. Nothing is lost.

This is why the stamp is a pair. While the month file is unreadable its stamp does not move, so a cache validated by the month file’s stamp alone would look fresh forever, and the cached process would write its stale recovery document over the other process’s records.

The first write that creates the recovery file is covered by the same check: the other process’s state has “no recovery file” in its stamp, the file now exists, the pair differs, the month is re-read.

A cached process after another one consolidated

Process A notices that the month file reads again, merges the recovery document and, on its next write to that month (or its next updateRecord / deleteRecord anywhere, through refreshIndexLocked), writes the month file and removes the recovery file. All of it under the lock.

Process B still has a state for that month that says “write to the recovery file”. Its next load compares the stamp pair: the month file’s stamp changed and the recovery file is gone. B rebuilds the month from disk, gets the plain “month file readable, no recovery file” state, and writes to the month file. It does not recreate the recovery file from its cache.

The same check covers the app between its own writes: hasExternalChanges() compares the pair, so the 5-second probe fires, and reloadFromExternalChanges() reloads.

If both processes find the pending consolidation at the same time, the lock orders them. The second one sees a changed pair, re-reads, and finds nothing left to merge. If a process dies between writing the month file and removing the recovery file, the next process merges the same recovery document again (the merge is idempotent, so the month file’s content does not change) and removes the file.

The recovery file is removed only if its stamp is still the one that was merged. If another writer or a sync service changed it in between, it stays and the next load merges the new content.

Older builds

A build from before this feature shares the folder without knowing about recovery files:

  • It ignores MM.recovery.automerge. Its month scan takes the file name without the extension and parses it as a number; 04.recovery is not one. It neither reads nor deletes the file, so the records written during an outage are invisible to it but safe, and a newer build consolidates them later.
  • It does not have the protections. A zero-byte month or tags file still reads as an empty document there, and its next save overwrites the file. A month file it cannot parse makes its whole load fail.
  • Its ConflictResolutionService leaves the recovery file alone for the same reason the current one does (below).

So during a rollout the guarantee “an unreadable file is never overwritten” holds only for processes running a current build. .minuta-version was not bumped for this: the format of the existing files did not change.

Two devices

File locks do not cross hosts. Two devices that both see MM.automerge unreadable each create their own MM.recovery.automerge, and the sync service turns the second one into a conflict copy, for example 04.recovery (conflicted copy).automerge.

ConflictResolutionService handles that with no special code: the conflict patterns strip the suffix, the base name is 04.recovery.automerge, and the copy is merged into the recovery file like any other conflict copy. The recovery file is a different base name from 04.automerge, so it is never grouped with the month file and never merged into it by the resolver; that merge belongs to AutomergeStorageService, which also applies the conflict fix-up.

AutomergeStorageService itself matches the recovery suffix exactly, so a conflict copy of a recovery file is not mistaken for a month’s file while it waits for the resolver.

The resolver’s merge is a plain Automerge merge, without the fix-up. If both devices saved the same record during the outage, each created its own object for it, and the merged recovery document serves one of the two whole.

The resolver follows the “never overwritten” rule itself. ConflictResolutionService loads the base file and every copy through AutomergeDocument.load(from:), so a zero-byte file is unreadable there as well. A base file that cannot be read, corrupt or empty, fails its group: the error names the file, nothing is written and no copy is removed until the file reads again.

A conflict copy of a recovery file can also be on disk with no recovery file next to it. The resolver promotes such a lone copy to 04.recovery.automerge, see Conflict resolution under the lock; storage then finds a recovery file and merges it into the month file like any other.

Tags

tags.automerge has no recovery file. While it is unreadable every process refuses tag writes before touching the file (AutomergeStorageError.tagsFileUnreadable), so there is nothing to coordinate: the file’s bytes stay as they are until the sync service or the user fixes it.

Version marker

Filename: <storage>/.minuta-version

Contents: a single integer (currently 1). Identifies a folder as initialized by a Phase-0-or-newer build (one that participates in the flock discipline).

  • The app writes the marker on launch and whenever storage location changes (StorageLocationManager.publishResolvedStoragePath).
  • The CLI auto-writes the marker on its first successful write if missing. This lets a CLI-only workflow (no app touched the folder yet) self-heal.
  • A stricter gate exists (strictRequireVersionMarker returns exit code 5 when absent) but isn’t currently used. It’s there for the future rollout scenario where an older app (without the flock discipline) could silently stomp on CLI writes.

Storage-path discovery

The app and CLI need to agree on which folder to operate on, even though the app is sandboxed (Mac Catalyst) and the CLI is a plain Unix binary.

The Catalyst container trap

Mac Catalyst apps are always containerized, regardless of sandbox entitlement. Inside the app:

  • NSHomeDirectory() resolves to ~/Library/Containers/tools.minuta.app/Data/
  • FileManager.default.urls(for: .applicationSupportDirectory, ...) also lives inside the container

That path is invisible to the CLI (a non-Catalyst binary that sees the real ~).

The bypass

CLIConfigService.configDirectory() calls getpwuid(getuid())->pw_dir to get the real user home directly from the OS, bypassing the Catalyst redirect. Both the app and CLI resolve the same physical path:

<realHome>/Library/Application Support/minuta/storage-path.txt

The file contains the absolute path of the user-selected storage folder, written as plain UTF-8.

Write triggers (app)

StorageLocationManager.publishResolvedStoragePath() writes the file on:

  • init() — app launch
  • saveBookmark(for:) — user picks a new folder
  • useDefaultLocation() — user reverts to the default

Exception: a sandboxed process (the Mac App Store build) never writes the pointer. StorageLocationManager sets isSandboxed when APP_SANDBOX_CONTAINER_ID is in the environment and skips the write, since the path is outside the app’s container. The .minuta-version marker is still written. The CLI then gets nothing from step 3 of the read order below (or a stale pointer left by an earlier non-sandboxed build) and falls back to ~/Documents/Minuta, which is not the container Documents folder the sandboxed app defaults to; users of that build pass --storage or set MINUTA_STORAGE.

Exception: launches with --test-storage= (UI tests) skip the pointer write — publishing a throwaway per-test temp folder would leave the CLI discovering a soon-deleted dir. The .minuta-version marker is still written, into the test folder itself. Regression-tested by StoragePointerTests (UI) and StorageLocationManagerTests (unit, via an injected StoragePathPublishing recorder).

Read order (CLI)

StorageResolver.resolve(...) — first match wins:

  1. --storage <path> flag
  2. MINUTA_STORAGE env var
  3. storage-path.txt pointer (from the app)
  4. ~/Documents/Minuta default

If step 3 points at an inaccessible folder (unmounted drive, stale Dropbox path), the CLI fails fast with exit code 2 rather than silently writing to the default — otherwise the two processes would diverge.

Cross-host (sync services)

flock(2) is kernel-local. Two machines editing the same Dropbox-synced folder do not see each other’s locks. That’s fine — Automerge handles it:

  • Each save is atomic locally (flock + atomic rename).
  • The sync service eventually propagates the bytes.
  • The next reader on the other machine detects the changed stamp, reloads, and the CRDT format means the document already represents the merge of all histories seen so far.
  • If the sync service itself detects a write-write conflict, it produces a (conflicted copy) file. ConflictResolutionService scans for these on app launch, on scene activation and after a detected external change, and merges them into the base document.

Conflict resolution under the lock

ConflictResolutionService.resolveConflicts() rewrites the primary document, renames and deletes files, so it is a writer like any other:

  • The scan (enumerate .automerge files, group by base name) takes no lock. A storage folder without conflict copies never waits on a writer.
  • Each group that has conflict copies is merged inside its own StorageFileLock.withExclusiveLock. Because the scan ran unlocked, the group is re-checked under the lock: files another resolver already merged and removed are skipped.
  • The primary is written (atomic replace) before any merged copy is deleted, so the data of a copy is on disk in the primary by the time the copy goes away.
  • A copy that cannot be read or merged (corrupt, truncated mid-sync, zero bytes) is left on disk and reported in ResolutionResult.errors; the other copies of the group still merge. A merged copy that cannot be removed is reported the same way.
  • A primary that exists but cannot be read fails the group: it is not changed and its copies are left in place.
  • A group with no primary is still processed. Whether the primary exists is checked under the lock, not taken from the scan, because a writer may have created it in between. If it is missing, the newest readable copy is renamed to the primary’s name (moveItem, which cannot replace an existing file) and the other copies are merged into it. This covers every .automerge file in the folder whose name matches a conflict pattern.

See ConflictResolutionService.

Sync-service-specific notes:

  • iCloud Drive: .nosync suffix on the lock file keeps it from syncing.
  • Dropbox: no per-file exclude. The lock file syncs harmlessly (it’s usually 0 bytes).
  • Google Drive for Desktop: same as Dropbox — harmless noise.
  • Shared/Sources/MinutaShared/Services/AutomergeStorageService.swift — the lock-protected save path, mtime cache, hasExternalChanges().
  • Shared/Sources/MinutaShared/Services/AutomergeStorageRecovery.swift — month states, the recovery document and its merge, UnreadableStorageFile, AutomergeStorageError.
  • Shared/Sources/MinutaShared/Services/StorageFileLock.swift — the flock wrapper.
  • Shared/Sources/MinutaShared/Services/ConflictResolutionService.swift — conflict-copy merge, one lock per group.
  • Shared/Sources/MinutaShared/Services/StorageVersionMarker.swift — the .minuta-version helper.
  • Shared/Sources/MinutaShared/Services/CLIConfigService.swift — the storage-path.txt helper and getpwuid resolution.
  • Minuta/Sources/Services/ExternalChangeMonitor.swift — app-side polling.
  • Minuta/Sources/Services/StorageLocationManager.swift — bookmark persistence and the publishResolvedStoragePath hook.
  • Shared/Sources/MinutaCLI/Runtime/StorageResolver.swift — CLI-side path discovery.