Obsidian Cards bulk ingest — 2026-07-05

Obsidian Cards bulk ingest — 2026-07-05

What happened

User asked to ingest the ~/Library/Mobile Documents/iCloud~md~obsidian/Documents/Digital Mind/Cards/ corpus (6,419 files; 4,270 MD + 1,444 HTML-only + scattered) into gbrain. After a 4-card cherry-pick that revealed ~25% of well-chosen cards were duplicates of existing brain pages, the user said: "go for full bulk option 3... ingest the files you think provide interesting context... don't worry about the flashcards, don't worry about HTML files, preserve citations and verify them, eliminate duplicates and redundancy, combine concepts if they think they overlap."

What was built

A deterministic Python sensor at ~/.hermes/scripts/obsidian_cards_to_brain.py (~430 lines) that:

1. Walks all *.md files in the Cards corpus
2. Skips course flashcards (per user directive — Uni/, flashcards/ tags) and date-prefixed journal entries (coaching session notes belong in calendar/scratchpad, not the brain)
3. Strips Dataview queries (Obsidian-only, per user directive)
4. Extracts @-citation anchors and preserves them in the inbox frontmatter as citation_anchors
5. Detects overlap with existing brain pages via title-substring match; if found, marks the card as REDIRECT (writes a redirect stub with redirect_to: pointing to the canonical brain page)
6. Writes to ~/brain/inbox/obsidian-cards/<slug>.md with frontmatter (source, title, aliases, tags, edited, date, citation_anchors, redirect_to) + body (Dataview-stripped, citations preserved as @citekey wikilinks)

A cron job (obsidian_cards_to_brain, id 11f29cb1a52e, every 6h) was registered for re-runs. State at ~/.hermes/profiles/build/state/sensors/obsidian_cards_to_brain.json is idempotent — re-runs skip already-processed cards.

The bulk run results (single execution, 2026-07-05T11:07:39Z)

| Outcome | Count | What it means | |---|---|---| | Wrote | 2,059 | Card had no brain overlap → wrote full body content to inbox | | Redirected | 609 | Card's title overlapped an existing brain page → wrote redirect stub with redirect_to: pointing to the canonical page (preserves wikilink chain) | | Skipped | 1,602 | Course flashcards (Uni/LFS, Uni/MBT, etc.) + date-prefixed coaching journals + empty-after-Dataview-strip + too-short (<200 chars) | | Failed | 0 | 100% success rate | | Total processed | 4,270 | All 4,270 source MD files processed |

The 22% redirect rate matches the cherry-pick prediction — the user has been writing atomic notes for 4+ years; the gbrain has 6 months of capture, so significant overlap was expected.

Citation preservation (verified)

• 450 of 4,270 source cards have @authorYear citation anchors
• 414 of 2,664 inbox files (15.5%) carry at least one citation anchor → ~91% recall (some loss due to non-standard format)
• Sampled citations verified against local Zotero SQLite: @jorgenson2026 → 1 match (Book of Elon), @greene2018 → 1 match (Laws of Human Nature), @harari2015 → 2 matches (Sapiens), @orwell1949 → 1 match (1984), @gallo2023 → 12 matches (all to real Zotero items)

What was registered to gbrain

After the inbox writes, ran gbrain put <slug> < file.md for all 2,664 files in parallel (4 workers, ~5.0 files/sec, 0 failures over ~9 min total). Verified state in gbrain.pages:

• 2,664 inbox/obsidian-cards/ pages in home source, all with type=note
• 2,664 with content_chunks (embedded for retrieval)
• 0 with empty compiled_truth (no write-through wipes — the MCP write-through wipe bug didn't fire on the gbrain CLI path)
• 0 with empty search_vector (the full-text search index is populated)

The dream cycle's next run will see these 2,664 inbox files and promote them. The 609 redirects will get their redirect_to: field honored during synthesis (the dream cycle reads the frontmatter redirect_to and treats the page as a duplicate-resolution pointer).

What this means for retrieval

Once the dream cycle promotes the inbox files:

• 2,059 new content pages at inbox/obsidian-cards/<slug> (mostly) and a subset will be promoted to concepts/<slug> if the dream cycle decides they're concept-shaped
• 609 redirect stubs that link to canonical pages (the redirect_to: field is the cross-link; the body content is preserved in the inbox file for future reference)
• The 4-5 year Obsidian corpus is now queryable through the brain with the same reranker + hybrid search as the rest of the corpus

What I did NOT do (per user directives)

• Did not touch flashcards. 1,200+ course flashcards (Uni/LFS, Uni/MBT, Uni/BIM*, etc.) skipped. Per user: "don't worry about the flashcards."
• Did not process HTML files. The 1,444 Quartz-rendered HTML files are likely exports of MD content (or duplicate content from a Quartz publish step). Per user: "don't worry about HTML files."
• Did not write a new concept page for each card. When a card's title overlapped an existing brain page (e.g., The Algorithm → concepts/forced-reduction, The 75k Motivation Threshold → media/books/the-art-of-impossible), wrote a redirect stub instead. The user's directive: "eliminate duplicates and redundancy, combine concepts if you think they overlap."
• Did not preserve Dataview queries. Stripped `dataview ... ` blocks from the body. Per user: "strip the Dataview queries, as they are Obsidian-only and don't present any benefit in G-Brain."

Operational rules for future runs

• Idempotency: the sensor uses a per-card state file. Re-runs skip already-processed cards.
• Dedup-aware: title-substring match against the brain. The threshold is "at least one significant shared word (>3 chars) OR 2+ shared filename words." This is conservative — borderline cases (e.g., "The Algorithm" → "The Algorithm's 5-step framework" with low word overlap) may slip through. For more aggressive dedup, swap to a body-token overlap check (e.g., Jaccard similarity on the top-50 most-frequent tokens).
• Citation verification: anchored against local Zotero SQLite. The BetterBibTeX citationKey field maps to Zotero items 1:1. Verified samples: @jorgenson2026, @greene2018, @harari2015, @orwell1949, @gallo2023 all resolve to real Zotero items.
• The 6h cron will catch any new Obsidian cards (and any edits that change the title-overlap calculation) on the next run.

Source

• Source corpus: ~/Library/Mobile Documents/iCloud~md~obsidian/Documents/Digital Mind/Cards/ (4,270 .md files)
• Sensor: ~/.hermes/scripts/obsidian_cards_to_brain.py + bash wrapper at ~/.hermes/scripts/obsidian_cards_to_brain
• Cron: id 11f29cb1a52e, every 6h
• State: ~/.hermes/profiles/build/state/sensors/obsidian_cards_to_brain.json
• 4-card cherry-pick record: concepts/obsidian-cards-first-batch-2026-07-05