DNC DE NORONHA CONSULTING

Blog / AI · · 14 min

How to Make Your Second Brain Dream at Night: The Setup Guide

A step-by-step guide to a nightly pipeline that turns your Obsidian notes into a wiki that consolidates itself and forgets what no longer matters.

The four nightly steps convert, ingest, consolidate and forget

In my previous post I described how my second brain “dreams” at night: during the day I only collect, and at night Claude organizes everything into a wiki. That wiki consolidates itself and forgets what no longer matters. In this post I explain, step by step, how to set it up yourself.

What you will build

At the end you have a single Obsidian vault with a fixed folder structure. Every night a script runs four steps: converting documents to Markdown, processing new sources into wiki pages, consolidating the wiki, and giving pages a temperature (hot, warm or cold).

What you need

You don’t need to be a programmer, but you do need to be comfortable opening a terminal.

A warning up front: you are giving an AI write access to your notes at night. Only do that with Git as a safety net (step 3), and start with a test on a copy.

Step 1: one vault, a fixed structure

I started with five separate vaults and went back to one. One vault means one place to sync, one Git repository and one nightly job. Links between clients and your own company simply work, too.

Create the vault in your home folder, so not in iCloud Drive or Documents. The structure:

~/Obsidian/
├── CLAUDE.md              # the rules for Claude (step 4)
├── scripts/               # nightly script, exporter, decay
├── Client-A/
│   ├── raw/               # new sources, unsorted
│   ├── meetings/
│   ├── notes/
│   ├── docs/              # Word, PDF, Excel, PowerPoint
│   └── _wiki/             # managed by Claude
│       ├── index.md
│       ├── log.md
│       └── _cold/
├── Client-B/              # same structure
├── Company/
└── Personal/

The most important rule: everything outside _wiki/ is yours, everything inside _wiki/ is Claude’s. You never write in the wiki, Claude never writes in your sources. That separation is what makes it safe to run at night.

Step 2: Obsidian Sync instead of iCloud

It is tempting to just put your vault in iCloud Drive. For a vault that scripts edit at night, I advise against it:

Obsidian Sync doesn’t have these problems. Files always stay local, the .git folder is not synced, and it is end-to-end encrypted.

How to set it up:

  1. Open the vault on the Mac that runs at night, and in Obsidian choose Settings → Sync → Create new remote vault.
  2. Under Selective sync, choose which file types to include. Turn on PDFs and Office files if you want to see them on your phone too.
  3. Connect your other devices to the same remote vault.

Run the nightly jobs on one machine, not several. That one machine is the source of truth; the others follow along through Sync.

Step 3: Git as a safety net

Git is what makes this safe. Every nightly change becomes a commit, so in the morning you can see exactly what Claude did, and roll everything back.

cd ~/Obsidian
git init
cat > .gitignore << 'EOF'
.obsidian/workspace*.json
.DS_Store
.trash/
.nightly.lock
EOF
git add -A
git commit -m "Starting point"

Want an off-site backup? Push the repository to a private remote (GitHub, Azure DevOps or your own NAS). Note: it contains client data, so never make it public.

Step 4: the rules in CLAUDE.md

Claude Code automatically reads the CLAUDE.md file in the folder where it starts. That is where you define the rules. It keeps your nightly prompts short, and Claude behaves the same way every night.

Create ~/Obsidian/CLAUDE.md with roughly this content:

# Rules for this vault

## Ownership
- Everything outside a _wiki/ folder belongs to the user. Read it, NEVER change it.
- You only write in <subfolder>/_wiki/.
- A wiki stays within its own subfolder. Refer to other subfolders only with a wikilink.

## Wiki pages
- One page per subject: person, organization, system, decision or concept.
- File name = subject in kebab-case, e.g. john-smith.md.
- Refer to other pages with [[wikilinks]].
- Every fact gets a source reference: (source: [[path/to/source]]).

## Frontmatter of every wiki page
---
type: person | organization | system | decision | concept
aliases: []
sources: []          # wikilinks to the sources
last_referenced:     # date of the newest source that refers to this page
temperature: hot     # hot | warm | cold, set by the decay script
pin:                 # optional: hot | never-cold | cold
---

## Log
- Every run appends a short report with the date to the bottom of _wiki/log.md.
- In doubt? Change nothing, but list it under "To review" in log.md.

The fields last_referenced, temperature and pin are needed for the decay step, so define them from the start.

Step 5: converting documents to Markdown

Claude reads Markdown quickly and cheaply. Word, PDF, PowerPoint and Excel are slower and more expensive. That is why a small script first converts every document to a Markdown file next to it, for example proposal.docx → proposal.docx.md.

I use MarkItDown for this, an open-source tool from Microsoft:

pip3 install 'markitdown[all]'

Save this as ~/Obsidian/scripts/export_md.py:

import hashlib, pathlib, sys
from markitdown import MarkItDown

VAULT = pathlib.Path.home() / "Obsidian"
TYPES = {".docx", ".pdf", ".pptx", ".xlsx"}
md = MarkItDown()

def sha(p):
    return hashlib.sha256(p.read_bytes()).hexdigest()

for src in VAULT.rglob("*"):
    if src.suffix.lower() not in TYPES or "_wiki" in src.parts:
        continue
    out = src.with_name(src.name + ".md")
    h = sha(src)
    if out.exists() and f"sha256: {h}" in out.read_text()[:300]:
        continue  # unchanged, skip
    try:
        text = md.convert(str(src)).text_content
    except Exception as e:
        print(f"ERROR {src}: {e}", file=sys.stderr)
        continue
    out.write_text(f"---\nsource: \"{src.name}\"\nsha256: {h}\n---\n\n{text}")
    print(f"converted: {src.relative_to(VAULT)}")

The hash makes sure only new or changed files are converted again. The first run takes a while; after that it is usually a handful of files per night.

Scanned PDFs without a text layer produce an empty file. Run those through OCR first, for example with ocrmypdf.

Step 6: pulling in email through a connector

Much of what happens lives in your email. Claude can read it through an MCP connector. I use one for Microsoft 365; similar connectors exist for Gmail.

Add the connector to Claude Code on the Mac that runs at night. The exact command is in your connector’s documentation, but usually looks like this:

claude mcp add --transport http --scope user mail <connector-url>

Then start claude interactively once and run /mcp to sign in. A nightly job can’t do that by itself.

In CLAUDE.md, define which email belongs in which subfolder, for example by sender domain:

## Mail routing
- @client-a.com → Client-A/raw/mail/
- @client-b.com → Client-B/raw/mail/
- other business mail → Company/raw/mail/
- newsletters and notifications: skip

The mail step writes the emails of the last 24 hours as a short Markdown summary into the right raw/mail/ folder. The ingest then picks them up like any other source.

Note: this puts email content and client data into your vault. Check that this fits your agreements with your clients.

Step 7: the prompts

The nightly job runs three prompts: mail, ingest and consolidation. Save them as text files in ~/Obsidian/scripts/prompts/. That way you can tweak them without changing the script. They can stay short, because the rules are already in CLAUDE.md.

mail.md

Using the mail connector, fetch all emails from the last 24 hours.
Route them according to "Mail routing" in CLAUDE.md.
Per subfolder, write one file raw/mail/YYYY-MM-DD.md with for each email:
sender, subject, date and a summary of at most 5 lines.
Copy decisions, agreements and deadlines verbatim.
Skip newsletters and notifications.

ingest.md

Run the ingest for subfolder {SUBFOLDER}.

1. Find all files outside _wiki/ that changed since the last run in
   {SUBFOLDER}/_wiki/log.md. For Office and PDF files: read the .md
   version next to them.
2. Extract the subjects from each source: people, organizations, systems,
   decisions and concepts.
3. Does a wiki page already exist (check aliases)? Extend it.
   If not: create a new page according to CLAUDE.md.
4. Add a source reference to every fact. Add [[wikilinks]] between pages.
5. Set last_referenced to the source's date if it is newer.
6. Update {SUBFOLDER}/_wiki/index.md.
7. Write a report in log.md: sources processed, pages new and updated.

consolidate.md

Run the consolidation for {SUBFOLDER}/_wiki/.

1. Find pages about the same subject (same person, system or concept;
   check aliases and spelling). Merge them into the page with the most
   incoming links. Put the other names in aliases. Turn the old page into
   a short redirect to the new one.
2. Find contradicting facts. Keep the value from the newest source.
   Put the old value below it as history:
   "(until YYYY-MM-DD: <old>, source [[...]])".
3. Replace relative dates ("next week") with absolute dates, counted from
   the date of the source.
4. Repair wikilinks that point nowhere, including links to pages in _cold/.
5. In doubt? Change nothing; list it under "To review" in log.md.
6. Write a report in log.md.

The nightly script fills in {SUBFOLDER} (step 9). That way the same prompt runs separately for each client, and every wiki stays within its own folder.

Step 8: forgetting with hot, warm and cold

I deliberately don’t let Claude do the decay step, but a plain Python script. It is pure arithmetic on dates, and you want that to go exactly the same way every night.

The rules:

First install pip3 install pyyaml, then save this as ~/Obsidian/scripts/decay.py:

import datetime as dt, pathlib, re, yaml

VAULT = pathlib.Path.home() / "Obsidian"
TODAY = dt.date.today()
FM = re.compile(r"^---\n(.*?)\n---\n", re.S)

def temperature(meta):
    pin = meta.get("pin")
    if pin in ("hot", "cold"):
        return pin
    last = meta.get("last_referenced")
    age = (TODAY - last).days if isinstance(last, dt.date) else 999
    t = "hot" if age < 30 else "warm" if age < 180 else "cold"
    return "warm" if (t == "cold" and pin == "never-cold") else t

count = {"hot": 0, "warm": 0, "cold": 0}
for wiki in VAULT.glob("*/_wiki"):
    cold_dir = wiki / "_cold"
    cold_dir.mkdir(exist_ok=True)
    for page in list(wiki.glob("*.md")) + list(cold_dir.glob("*.md")):
        if page.name in ("index.md", "log.md"):
            continue
        text = page.read_text()
        m = FM.match(text)
        if not m:
            continue
        meta = yaml.safe_load(m.group(1)) or {}
        t = temperature(meta)
        count[t] += 1
        if meta.get("temperature") != t:
            meta["temperature"] = t
            fm = yaml.safe_dump(meta, allow_unicode=True, sort_keys=False)
            text = f"---\n{fm}---\n" + text[m.end():]
            page.write_text(text)
        target = (cold_dir if t == "cold" else wiki) / page.name
        if target != page:
            page.rename(target)

print(f"decay: {count}")

Because Obsidian resolves wikilinks by file name, links to a page in _cold/ keep working. A cold page is not gone: it is searchable and in Git. It just no longer counts towards what Claude reads first.

Want Claude to ignore cold pages entirely when answering questions? Add this to CLAUDE.md: “Only read _cold/ when the user explicitly asks for it or when a hot page links to it.”

Step 9: the nightly script and the scheduler

Now everything comes together in one script that runs the steps in the right order:

  1. Convert to Markdown
  2. Fetch email
  3. Ingest, per subfolder
  4. Consolidate, per subfolder
  5. Decay

Decay is deliberately last, so it judges the cleaned-up version of the wiki.

The nightly script

Save this as ~/Obsidian/scripts/nightly.sh:

#!/bin/zsh
set -u
VAULT="$HOME/Obsidian"
P="$VAULT/scripts/prompts"
CLAUDE="$HOME/.local/bin/claude"   # check with: which claude
SUBFOLDERS=(Client-A Client-B Company Personal)
LOCK="$VAULT/.nightly.lock"
LOG="$VAULT/scripts/nightly.log"

cd "$VAULT" || exit 1
exec >> "$LOG" 2>&1
echo "=== $(date) ==="

# Never run twice at the same time
if [ -e "$LOCK" ]; then echo "lock exists, stopping"; exit 0; fi
touch "$LOCK"; trap 'rm -f "$LOCK"' EXIT

commit() { git add -A && git commit -qm "nightly: $1" || true; }
run() {   # $1 = prompt file, $2 = subfolder
  prompt=$(sed "s/{SUBFOLDER}/$2/g" "$P/$1")
  "$CLAUDE" -p "$prompt" \
    --permission-mode acceptEdits \
    --allowedTools "Read,Write,Edit,Glob,Grep,mcp__mail"
}

commit "before the run"

/usr/bin/python3 scripts/export_md.py;  commit "convert"
run mail.md "";                          commit "mail"
for s in $SUBFOLDERS; do run ingest.md "$s";      commit "ingest $s"; done
for s in $SUBFOLDERS; do run consolidate.md "$s"; commit "consolidate $s"; done
/usr/bin/python3 scripts/decay.py;      commit "decay"

echo "done"

Make it executable with chmod +x scripts/nightly.sh and run it by hand once first. Then use git log and git diff HEAD~5 to see what happened.

Every step gets its own commit. If the consolidation for one client goes wrong, you roll back exactly that one commit with git revert.

The scheduler

I use cron. Open your crontab with crontab -e and add:

0 22 * * * /Users/<your-name>/Obsidian/scripts/nightly.sh

Three things that often go wrong on a Mac:

Prefer something that catches up on a missed run after sleep? Use launchd with a .plist in ~/Library/LaunchAgents/. For a Mac that is always on, cron is simpler.

Step 10: checking in the morning

For the first two weeks I take five minutes every morning:

If it goes well for a few weeks, it becomes a weekly glance. Adjust the prompts when you keep seeing the same mistake. That is the beauty of separate prompt files: you improve the system without changing code.

Wrapping up

It sounds like a lot of work, but it is a few small scripts and three prompts. What delivered the most for me was the separation: collect by day, process by night, and everything Claude does in its own folder with Git next to it.

Start small. One subfolder, ingest only, for a week. Then add consolidation, then decay, and finally email. That way you see what each step adds.

#obsidian ·#claude ·#claude-code ·#second-brain ·#automation ·#how-to

← Alle posts