Refreshes SQLite from one or both archive sources.

By default, sync runs both live/local sources and does not import the Git snapshot first:

• Discord bot-token sync for bot-visible guild data
• local Discord Desktop cache import for classifiable cached messages and proven DMs

Use update when you want to pull/import the shared Git snapshot. Routine imports upsert changed shards without deleting local cache rows. If you intentionally want a sync run to import the snapshot before live deltas, pass --update=auto for the configured stale update mode (merge by default) or --update=force for an exact replacement. --no-update is accepted as an explicit no-op alias for the default.

Run one explicit --full pass when you want a complete historical guild archive. Use plain sync afterward for frequent latest-message and desktop-cache refreshes.

discrawl sync
discrawl sync --update=auto
discrawl sync --update=force
discrawl sync --no-update
discrawl sync --full
discrawl sync --full --all
discrawl sync --guild 123456789012345678
discrawl sync --guilds 123,456 --concurrency 8
discrawl sync --source both      # default: bot API + desktop cache
discrawl sync --source discord   # bot API only; aliases: key, bot, api
discrawl sync --source wiretap   # desktop cache only; aliases: desktop, cache
discrawl sync --guild 123456789012345678 --all-channels
discrawl sync --channels 111,222 --since 2026-03-01T00:00:00Z
discrawl sync --with-embeddings
discrawl sync --with-media

• --source <both|discord|wiretap> - which archive sources to read
• --update <auto|force|never> - apply the configured stale snapshot update mode, force an exact replacement, or skip snapshot import before live deltas
• --full - historical backfill (slow on large guilds)
• --all-channels - broader incremental sweep across every stored channel/thread
• --latest-only - explicit latest-only run (also the default for untargeted sync)
• --all - ignore default_guild_id and fan out across every discovered guild
• --guild <id> / --guilds <id,id> - target specific guilds
• --channels <id,id> - target specific channels (forum ids expand to threads)
• --include-stored-threads - with --source discord --channels, also target stored direct threads of the selected text and announcement channels; does not discover new threads or change ordinary channel selection
• --since <RFC3339> - limit initial history and --full backfill to messages at or after this timestamp
• --concurrency <n> - override worker count (default auto-sized: floor 8, cap 32)
• --skip-members - refresh guild/channel/message data without crawling members
• --with-members - refresh guild members even during the default latest-only sync; fail if the member crawl cannot complete
• --with-embeddings - also enqueue changed messages into embedding_jobs
• --with-media - after sync, download missing attachment media into cache_dir/media

• --latest-only is the default for untargeted sync. Use --all-channels to opt out without doing a full historical crawl.
• --with-media records expired or removed Discord CDN URLs as failed fetches with the HTTP status, commonly 404.
• --with-media updates the local cache only; run publish --push afterward to include cached non-DM media in the Git backup as gzip-compressed files.
• --since does not mark older history as complete, so a later sync --full without --since can continue the backfill.
• --full --since <RFC3339> reads from each channel's current head to that boundary and verifies current history permissions before and after the scan. Successful scans record their [since, until) coverage even when no messages exist in that window. The channel:<id>:history_window:v1:<since> sync-state record binds the guild, channel, native type, UTC bounds, completion time, and terminal pagination boundary. Distinct lower bounds retain separate records; routine latest-only syncs do not extend them. These records do not mark older history complete.
• Voice and stage text chats use the same history path as text channels. Verifying their history requires VIEW_CHANNEL, READ_MESSAGE_HISTORY, and CONNECT.
• Targeted bot syncs (--source discord --channels <ids>) refresh Desktop-imported channel placeholders and their parent metadata from Discord before crawling messages. Unavailable metadata or missing parent or permission evidence fails the sync visibly; ordinary cached channels keep their existing behavior. Ancestor metadata refreshes do not crawl ancestor messages or reset history and archive cursors. This is still a message sync: --since limits initial history and backfill, not fetching new messages after an existing cursor.
• Long runs emit periodic progress logs to stderr.
• Heartbeat logs (message sync waiting) name the oldest active channel and per-channel page activity if in-flight channels stop completing for a while.
• Every run ends with a message sync finished summary.
• Each channel crawl has a bounded runtime budget; pathological channels are deferred and retried next sync.
• Guild and archived-thread pagination reports a cursor error if Discord repeats a page instead of continuing indefinitely.
• Full message pages with missing or repeated cursors stop the channel crawl with an error and preserve its last usable backfill checkpoint.
• Retryable failures and unavailable-channel markers are tracked per channel; stale unavailable markers are cleared after a later successful crawl.
• A channel that fails with missing access carries an unavailable marker, and routine syncs pass over it while the marker is inside its seven-day window. The next routine sync after the marker reaches seven days retries the channel. To retry sooner: sync --guild <id> --full attempts every channel in the guild immediately, marked ones included, and sync --channels <id> retries a single channel immediately. doctor reports unavailable_markers_active (inside the window, so currently passed over) and unavailable_markers_expired (due for another attempt).
• Full sync retries marked channels even when their older history is complete. When only unavailable channels remain in the cached backlog, it also discovers new channels; the retry and discovery phases do not request the same channel twice.
• Marker cleanup is best-effort, so one missing local sync-state row cannot crash the run.
• Member refresh is best-effort and gives up after five minutes without a caller-supplied deadline. Routine latest-only syncs skip it unless --with-members is set.
• Routine refreshes keep a per-parent archived-thread cursor, so they discover threads archived between runs without rescanning the historical thread catalog.
• When the archive is already complete, sync --full reuses backlog markers and the same incremental thread discovery instead of revisiting every stored archived thread.
• Completed channels with no stored messages are verified with a full crawl when Discord still reports a last message. Routine sync skips channels verified empty; --full rechecks them. A --since window does not initiate this recovery.
• Interrupted recovery resumes from its own saved page checkpoint on the next unwindowed sync, including after cancellation or process termination. Messages ingested in the meantime do not mark the older history complete.
• Failed recovery leaves partial history resumable. After cancellation, restoring a still-empty channel's completion marker uses the existing five-second failure-cleanup budget.

Discrawl versions before Components V2 support stored messages sent with the IS_COMPONENTS_V2 flag (32768), such as app-posted GitHub cards, without their component tree or text. This affects both bot sync and Desktop cache imports. Those rows cannot be rebuilt locally, and routine runs don't re-read them: sync, --all-channels, --full and --channels only request messages after each channel's stored cursor, and wiretap skips cache files that haven't changed. Repair them with the fixed binary:

0. Back up the archive. sqlite3 .backup takes a consistent copy even while a writer is running:

   DB="$(discrawl status | sed -n 's/^db=//p')"
   sqlite3 "$DB" ".backup '$DB.before-components-v2'"

   To roll back later, stop every sync/tail, then run cp "$DB.before-components-v2" "$DB" && rm -f "$DB-wal" "$DB-shm".

1. Stop every sync, tail and scheduled sync that writes to this archive.

2. List the affected channels, their broken-row counts and oldest broken message ids, and save the channel ids:

   BROKEN="deleted_at is null and json_extract(raw_json, '\$.flags') & 32768 and json_type(raw_json, '\$.components') is null"
   sqlite3 -header "$DB" "select channel_id, count(*) as broken, min(cast(id as integer)) as oldest_id from messages where $BROKEN group by channel_id"
   CHANNELS="$(sqlite3 "$DB" "select group_concat(distinct channel_id) from messages where $BROKEN")"

3. Rewind each affected channel's latest_message_id cursor to just before its oldest broken message:

   sqlite3 "$DB" "
     update sync_state
        set cursor = (select cast(min(cast(id as integer)) - 1 as text) from messages
                       where $BROKEN and sync_state.scope = 'channel:' || channel_id || ':latest_message_id')
      where scope in (select distinct 'channel:' || channel_id || ':latest_message_id' from messages where $BROKEN)"

4. Re-fetch those channels. The crawl moves forward from the rewound cursor and rewrites every message it receives, including raw_json, normalized_content and the search indexes. When it finishes, the cursor is back at the channel head:

   discrawl sync --source discord --channels "$CHANNELS"

5. Verify. The broken-row count should be 0, and a search for the text of a known card should find it:

   sqlite3 "$DB" "select count(*) from messages where $BROKEN"
   discrawl search "<text from a known card>"

   Any rows that remain are messages Discord no longer returns, for example because they were deleted. List them with the step 2 query.

Desktop imports keep a sanitized raw_json without flags or components, so broken Desktop rows can't be counted exactly. The closest check is Desktop rows that have no text and no attachments. It also matches messages that are genuinely empty, such as sticker-only messages.

6. Count the candidates:

   DESKTOP="deleted_at is null and json_extract(raw_json, '\$.source') = 'discord_desktop' and trim(coalesce(normalized_content, '')) = '' and has_attachments = 0"
   sqlite3 "$DB" "select count(*) from messages where $DESKTOP"

7. Clear the Desktop file index so the next import rescans every cache file instead of skipping unchanged ones, then re-import:

   sqlite3 "$DB" "delete from sync_state where scope like 'wiretap:file_index:%'"
   discrawl wiretap

8. Verify. The candidate count should drop by the number of recovered messages, and a search for the text of a known card should find it:

   sqlite3 "$DB" "select count(*) from messages where $DESKTOP"
   discrawl search "<text from a known card>"

This only recovers messages that Discord Desktop still has cached. If a remaining candidate is in a guild channel the bot can read, repeat steps 2-5 with BROKEN="$DESKTOP and guild_id <> '@me'". Keep only those channels in CHANNELS, because a targeted sync fails on channels the bot cannot read. The rewind and --channels sync then fetch those messages through the bot.

9. Publish the repaired archive:

   discrawl publish --push

• Sync sources
• tail
• update