External-source connectors

​

Shared base

def iter_records(self, cursor: dict) -> Iterator[tuple[ConnectorRecord, dict]]:
    """Yield (record, advanced_cursor) pairs."""

• Deterministic identity — kh-id is kh-<26-char Crockford32> derived from a stable source_id (SHA-256, first 16 bytes encoded). Same upstream record always yields the same id, so updates rewrite the same chunk rather than spawning duplicates.
• Idempotent writes — when an existing file’s canonical frontmatter + body match the new render, the write is skipped entirely. Existing id and created_at are preserved across rewrites.
• Cursor persistence — the Postgres connector_state table (one row per connector) stores a JSON cursor so the next run picks up where this one left off.
• Schema-compliant frontmatter — every emitted document validates against knowledge-hub/.kb/schema/frontmatter.schema.json (required fields, sensitivity enum, source_hash format).

​

Trigger model

1. Init container clones the curated hub into an emptyDir (same pattern as cerebrum-hub-ingest).
2. Main container runs python -m ingest.run_connector --connector <name>.
3. The connector writes new / changed markdown into the emptyDir under sources/<service>/....
4. The same entrypoint immediately calls ingest_hub.reindex_paths() on the changed files, so Qdrant has the new chunks before the pod exits.
5. Pod exits, emptyDir wiped. Durable state lives in Postgres (cursor) and Qdrant (content).

​

Notion connector

• POST /v1/search with filter={object:page} sorted descending by last_edited_time.
• For each page newer than the cursor: GET /v1/blocks/{id}/children recursively to build the block tree.

• “What did we decide about retention in the Atlas spec?” — product docs in Notion.
• Vendor and project runbooks that don’t get migrated to the curated hub.
• OKRs, weekly reviews, meeting notes.

​

Slack connector

1. conversations.history with oldest=<cursor>, paginated via next_cursor. Returns messages descending by ts; the connector sorts ascending before yielding.
2. For each message where ts == thread_ts AND reply_count > 0: conversations.replies to fetch the thread tail. Replies already in history are deduplicated by ts.

1. User mentions <@U001> → @<display-name>. Resolved lazily via users.info, cached for the run. Unknown users render as @U001.
2. Channel mentions <#C111|design> → #design. Falls back to #C111 if Slack didn’t include the name.
3. Links <https://x.test|label> → [label](https://x.test); bare <https://x.test> → https://x.test.
4. html.unescape collapses &, <, etc.

**@user** (HH:MM UTC) — [permalink](https://workspace/archives/CXX/pTSTS)

<message text after transforms>

- 📎 `filename.png`

_Reactions: :rocket: ×3, :fire: ×1_

• “Why did we pick UUID v7 for the workflows table?” — the decision conversation in #general.
• Async standup context: who’s blocked on what, surfaced when an agent picks up a half-finished task.
• For agents: cross-referencing diary entries with the Slack thread that triggered them — same task_id tag.

​

GitHub connector

• For each repo in the allowlist: git clone --depth 1 --branch <branch> --single-branch into a temp dir (auth via token-in-URL when configured).
• git rev-parse HEAD → current commit SHA.
• If cursor[<owner/repo>] == sha: skip the whole repo. Repos that haven’t moved cost one clone + one rev-parse.

1. Compute repo-relative posix path.
2. Match against exclude_globs → skip if any match.
3. Match against include_globs → skip if none match.
4. stat().st_size > max_file_bytes (default 256 KB) → skip.
5. read_text(encoding="utf-8") — UnicodeDecodeError is treated as binary and skipped.
6. Emit a record.

# <owner>/<repo>:<path>

Source: [<path>](https://github.com/<owner>/<repo>/blob/<sha>/<path>)
Commit: `<sha-prefix>`. Branch: `<branch>`.

```<language>
<file contents verbatim>

Language tag is inferred from the file extension via a table covering ~30 common languages plus filename-keyed matches for `Dockerfile` and `Makefile`. Unknown extensions get an empty fence (still a valid markdown code block).

**Per-repo policy** (`GitHubRepoConfig`)

```json
{
  "owner": "gdi-labs",
  "repo": "cerebrum",
  "branch": "master",
  "include_globs": ["**/*.py", "**/*.rs", "**/*.ts", "**/*.tsx", "**/*.md"],
  "exclude_globs": [".git/**", "**/node_modules/**", "**/target/**", "**/.venv/**"],
  "sensitivity": "internal",
  "tags": ["self"]
}

• “Find the function that resolves Mother AI URLs.” — literal-jargon queries where dense embeddings blur token boundaries.
• “Where do we set the Qdrant collection name?” — config-string lookup across repos.
• Cross-repo references: ingest cerebrum and cerebrum-templates together so workers can reason about templates while looking at how they get applied.
• A motivating workload for the hybrid retrieval (BM25 + dense + RRF) work tracked next: identifier-style code tokens match literally, which is exactly where pure dense embeddings underperform.

​

Configuration

notion_enabled        = true
notion_schedule       = "0 */2 * * *"
notion_auth_secret_id = "cerebrum/connectors/notion"
notion_workspace_slug = "gdi-labs"
notion_sensitivity    = "internal"

slack_connector_enabled  = true
slack_connector_schedule = "*/30 * * * *"
slack_auth_secret_id     = "cerebrum/connectors/slack"
slack_workspace_host     = "gdi-labs.slack.com"
slack_channels_json      = "[{\"id\":\"C0\",\"name\":\"general\",\"scope\":\"sources/slack/general\",\"sensitivity\":\"internal\",\"tags\":[\"company-wide\"]}]"

github_connector_enabled        = true
github_connector_schedule       = "0 */6 * * *"
github_connector_auth_secret_id = "cerebrum/connectors/github"
github_connector_repos_json     = "[{\"owner\":\"gdi-labs\",\"repo\":\"cerebrum\",\"branch\":\"master\", ...}]"

​

Manual trigger

kubectl -n cerebrum create job --from=cronjob/cerebrum-notion-ingest notion-once
kubectl -n cerebrum logs job/notion-once -f

​

Invariants

• Source-system credentials never leave AWS Secrets Manager. The Python entrypoint resolves them at startup; no token lives in a manifest or env-fallback in production.
• Cursor + Qdrant carry the durable state. A connector pod’s emptyDir is wiped on exit; this is intentional. The cursor in Postgres prevents re-fetching upstream; Qdrant payload carries chunk content.
• Sensitivity is per-record, not per-connector. A Slack channel can be partner; a GitHub repo can be internal; a Notion page tier is the connector default. Retrieval-time tier enforcement (when it lands) filters on this.
• Idempotent writes. The same upstream record on a re-run produces the same kh-id and skips the write when content matches. Reseeding a connector is safe.