Architecture and Core Functionality

Overview

Kortex is a knowledge management and retrieval system designed to provide information retrieval from trusted sources. By leveraging a Graph RAG architecture, Kortex prioritises the preservation of document relationships, especially those inherent in the Zettelkasten method over isolated data points.

The initial targeted user is me alone. Trusted sources are: my owner’s Zettelkasten vault (org-roam), Wikipedia, and a curated set of blogs.

Core Goals

  • Retrieval must be strictly anchored in sources explicitly chosen and trusted by the user.
  • Kortex must exploit and preserve the links between documents. This is critical because the value of a Zettelkasten lies in its interconnect rather than individual notes.
  • The system aims for a streamlined experience, providing at minimum a graph-based RAG organised by collections, with optional user-facing features like folder-based chat interfaces.

Functional Requirements

Ingestion Pipeline

The system will handle the full lifecycle of raw data into the knowledge base:

  • Fetch & Parse: Retrieve and structure raw content.
  • Chunk & Embed: Segment content into manageable units. Because notes in Zettelkasten vault are short and atomic, therefore, these files should be embedded whole, while semantic chunking is only triggered above a defined token threshold to avoid splitting an already-atomic idea.
  • Upsert: Simultaneously update the vector store and the graph store, with optional detecting and recovering from partial failures.

Wikipedia Ingestion Strategy

A full English Wikipedia dump exceeds 20+ GB, making naive full ingestion impractical for a single-user local system. Two pure strategies were considered and rejected in favour of a hybrid:

  • On-demand fetch at query time: avoids storage bloat but adds latency to every query touching un-ingested content, and is bound by Wikipedia’s live API rate limits.
  • Pre-pull by category only: bounds the corpus to relevant topics, but raises the hard question of distinguishing “article missing from KB” vs. “article does not exist” without a live lookup.

Adopted hybrid approach:

  • Pre-pull: fetch articles under user-selected categories (recursively, to a configurable depth), building a deterministic seed corpus scoped to actual interest.
  • Existence check at query time: when a query references an article not found in the KB, issue a lightweight metadata call (not a full content fetch) to determine whether the article exists at all.
  • Miss handling: if the article exists but is not yet ingested, the current query is answered using only what is already in the KB. The missing article is queued for asynchronous fetch, parse, chunk, embed, and upsert.
  • Background sync: a scheduled job re-walks the category tree for new or changed pages, batching requests within Wikimedia rate limits, decoupling ingestion latency from query latency entirely.

Retrieval Engine

Kortex will employ a multi-stage retrieval strategy to maximise accuracy:

  • Hybrid search: Execute lexical (BM25) and vector search in parallel. Lexical search ensures precision for specific terms and acronyms, while semantic search captures broader intent and nuances.
  • Fusion: Results are merged using Reciprocal Rank Fusion. Fusion parameters (rank constant k, per-lane candidate window) are explicit configuration values, not hardcoded defaults, to allow empirical tuning.
  • Graph Expansion: Results are expanded along the document graph to capture relational context that vector embeddings alone might overlook. Expansion depth defaults to one hop but is degree-aware: hub notes with very high link counts are capped in how many neighbours they contribute, preventing context-window flooding from a single popular note.
  • Re-ranking & Pruning: Final results are re-ranked by cosine similarity and a dedicated reranker (cross-encoder), then pruned to fit the maximum context window.

Knowledge Management

  • Collection management: Users can organise knowledge into distinct collections.
  • Inspection: Tools to inspect documents and visualise the underlying graph.
  • Synchronisation: A dedicated mechanism to trigger data updates.

Chat Interface

  • Conversation management: Handle multi-turn dialogues.
  • Folder organisation: Provide a hierarchical structure for organising distinct conversation threads.

Technical Rationale and Design Principles

Why Graph RAG?

Relying solely on semantic (vector) retrieval has inherent limitations regarding structural and relational knowledge. Vector embeddings are excellent for topical similarity but often struggle to represent specific network-based relationships. By incorporating a graph structure, Kortex ensures that explicit links and structural dependencies are not lost during the retrieval process.

Hybrid Search Strategy

To achieve a robust retrieval process that satisfies both literal and conceptual intents, Kortex utilises hybrid search:

  • Lexical Matching: Highly effective for capturing specific terms, acronyms, and exact identifiers that vector models might "smooth over”.
  • Semantic Search: Complements lexical matching by capturing broader intent and conceptual nuances.

Performance Considerations

Lexical indexing (Inverted Indices) will be leveraged to maintain high throughput and minimal CPU/memory overhead across massive datasets, balancing the higher resource intensity required by high-dimensional vector calculations. Where Applicable, parallel processing will be utilised to reduce end-to-end time by executing independent computations simultaneously.

Scoping and Consistency Notes

  • Single-user scope: Collections and the graph store operate in a single namespace. No auth/multi-tenancy is required at this stage, but this assumption should be revisited before any future multi-user migration.
  • All ingestion paths (Vault, Wikipedia, Blogs) pass through the same trust-filtering and upsert-consistency mechanisms described above, ensuring uniform behaviour regardless of source type.