A single registry keyed by a composite (transport, format, category) tuple, mapping directly to a fully constructed pipeline.

• Pro: Only one registry to maintain and no intermediate composition step.
• Con: Combinatorial explosion of registrations and duplicating logic across combinations that share the same Transport or format. Furthermore, it does not reuse the existing per-axis metadata split already established in the Datasource doc.

A single factory function takes the Datasource record and internally branches (via type-switch) on transport, format, and category to construct the pipeline.

• Pro: Simple call site, no registries.
• Con: Can lead to “Factory Bloat” if there are many unique combinations.

• Pro: New transport, format, or category values are added via registration, not by editing core dispatch logic.
• Con: format/category compatibility is enforced only implicitly, at the type-assertion site, not by the type system.

• The orchestrator must resolve transport, format, and category independently from a single datasource record.
• The composed fetcher must satisfy the fetcher role defined in the Synchronisation Subsystem Architecture.
• The category registry’s pipeline factory must return a type satisfying the existing Sync interface.

• Adding a new transport, format, or category must not require modifying the orchestrator’s core composition logic, only registering new entries.
• The type assertion between decoder output and mapper input must fail with a typed, catchable error rather than panicking.

Currently, a mismatched format/category pairing (e.g. a CSV format registered under a category whose mapper expects iCal structs) is only caught at the type assertion inside the pipeline factory, at sync time. Two follow-up validation layers are being considered:

• Registration-time validation: check the (format, category) pair against a known-compatible-pairs table when a datasource record is created or updated, rejecting invalid combinations before the first sync attempt.
• Sync-time validation: ensure the type assertion inside the pipeline factory returns a descriptive error (not a panic), surfaced through the existing per-datasource logging hooks defined in the Datasources doc.

This is deferred to a later revision once real-world format/category combinations are better understood.

Revisit whether a generic Decoder[T] is worthwhile once the number of format implementations grows large enough that the single untyped assertion point becomes a maintenance burden.

Due to the fact already mentioned above, each service will become very complex with lots of different logic for different data types. This may be fine if the list of supported data types is known and the logic could be shared, but considering this is not the case, making this option not suitable.

Given each data type its own fetcher, mapper, and repository, will isolate the edge cases of one data type from the other ones. This means each component could focus on doing one thing well without caring about the other data types, making the implementation in most cases small enough to comprehend. However, this also means every new data type must implement all components, even if some are trivial. But because different data types will require different ways of fetching and mapping, the duplication of this option is minimal. Therefore, this is a better option for the long term.

To keep higher layers simple, an orchestrator service per data type is required to manage the Fetch → Validate → Map → Persist flow and handle edge cases internally.

Since the main loop must execute sync for all data types uniformly, these services need a shared contract. Three options were considered:

• Function callback func(ctx context.Context) error: Lightweight, but loses the ability to attach additional methods (e.g. Name(), IsEnabled()) to the contract later without breaking all call sites.
• Base struct with embedding: Allows shared behaviour, but Go’s composition model means the base struct cannot enforce that subtypes implement core logic, weakening the contract.
• Interface: Explicit contract, zero-cost abstraction, idiomatic Go. New data types are forced to satisfy the contract at compile time, and the registry pattern becomes trivial to implement.

Therefore, an interface is chosen:

type Sync interface{
  Run(ctx context.Context) error
}

That leaves one question left to answer: How to organise these components?

Datasources are described as flat config files on disk.

• Pro: Human-readable, easy to diff in version control.
• Con: No lifecycle management (active/inactive, versioning). Credential handling is insecure by default. Operationally impractical for dynamic addition or removal at runtime.

Datasources are stored as versioned rows in the persistence layer with an encrypted config blob.

• Pro: Supports runtime addition/modification without redeployment. Encryption is enforced at the model layer. Versioning enables non-destructive schema migrations.
• Con: Requires a migration strategy for config schema evolution. Adds a decryption step in the hot path (acceptable given ChaCha20-Poly1305 performance characteristics).

• The system should support at minimum two protocol families: HTTP and Database.
• The configuration must provide enough metadata (e.g., source type, endpoint, credentials, etc.) for a higher-level orchestrator to instantiate the correct pipeline components, as outlined in Synchronisation Subsystem Architecture.
• Each datasource must at least support lifecycle metadata such as refresh interval, scheduling priority, and active status.

• All sensitive configuration fields must be encrypted at rest using ChaCha20-Poly1305.
• Input validation must be performed on all configuration fields to prevent injection attacks (e.g., SQL injection in DB connection strings).
• The architecture must allow for the addition of new source types (e.g., gRPC, File System, WebSockets) with minimal friction, ideally via a registry/factory pattern rather than conditional branching.
• The system should provide hooks for logging sync status, errors, and latency per datasource.

New data types, e.g. finance, health, etc., require a new adapter. The role of this adapter is to persist raw input from a specified datasource to a specific table in DB. Keep in mind, the adapter does not handle how to fetch data from datasouce, since this is the responsibility of a separated layer.

While the alternative, which is allowing runtime-defined transformation logic such as user-supplied SQL expressions evaluated during ingestion, offers better flexibility when integrating with new data types, but introduces an unacceptable security surface. Because the queries could contain anything, this may lead to at worst SQL injection and at best loss of data because of side effects in queries. By following this alternative, the complexity of validation of the runtime-defined logic also increases to make sure it is safe to be executed.

Each adapter takes a raw input, transform this data, and persist to DB. That means, an adapter is composed of two parts:

• Mapper: maps the raw input into a specified domain record
• Repository: handles the interaction between the application and the DB

The analysis subsystem is modelled on three principles: SQL-first, query-centric, and decoupled from ingestion concerns. User interact with DB tables directly through the UI by writing queries, defining transformation views, configuring charts, and composing dashboards.

The system will handle the full lifecycle of raw data into the knowledge base:

• Fetch & Parse: Retrieve and structure raw content.
• Chunk & Embed: Segment content into manageable units. Because notes in Zettelkasten vault are short and atomic, therefore, these files should be embedded whole, while semantic chunking is only triggered above a defined token threshold to avoid splitting an already-atomic idea.
• Upsert: Simultaneously update the vector store and the graph store, with optional detecting and recovering from partial failures.

Kortex will employ a multi-stage retrieval strategy to maximise accuracy:

• Hybrid search: Execute lexical (BM25) and vector search in parallel. Lexical search ensures precision for specific terms and acronyms, while semantic search captures broader intent and nuances.
• Fusion: Results are merged using Reciprocal Rank Fusion. Fusion parameters (rank constant k, per-lane candidate window) are explicit configuration values, not hardcoded defaults, to allow empirical tuning.
• Graph Expansion: Results are expanded along the document graph to capture relational context that vector embeddings alone might overlook. Expansion depth defaults to one hop but is degree-aware: hub notes with very high link counts are capped in how many neighbours they contribute, preventing context-window flooding from a single popular note.
• Re-ranking & Pruning: Final results are re-ranked by cosine similarity and a dedicated reranker (cross-encoder), then pruned to fit the maximum context window.

• Collection management: Users can organise knowledge into distinct collections.
• Inspection: Tools to inspect documents and visualise the underlying graph.
• Synchronisation: A dedicated mechanism to trigger data updates.

• Conversation management: Handle multi-turn dialogues.
• Folder organisation: Provide a hierarchical structure for organising distinct conversation threads.

Relying solely on semantic (vector) retrieval has inherent limitations regarding structural and relational knowledge. Vector embeddings are excellent for topical similarity but often struggle to represent specific network-based relationships. By incorporating a graph structure, Kortex ensures that explicit links and structural dependencies are not lost during the retrieval process.

To achieve a robust retrieval process that satisfies both literal and conceptual intents, Kortex utilises hybrid search:

• Lexical Matching: Highly effective for capturing specific terms, acronyms, and exact identifiers that vector models might "smooth over”.
• Semantic Search: Complements lexical matching by capturing broader intent and conceptual nuances.

Lexical indexing (Inverted Indices) will be leveraged to maintain high throughput and minimal CPU/memory overhead across massive datasets, balancing the higher resource intensity required by high-dimensional vector calculations. Where Applicable, parallel processing will be utilised to reduce end-to-end time by executing independent computations simultaneously.

• Single-user scope: Collections and the graph store operate in a single namespace. No auth/multi-tenancy is required at this stage, but this assumption should be revisited before any future multi-user migration.
• All ingestion paths (Vault, Wikipedia, Blogs) pass through the same trust-filtering and upsert-consistency mechanisms described above, ensuring uniform behaviour regardless of source type.