Datasources

Overview

This document outlines the design and requirements for the datasource management within the synchronisation subsystem. A Datasource is a configuration entity that describes “where” and “how” of raw data ingestion. The core idea is to treat datasources as configuration-driven entities, allowing the synchronisation engine to dynamically assemble pipelines based on provided metadata rather than hard-coded logic. The goal of datasource is to provide enough information and metadata for higher layer to construct correct pipeline.

Context

The synchronisation subsystem must ingest raw data from various external sources (HTTP APIs, relational databases, and prospectively gRPC services, file systems, and WebSockets). Without a unified abstraction, each new source type would require changes to the core engine, creating a maintenance burden. The datasource entity solves this by encapsulating all source-specific configuration behind a unified interface, so the orchestrator can drive behaviour purely from metadata. Furthermore, the datasource component is responsible solely for acquisition and initial mapping, as it does not handle business logic, data transformation for analysis, or long-term storage optimisation.

Decision

Datasource is a first-class, versioned configuration entity stored in the persistence layer (as per Data Pipeline Persistence Design). The orchestrator reads datasource records at runtime to instantiate the appropriate pipeline. components, as described in the Synchronisation Subsystem Architecture. Furthermore, the configuration blob should be encrypted using ChaCha20-Poly1305, because it may contain sensitive fields.

Alternatives Considered

Plain-text configuration files (e.g., YAML/TOML)

Datasources are described as flat config files on disk.

  • Pro: Human-readable, easy to diff in version control.
  • Con: No lifecycle management (active/inactive, versioning). Credential handling is insecure by default. Operationally impractical for dynamic addition or removal at runtime.

Chosen: Versioned, encrypted database records

Datasources are stored as versioned rows in the persistence layer with an encrypted config blob.

  • Pro: Supports runtime addition/modification without redeployment. Encryption is enforced at the model layer. Versioning enables non-destructive schema migrations.
  • Con: Requires a migration strategy for config schema evolution. Adds a decryption step in the hot path (acceptable given ChaCha20-Poly1305 performance characteristics).

Requirements

Functional Requirements

  • The system should support at minimum two protocol families: HTTP and Database.
  • The configuration must provide enough metadata (e.g., source type, endpoint, credentials, etc.) for a higher-level orchestrator to instantiate the correct pipeline components, as outlined in Synchronisation Subsystem Architecture.
  • Each datasource must at least support lifecycle metadata such as refresh interval, scheduling priority, and active status.

Non-Functional Requirements

  • All sensitive configuration fields must be encrypted at rest using ChaCha20-Poly1305.
  • Input validation must be performed on all configuration fields to prevent injection attacks (e.g., SQL injection in DB connection strings).
  • The architecture must allow for the addition of new source types (e.g., gRPC, File System, WebSockets) with minimal friction, ideally via a registry/factory pattern rather than conditional branching.
  • The system should provide hooks for logging sync status, errors, and latency per datasource.

Data Model

The datasource entity should consist of:

  • ID: Unique identifier
  • Version: Schema version for the configuration blob (to handle migrations)
  • Name: Human-readable name
  • Transport: Type of transport, e.g. HTTP, Database, etc.
  • Format: Payload decoding, e.g. iCal, JSON, CSV, native SQL rows, etc.
  • Category: Synchronisation pipeline type, e.g. timeline, finance, etc.
  • Config: Encrypted blob containing connection details
  • Sync policy: Metadata regarding frequency and priority
    • Interval: Duration (e.g., 5m, 1h).
    • Priority: Integer for scheduling order.
    • Active: Boolean flag.

Future Considerations

  • Rate limiting: For HTTP sources, implementing per-datasource rate limiting to respect provider constraints.
  • Circuit Breaker: Implementing a circuit breaker to prevent failing datasources from exhausting system resources.
  • Partial Sync/Checkpointing: Store “last synced” offsets (e.g., timestamps or IDs) to allow resuming interrupted syncs.