Synchronisation Orchestrator

Overview

The Datasources doc outlines three independent metadata axes on each record, e.g. transport, format, and category metadata. The Synchronisation Subsystem Architecture doc defines the per-category pipeline components (fetcher, mapper, repository, and service) but does not specify how a fetcher is assembled from transport- and format-specific pieces. This lack of connection makes the coordination within the subsystem impossible. Therefore, in order to solve this problem, this document aims to design the orchestrator, which is the missing layer that reads a datasource record and wires the two together.

Decision

The orchestrator uses three independent registries, one per metadata axis, composed in sequence:

  • Transport registry: maps transport to a fetch function constructor.
  • Format registry: maps format to a decoder.
  • Category registry: maps category to a pipeline factory that accepts a composed fetcher and returns a wired pipeline.

The transport fetch function and format decoder are composed into a single fetcher, satisfying the fetcher role already defined in the architecture doc.

Alternatives Considered

Single-tier registry

A single registry keyed by a composite (transport, format, category) tuple, mapping directly to a fully constructed pipeline.

  • Pro: Only one registry to maintain and no intermediate composition step.
  • Con: Combinatorial explosion of registrations and duplicating logic across combinations that share the same Transport or format. Furthermore, it does not reuse the existing per-axis metadata split already established in the Datasource doc.

Factory with inline branching

A single factory function takes the Datasource record and internally branches (via type-switch) on transport, format, and category to construct the pipeline.

  • Pro: Simple call site, no registries.
  • Con: Can lead to “Factory Bloat” if there are many unique combinations.

Chosen: Multi-tier registry with thin factory wrapper

  • Pro: New transport, format, or category values are added via registration, not by editing core dispatch logic.
  • Con: format/category compatibility is enforced only implicitly, at the type-assertion site, not by the type system.

Requirements

Functional Requirements

  • The orchestrator must resolve transport, format, and category independently from a single datasource record.
  • The composed fetcher must satisfy the fetcher role defined in the Synchronisation Subsystem Architecture.
  • The category registry’s pipeline factory must return a type satisfying the existing Sync interface.

Non-Functional Requirements

  • Adding a new transport, format, or category must not require modifying the orchestrator’s core composition logic, only registering new entries.
  • The type assertion between decoder output and mapper input must fail with a typed, catchable error rather than panicking.

Future Considerations

Format/Category compatibility validation

Currently, a mismatched format/category pairing (e.g. a CSV format registered under a category whose mapper expects iCal structs) is only caught at the type assertion inside the pipeline factory, at sync time. Two follow-up validation layers are being considered:

  • Registration-time validation: check the (format, category) pair against a known-compatible-pairs table when a datasource record is created or updated, rejecting invalid combinations before the first sync attempt.
  • Sync-time validation: ensure the type assertion inside the pipeline factory returns a descriptive error (not a panic), surfaced through the existing per-datasource logging hooks defined in the Datasources doc.

This is deferred to a later revision once real-world format/category combinations are better understood.

Generic Decoder interface

Revisit whether a generic Decoder[T] is worthwhile once the number of format implementations grows large enough that the single untyped assertion point becomes a maintenance burden.