Synchronisation Subsystem Architecture

Overview

This document outlines the architectural design for the synchronisation subsystem within the data ingestion pipeline. The primary objective is to fetch raw data from external sources, transform it into domain records, and persist it into the database.

The synchronisation subsystem needs to interact with different data types or more specifically different external systems. This means the raw data may come via REST API, or gRPC, or simply a database with the potential of different authorisation and methods.

Data Pipeline

One of the major challenges when fetching data from multiple data sources for the same data type is that the raw input may have slightly different formats. This increases the friction in the analysis layer when interacting with these data. This means each data type should have its own “universal” structure, so it removes the “where do the data come from” in subsequent layers. This approach does have some friction when integrating new data sources, but it is still better to pay a small price in one single place rather than along the way up to the visualisation. Therefore, the data flow in syncrhonisation subsytem becomes: Fetch → Validate → Map → Persist.

System Decomposition

Based on the data flow, there are three primary components in the pipeline:

  • Fetcher: Responsible for fetching raw data.
  • Mapper: Transforms raw input from the fetcher into specified domain records.
  • Repository: Manages the interaction between the application domain and the database.
  • Service: Orchestrates the flow between the above components (derived below).

However, there are several choices of how to organise these for easy maintainability and extendability:

One global service for each component

Due to the fact already mentioned above, each service will become very complex with lots of different logic for different data types. This may be fine if the list of supported data types is known and the logic could be shared, but considering this is not the case, making this option not suitable.

Each data type has its onw components

Given each data type its own fetcher, mapper, and repository, will isolate the edge cases of one data type from the other ones. This means each component could focus on doing one thing well without caring about the other data types, making the implementation in most cases small enough to comprehend. However, this also means every new data type must implement all components, even if some are trivial. But because different data types will require different ways of fetching and mapping, the duplication of this option is minimal. Therefore, this is a better option for the long term.

To keep higher layers simple, an orchestrator service per data type is required to manage the Fetch → Validate → Map → Persist flow and handle edge cases internally.

Since the main loop must execute sync for all data types uniformly, these services need a shared contract. Three options were considered:

  • Function callback func(ctx context.Context) error: Lightweight, but loses the ability to attach additional methods (e.g. Name(), IsEnabled()) to the contract later without breaking all call sites.
  • Base struct with embedding: Allows shared behaviour, but Go’s composition model means the base struct cannot enforce that subtypes implement core logic, weakening the contract.
  • Interface: Explicit contract, zero-cost abstraction, idiomatic Go. New data types are forced to satisfy the contract at compile time, and the registry pattern becomes trivial to implement.

Therefore, an interface is chosen: 

type Sync interface{
  Run(ctx context.Context) error
}

That leaves one question left to answer: How to organise these components?

Option 1: Per role

That means all fetchers in the same subpackage, as well as all mappers and repositories in their own subpackages. This approach makes it easy to discover all code sharing the same responsibility in one place, but also makes the code for one data type scatter around the code base, making it hard to discover and maintain. For that reason, this option is rejected.

Option 2: Per data type

Each data type is its own package and contains all four components: fetcher, mapper, repository, and service. This approach follows the idea of “what uses together, comes together”. This makes discoverability of all related code for one specific data type easy, while not preventing share logic between components between data types. Therefore, this option is chosen.