Developer Guide
Doppelgänger is mainly a Rust based project, with code located in the same repository. It uses a workspace layout, splitting up the different component into different crates. The following sections should give you an overview and enable you to navigate the project.
Basic concepts & architecture
In the Overview section, you will find chapters about the Concepts and High Level Architecture. Be sure to get a basic understanding of Doppelgaenger before reading this document.
Internal Architecture
The illustration in the High Level Architecture section showed the main components of the default deployment. However, internally Doppelgaenger is a bit more modular, which allows for other deployment models, such as the "single server binary".
Most of the internal components are located in the core crate. The other projects only set up a suitable process and
container image.
- Processor
-
The processor simply consumes events from the Kafka stream and executes the commands using the "Service". It will perform the acknowledgement handling with Kafka and re-try in cases of failure of broken optimistic locks.
- Service
-
The main service, taking care of processing requests. It will load the current state from the "Storage", apply the required operation, and then use the "Machine" to reconcile the state. If that was successful, it will[1]:
-
Persist the new state
-
Send internal events
-
Send commands
-
Send change notifications (if the thing was changed)
-
- Listener
-
The listener which receive change events from the "notifications" topic, and send out notifications to its listeners. The listener will also handle things like sending initial state or differential send.
- Management
-
Execute management operations on the things, as they get requested through the REST API. The process isn’t that different from the Processor’s. However, the REST API provides a different API, suitable for external consumption.
Assumptions
Doppelgaenger makes a few assumptions:
Dominance of ingress traffic
It is expected that the traffic flowing into the system is much more than the traffic flowing out of Doppelgaenger.
This is a typical IoT scenario, where most data (telemetry) gets sent by devices towards the cloud.
Kafka key based on thing ID
Kafka can guarantee the order of events only for a key. Therefore, the (Kafka) key for events must be ID of the thing.
This guarantees that all events for a thing are processed in the correct order, and also makes breaking the optimistic lock during a thing update much less likely.
Doppelgaenger holds an optimistic lock on things while processing events or backend requests. When the lock breaks, it will re-try. However, this costs additional resources and should be avoided.
Still, it cannot be avoided completely when using the backend API for mutating requests.
Noteworthy
Traits and implementations
Currently, Doppelgaenger uses Rust "traits" for abstracting from actual implementations. However, it is currently not possible to swap out implementations during runtime.
It would be possible to create a new "Storage" implementation based on a database which is not PostgreSQL. However, it currently would not be possible to choose an implementation during runtime. It has to be done during compilation and required some small changes in the code.