Data plane

A core feature of Drogue Cloud is the abstraction of protocols. Typical IoT protocols are MQTT, CoAP, OPC UA, but also HTTP is being used by some devices.

Ideally, your cloud side application only needs to think about implementing one protocol, in order to interact with your devices. However, different IoT protocols provide benefits on the device side. As the devices are more constrained than the cloud side, Drogue Cloud tries to leverage the benefits of IoT protocols, while providing a normalization layer on the cloud side, that makes it easy for your applications to consume the data.

But also on the cloud side, different ways to consume data exists. May that be a direct connection to a Kafka topic, again MQTT, or a simple Web socket connection. Drogue Cloud tries to be integrated as good as possible, and not force you into a specific API.

Drogue Cloud is based on Knative, and internally uses the Cloud Events specification to forward messages. Cloud events on the other side have different mappings to transport channels, such as MQTT, Kafka, HTTP, etc.. And while Cloud Events can also be used on the external interfaces of Drogue Cloud, it isn’t a requirement, and you can use your custom protocols as you need.

Devices communicate with the Protocol endpoints, which implement the protocol specific mapping between the actual protocol and the internal Cloud Events based message format.

The protocol endpoints also authenticates and authorizes devices and its configured gateways. Validate credentials, or X.509 client certificates and ensure that the devices are enabled in the system.

Once the device is granted access to publish (or receive) data, the protocol endpoint will hook up the device to the internal message flow. It subscribes to commands targeted towards the device, and forwards messages from the device to an internal Knative eventing endpoint which takes care of processing and delivering the messages.

The actual payload of the messages is not processed, and just passed on the Knative eventing destination.

When a protocol endpoint drops off an incoming message to the Knative eventing service, this service is backed by a Kafka topic^[1].

The responsibility of Kafka is to deliver messages to the target consumer. Persisting messages if needed, and handling shared consumer groups if requested. This allows Drogue Cloud to buffer messages when the consumer is unavailable or if there is a peak of messages that the consumer currently cannot handle. Also, this allows to drive a shared consumer model, where in a group of consumers only one receives a messages. This concept is used to share the load for messages to multiple instances of a consumer.

As mentioned before, Drogue Cloud relies on a Knative eventing destination which provides these guarantees, and uses Kafka to implement this. However, it is possible to replace the Kafka implementation with a different implementation.

Applications, like devices, are provided by the user and live outside the Drogue Cloud instance. They still may run on the same cluster. By "consuming" messages, we mean that messages, sent by a device, are forwarded to the target application. It also implies a channel back to the device, for sending commands. However, as device-to-cloud messages are much more frequent, we simply focus on "consuming" messages.

In general there are "push" or "pull" modes. Messages are always sent to the consumer in an event oriented manner, "push" and "pull" only refers on who initiates the process.

Applications also have the ability to send back messages to the device, called commands. Commands are best-effort, and are not guaranteed to be delivered. If it is important to the application that a command got delivered, it needs to implement a reconciliation process on top of the command functionality.

This is necessary as it isn’t always clear if and how it makes sense to buffer commands. Assuming a decision is made, and a command is executed. It is not necessarily correct to buffer the command for a longer period of time, and then just resend it. Also, just because a command was delivered to a device attached to Drogue Cloud, doesn’t mean the action of the command was actually performed. A command could get lost in the next step of the processing. Therefore, it is important to have some feedback signal end-to-end. Some commands also cannot simply be retried, as that could be interpreted by a receiving device as a new request to process the command.

Therefore, doesn’t Drogue Cloud enforce any semantics on commands, and let the application deal with the case of undelivered commands.

Authentication (authn) and authorization (authz) happens in two different realms^[2] of the system. On the device facing side and on the user/application facing side. And while it may be technically incorrect, in the next few paragraphs "authentication" implies authentication and authorization, unless noted otherwise.

The device facing authentication is backed by the device registry, which is the central storage for device information and configuration. It also stores credentials of devices, like passwords or X.509 trust anchors.

The user facing authentication is backed by an Open ID connect, single sign-on system. By default, we use Keycloak, however it should be possible to use any other Open ID connect compatible solution.

Services internally also use the same SSO solution as external users.