Data plane

A core feature of Drogue Cloud is the abstraction of protocols. Typical IoT protocols are MQTT, CoAP, OPC UA, but also HTTP is being used by some devices.

Ideally, your cloud side application only needs to think about implementing one protocol, in order to interact with your devices. However, different IoT protocols provide benefits on the device side. As the devices are more constrained than the cloud side, Drogue Cloud tries to leverage the benefits of IoT protocols, while providing a normalization layer on the cloud side, that makes it easy for your applications to consume the data.

But also on the cloud side, different ways to consume data exists. May that be a direct connection to a Kafka topic, again MQTT, or a simple Web socket connection. Drogue Cloud tries to be integrated as good as possible, and not force you into a specific API.

Drogue Cloud uses the Cloud Events specification to forward messages internally. Cloud events have different mappings to transport channels, such as MQTT, Kafka, HTTP, etc.. And while Cloud Events can also be used on the external interfaces of Drogue Cloud, it isn’t a requirement, and you can use your custom protocols as you need.

Devices communicate with the Protocol endpoints, which implement the protocol specific mapping between the actual protocol and the internal Cloud Events based message format.

The protocol endpoints also authenticates and authorizes devices and its configured gateways. Validate credentials, or X.509 client certificates and ensure that the devices are enabled in the system.

Once the device is granted access to publish (or receive) data, the protocol endpoint will hook up the device to the internal message flow. It forwards messages from the device to an internal Kafka cluster, which takes care of storing the messages.

The responsibility of Kafka is to store the message until it can be fetched by a consumer. This allows Drogue Cloud to store messages when the consumer is unavailable or if there is a peak of messages that consumers currently cannot handle. Using Kafka allows for a wide range of consumption patterns.

Applications, like devices, are provided by the user and live outside the Drogue Cloud instance. They still may run on the same cluster. By "consuming" messages, we mean that messages, sent by a device, are forwarded to the target application. It also implies a channel back to the device, for sending commands. However, as device-to-cloud messages are much more frequent, we simply focus on "consuming" messages.

In general there are "push" or "pull" modes. Messages are always sent to the consumer in an event oriented manner, "push" and "pull" only refers on who initiates the process.

Applications also have the ability to send back messages to the device, called commands. Commands are best-effort, and are not guaranteed to be delivered. If it is important to the application that a command got delivered, it needs to implement a reconciliation process on top of the command functionality.

This is necessary as it isn’t always clear if and how it makes sense to buffer commands. Assuming a decision is made, and a command is executed. It is not necessarily correct to buffer the command for a longer period of time, and then just resend it. Also, just because a command was delivered to a device attached to Drogue Cloud, doesn’t mean the action of the command was actually performed. A command could get lost in the next step of the processing. Therefore, it is important to have some feedback signal end-to-end. Some commands also cannot simply be retried, as that could be interpreted by a receiving device as a new request to process the command.

Therefore, Drogue Cloud doesn’t enforce any semantics on commands, and let the application deal with the case of undelivered commands.

Authentication (authn) and authorization (authz) happens in two different realms^[1] of the system. On the device facing side and on the user/application facing side. And while it may be technically incorrect, in the next few paragraphs "authentication" implies authentication and authorization, unless noted otherwise.

The device facing authentication is backed by the device registry, which is the central storage for device information and configuration. It also stores credentials of devices, like passwords, pre-shared keys or X.509 trust anchors.