Designing a Layered Databus System with Connext DDS

In the ever-evolving world of system design, layered databus architecture is quite common and presently deployed in a wide variety of industries. But, as it’s not always fully understood or exploited, I wanted to spend a little time discussing the layered databus architecture from a practical point-of-view. I will be talking more about this in my April 30 TechTalk (available on demand afterwards).

In my experience, there are very few use cases that wouldn’t benefit from this highly scalable architecture. Conversely, I’ve encountered some deployments without a layered databus where customers were eager to understand the best approach for retrofitting layers into their system architecture.

The key takeaways the reader should get from this blog are an understanding of what the layered databus architecture looks like, how they could apply it to their system architecture and the potential benefits they may realize.

What is Layered Databus Architecture?

The layered databus architecture separates, or logically segments, Data Distribution Service™ (DDS) databuses in the global dataspace. What does separating the databuses mean? First, consider that the DDS API provides a number of obvious approaches to separating the databus - these methods include DDS Domains and Partitions. Topics are also sometimes used, but the danger of relying on Topics to separate the data streams is that it might hinder future system expandability.

Other less-obvious methods exist for separating the layers, and these include both transports or multicast groups. Transports are usually dictated by the system hardware configuration or communication requirements. Examples of this include UDP versus the TCP transport, or event UDPv4 versus UDPv6. By contrast, multicast groups are joined by the datareader and communicated during discovery.

Additionally, Connext® DDS Professional provides the ability to configure a multichannel datawriter to publish data to multiple multicast groups. It enables the datawriter to partition its data based on the content of the data. While this feature is quite powerful, keep in mind that it is a unique extension provided by RTI to the DDS standard.

Connecting the Layers

An important component of the layered databus architecture is the Bridging function, which is provided by RTI’s Routing Service. At a high level, the Routing Service simply moves data from one layer to another, but its functionality actually goes much deeper. For example, it can be used to process, or apply some type of transformation, to the data as it passes through. Alternatively, an adapter can be added to an input or output to merge DDS data with other I/O (e.g., RS-232).

Other services that might be useful are Persistence Service or Queuing Service. Persistence Service complements the original data publisher, storing its published data so that it remains available for late-joining applications after the original publisher exists. This functionality maps to the TRANSIENT or PERSISTENT setting of the datawriter’s and datareader’s durability Quality of Service (QoS).

Queuing Service guarantees “at-least once” and “at-most once” delivery of samples to a pool of datareaders. This feature is useful for load-balancing of samples for processing. Queuing Service also guarantees that delivered samples are processed, in that it waits for the datareader application to explicitly acknowledge receipt of the sample before considering it delivered.

These three services complement one another and can be applied to any given use case, as needed. Of the three, Routing Service is the most commonly deployed.

What Problems Does Layered Databus Architecture Solve?

To address this issue, let’s consider a case where a layered databus architecture has not been used. As that system grows, the number of Domain Participants and DDS endpoints would scale proportionally. But if every Domain Participant is able to discover every other Domain Participant and every DDS endpoint attempts to match, the length of time it takes for discovery to complete would grow exponentially. Furthermore, the database of discovered Domain Participants and DDS endpoints that each Domain Participant maintains can grow large. Depending on system resources, the Domain Participant’s database consumption could begin to create problems.

Other problems may arise too, such as load on the network when a late-joiner comes alive. Imagine the system is in a steady-state: discovery has completed and data is flowing smoothly. Suddenly a late-joiner arrives and, as a result, discovery traffic begins to spike on the network. The discovery traffic throttles, at best, the steady-state traffic. The problem is exacerbated if the Durability QoS is used and a large amount of historical data must be delivered to the late-joiner.

What Problems Could a Layered Databus Architecture Introduce?

A resilient system must also consider the potential drawbacks of a layered databus architecture. It is my firm belief that the benefits far outweigh any potential drawbacks - and if the potential issues are understood up front, they can be accounted for. So what are the issues that must be considered?

Concentrated Points-of-Failure

Routing Service functions as the bridge between the layers. This means that all traffic from each layer must flow through Routing Service, effectively making it a pinch point. What if Routing Service fails, or becomes overloaded? Luckily, this problem has a simple solution if it’s a potential issue: Redundant Routing Services, each servicing a single pair of domains or a set of topics. The ultimate configuration will be system-dependent, but if you’re aware from the beginning, the design will be much more straightforward.

Added Latency

The latency added by Routing Service, while minimal, does exist. If your system is sensitive to additional latency, then a resolution should be considered during the system design. Additional processing resources can be added. For example, on a CPU with many cores, Routing Service can be configured to add a thread per route. Or, similar to the resolution to the concentrated point-of-failure problem, multiple Routing Services could be deployed, with the different routes load-balanced among them.

Additional Resource Requirements

Similar to added latency, as Routing Services are introduced into the system design, they will consume resources. This problem must be accounted for by having sufficient hardware resources available. This could mean adding additional CPU or memory, or the problem might be able to be resolved by simply shuffling where Routing Service is run.

The layered databus is a common architecture across Industrial IoT (IIoT) systems in multiple industries. This architecture provides low-latency, secure, peer-to-peer data communications across logical layers of the system. It is most useful for systems that must manage direct interactions between applications in the field, such as control, local monitoring and edge analytics.

Best Practices for Applying the Layered Databus Architecture

Often, the question asked is about how to best define the layers. Or, how many layers is enough? When are there too many layers? Answers to these questions are, unfortunately, not straightforward. No firm rule exists to indicate whether a new layer is needed or not. Different strategies do exist, however, on how to approach this problem. Keep in mind that it’s important to stick with the strategy once it is adopted, and to make sure it's consistently applied. This process will most likely be iterative. Be prepared to work though at least a few attempts before the ideal result is identified.

What are some possible strategies to separate the layers?

• By function - System status information, command and control functions, supervisory analysis, etc.
• By Scope - Scope of interest, scope of control, or scope of data.
• By physical location - Control room, vehicle, patient’s bed, data center, etc.

Conclusion

The design of a layered databus system can be complex. But if the considerations are understood at the beginning, the design process is easily manageable. The end result can be a scalable, resilient and reliable system that operates rapidly and efficiently. This fact has been proven time and time again across various industries and in different use cases, and in systems that require massive scalability.

I’ll also be presenting a Tech Talk on the topic of designing a layered databus system on April 30 at 9:00 a.m. PDT / 6:00 p.m. CET. If this effort is top-of-mind for you or your team, I encourage you to register for this free session now. Can't join us live? It will be available for on-demand replay afterwards. 

I hope to see you there!

About the author

John Patchin, Senior Applications Engineer, RTI

John Patchin is a Senior Applications Engineer for Real-Time Innovations (RTI). He received his BS and MS in Mechanical Engineering from the University of Kansas. John has over 25 years of experience in industrial automation, embedded software development and networking. Prior to RTI, he developed machine tool and robot control systems at Hewlett-Packard and KUKA robotics, where he specialized in sensor feedback applications. He also spent time at Lynx Software Technologies, where he focused on real-time embedded software for aerospace and military markets.  John works on the Professional Services team at RTI, helping customers successfully integrate and deploy their DDS-based systems.