Eclipse zenoh: The Edge Data Fabric

With the rise of edge computing, efficient and effective data management is of paramount importance for every modern application. Eclipse zenoh fulfills this need, providing a data fabric that addresses the unique requirements of edge environments and modern edge applications.

To fully understand the role that Eclipse zenoh plays, it’s important to first understand the architecture and data management model needed for edge computing.

Edge Computing Requires a Decentralized Architecture

The past decade has been characterized by the rise of the data center, the relentless dominance of the cloud, and an overall trend towards centralized systems. This model has helped to propel the growth of software as a service (SaaS), mutualize IT infrastructures, and bootstrap consumer IoT. However, the centralized model has started to crack under the pressure, and has shown clear limitations in its ability to support increasing numbers of IoT and Industrial IoT (IIoT) use cases.

At the same time, concerns around sovereignty, confidentiality, and energy efficiency started to increase, and eventually led to the realization that we need to get back to the future. As Doc relied on a DeLorean to go back in time in the famous movie series, edge computing is our machine to get back to the future of digital systems: decentralization.

The Eclipse Edge Native Working Group recently published its vision for edge computing in a white paper, and if you haven’t read it yet, we recommend that you do. As the paper explains, edge computing comprises a system that allows computing power and storage capabilities to be distributed anywhere along an edge-to-cloud continuum while supporting any type of edge deployments.

At the edge, each use case and scenario requires the right mix of private and public locations, decentralization patterns, and resources — from tiny sensors to powerful servers. With this flexibility, computations and data can be distributed where it makes the most sense for the application with no artificial constraints imposed by the underlying edge system.

Data Drives the Need for Decentralization

In some cases, data contains sensitive information that should not be in a public domain. In other cases, the large amount of data being generated is very challenging to push to the cloud. There are sometimes considerations around data sovereignty, especially in highly regulated industries. In addition, you may want to process time-sensitive data as close as possible to the system dynamics, for example, when an application controls a robotic arm.

If you look at any modern system, it’s easy to see that data is constantly produced, distributed, computed, and eventually stored somewhere. Once data is computed or stored, newly produced data is often redistributed in the system to perform additional computations, or to be stored somewhere else (Figure 1). These concepts are referred to as data in motion and data at rest.

Figure 1: Data in Motion and Data at Rest

Historically, technologies for dealing with data in motion and data at rest have belonged to different families:

• Publish/subscribe, which is the leading paradigm for dealing with data in motion.
• Databases, such as SQL and NoSQL, which are the leading paradigms for dealing with data at rest.

Data in motion and data at rest can also be distinguished in another way:

• Data in motion is pushed to interested parties.
• Data at rest is pulled when needed.

Existing technologies for data in motion and data at rest were developed for a centralized model, which works very well in the cloud. However, the decentralized nature of edge computing adds requirements those technologies struggle to support. More importantly, these technologies also fail to provide unified data management, most notably along the edge-to-cloud continuum.

Eclipse zenoh Fills the Gaps in Today’s Data Management Technologies

Eclipse zenoh enables decentralized, yet unified, data management for edge computing. It provides unified abstractions for data in motion, data at rest, and computations in the edge-to-cloud continuum to create a data fabric that’s very well-suited for edge computing.

Since its inception, zenoh has supported heterogeneous sets of applications, environments, and networks, from constrained scenarios where a sensor is communicating over Bluetooth, to mobile scenarios where autonomous robots and vehicles communicate over 5G, to mission-critical scenarios with decentralized infrastructure management.

To deal with the heterogeneity of the edge, zenoh provides extremely good time and space efficiency. It is also very good at handling asymmetric systems and varying interaction models, such as devices with duty cycles that mean they may be in sleep mode.

To provide these capabilities, zenoh blends traditional publish/subscribe technologies with geo-distributed storage, queries, and computations in a unified and location-transparent set of programming primitives. It provides:

• Efficient publish/subscribe technologies that support multiple levels of reliability, dynamic discovery, fragmentation, and wire-level batching.
• A streamlined definition of geo-distributed storage.
• Well-defined semantics for querying and aggregating tasks.

In addition, zenoh allows applications to register computations that are triggered by queries, allowing many patterns, including remote procedure calls (RPCs) and MapReduce to be implemented.

Developers Don’t Need to Keep Track of Data

With zenoh, developers can focus on the data of interest rather than where it is, or how to retrieve it. This is very important in edge computing where applications, devices, and data move around. Keeping track of their location can be very difficult and subject to many issues related to data distribution, concurrency, and consistency. zenoh considers these aspects at its very core so developers don’t need to worry about them.

The Eclipse zenoh protocol stack includes two layers (Figure 2):

• zenoh.net implements a networking layer that can run above a data link, network, or transport layer. This layer provides primitives for efficient pub/sub and distributed queries. It also supports fragmentation and ordered reliable delivery.
• zenoh provides a high-level API for pub/sub and distributed queries, data representation transcoding, geo-distributed storage, and distributed computed values.

Figure 2: Eclipse zenoh Protocol Stack Layers

In both layers, data is represented by (name, value) pairs, such as (/home/sensor/temperature, 30). Developers always work with data names and values, leaving zenoh to deliver the right contents to the right locations (Figure 3).

Figure 3: Eclipse zenoh Delivers the Right Data to the Right Places

zenoh provides location transparency through its ability to route data and queries based on system status. This is made possible by a scalable and decentralized routing infrastructure that allows application components to become seamlessly decentralized so they can be easily replicated or migrated at internet scale.

zenoh also enables efficient scale-down capabilities. A minimal wire overhead of just five bytes allows support for constrained transport mechanisms, such as low-power WANs (LPWANs), low-power wireless personal area networks (LoWPANs), and Bluetooth Low Energy (BLE), as well as extremely resource-constrained devices, such as eight-bit microcontrollers.

Additionally, zenoh can effectively exploit the parallel processing in modern CPUs to achieve high throughput, on the order of tenths of a gigabit per second, in sustained conditions. Eclipse zenoh exploits such parallelism to provide capabilities such as automatic batching — messaging grouping that results in lower overhead on the wire and is reflected in higher throughput.

Finally, zenoh offers extremely low latency, on the order of microseconds, making it appropriate for applications that require exceptionally low overhead and reaction times.

Learn More

To learn more about Eclipse zenoh and get started with the software, visit the project website and join the zenoh community.