July 02, 2021 12 min read
Chief Technology Officer, PhD in Physics and Mathematics
Effective data transmission is essential for smart networks. There is a wide range of communication protocols that can provide smooth interaction between connected devices. The alternative solution designed by our engineers can compete with existing technologies such as HTTP and MQTT. Here, we will tell you about Integra’s messaging library, its structure, functionality, and competitive strengths.
The Internet of Things (IoT) relies on the interoperation of network nodes, including the collection and exchange of data. To implement a messaging system in your product, you can use a suitable out-of-the-box technology. However, ready-made solutions are not universal so they cannot fit every project’s needs.
In one of the IoT systems developed by the Integra team, we needed to connect the client devices with the database and peripheral modules. Our developers were looking for a highly flexible solution that could meet all the project requirements.
Finally, we created a custom technology that improved the efficiency of the client-server interaction. Our team integrated several components, uniting them into Integra Sources Messaging Library, or ISML. In this article, we’ll elaborate on the design and capabilities of our tailor-made solution.
In one of our recent IoT projects, we built an API on the server’s side of a device for its interaction with the peripherals. During the project development, we had to face some common challenges. Initially, we tried to meet them using standard networking technologies.
Thus, we used HTTP to interact with the web dashboard of the IoT system. To establish the communication between the server and the client device, which is a charging station, we tried to use the MQTT protocol. This lightweight networking standard is widely used for client-server communication in low-bandwidth systems.
However, it didn’t appear suitable for our project, as it couldn’t guarantee efficient data transfer and a stable connection with the charging devices (some of them worked on 2G networks that provided poor throughput and irregular communication).
So we needed more reliable technology to keep a steady connection with the chargers. That’s why we decided to develop our own messaging solution, which optimized communication between the components of the network.
ISML has much in common with similar IoT messaging protocols. It is also a lightweight solution intended for smart networks that require fast and efficient data exchange. It uses a communication model similar to the messaging mechanism of MQTT and has easy implementation like the POrtable COmponents C++ Libraries (POCO).
However, we can point out several advantages of our custom IoT messaging product that can optimize the interaction of components within your system.
Our messaging library has wide use and can fit a number of projects connected with data exchange and serialization, session management, and inter-networking. ISML is one of the solutions that the Integra engineering team creates within our system and application software development services. We build network applications and protocols that provide smooth communication between the components of IoT and embedded systems.
ISML is a set of components designed to integrate message-oriented middleware (MOM). MOM is the software that supports communication within a distributed environment. The message-oriented infrastructure of the ISML library facilitates the implementation of a dispatcher, router, and other components related to data delivery.
You can use the library to develop applications with client-server and peer-to-peer (P2P) architectures. ISML provides interfaces to create, send, receive, and store communication data. It is designed for the efficient transmission of binary files. There are several classes used to generate different types of messages and deliver them.
ISML works with sessions so it doesn’t support connectionless protocols, such as UDP and IP. At the present moment, it is compatible with only one network protocol, which is TCP.
However, we singled out the transport layer as a separate class. It helps the client device interact with the peripherals, for example, via RS-485. So, in the future, you can build a data exchange system using any bus and transport protocol - either connectionless or connection-oriented.
ISML uses a publish/subscribe pattern which differs slightly from the model applied in similar messaging technologies. For example, MQTT employs a broker that serves as a mediator between the network nodes. In our communication model, clients can interact with the server directly.
There is no need for topics as well since the server has a list of clients subscribed to certain sessions. To simplify the data transfer, you can also create a message channel for each of the sessions.
The architecture of the ISML library enables you to add any encryption and authentication systems and cryptographic protocols. By customizing it for your particular project, you can achieve the required security level.
ISML is a highly flexible and customizable solution that enables users to implement their own logic, leveraging the components of the library.
ISML comprises interrelated components or classes used to implement the functionality of a data exchange system. The basic classes include Session, Transport, SessionManager, and Message.
A communication session introduced as the Session class is one of the key components of the library. It helps identify the messaging client in the system. This class represents a high-level interface for sending and receiving messages.
You can also use Session to store various data necessary for the management of the latest session. The data includes session lifetime, latest activity time, the number of transmitted messages, etc.
The user can expand and customize the list of the stored parameters with the help of a built-in mechanism for the Properties class - a heterogeneous container for data storage.
Session also provides a high-level interface for sending requests and replies. It differs from the standard sending and receiving of messages in its synchronized nature. It allows the requestor to send a message and switch to a waiting mode without changing the execution context.
This mechanism can be helpful for sending confirmations or notifications. It assists in implementing the Remote Procedure Call (RPC) and requests with the use of the messaging system (for example, a request to the network node granting access to the database).
In an ordinary use case, sessions appear in the system during the setup of the connection with a remote client (on the server’s side) and upon server connectivity (on the client’s side). In both cases, the socket connection gets established regardless of the applied protocol - whether it is TCP or WebSocket.
ISML offers the Transport component that encapsulates all the necessary resources to manage the connection and a part of the logic responsible for sending and receiving messages.
Transport equips the library user with a high-level interface that allows for messaging without reference to all low-level details specific to the used protocol. For example, we use the Asio library that provides an input/output context and a set of objects to work with TCP. The Transport class implements the messaging mechanism on the basis of this protocol hiding the TCP/IP stack routine from the user.
Along with that, Transport is responsible for storing the messages sent and received. It releases users from building their own mechanisms that implement this functionality. The messages are stored in a queue that can operate either in a single-thread system or under concurrency.
Thus Transport makes it possible to work with a queue in a multithreaded environment feeling secure about data or API races. The queue interface eliminates the risk of the races using the optional types. It also excludes all possible methods that could lead to such situations, for example, calculations of the queue size.
Creating a session, you should pass a class instance that implements message Transport. The session uses it to send and receive messages.
ISML contains the SessionManager component that manages sessions and provides the interface to create them. For the component’s usability, you can register a callback in the network sub-system on one of the possible events - connection accepted or connected.
An Acceptor or ConnectionListener is a low-level component of the library. It is responsible for receiving the connection events and notifying SessionManager of the necessity to create a session.
Thus, in case of a successful connection, an instance of the message Transport will be passed to SessionManager. The latter will create a new session and connect it with the specified Transport.
The Message class introduced by ISML is a data unit of a messaging system. Message is a set of fields used to store valuable data that builds up the message body. It also includes helper fields used to identify a message, determine its type and owner.
The MessageFactory class is a special component that creates messages out of the raw data. Building messages outside MessageFactory is unacceptable and will lead to system errors. To create a message, you should know its type and session that acts as a context.
The type of message defines its structure. There can be any number of message types registered in the system. What is important is that they should be unique. A message type itself is an ordinary numerical identifier. Its underlying type is determined by MessageType.
As soon as a message is created, it is then put into the message channel or queue. Dispatcher is the library component that registers handlers for different message types. Thus, handler 1 is invoked to process MessageType 1, etc.
To create a reply, we should provide MessageFactory with a specific MessageType and a Session.
One numerical identifier is obviously not enough to describe the message structure. ISML comprises the Field class, which is a basic type of message field, and FieldDescriptor. The latter provides an interface to receive additional information about the field, for example, its name, and assists in building the desired field type.
Both components belong to abstract classes, and they can’t serve the purposes mentioned above directly. To store data, you can use the descendant classes - DataField and DataFieldDescriptor respectively. This method is called type erasure. It is used to store any data, including built-in types of the C++ language and user-defined types (UDTs).
FiledDescriptors get united into a message descriptor that is passed to MessageFactory as a prototype. This prototype will be the basis for an instant message of a specific type.
Schematically, we can show it as follows:
So, each message has a descriptor that contains information about the message type (also called a numeric identifier) and its fields. These fields are not stored in the message descriptor itself, but field descriptors are used - they create an actual field of the desired type with the desired name.
Thus, the message descriptor is the prototype of the message, when using information about the message type and fields, messages can be created from its descriptor an unlimited number of times.
In other words, MessageDescriptor contains descriptors of all the fields that the message of this type will include. Thus,
Message is a heterogeneous container and it can embrace the fields that store values of any type of data.
The widespread use of the Internet of Things is closely connected with the growth of energy-efficient communication technologies. They make it possible to create networks of smart devices that effectively interact with each other.
Such standards as MQTT, HTTP, and POCO are broadly used in IoT development. However, they are not multipurpose and you can’t implement them in any scenario. Some projects require much more flexible and easy-to-use solutions.
Integra Sources Messaging Library is a custom technology that provides strong and stable communication within a network of connected devices. It can make your project development easy, timely, and cost-friendly. If you want to learn more about our IoT messaging solution or customize it according to your needs, feel free to get in touch with our engineering team.
Copyright © 2021 Integra Sources. All rights reserved.
This library is an internal development of Integra Sources Ltd.
This library is published with usage restrictions.
If you are interested in using this library in your project, please contact us and we will permit you to use it on your personal terms.
ISML on GiTHub
Share this article
June 16, 2021 13 min read
OpenCV is an open-source computer vision library widely used in IoT home automation. Detecting people with OpenCV algorithms can be part of security systems and lifesaving solutions. Learn more about the library and our personal experience with real-time human detection.
By Andrey Solovev & Anna Petrova
March 22, 2021 15 min read
A fully electronic control system of a brushless DC motor creates opportunities for its widespread use in electronics of various power and size. High efficiency, reliability, and low maintenance are among the key benefits of a brushless DC motor controller. Learn how it works and what design techniques you can use for its implementation.
By Andrey Solovev & Anna Petrova
October 09, 2020 17 min read
A battery management system or BMS performs many functions. It can help you estimate the charging level of your battery and protect it from fires. By reading this article, you will get an idea of a BMS, its functions, and the software it might use.
By Andrey Solovev & Anna Petrova