Low-latency real-time

EVA ICS v4 supports low-latency real-time mode. With this functionality, certain or even all fieldbus tasks can be replaced with EVA ICS services.

This allows to:

  • Replace traditional fieldbus controllers, reducing the number of devices, costs of ownership and maintenance

  • Significantly increase the overall system reliability

  • Increase overall flexibility and bring modern development and maintenance approaches to control fieldbus devices

  • Use the same nodes for both real-time and high-load tasks

The ultimate design of the platform allows to have guaranteed low latency (<100us) for the majority of real-time-critical tasks.

Requirements

  • For commercial purposes, EVA ICS Enterprise is required to be purchased.

  • For non-commercial purposes, real-time functionality can be used for free. Real-time monitor is not available.

  • Real-time mode is available in EVA ICS 4.0.2 build 2024101301 or above. Earlier versions do not provide low-latency real-time functionality.

System preparation

It is highly recommended to isolate CPU cores for real-time tasks. This can be done with isolcpus kernel parameter. For example, to isolate cores 2 and 3, add the following line to the kernel command line:

isolcpus=2,3

Refer to your Linux distribution documentation for more information.

Note

EVA ICS core always uses CPU 0 for certain general tasks, so it is not recommended to isolate CPU 0 or use it for real-time.

General setup

Note

To achieve maximum performance, it is recommended to apply the instructions from High-load and mission-critical environments section.

System parameters

If certain system parameters need to be adjusted, use etc/system-config.map file:

# used to set /proc and /sys parameters at start

#/proc/sys/kernel/sched_rt_runtime_us = -1 # no limit for real-time tasks (USE WITH CARE!)

#/sys/devices/system/cpu/cpu1/cpufreq/scaling_governor = performance # CPU 1
#/sys/devices/system/cpu/cpu2/cpufreq/scaling_governor = performance # CPU 2
#/sys/devices/system/cpu/cpu3/cpufreq/scaling_governor = performance # CPU 3

The specified parameters are applied on EVA ICS node startup and rolled back on shutdown.

Service parameters

To launch a EVA ICS service in real-time mode, it is required to have EVA ICS 4.0.2 build 2024101301 or above. The build brings the following additional service parameters:

realtime:
 cpu_ids: []
 prealloc_heap: null
 priority: null

  • cpu_ids - list of CPU cores to run the service on. If not set, the service runs on all available cores. When CPU isolation is used, it is recommended to assign real-time services to isolated cores only. If not set, the default affinity is used.

  • prealloc_heap - preallocate heap memory for the service. Use with caution, available in native packages only (services, built with target gnu).

  • priority - set the service priority. The value is a number from 0 to 99 (higher number means higher priority). If set to zero, the service runs with no real-time priority. If set to 1-99, the service runs with real-time priority with FIFO scheduling policy. If not set, the default priority is used.

Note

The best practice is to run in real-time mode only fieldbus interfaces and decision-making services.

Do not launch general-purpose services in real-time mode (such as HMI, authentication, database interfaces etc.) except for certain very specific cases or testing purposes.

Real-time core

Running the core in real-time mode may be useful if real-time services use local node bus for inter-process communication.

To enable real-time mode for the core, specify REALTIME parameter in eva_config. The parameter sets CPU affinity and real-time priority for the core threads.

When the core is running in real-time mode, all services are started as real-time by default, inheriting the core parameters. To disable real-time mode for a service, forcibly set priority to 0 and set cpu_ids to non-real-time CPU cores.

Secondary points do not need to have real-time parameters in eva-config, however they may be specified to set the default parameters for the services launched.

Real-time EAPI bus

When running in real-time mode, it is important to keep the bus latency as low as possible. The eva_config parameter DIRECT_ALLOC_LIMIT is used to limit memory allocation for the incoming messages.

  • In case, if the message payload is less than DIRECT_ALLOC_LIMIT, the message buffer is allocated directly in the real-time core thread and the message is processed with the lowest possible latency.

  • If the message payload is greater than DIRECT_ALLOC_LIMIT, the message buffer is allocated in the non-real-time thread and the message is processed with a higher latency. This allows to successfully combine both real-time and high-load tasks on the same node for the most of the typical cases. As soon as some message reaches the limit, the warning is logged.

Warning

It is not recommended to combine high-load and real-time functionality for mission-critical systems, unless the configuration is thoroughly tested and the real-time tasks either can deal with latency 500us and more (depends on CPU model), use own inter-process communication methods or do not require IPC at all.

Monitoring

Real-time tasks can be monitored with Real-time monitor. Deploy the service instance:

eva svc create eva.svc.rtmon /opt/eva4/share/svc-tpl/svc-tpl-rtmon.yml

Node system dashboard

Real-time tasks can be viewed in Node system dashboard, section “Realtime”:

Real-time tasks in the system dashboard

eva-shell

To view real-time tasks with eva-shell, use the following command:

eva rt
Real-time tasks in EVA shell

Limitations

  • When the core runs in real-time mode, the internal memory monitoring is disabled, memory warning settings are ignored.

  • Currently Real-time monitor does not support monitoring of secondary points.

  • Latency may significantly increase during certain core tasks, such as deploying items / services or replicating a large amount of items from remotes. It is highly recommended to modify production node inventory with putting affected fieldbus components into maintenance and limit replication tasks to outgoing only.

  • It is not recommended to use Python and TypeScript-based services in real-time mode for mission-critical tasks. The behavior may be unpredictable due to garbage collection and other factors.

  • Real-time parameters are applied only on services, built with EVA ICS Rust SDK version 0.3.58 or above and for services, started with FFI launcher (C++ SDK and other binary-compiled). To apply real-time parameters on other services (including Python and TypeScript), use the bundled rt-launch tool:

# service parameters
command: svc/rt-launch path/to/svc

The tool automatically applies required real-time parameters and launches the service executable.

EVA ICS vs RoboPLC

EVA ICS real-time mode solves the majority of typical fieldbus tasks, however for strict thread control, it is still recommended to use RoboPLC.

  • EVA ICS is better when a real-time service need to be deployed inside the existing platform setup.

  • RoboPLC is better when there is an embedded machine which performs mission-critical real-time tasks only and it must be physically isolated from other processes.

  • EVA ICS real-time mode is much easier to setup and maintain. RoboPLC requires strong knowledge of Linux, real-time systems and Rust, however it can bring much better performance.

  • EVA ICS services are asynchronous by design, combining best sides of high-load and real-time. RoboPLC programs are focused mainly on synchronous tasks, which can be controlled with much higher precision.

  • EVA ICS real-time mode requires EVA ICS Enterprise license for commercial purposes but brings functionality to run both EVA ICS core and any existing EVA ICS service in real-time mode. RoboPLC is free for commercial use, however the majority of the code must be written by the user.