Replication service

Contents

Replication service

The replication service allows V4 nodes to replicate events and interact with each other.

Note

There can be only a single replication service instance per pub/sub server deployed on each EVA ICS node.

The nodes must be connected via either PSRT or MQTT pub/sub.

PSRT is an in-house pub/sub protocol, designed especially for industrial needs, which can efficiently replicate lots of events per seconds and deal with large payloads via slow channels.
MQTT is popular standard IoT protocol, widely used and provided by lots of free and commercial service applications, as well as hosted versions from various cloud providers.

To communicate, both local and remote node must share the same API key (a key service is required, e.g. Local user/key authentication service. In trusted environment it is possible to use the same “default” key for all nodes. In untrusted ones it is recommended to have a dedicated key for each node pair.

Warning

All key fields MUST contain API key ID, not the key value.

The API key specifies ACL(s) for the remote node to handle RPC-via-pub/sub calls.

To enable all remote bus calls, nodes must also share the same API key with admin privileges, which must be set in “admin_key_id” configuration field on the primary (manager) node. The security policy is the same as for regular keys.

When the “admin_key_id” is set, the remote node becomes “managed”.

Note

Remote bus calls are required for IaC and deployment if a deployment configuration contains sections for remote nodes.

If discovery is enabled, newly discovered nodes are automatically connected with the “default_key”. Discovery feature is recommended for trusted and semi-trusted environments.

One node can have multiple replication services deployed. However all of the must replicate with the own set of remote nodes. Having same remote nodes replicated by different services leads to abnormal system behavior.

Pub/sub security

In production systems a pub/sub server can be securely shared between customers in case of the following conditions:

no node discovery (as node states can be forged)
bulk telemetry, bulk encryption

All RPC calls in EVA ICS are AES-256-encrypted by default.

Replicating untrusted nodes

By default, all remote nodes are trusted, which means any one can push/provide telemetry data for any item.

If there is an untrusted node connected, it can provide fake telemetry data for certain items, also it is not possible to use pub/sub security for bulk topics.

In this case, the node must be marked as untrusted in its configuration, which can be done either with eva-shell command “node append” with --untrusted argument, “node edit” for existing nodes (config field: trusted) or in the node deployment configuration.

Untrusted nodes should use dedicated API Keys only. To let a remote untrusted node provide telemetry data, the configured API key must have ACL with “write” permission for the allowed items, otherwise the telemetry is ignored in both push and pull.

Untrusted nodes should provide their telemetry via bulk topics only, in the encrypted way only. Such topics must be configured in the replication service as “secure topics”:

bulk:
  receive:
    secure_topics:
      - all

Regular bulk topics do not check senders’ ACLs and should be used for trusted nodes only.

Setup

Use the template EVA_DIR/share/svc-tpl/svc-tpl-replication.yml:

# EVA ICS v4-v4 replication service
command: svc/eva-repl
workers: 1
bus:
  path: var/bus.ipc
config:
  pubsub:
    # mqtt or psrt
    proto: psrt
    # path to CA certificate file. Enables SSL if set
    ca_certs: null
    # single or multiple hosts
    host:
      - 127.0.0.1:2873
    # if more than a single host is specified, shuffle the list before connecting
    cluster_hosts_randomize: false
    # user name / password auth
    username: null
    password: null
    ping_interval: 10
    # pub/sub queue size
    queue_size: 1024
    # pub/sub QoS (not required for PSRT)
    qos: 1
  # the local key service, required both to make and process API calls via PubSub
  key_svc: eva.aaa.localauth
  # set null to turn off announcements
  announce_interval: 10
  # respond to API calls
  api_enabled: true
  # enable discovery of secondary nodes
  discovery_enabled: true
  # key (id) to use for auto-discovered nodes
  default_key_id: default
  # subscription logic
  #
  # each: subscribe to individual topics for each item, plus bulks if defined
  # all: accept states for all items, move filtering from the Pub/Sub server
  #       to the local core
  # bulk_only: subscribe to bulk topics only
  subscribe: each
  #bulk:
    #send:
      #topic: all
      #compress: true # set to enable bulk compression
      #encryption_key: null # set (key id) to enable bulk encryption
      #buf_ttl_sec: 0.5
    #receive:
      #topics:
        #- all_pub
      #secure_topics:
        # - all
  # subscribed OIDs
  oids:
    - "#"
  oids_exclude: []
  # DANGEROUS, enable for multi-level clusters only
  #replicate_remote: true
user: nobody

Create the service using eva-shell:

eva svc create eva.repl.1 /opt/eva4/share/svc-tpl/svc-tpl-replication.yml

or using the bus CLI client:

cd /opt/eva4
cat DEPLOY.yml | ./bin/yml2mp | \
    ./sbin/bus ./var/bus.ipc rpc call eva.core svc.deploy -

(see eva.core::svc.deploy for more info)

EAPI methods

See EAPI commons for the common information about the bus, types, errors and RPC calls.

bus::<SVC_ID>::<METHOD>

Description	Calls any remote method (requires management access)
Parameters	required
Returns	RPC result as-is

Parameters
Name	Type	Description	Required
node	String	Remote node ID	yes

node.append

Description	Appends remote node with the default configuration
Parameters	required
Returns	nothing

Parameters
Name	Type	Description	Required
i	String	Node name	yes

node.deploy

Description	Deploys remote nodes and their configuration
Parameters	required
Returns	nothing

Parameters
Name	Type	Description	Required
nodes	Vec<struct>	Node configurations (same as got in node.export)	no

node.export

Description	Exports node deployment configurations
Parameters	required
Returns	Node deploy config

Parameters
Name	Type	Description	Required
i	String	Node ID (accepts wildcards)	yes

Return payload example:

{
    "nodes": [
        {
            "admin_key_id": "admin",
            "compress": true,
            "enabled": true,
            "key_id": "default",
            "name": "rtest1",
            "ping_interval": 1.0,
            "reload_interval": 60.0,
            "timeout": 30.0
        }
    ]
}

node.get

Description	Gets state of a single remote node
Parameters	required
Returns	Node state (struct)

Parameters
Name	Type	Description	Required
i	String	Node name	yes

Return payload example:

{
    "build": 2022050902,
    "compress": true,
    "enabled": true,
    "link_uptime": 183.26962651,
    "managed": true,
    "name": "rtest1",
    "online": true,
    "ping_interval": 1.0,
    "reload_interval": 60.0,
    "static": true,
    "timeout": 30.0,
    "version": "4.0.0"
}

node.get_config

Description	Gets configuration of a single node
Parameters	required
Returns	Node config (struct)

Parameters
Name	Type	Description	Required
i	String	Node ID	yes

Return payload example:

{
    "admin_key_id": "admin",
    "compress": true,
    "enabled": true,
    "key_id": "default",
    "name": "rtest1",
    "ping_interval": 1.0,
    "reload_interval": 60.0,
    "timeout": 30.0
}

node.list

Description	Lists remote nodes
Parameters	none
Returns	List (struct)

Return payload example:

[
    {
        "build": 2022050902,
        "compress": true,
        "enabled": true,
        "link_uptime": 85.004101572,
        "managed": true,
        "name": "rtest1",
        "online": true,
        "ping_interval": 1.0,
        "reload_interval": 60.0,
        "static": true,
        "timeout": 30.0,
        "version": "4.0.0"
    },
    {
        "build": 2022050902,
        "compress": true,
        "enabled": true,
        "link_uptime": 122.0092,
        "managed": true,
        "name": "rtest2",
        "online": true,
        "ping_interval": 1.0,
        "reload_interval": 60.0,
        "static": true,
        "timeout": 30.0,
        "version": "4.0.0"
    }
]

node.mtest

Description	Tests management call to a remote node
Parameters	required
Returns	nothing

Parameters
Name	Type	Description	Required
i	String	Node ID	yes

node.reload

Description	Forces reload of a remote node
Parameters	required
Returns	nothing

Parameters
Name	Type	Description	Required
i	String	Node ID	yes

node.remove

Description	Removes a single node
Parameters	required
Returns	nothing

Parameters
Name	Type	Description	Required
i	String	Node ID	yes

node.test

Description	Tests connection to a remote node
Parameters	required
Returns	nothing

Parameters
Name	Type	Description	Required
i	String	Node ID	yes

node.undeploy

Description	Deploys remote nodes and their configuration
Parameters	required
Returns	nothing

Parameters
Name	Type	Description	Required
nodes	Vec<struct/String>	Node configurations or IDs	no