Dynamically Adding WAN Publishers

4.2 4.1 4.0 3.12 3.11 3.10 3.9 3.8 3.7 3.6 3.5

When running clusters for an extensive period, you might need to dynamically change the configuration while the cluster is running. This includes dynamically adding new WAN replication publishers (new target clusters) and replicating the subsequent map and cache updates to the new publishers without any manual intervention.

You can add new WAN publishers to an existing WAN replication using almost all of the configuration options that are available when configuring the WAN publishers in the static configuration (including using Discovery SPI). The new configuration is not persisted but it is replicated to all existing and new members. Once the cluster is completely restarted, the dynamically added publisher configuration is lost and the updates are not replicated to the target cluster anymore until added again.

If you wish to preserve the new configuration over cluster restarts, you must add the exact same configuration to the static configuration file after dynamically adding the publisher configuration to a running cluster.

You cannot remove the existing configurations but can put the publishers into a STOPPED state which prevents the WAN events from being enqueued in the WAN queues and prevents the replication, rendering the publisher idle. The configurations also cannot be changed.

Synchronization can be initiated through Management Center and Hazelcast’s REST API.

Below is the URL for the REST call:

http://member_ip:port/hazelcast/rest/mancenter/wan/addWanConfig

You need to add the configuration for the WAN publishers as a JSON parameter. You can, at any point, even when maps and caches are concurrently mutated, add a new WAN publisher to an existing WAN replication configuration. The limitation is that there must be an existing WAN replication configuration but it can be empty, without any publishers (target clusters). For instance, this is an example of an XML configuration to which you can dynamically add new publishers:

<hazelcast>
    ...
    <wan-replication name="wanReplication"></wan-replication>
    <map name="my-map">
        <wan-replication-ref name="wan-replication">
            <merge-policy>com.hazelcast.map.merge.PassThroughMergePolicy</merge-policy>
            <republishing-enabled>false</republishing-enabled>
       </wan-replication-ref>
    </map>
    ...
</hazelcast>

Note that the map has defined WAN replication but there is no target cluster yet. You can then add the new WAN replication publishers (target clusters) by performing an HTTP POST as shown below:

curl -H "Content-type: text/plain" -X POST -d "{...}" --URL http://127.0.0.1:5701/hazelcast/rest/mancenter/wan/addWanConfig

You can provide the full configuration as JSON as a parameter. Any WAN configuration supported in the XML and programmatic configurations is also supported in this JSON format. Below are some examples of JSON configuration for a WAN publisher using the Discovery SPI and static IP configuration. Here are the integer values for initialPublisherState, queueFullBehavior and consistencyCheckStrategy:

  • initialPublisherState:

    • 0: REPLICATING

    • 1: PAUSED

    • 2: STOPPED

  • queueFullBehavior:

    • 0: DISCARD_AFTER_MUTATION

    • 1: THROW_EXCEPTION

    • 2: THROW_EXCEPTION_ONLY_IF_REPLICATION_ACTIVE

  • consistencyCheckStrategy:

    • 0: NONE

    • 1: MERKLE_TREES

Below is an example using Discovery SPI (AWS configuration):

{
   "name":"wanReplication",
   "publishers":[
      {
         "groupName":"tokyo",
         "queueCapacity":10000,
         "queueFullBehavior":0,
         "initialPublisherState":0,
         "properties":{
            "group.password":"pass"
         },
         "className":"com.hazelcast.enterprise.wan.replication.WanBatchReplication",
         "discovery":{
            "nodeFilterClass":null,
            "discoveryStrategy":[
               {
                  "className":"com.hazelcast.aws.AwsDiscoveryStrategy",
                  "properties":{
                     "security-group-name":"hazelcast",
                     "tag-value":"cluster1",
                     "host-header":"ec2.amazonaws.com",
                     "tag-key":"aws-test-cluster",
                     "secret-key":"my-secret-key",
                     "iam-role":"s3access",
                     "access-key":"my-access-key",
                     "hz-port":"5701-5708",
                     "region":"us-west-1"
                  }
               }
            ]
         }
      }
   ]
}

Below is an example with Discovery SPI (the new AWS configuration)

{
   "name":"wanReplication",
   "publishers":[
      {
         "groupName":"tokyo",
         "queueCapacity":1000,
         "queueFullBehavior":0,
         "initialPublisherState":0,
         "properties":{
            "group.password":"pass"
         },
         "className":"com.hazelcast.enterprise.wan.replication.WanBatchReplication",
         "aws":{
            "enabled":true,
            "usePublicIp":false,
            "properties":{
               "security-group-name":"hazelcast-sg",
               "tag-value":"hz-nodes",
               "host-header":"ec2.amazonaws.com",
               "tag-key":"type",
               "secret-key":"my-secret-key",
               "iam-role":"dummy",
               "access-key":"my-access-key",
               "region":"us-west-1"
            }
         },
         "sync":{
            "consistencyCheckStrategy":0
         }
      }
   ]
}

Below is an example with static IP configuration (with some optional attributes):

{
   "name":"wanReplication",
   "publishers":[
      {
         "groupName":"tokyo",
         "queueCapacity":1000,
         "queueFullBehavior":0,
         "initialPublisherState":0,
         "properties":{
            "response.timeout.millis":"5000",
            "group.password":"pass",
            "endpoints":"10.3.5.1:5701, 10.3.5.2:5701",
            "batch.max.delay.millis":"3000",
            "batch.size":"50",
            "snapshot.enabled":"false",
            "ack.type":"ACK_ON_OPERATION_COMPLETE"
         },
         "className":"com.hazelcast.enterprise.wan.replication.WanBatchReplication",
         "sync":{
            "consistencyCheckStrategy":0
         }
      }
   ]
}

Below is an XML configuration with two publishers and several (disabled) discovery strategy configurations:

{
   "name":"wanReplication",
   "publishers":[
      {
         "groupName":"tokyo",
         "queueCapacity":1000,
         "queueFullBehavior":0,
         "initialPublisherState":0,
         "properties":{
            "group.password":"pass"
         },
         "className":"com.hazelcast.enterprise.wan.replication.WanBatchReplication",
         "aws":{
            "enabled":true,
            "usePublicIp":false,
            "properties":{
               "security-group-name":"hazelcast-sg",
               "tag-value":"hz-nodes",
               "host-header":"ec2.amazonaws.com",
               "tag-key":"type",
               "secret-key":"my-secret-key",
               "iam-role":"dummy",
               "access-key":"my-access-key",
               "region":"us-west-1"
            }
         },
         "gcp":{
            "enabled":false,
            "usePublicIp":true,
            "properties":{
               "gcp-prop":"gcp-val"
            }
         },
         "azure":{
            "enabled":false,
            "usePublicIp":true,
            "properties":{
               "azure-prop":"azure-val"
            }
         },
         "kubernetes":{
            "enabled":false,
            "usePublicIp":true,
            "properties":{
               "k8s-prop":"k8s-val"
            }
         },
         "eureka":{
            "enabled":false,
            "usePublicIp":true,
            "properties":{
               "eureka-prop":"eureka-val"
            }
         },
         "discovery":{
            "nodeFilterClass":null,
            "discoveryStrategy":[

            ]
         },
         "sync":{
            "consistencyCheckStrategy":0
         }
      },
      {
         "groupName":"london",
         "queueCapacity":1000,
         "queueFullBehavior":0,
         "initialPublisherState":0,
         "properties":{
            "response.timeout.millis":"5000",
            "group.password":"pass",
            "endpoints":"10.3.5.1:5701, 10.3.5.2:5701",
            "batch.max.delay.millis":"3000",
            "batch.size":"50",
            "snapshot.enabled":"false",
            "ack.type":"ACK_ON_OPERATION_COMPLETE"
         },
         "className":"com.hazelcast.enterprise.wan.replication.WanBatchReplication",
         "aws":{
            "enabled":false,
            "usePublicIp":false
         },
         "gcp":{
            "enabled":false,
            "usePublicIp":false
         },
         "azure":{
            "enabled":false,
            "usePublicIp":false
         },
         "kubernetes":{
            "enabled":false,
            "usePublicIp":false
         },
         "eureka":{
            "enabled":false,
            "usePublicIp":false
         },
         "discovery":{
            "nodeFilterClass":null,
            "discoveryStrategy":[

            ]
         },
         "sync":{
            "consistencyCheckStrategy":1
         }
      }
   ]
}