A newer version of Hazelcast Platform is available.

View latest

Store Persistence Data on Intel® Optane™ DC Persistent Memory

6.0-SNAPSHOT 5.5 5.4 5.3 5.2 5.1 5.0

In this tutorial, you’ll use Intel® Optane™ DC Persistent Memory (PMem) to improve the performance of disk reads and writes, allowing your cluster to access data on disk faster.

Before you Begin

You need the following configuration tools:

Step 1. Configure the Persistent Memory as a File System

The Dual In-Line Memory Modules (DIMMs) can operate in two modes: MemoryMode or AppDirect.

To use PMem with Persistence in Hazelcast, DIMMs should be configured with AppDirect mode to allow you to mount them as a file system.

  1. First, check the current setup of the system:

    [root@localhost builder]# ipmctl show -socket

 SocketID | MappedMemoryLimit | TotalMappedMemory
==================================================
 0x0000   | 4096.0 GiB        | 95.0 GiB
 0x0001   | 4096.0 GiB        | 852.0 GiB

    The output shown above provides the CPU sockets of the system. You can print the DIMMs of each socket by using its ID, as shown below.

    [root@localhost builder]# ipmctl show -dimm -socket 0x0000

 DimmID | Capacity  | HealthState | ActionRequired | LockState | FWVersion
==============================================================================
 0x0011 | 126.4 GiB | Healthy     | 0              | Disabled  | 01.00.00.4877
 0x0021 | 126.4 GiB | Healthy     | 0              | Disabled  | 01.00.00.4877
 0x0001 | 126.4 GiB | Healthy     | 0              | Disabled  | 01.00.00.4877
 0x0111 | 126.4 GiB | Healthy     | 0              | Disabled  | 01.00.00.4877
 0x0121 | 126.4 GiB | Healthy     | 0              | Disabled  | 01.00.00.4877
 0x0101 | 126.4 GiB | Healthy     | 0              | Disabled  | 01.00.00.4877

    You can also see the current configuration of the system, as shown below:

    [root@localhost builder]# ipmctl show -region

 SocketID | ISetID             | PersistentMemoryType | Capacity  | FreeCapacity | HealthState
===============================================================================================
 0x0001   | 0xb5b67f48a7c32ccc | AppDirect            | 756.0 GiB | 0.0 GiB      | Healthy

    The above example output shows that the DIMMs of the socket with the SocketID 0x0000 is not in use. So, let’s configure 0x0000 for Persistence following the steps below.

  2. Use the following command for the socket 0x0000:

    [root@localhost builder]# ipmctl create -goal -socket 0x0000 PersistentMemoryType=AppDirect

The following configuration will be applied:
 SocketID | DimmID | MemorySize | AppDirect1Size | AppDirect2Size
==================================================================
 0x0000   | 0x0011 | 0.0 GiB    | 126.0 GiB      | 0.0 GiB
 0x0000   | 0x0021 | 0.0 GiB    | 126.0 GiB      | 0.0 GiB
 0x0000   | 0x0001 | 0.0 GiB    | 126.0 GiB      | 0.0 GiB
 0x0000   | 0x0111 | 0.0 GiB    | 126.0 GiB      | 0.0 GiB
 0x0000   | 0x0121 | 0.0 GiB    | 126.0 GiB      | 0.0 GiB
 0x0000   | 0x0101 | 0.0 GiB    | 126.0 GiB      | 0.0 GiB
Do you want to continue? [y/n] y

Created following region configuration goal
 SocketID | DimmID | MemorySize | AppDirect1Size | AppDirect2Size
==================================================================
 0x0000   | 0x0011 | 0.0 GiB    | 126.0 GiB      | 0.0 GiB
 0x0000   | 0x0021 | 0.0 GiB    | 126.0 GiB      | 0.0 GiB
 0x0000   | 0x0001 | 0.0 GiB    | 126.0 GiB      | 0.0 GiB
 0x0000   | 0x0111 | 0.0 GiB    | 126.0 GiB      | 0.0 GiB
 0x0000   | 0x0121 | 0.0 GiB    | 126.0 GiB      | 0.0 GiB
 0x0000   | 0x0101 | 0.0 GiB    | 126.0 GiB      | 0.0 GiB
A reboot is required to process new memory allocation goals.

  3. Reboot your system. After the reboot, check the regions and namespaces in the system as shown below:

    [root@localhost builder]# ndctl list --regions --human -N
[
  {
    "dev":"region1",
    "size":"756.00 GiB (811.75 GB)",
    "available_size":0,
    "max_available_extent":0,
    "type":"pmem",
    "iset_id":"0xb5b67f48a7c32ccc",
    "persistence_domain":"memory_controller",
    "namespaces":[
      {
        "dev":"namespace1.0",
        "mode":"fsdax",
        "map":"dev",
        "size":"744.19 GiB (799.06 GB)",
        "uuid":"65121d0e-a8a0-40f1-aed5-8a8ada13b6c7",
        "blockdev":"pmem1"
      }
    ]
  },
  {
    "dev":"region0",
    "size":"756.00 GiB (811.75 GB)",
    "available_size":"756.00 GiB (811.75 GB)",
    "max_available_extent":"756.00 GiB (811.75 GB)",
    "type":"pmem",
    "iset_id":"0x63f47f485dd02ccc",
    "persistence_domain":"memory_controller"
  }
]

    You can see “region0” has been created with the DIMMs of the socket (ID = 0x0000) in the above output.

  4. Create a namespace for “region0” as shown below:

    [root@localhost builder]# ndctl create-namespace --mode fsdax --region region0
{
  "dev":"namespace0.0",
  "mode":"fsdax",
  "map":"dev",
  "size":"744.19 GiB (799.06 GB)",
  "uuid":"87449768-1cc7-4c1b-b138-ea79bc4ee68e",
  "raw_uuid":"6756ef99-744f-4467-90f7-591c0ae162ec",
  "sector_size":512,
  "blockdev":"pmem0",
  "numa_node":0
}

  5. Make sure you can see the device.

    [root@localhost builder]# ll /dev/pmem0
brw-rw----. 1 root disk 259, 0 Mar 4 02:35 /dev/pmem0

  6. Format the partition with ext4 file system using the following command:

    [root@localhost builder]# mkfs.ext4 /dev/pmem0

  7. Create a mount point and mount the new filesystem to that mount point using the following commands:

    [root@localhost builder]# mkdir /mnt/pmem0
[root@localhost builder]# mount -o dax /dev/pmem0 /mnt/pmem0

Step 2. Configure Hazelcast to Use PMem for Persistence Storage

In Hazelcast, you must configure Persistence to tell your cluster where to save data. To improve performance, you can also adjust the number of I/O threads that Hazelcast can use to access persisted data.

  1. Create a new directory inside your /mnt/pmem0 directory.

    [root@localhost builder]# mkdir /mnt/pmem0/persistence

  2. Configure Hazelcast to use this directory, and for best performance set the parallelism option to 8 or 12.

    <persistence enabled="true">
    <base-dir>/mnt/pmem0/persistence</base-dir>
    <parallelism>12</parallelism>
</persistence>

Next Steps

For test results, which show why increasing parallelism improves performance, see our blog on https://builders.intel.com/datacenter/blog/hazelcast-fast-restart-optane-dc-persistent-memory.

For details about persistence configuration, see Configuring Persistence.

lifebuoy

Help and support