This is a prerelease version.

View latest

Storing Persistence Data on Intel® Optane™ DC Persistent Memory

Storing data in Intel® Optane™ DC Persistent Memory (PMem) improves the performance of reads and writes, allowing your cluster to access data on disk faster.

Before you Begin

You need the following configuration tools:

Step 1. Configure the Persistent Memory as a File System

The Dual In-Line Memory Modules (DIMMs) can operate in two modes: MemoryMode or AppDirect.

To use PMem with Persistence in Hazelcast, DIMMs should be configured with AppDirect mode to allow you to mount them as a file system.

  1. First, check the current setup of the system:

    [root@localhost builder]# ipmctl show -socket
    
     SocketID | MappedMemoryLimit | TotalMappedMemory
    ==================================================
     0x0000   | 4096.0 GiB        | 95.0 GiB
     0x0001   | 4096.0 GiB        | 852.0 GiB

    The output shown above provides the CPU sockets of the system. You can print the DIMMs of each socket by using its ID, as shown below.

    [root@localhost builder]# ipmctl show -dimm -socket 0x0000
    
     DimmID | Capacity  | HealthState | ActionRequired | LockState | FWVersion
    ==============================================================================
     0x0011 | 126.4 GiB | Healthy     | 0              | Disabled  | 01.00.00.4877
     0x0021 | 126.4 GiB | Healthy     | 0              | Disabled  | 01.00.00.4877
     0x0001 | 126.4 GiB | Healthy     | 0              | Disabled  | 01.00.00.4877
     0x0111 | 126.4 GiB | Healthy     | 0              | Disabled  | 01.00.00.4877
     0x0121 | 126.4 GiB | Healthy     | 0              | Disabled  | 01.00.00.4877
     0x0101 | 126.4 GiB | Healthy     | 0              | Disabled  | 01.00.00.4877

    You can also see the current configuration of the system, as shown below:

    [root@localhost builder]# ipmctl show -region
    
     SocketID | ISetID             | PersistentMemoryType | Capacity  | FreeCapacity | HealthState
    ===============================================================================================
     0x0001   | 0xb5b67f48a7c32ccc | AppDirect            | 756.0 GiB | 0.0 GiB      | Healthy

    The above example output shows that the DIMMs of the socket with the SocketID 0x0000 is not in use. So, let’s configure 0x0000 for Persistence following the steps below.

  2. Use the following command for the socket 0x0000:

    [root@localhost builder]# ipmctl create -goal -socket 0x0000 PersistentMemoryType=AppDirect
    
    The following configuration will be applied:
     SocketID | DimmID | MemorySize | AppDirect1Size | AppDirect2Size
    ==================================================================
     0x0000   | 0x0011 | 0.0 GiB    | 126.0 GiB      | 0.0 GiB
     0x0000   | 0x0021 | 0.0 GiB    | 126.0 GiB      | 0.0 GiB
     0x0000   | 0x0001 | 0.0 GiB    | 126.0 GiB      | 0.0 GiB
     0x0000   | 0x0111 | 0.0 GiB    | 126.0 GiB      | 0.0 GiB
     0x0000   | 0x0121 | 0.0 GiB    | 126.0 GiB      | 0.0 GiB
     0x0000   | 0x0101 | 0.0 GiB    | 126.0 GiB      | 0.0 GiB
    Do you want to continue? [y/n] y
    
    Created following region configuration goal
     SocketID | DimmID | MemorySize | AppDirect1Size | AppDirect2Size
    ==================================================================
     0x0000   | 0x0011 | 0.0 GiB    | 126.0 GiB      | 0.0 GiB
     0x0000   | 0x0021 | 0.0 GiB    | 126.0 GiB      | 0.0 GiB
     0x0000   | 0x0001 | 0.0 GiB    | 126.0 GiB      | 0.0 GiB
     0x0000   | 0x0111 | 0.0 GiB    | 126.0 GiB      | 0.0 GiB
     0x0000   | 0x0121 | 0.0 GiB    | 126.0 GiB      | 0.0 GiB
     0x0000   | 0x0101 | 0.0 GiB    | 126.0 GiB      | 0.0 GiB
    A reboot is required to process new memory allocation goals.
  3. Reboot your system. After the reboot, check the regions and namespaces in the system as shown below:

    [root@localhost builder]# ndctl list --regions --human -N
    [
      {
        "dev":"region1",
        "size":"756.00 GiB (811.75 GB)",
        "available_size":0,
        "max_available_extent":0,
        "type":"pmem",
        "iset_id":"0xb5b67f48a7c32ccc",
        "persistence_domain":"memory_controller",
        "namespaces":[
          {
            "dev":"namespace1.0",
            "mode":"fsdax",
            "map":"dev",
            "size":"744.19 GiB (799.06 GB)",
            "uuid":"65121d0e-a8a0-40f1-aed5-8a8ada13b6c7",
            "blockdev":"pmem1"
          }
        ]
      },
      {
        "dev":"region0",
        "size":"756.00 GiB (811.75 GB)",
        "available_size":"756.00 GiB (811.75 GB)",
        "max_available_extent":"756.00 GiB (811.75 GB)",
        "type":"pmem",
        "iset_id":"0x63f47f485dd02ccc",
        "persistence_domain":"memory_controller"
      }
    ]

    You can see “region0” has been created with the DIMMs of the socket (ID = 0x0000) in the above output.

  4. Create a namespace for “region0” as shown below:

    [root@localhost builder]# ndctl create-namespace --mode fsdax --region region0
    {
      "dev":"namespace0.0",
      "mode":"fsdax",
      "map":"dev",
      "size":"744.19 GiB (799.06 GB)",
      "uuid":"87449768-1cc7-4c1b-b138-ea79bc4ee68e",
      "raw_uuid":"6756ef99-744f-4467-90f7-591c0ae162ec",
      "sector_size":512,
      "blockdev":"pmem0",
      "numa_node":0
    }
  5. Make sure you can see the device.

    [root@localhost builder]# ll /dev/pmem0
    brw-rw----. 1 root disk 259, 0 Mar 4 02:35 /dev/pmem0
  6. Format the partition with ext4 file system using the following command:

    [root@localhost builder]# mkfs.ext4 /dev/pmem0
  7. Create a mount point and mount the new filesystem to that mount point using the following commands:

    [root@localhost builder]# mkdir /mnt/pmem0
    [root@localhost builder]# mount -o dax /dev/pmem0 /mnt/pmem0

Step 2. Configure Hazelcast to Use PMem for Persistence Storage

In Hazelcast, you must configure Persistence to tell your cluster where to save data. To improve performance, you can also adjust the number of I/O threads that Hazelcast can use to access persisted data.

  1. Create a new directory inside your /mnt/pmem0 directory.

    [root@localhost builder]# mkdir /mnt/pmem0/persistence
  2. Configure Hazelcast to use this directory, and for best performance set the parallelism option to 8 or 12.

    <persistence enabled="true">
        <base-dir>/mnt/pmem0/persistence</base-dir>
        <parallelism>12</parallelism>
    </persistence>

Further Reading

For test results, which show why increasing parallelism improves performance, see our blog on https://builders.intel.com/datacenter/blog/hazelcast-fast-restart-optane-dc-persistent-memory.

For details about configuration, see Persistence Configuration Options.