Migration of profile backups to nested mode

This article is aimed at platform administrators and developers. It describes the migration of an allocation unit to the Nested mode of the profile backup handler. It also explains the operational impact of this change.

The platform switches the storage of profile backups to a nested directory structure. Nested mode distributes the backup files across 100 subdirectories per bucket. This makes parallel write operations significantly faster. A fallback mechanism keeps jobs from the old structure readable. You can activate the mode via a JVM system property. Manual data migration is not required.

Background: Directory locking with shared backup storage

The directory structure for profile backups has been revised. The previous structure stores all backup files directly in the profile directory. Optionally, they are split into buckets of 50,000.

As a transition, the platform introduces a Fallback mode. The platform writes new jobs to the new, nested structure. Existing jobs in the old structure remain readable via the fallback mechanism.

NOTE:

In high availability and load balancing environments, the backup root directory resides on a shared cloud file system, for example, AWS EFS. Multiple cluster nodes write to it simultaneously. Network file systems serialize changes to directory entries. Every write, rename, or delete operation briefly locks the entire directory. With many parallel threads, this results in directory lock contention. As a result, the old structure barely scales beyond a single thread.

The new structure distributes the files across 100 subdirectories. The subdirectory results from job number % 100. Parallel write operations therefore end up in different subdirectories. They no longer compete for the same lock. Benchmarks with ProfileBackupHandlerPerfMain show up to 8.5 times the throughput compared to the old structure when running 50 threads.

Lobster recommends the migration for all HA and load-balancing environments with a shared backup root and high parallel write load. Single-node installations on local storage are not affected by this issue.

Directory structures compared

Previous structure (Legacy)

You can enable bucketing in two ways. Either set the JVM system property hub.datawizard.subdirs. Or set MultipleBackupSubDirs in ./etc/startup.xml. If both values are set and conflict, the value from startup.xml takes precedence.

ENV files are always located next to the job files: ENV_<jobnr>. The bucket calculation is done in increments of 50,000: Jobs 1 to 50,000 are placed in a bucket 50000, jobs 50,001 to 100,000 in a bucket 100000, and so on.

New structure (Nested)

The new structure is always two levels deep. The subdirectory results from the remainder of dividing the job number by 100.

.../datawizard/backup/
  └── /
      └── 50000/
          ├── 00/
          │   ├── ENV_25000    (25000 % 100 = 0)
          │   └── Job_25000
          └── 08/
              ├── ENV_25008
              └── Job_25008

Immediately after the migration, both structures exist in parallel, with older jobs in the previous structure and new jobs in the nested structure.

The following folders are located directly under the backup root and do not belong to any profile. Both structures ignore them: commonqueue, ci, oftp, queue, reference, rules, suspend, temp, tmp, unknown, payload, unresolved.

Operating modes via the JVM system property

You select the implementation at platform startup via the JVM system property profile.backup.handler.implementation.

NOTE:

The values are case-insensitive. An unknown, empty, or invalid value always results in a fallback to legacy. Use only legacy or nested.

If an invalid value is used, the platform logs a setup error in services/error.log at startup:

SYSTEM:DATAWIZARD:SETUP Unknown value for -Dprofile.backup.handler.implementation='<value>' — falling back to 'legacy'. Valid values: legacy, nested, nested_only

This message indicates a misconfiguration. Correct the value before operating the system in production.

Fallback handler behavior

The fallback handler checks each job for its structure. It stores the result in the memory to avoid repeated file system access. The platform always reads the payload file and ENV file of a job from the same structure.

• Write: The platform writes new jobs exclusively to the new structure. A failed write operation is reported as an error.

• Read: The platform checks the new structure first. If it does not find the job there, it falls back to the old structure.

• Delete: A recursive delete operation removes files from both structures since they are located in the same profile directory.

Perform migration

Prerequisites

• The backup root directory is writable. The platform automatically creates the subdirectories of the new structure on the first write operation.

• Existing files from the old structure remain unchanged. Manual migration is not necessary.

Procedure

1. Open ./bin/platform.json and add the parameter to the jvmArgs array: -Dprofile.backup.handler.implementation=nested. If you are not yet using an application wrapper, set the parameter instead in ./etc/wrapper.conf, ./bin/hub.bat, or ./bin/execute.sh, depending on the startup mechanism. For details, see Lobster Application Wrapper.

2. Restart the platform.

3. From this point on, the platform stores all new jobs in the nested structure.

Monitor migration

You can monitor the progress of the virtual machine migration using Insight data. The fallbackRatio attribute indicates the proportion of jobs that still originate from the old structure.

<profileBackupFilesStorageMode mode="nested" fallbackRatio="0.0" />

In legacy mode, the platform shows instead:

<profileBackupFilesStorageMode mode="legacy" />

Emergency: Reverting to legacy mode

In exceptional cases, you can revert the backup handler back to legacy mode, but you must consider the impact on jobs already saved in the new structure.

IMPORTANT:

Jobs that have already been saved in the nested structure are no longer accessible to the platform after reverting to legacy. The files remain on the storage medium. However, the legacy handler is not aware of the new directory structure.

Reverting to legacy is only safe if no new jobs have been processed since the conversion. Otherwise, ensure that affected jobs are available in the old structure beforehand or accept the data loss.

Set the parameter in the same place as when switching to nested mode (see Perform migration above), but this time with the value legacy:

-Dprofile.backup.handler.implementation=legacy

Operational impact overview

The following overview shows the most important differences between the previous legacy behavior and the new fallback mode.