Page History

...

The ecs-sync program is specifically designed for moving data from one storage technology to another.  It comes from the Dell/EMC support labs, and is what EMC support engineers use for migrating data.  Due to security issues, we do not offer the job-oriented service mode described in the ecs-sync documentation.  (If the ecs-sync maintainers ever fixed the security holes, we could reconsider.)  Instead we only support running ecs-sync in what its documentation calls "Alternate (legacy) CLI execution", where a user runs ecs-sync as a command, instead of queuing up a job.  This allows the command to be run in the context of the user's login, honoring permissions.  One side effect is that the copied data could end up being owned by the user running the command, regardless of who owned the source data, which might not be desirable for all cases.

A command alias exists on Grace's login and compute nodes for running ecs-sync.  The alias actually runs the command on a data transfer node, so does not bog down the node on which the command is issued.  In other words, feel free to use ecs-sync from the login node, from the command prompt in Grace's Open OnDemand portal, or even from a compute job, if needed, though is it somewhat wasteful of resources to run ecs-sync from a compute job. 

...

Again, replace <config-file> and <log-file> with any name you wish.

The most complicated task in using ecs-sync is setting up the xml config file.  The config file consists of three parts: 

• the global settings define various parameters of how the sync should be run, such as buffer sizes, number of retries, number of parallel threads, etc. that are not specific to either the source or the target
• the source settings define where the data to be copied is coming from, including the type of storage, the location of the data, and access information
• the target settings defing where the data being copied should be stored, including the type of storage, the destination bucket or directory, and other access related information

The basic layout for the xml config file is as follows:

<syncConfig xmlns="http://www.emc.com/ecs/sync/model">  
    <options>
        <!--- ... the global settings ... --->
    </options>

    <source>
        <filesystemConfig>
            <!--- ... file copy settings ... --->
        </filesystemConfig>
    </source>

    <target>
        <ecsS3Config>
            <!--- ... ecs s3 copy settings ... --->
        </ecsS3Config>         
    </target>
</syncConfig>

Note that the contents of <source> and <target> are interchangeable, and the order of the blocks doesn't matter.  So the direction of the copy can be switched by simply swapping the <source> and <target> keywords.

Although ecs-sync knows abaoaut other types of storage, since this guide is for moving data between Grace's storage and ROSS, we will only describe the filesystem (for Grace's storage) and ecsS3 (for ROSS's storage) types, since those are the 2 needed.  If you are interested in other storage types, please consult the ecs-sync documentation.

The easiest way to create an xml config file is to copy an exiting template, and just change the pertinent values.  To help guide, here are examples of the 3 sections with descriptions of the parameters

Global Options

This example shows available global parameter options.  These are typical values, and may be changed.  The important options are deleteSource, fourceSync, recursive, retryAttempts, threadcount, verify and verifyOnly,

<syncConfig xmlns="http://www.emc.com/ecs/sync/model">
    <options><syncConfig xmlns="http://www.emc.com/ecs/sync/model">
        <bufferSize>524288</bufferSize>
        <dbConnectString>dbConnectString</dbConnectString>
        <dbFile>ecsdbFile</dbFile>
        <dbTable>dbTable</dbTable>
        <deleteSource>false</deleteSource>
        <forceSync>false</forceSync>
        <ignoreInvalidAcls>false</ignoreInvalidAcls>
        <logLevel>quiet</logLevel>
        <monitorPerformance>true</monitorPerformance>
        <recursive>true</recursive>
        <rememberFailed>false</rememberFailed>
        <retryAttempts>2</retryAttempts>
        <sourceListFile>sourceListFile</sourceListFile>
        <syncAcl>false</syncAcl>
        <syncData>true</syncData>
        <syncMetadata>true</syncMetadata>
        <syncRetentionExpiration>false</syncRetentionExpiration>
        <threadCount>16</threadCount>
        <timingWindow>1000</timingWindow>
        <timingsEnabled>false</timingsEnabled>
        <verify>false</verify>
        <verifyOnly>false</verifyOnly>
</options>

File System Options

This example shows the file system as a source.  It could also be a target.  The example shows typical values, which generally need to change to fit your situation.

    <source>
        <filesystemConfig>
            <path>/home/user/the/subdirectory/to/copy</path>
            <followLinks>true</followLinks>
            <storeMetadata>false</storeMetadata>
        </filesystemConfig>
    </source>

Ross (ECS S3) Options

ROSS uses the ECS S3 storage system type.  This example shows ROSS as a destination.  It could also be a target.  The example shows typical values, some of which must be changed for the sync to work, such as accessKey, bucketName, keyPrefix, and secretKey.

    <target>
        <ecsS3Config>
            <accessKey>accessKey</accessKey>
            <apacheClientEnabled>false</apacheClientEnabled>
            <bucketName>bucketName</bucketName>
            <createBucket>false</createBucket>
            <decodeKeys>false</decodeKeys>
            <enableVHosts>false</enableVHosts>
            <geoPinningEnabled>false</geoPinningEnabled>
            <host>host</host>
            <includeVersions>false</includeVersions>
            <keyPrefix>keyPrefix</keyPrefix>
            <mpuDisabled>false</mpuDisabled>
            <mpuPartSizeMb>128</mpuPartSizeMb>
            <mpuThreadCount>4</mpuThreadCount>
            <mpuThresholdMb>512</mpuThresholdMb>
            <port>0</port>
            <preserveDirectories>false</preserveDirectories>
            <protocol>protocol</protocol>
            <secretKey>secretKey</secretKey>
            <smartClientEnabled>true</smartClientEnabled>
            <socketConnectTimeoutMs>15000</socketConnectTimeoutMs>
            <socketReadTimeoutMs>60000</socketReadTimeoutMs>
            <vdcs>vdcs</vdcs>
        </ecsS3Config>
     </target>