utransfer

Introduction

Setting up a test or staging environment can require a great deal of effort. Unblu contains numerous configuration properties and text properties that allow you to customize it to meet your needs. You probably also created entities such as conversation templates, named areas, and teams that are similar or even identical to the ones you want to use in production.

It would therefore be helpful if, when the time comes to put Unblu into production, you were able to transfer as many settings as possible to the new environment. However, this is not as straightforward as you might think:

The account and global server configuration interfaces offer ways to export the configuration and text properties that you customized. But Unblu accounts also have other attributes, such as a name or an avatar, as well as metadata associated with them, that cannot be exported via the configuration interfaces.
The Web API’s read endpoints with the appropriate expand query parameter provides a means to export a JSON representation of an entity or set of entities, but there are no corresponding endpoints to reimport the JSON into an account.
Entities are interrelated. For example, a user may be a member of a particular team, or a conversation template may be linked to a particular named area. These relationships are expressed using references based on entity IDs, which are specific to a setup and connot be transferred.

utransfer solves these issues.

What is `utransfer`?

In a nutshell, utransfer is a utility to transfer entities and configurations from one Unblu account to another. It relies in part on the Unblu Web API, but includes additional features such as ID modification and filtering.

You can:

Export data from an account to multiple JSON files, zipped JSON files, or a single JSON file
Import data from exports into one or more accounts on one or more servers
Export and import global configuration properties and text properties if you provide superadministrator credentials

For transfers between Unblu 6 accounts, utransfer uses version 3 of the Unblu Web API. Versions for transfers between Unblu 5 accounts use version 2 of the Unblu Web API.

It is not possible to transfer entities between two different major versions of Unblu.

We strongly recommend that you use the most recent version of utransfer for the major version that corresponds with your server installation, although it is not a strict requirement. For example, you can transfer entities from an installation of Unblu 5.13.1 to an installation of Unblu 5.23.3 using version 5.22.1 of utransfer.

If you are using Java 11, use at least version 6.5.1 of utransfer.

The `utransfer` CLI

The utransfer command line interface (CLI) comes bundled as a JAR file. Running the JAR file with the --help option provides an overview of the options available:

Listing 1. The output of running utransfer with the --help option

$ java -jar <name-of-utransfer-jar-file>.jar --help (1)
Usage: utransfer [-hjkVz] [--fromNoAuth] [--keepApiKeys] [--noAuth]
                 [--toNoAuth] [--fromAccount=<fromAccountId>]
                 [--fromAdmin=<fromAdmin>] [--fromPassword=<fromPassword>]
                 [--fromSuperadmin=<fromSuperadmin>]
                 [--idConversion=<idConversionStrategy>] [--toAdmin=<toAdmin>]
                 [--toPassword=<toPassword>] [--toSuperadmin=<toSuperadmin>]
                 [-a=<admin>] [-p=<password>] [-s=<superadmin>]
                 [--fromHeader=<String=String>]... [--skip=<skipEntities>]...
                 [--toHeader=<String=String>]... [-d=<String=String>]... FROM TO

Transfer Unblu entity configuration and localization from FROM to TO where
FROM, TO are either an Unblu server URL or a path

      FROM                  path or URL. If a path is provided, it's assumed to be a
                              directory where previously exported utransfer files
                              are located. If a URL is provided, it is assumed to be
                              an Unblu server web API endpoint from which data is
                              exported. URL must be fully qualified including unblu
                              prefix (trusted entry-path). Example: https:
                              //unbluserver:1234/unblu
      TO                    path or URL. If a path is provided, it should be a file
                              path ending with ('.zip' or '.json') or directory
                              where the export should be stored. If a URL is
                              provided, it is assumed to be an Unblu server web API
                              endpoint where data is imported into. URL must be
                              fully qualified including unblu prefix (trusted
                              entry-path). Example: https://unbluserver:1234/unblu
  -a, --admin=<admin>       Unblu username with admin (or superadmin) role to use
                              when only one of FROM or TO is an Unblu url
  -s, --superadmin=<superadmin>
                            Unblu username with superadmin role to use when only one
                              of FROM or TO is an Unblu url
  -p, --password=<password> Password to use for the --admin or --superadmin
  -d, --header=<String=String>
                            Headers sent when only one of FROM or TO is an Unblu url
      --noAuth              Work without basic auth when one of FROM or TO is an
                              Unblu url
      --fromAdmin=<fromAdmin>
                            Unblu username with admin (or superadmin) role to use
                              when FROM is an Unblu url
      --fromSuperadmin=<fromSuperadmin>
                            Unblu username with superadmin role to use when FROM is
                              an Unblu url
      --fromPassword=<fromPassword>
                            Password to use for the --fromAdmin or --fromSuperadmin
      --fromAccount=<fromAccountId>
                            Unblu account to use in case superadmin is provided and
                              account should not be the superadmin's when FROM is an
                              Unblu url
      --fromHeader=<String=String>
                            Headers sent when FROM is an Unblu url
      --fromNoAuth          Work without basic auth when FROM is an Unblu url
      --toAdmin=<toAdmin>   Unblu username with admin (or superadmin) role to use
                              when TO is an Unblu url
      --toSuperadmin=<toSuperadmin>
                            Unblu username with superadmin role to use when TO is an
                              Unblu url
      --toPassword=<toPassword>
                            Password to use for the --toAdmin or --toSuperadmin
      --toHeader=<String=String>
                            Headers sent when TO is an Unblu url
      --toNoAuth            Work without basic auth when TO is an Unblu url
  -k, --keepMarkers         By default database markers like creation date of
                              entities are removed on export to have clean data.
                              With this option set, markers are kept in the export
      --keepApiKeys         By default api keys values are removed on export to have
                              clean data. With this option set, api keys are kept in
                              the export
  -z                        Create a zip file in TO instead of creating a file per
                              entity in the folder
  -j                        Create a json file in TO instead of creating a file per
                              entity in a folder
      --idConversion=<idConversionStrategy>
                            How IDs should be converted on export. NAME: use unique
                              entity name, ID: use original id, HASH: create hash
                              based on unique entity name, COUNTER: replace the id
                              based on a counter. Default is NAME
      --skip=<skipEntities> A comma separated list of entities that should not be
                              transferred. By default, USERS, TEAMS is set as skipped
  -h, --help                Show this help message and exit.
  -V, --version             Print version information and exit. (2)

1	Replace `<name-of-utransfer-jar-file>.jar` with the name of your `utransfer` JAR file.
2	The `-V` and `--version` options currently don’t work.

The sections below go into how to use utransfer in more detail.

Exporting data

The command for an administrator to export the account from https://foo.com to the folder /tmp/unblu-export would be:

Listing 2. Exporting an account as separate JSON files

$ java -jar <name-of-utransfer-jar-file>.jar \ (1)
    https://foo.com/co-unblu \ (2)
    /tmp/unblu-export \ (3)
    --fromAdmin=<admin-username> \ (4)
    --fromPassword=<admin-password> (5)

1	Replace `<name-of-utransfer-jar-file>.jar` with the name of your `utransfer` JAR file.
2	`FROM`
3	`TO`
4	Replace <admin-username> with you admin username. Since only `FROM` is an Unblu URL, you could use `-a` instead of `--fromAdmin`.
5	Replace <admin-password> with your admin password. Since only `FROM` is an Unblu URL, you could use `-p` instead of `--fromPassword`.

When you execute this command, utransfer will generate a folder named after the account in the TO folder you specified, with each exported entity in a separate JSON file:

Listing 3. Default utransfer export format

Main_Account/
├── account_Main_Account.json
├── apiKey_Default.json
├── avatar_1094328050070469361.json
├── avatar_9060334214433969142.json
├── conversationTemplate_default_audio_request_template.json
├── conversationTemplate_default_chat_request_template.json
├── conversationTemplate_default_chat_template.json
├── conversationTemplate_default_domcap_browser_request_template.json
├── conversationTemplate_default_embedded_template.json
├── conversationTemplate_default_headless_template.json
├── conversationTemplate_default_mobile_cobrowsing_request_template.json
├── conversationTemplate_default_mobile_template.json
├── conversationTemplate_default_offline_chat_request_template.json
├── conversationTemplate_default_video_request_template.json
├── namedArea_AA.json
├── person_concierge-bot-person-id.json
├── webhook_AA.json
└── webhook_BB.json

utransfer offers two alternative export formats:

A single ZIP file containing the same set of individual JSON files for each entity as the default export format. The ZIP file will be generated in the TO folder (/tmp/unblu-export in the example above) with missing parent directories being created as needed. The name of the ZIP file will be <account_ID>-<export_date>-<export_time>.zip.

Use the -z option in the command above to choose this export format.
A single JSON file containing all of the exported entities. The JSON file will be saved in the TO folder. Again, missing parent directories will be created as needed. The name of the JSON file will be <account_ID>-<export_date>-<export_time>.json.

Having all entities in a single JSON file makes it particularly easy to compare two or more installations.

Use the -j option in the command above to choose this export format.

If you run the export command with the option --fromSuperadmin, the data will include global configuration and text properties.

Transferring persons

The only person entities that can be transferred are those of dialog bots or the concierge. Other persons belong to conversations. Since it is not possible to transfer conversations, there is no point in transferring other persons, either.

Skipping entities

You may not want or need to transfer all types of entity. In that case, you can exclude entities using the --skip option with a comma-separated list of the types of entity you don’t want to transfer.

The list of possible values to exclude is:

API_KEYS
BOTS
CANNED_RESPONSES
CONVERSATION_TEMPLATES
DOMAINS
EXTERNAL_MESSENGERS
GLOBAL (only relevant when utransfer is called with superadministrator credentials)
NAMED_AREAS
NONE
TEAMS (excluded by default)
USERS (excluded by default)
WEBHOOKS

This can be useful if, for example, you exported all of an account’s entities but only want to import, say, its conversation templates and named areas into a new account. In that case, you would run utransfer with the option --skip=API_KEYS,BOTS,CANNED_RESPONSES,DOMAINS,EXTERNAL_MESSENGERS,TEAMS,USERS,WEBHOOKS.

By default, utransfer does not transfer users and teams. The reason for this is that it is not possible to export users with their credentials, and teams without users often make little sense. If you would like to transfer users and teams anyway, include the option --skip=NONE when you run utransfer. Including NONE in a list with actual entities will result in an error.

The license key is never included in a utransfer export.

Importing data

To import data into an account, use a command such as the following one:

Listing 4. Importing data from a ZIP file into an account

$ java -jar <name-of-utransfer-jar-file>.jar \ (1)
    /tmp/unblu-export \ (2)
    https://foo.com/co-unblu \ (3)
    --toAdmin=<admin-username> \ (4)
    --toPassword=<admin-password> (5)

1	Replace `<name-of-utransfer-jar-file>.jar` with the name of your `utransfer` JAR file.
2	`FROM`
3	`TO`
4	Replace <admin-username> with you admin username. Since only `TO` is an Unblu URL, you could use `-a` instead of `--toAdmin`.
5	Replace <admin-password> with your admin password. Since only `TO` is an Unblu URL, you could use `-p` instead of `--toPassword`.

This command wil merge the transferred data into the account of the administrator whose credentials were provided.

When you run the command with the option --toSuperadmin (and the corresponding credentials), the behavior depends on the name of the account whose entities you are importing.

If no account with that name exists at the TO location, utransfer will create a new account.
If an account of the same name exists, utransfer will merge the transferred entities into the existing account of the same name.

Tf the exported data contains global configuration and text properties, these, too, will be imported.

Importing global configuration and text properties can potentially affect all of the accounts on the target installation. You should therefore give careful thought to the effect of including global entities when using utransfer.

Import hints

If the target of your transfer isn’t a brand-new account, it may contain a significant number of entities. Even if it is new, it will already contain a number of default entities. Thus the question arises of how to deal with existing entities at the import target. utransfer uses hints to determine how to proceed.

If you export a ZIP file or a folder with a JSON for each entity, your export will include a file named hints.json like the one below:

Listing 5. An example hints.json file generated with --skip=CONVERSATION_TEMPLATES,NAMED_AREAS

{
  "apiKeysStrategy" : "DELETE",
  "cannedResponsesStrategy" : "DELETE",
  "conversationTemplatesStrategy" : "KEEP",
  "domainsStrategy" : "DELETE",
  "namedAreasStrategy" : "KEEP",
  "teamsStrategy" : "DELETE",
  "usersStrategy" : "DELETE",
  "webhookRegistrationsStrategy" : "DELETE",
  "dialogBotsStrategy" : "DELETE",
  "externalMessengersStrategy" : "DELETE"
}

If you use the -j option to export a single JSON file containing all entities, the JSON file will contain an object named "hints" whose value will mirror that of the hints.json file above.

As you can see in the example above, there are two different import strategies, DELETE and KEEP.

DELETE: Any entities at the target that have no corresponding entity with the same name in the imported data are deleted.
KEEP: Entities at the target that have no corresponding entity with the same name in the imported data remain unchanged.

Suppose you exported the named areas NA01, NA02, and NA03 from account A and want to import them into account B, which already has the named areas NA02, NA03, and NA04.

With the import hint "namedAreaStrategy" set to KEEP, you would end up with four named areas in account B: NA01, NA02, NA03, and NA0,
If the import hint were set to DELETE, account B would end up with only three named areas: NA01, NA02, and NA03.
In both cases, the named areas NA02 and NA03 in account B would be updated to reflect the named area exported from account A.

Some entities, such as the default team and default conversation templates, are never be deleted, regardless of the strategy specified in the import hints.

Which strategy utransfer defaults to for each type of entity depends on whether that type of entity was skipped or not during the export.

Entity types that were skipped will be assigned the KEEP strategy.
Entity types that were included in the export will be assigned the DELETE strategy.

Data modification

utransfer can carry out three types of modification on the data you are transferring: API key value removal, database marker removal, and ID conversion.

By default, the values of the API keys are removed when they are exported. (The other values of API key entities are exported.) To disable this behavior and export API keys with their key values, run utransfer with the option --keepApiKey.

Database markers, too, are removed by default during utransfer data exports. You can retain them by including the option --keepMarkers

To give you a better idea of the options' effect, compare the two excerpts below, taken from an export to a single JSON file. The first example was exported with the options --keepApiKeys and --keepMarkers. The second export was run without these options.

As you can see, the first example includes various values — "apiKey", "creationTimestamp", "modificationTimestamp", and a number of "$_version" entries — that are missing from the second example. Apart from "apiKey", the missing values are all database markers.

Listing 6. Extract of an export with the options -j --keepApiKeys --keepMarkers

"mobile-agent" : {
  "$_type" : "ApiKey",
  "id" : "mobile-agent",
  "creationTimestamp" : 1605607944974,
  "modificationTimestamp" : 1605607944974,
  "version" : 1,
  "accountId" : "The Bank",
  "apiKey" : "HGXXQQI1TA6OFiE_OzNg8g",
  "name" : "mobile-agent",
  "description" : "Demo api key for usage in area: mobile-agent",
  "configuration" : {
    "$_version" : "50d60323f704eb76a5cc181039d113f8",
    "com.unblu.conversation.ui.individualUiActionBarTheme" : "DARK"
  },
  "text" : {
    "$_version" : {
      "$_version" : "d41d8cd98f00b204e9800998ecf8427e"
    }
  },
  "metadata" : {
    "$_version" : "2fe774b6fd245e8c80d136f67a4c344b",
    "demo.data" : "true"
  }
}

Listing 7. Extract of an export with the option -j

"mobile-agent" : {
  "$_type" : "ApiKey",
  "id" : "mobile-agent",
  "accountId" : "The Bank",
  "name" : "mobile-agent",
  "description" : "Demo api key for usage in area: mobile-agent",
  "configuration" : {
    "com.unblu.conversation.ui.individualUiActionBarTheme" : "DARK"
  },
  "text" : { },
  "metadata" : {
    "demo.data" : "true"
  }
}

When you import data that includes API keys or database markers, run utransfer with the same option to include the data that’s normally removed. For example, to include API keys in your import, run utransfer with the option --keepApiKeys.

ID conversion

The default entity IDs in Unblu are random character sequences, automatically generated when the entity is created. This makes them hard to remember and to find when one wants to compare entities from two or more different environments. It also makes it hard to compare two or more different installations.

Transforming IDs into something more legible to humans makes it easier to find one’s way around an export. Transforming IDs in a stable fashion also makes comparisons between different installations simply a matter of performing a diff.

To transform entity IDs during an export, run utransfer with the option --idConversion=<strategy>, where <strategy> is one of the following:

COUNTER: An entity’s ID is replaced with an id based on a counter and its entity type. The result depends on the order in which
HASH: id is a hash based on the business logic attribute (title, name, …).
ID: The entity keeps its original id, i.e. no transformation occurs.
NAME: This is the default strategy for the utransfer CLI. The entity’s name or title is used as its ID.

The strategies NAME and HASH are stable across installations,that is, the same entity exported from two or more environments will have the same ID.

An ID clash occurs when an ID transformation results in the same output ID for two different input entities. If an ID clash occurs, utransfer will raise an error and exit. In such cases, we recommend that you use a transformation strategy that cannot create clashes, i.e. ID or COUNTER.

Consider the API key used in the examples of the previous section. This is the exported data with the original IDs, i.e. using the conversion strategy ID:

Listing 8. API key exported via utransfer with the options -j --idConversion=ID --keepApiKeys --keepMarkers

"QA9tsi5QRZ2dFMo7JOUwHQ" : {
  "$_type" : "ApiKey",
  "id" : "QA9tsi5QRZ2dFMo7JOUwHQ",
  "creationTimestamp" : 1610359268499,
  "modificationTimestamp" : 1610359268499,
  "version" : 1,
  "accountId" : "T-fc_kUHQ_WNvY1ZX9ZA0w",
  "apiKey" : "JmvURvDWQdywtaN0sTFZYQ",
  "name" : "mobile-agent",
  "description" : "Demo api key for usage in area: mobile-agent",
  "configuration" : {
    "$_version" : "50d60323f704eb76a5cc181039d113f8",
    "com.unblu.conversation.ui.individualUiActionBarTheme" : "DARK"
  },
  "text" : {
    "$_version" : {
      "$_version" : "d41d8cd98f00b204e9800998ecf8427e"
    }
  },
  "metadata" : {
    "$_version" : "2fe774b6fd245e8c80d136f67a4c344b",
    "demo.data" : "true"
  }
}

This is what the entity looks like when exported with the ID conversion strategy COUNTER:

Listing 9. API key exported with the options -j --idConversion=COUNTER --keepApiKeys --keepMarkers

"apiKey030" : {
  "$_type" : "ApiKey",
  "id" : "apiKey030",
  "creationTimestamp" : 1610359268499,
  "modificationTimestamp" : 1610359268499,
  "version" : 1,
  "accountId" : "account001",
  "apiKey" : "JmvURvDWQdywtaN0sTFZYQ",
  "name" : "mobile-agent",
  "description" : "Demo api key for usage in area: mobile-agent",
  "configuration" : {
    "$_version" : "50d60323f704eb76a5cc181039d113f8",
    "com.unblu.conversation.ui.individualUiActionBarTheme" : "DARK"
  },
  "text" : {
    "$_version" : {
      "$_version" : "d41d8cd98f00b204e9800998ecf8427e"
    }
  },
  "metadata" : {
    "$_version" : "2fe774b6fd245e8c80d136f67a4c344b",
    "demo.data" : "true"
  }
}

Note that the reference to the account this entity belongs to, the value of "accountId", has been changed to reflect the new value of the account’s ID. This shows how utransfer keeps track of all cross-references between the entities it processes.

Here is the same entity exported with the HASH strategy:

Listing 10. API key exported via utransfer with the options -j --idConversion=HASH --keepApiKeys --keepMarkers

"4085760720999967238" : {
  "$_type" : "ApiKey",
  "id" : "4085760720999967238",
  "creationTimestamp" : 1610359268499,
  "modificationTimestamp" : 1610359268499,
  "version" : 1,
  "accountId" : "7303578532599678026",
  "apiKey" : "JmvURvDWQdywtaN0sTFZYQ",
  "name" : "mobile-agent",
  "description" : "Demo api key for usage in area: mobile-agent",
  "configuration" : {
    "$_version" : "50d60323f704eb76a5cc181039d113f8",
    "com.unblu.conversation.ui.individualUiActionBarTheme" : "DARK"
  },
  "text" : {
    "$_version" : {
      "$_version" : "d41d8cd98f00b204e9800998ecf8427e"
    }
  },
  "metadata" : {
    "$_version" : "2fe774b6fd245e8c80d136f67a4c344b",
    "demo.data" : "true"
  }
}

Finally, here is the entity exported with the NAME strategy:

Listing 11. API key exported via utransfer with the options -j --idConversion=NAME --keepApiKeys --keepMarkers

"mobile-agent" : {
  "$_type" : "ApiKey",
  "id" : "mobile-agent",
  "creationTimestamp" : 1610359268499,
  "modificationTimestamp" : 1610359268499,
  "version" : 1,
  "accountId" : "The Bank",
  "apiKey" : "JmvURvDWQdywtaN0sTFZYQ",
  "name" : "mobile-agent",
  "description" : "Demo api key for usage in area: mobile-agent",
  "configuration" : {
    "$_version" : "50d60323f704eb76a5cc181039d113f8",
    "com.unblu.conversation.ui.individualUiActionBarTheme" : "DARK"
  },
  "text" : {
    "$_version" : {
      "$_version" : "d41d8cd98f00b204e9800998ecf8427e"
    }
  },
  "metadata" : {
    "$_version" : "2fe774b6fd245e8c80d136f67a4c344b",
    "demo.data" : "true"
  }
},

When you reimport the data of entities whose IDs were transformed, the entities will be assigned random character sequences as IDs again.

Using `utransfer` with different user roles

utransfer has options to specify the username and password of either an admin or a superadmin. Which of the two roles you use utransfer with will affect how imports and exports of entities work.

Administrator

If you run utransfer with the option --fromAdmin, then that administrator’s account will be exported.

When you run utransfer with the --toAdmin option, the data will be imported into that administrator’s account. Any GLOBAL entities in the data will be ignored.

Technical administrator

If the credentials you provide belong to a technical administrator, utransfer will run as if it had been called with the credentials of a normal account administrator. However, not all entities will be transferred. The following types of entity will always be skipped, regardless of any --skip option you may include:

BOTS
EXTERNAL_MESSENGERS
PERSONS
WEBHOOKS

Superadministrator

When you run utransfer with the option --fromSuperadmin, then the superadministrator’s account will be exported. You can, however, include the --fromAccount option to specify a different account. If the superadministrator does not belong to the account specified, utransfer will create a temporary administrator. The temporary administrator carries out the export and is deleted once the export has finished.

To avoid this, you can provide the superadministrator’s username in the --fromAdmin option instead.

Exports carried out by superadministrators include global configuration and text properties. To prevent this, include the value GLOBAL in the --skip option.

Running `utransfer` with ID propagation

You can use <<(root)knowledge-base/authentication.adoc#authentication-source-propagated,ID propagation>> when you run utransfer. To do so, you can use the appropriate --header, --FromHeader or --ToHeader option:

Listing 12. Example of exporting data using authentication with ID propagation

$ java -jar utransfer.jar \ (1)
    http://localhost:7777/co-unblu \ (2)
    /tmp/unblu-export \ (3)
    -j (4)
    -d x-propagated-trusted-user-id=utransferUser -d x-propagated-trusted-user-roles=ADMIN (5)

1	Replace `<name-of-utransfer-jar-file>.jar` with the name of your `utransfer` JAR file.
2	`FROM`
3	`TO`
4	Generate a single JSON file.
5	`-d` is the shorthand notation for the `--header`. Since only `FROM` is an Unblu URL, there is no need to use the more specific `--fromHeader`. Replace the header with the header(s) you need.

The headers you have to include will depend on you particular setup. Suppose you need to include a header such as x-awesome-proxy-header: YesDefinitely for the reverse proxy, which then carries out authentication. In that case, you would call utransfer with the option -d x-awesome-proxy-header=YesDefinitely.

If necessary, you can remove the basic authentication headers from the request with the options --noAuth, --noFromAuth or --noToAuth.

ID propagation was introduced in version 6.21.1 of utransfer. The option to remove basic authentication headers was added in utransfer version 6.23.0.