Upgrade Migrations

Migrating to v0.5.0

The only major change in v0.5.0 was the removal of the HelloWorld Tools from the main Gladier package. The following are no longer present:

  • gladier.tools.hello_world.HelloWorld

  • gladier.tools.manifest.ManifestTransfer

  • gladier.tools.manifest.ManifestToFuncXTasks

There currently aren’t plans to rewrite them in the Gladier Tools package, but open an issue if you would like us to consider changing that!

Migrating to v0.4.0

Gladier v0.3.x depended on FuncX v0.0.5 and FuncX Endpoint v0.0.3. Gladier v0.4.x now uses Funcx v0.2.3-v0.3.0+ (funcx-endpoint v0.2.3-v0.3.0+). There are a number of breaking changes between these two versions of FuncX, including funcx endpoints, flow definitions, and backend services.

FuncX Endpoints

All FuncX endpoints will need to be recreated with the never version of FuncX. Gladier typically names these endpoints as the following:

  • funcx_endpoint_non_compute

  • funcx_endpoint_compute

Since these use different backend services, using endpoints that don’t match the FuncX version will result in errors. Using 0.0.3 endponits on 0.2.3+ will result in permission denied, using 0.2.3+ on 0.0.3 will result in Server 500s.

Argument Passing and Function Definitions

Previously, all arguments in a Flow were passed to FuncX functions as a dict. It looked like the following:

'Parameters': {'tasks': [{'endpoint.$': '$.input.funcx_endpoint_non_compute',
                          'function': '8227609b-4869-4c6f-9a1b-87dc49fcc687',
                          'payload.$': '$.input'}]},

def my_function(data):
    ...

In the above, data would get the entire dict from $.input, which was typically whatever input was passed to start the flow. In the new version of FuncX, this has changed. All arguments are either positional or keyword arguments and should be named. This is difficult in automate, since naming arguments requires specifying them explicitly in the flow definition. An easy migration path is the following:

'Parameters': {'tasks': [{'endpoint.$': '$.input.funcx_endpoint_non_compute',
                          'function': '8227609b-4869-4c6f-9a1b-87dc49fcc687',
                          'payload.$': '$.input'}]},

def my_function(**data):
    ...

Changing data to a keyword argument will allow re-creating the same behavior as before.

FuncX Functions

Like FuncX Endpoints, FuncX Functions also need to be changed between versions. This is an automatic process in most cases if you are running the latest version of Gladier and saw a big giant warning when upgrading. Gladier will automatically delete funcx functions that don’t match the newly supported version of FuncX Gladier uses.

However, it’s necessary to do a manual upgrade to remove these functions in some cases. To upgrade manually, edit the file ~/.gladier-secrets.cfg, and remove all config items that end in funcx_id and funcx_id_checksum:

hello_world_funcx_id = 3bccfcdb-bc0e-4549-9297-8e08c6f50bd5
hello_world_funcx_id_checksum = c590423de52051e7b7bb044dc173673d2c9ad965f7f71bee665494815b3a2046

Flow Definitions

Some items in Automate flow definitions also changed. See below for a list of the attributes.

FuncX Version 0.0.5 flow definitions:

FuncX Version 0.2.3+ flow definitions:

Additionally for FuncX Payloads, Function UUIDs are passed with a different name.

‘func.$’: ‘$.input.’

Needs to be changed to:

‘function.$’: ‘$.input.’

FuncX Flow Result Format

The format of the return value from FuncX functions has changed format. This only affects Flow states that depend on the output of a FuncX function/flow state.

Previous flow states were not returned in a list, and were referenced with the following:

'InputPath': '$.MyFuncXFunctionOutput.details.result',

FuncX now returns these in a list, and they need to be index. The above needs to be changed to the following:

'InputPath': '$.MyFuncXFunctionOutput.details.result[0]',