Flow Generation
For simple flows, Gladier can handle the process of stitching together the flows of several tools into one big flow, or even generating a flow from scratch for a simple Gladier tool. Flow generation does not replace manually authored flows, but provides a path for automating simple flows that don’t require a lot of branching.
Flow generation is available for both clients and tools with the @generate_flow_definition
decorator. Although it’s applied the same way, the decorator behaves a little differently
for a client and a tool.
Gladier Tool – Generates a flow for each function defined in
compute_functions
Gladier Client – Combines flows for each Gladier Tool defined in
gladier_tools
Flow Generation on Gladier Tools
For Gladier Tools that only need one flow step per Compute Function, flow generation can be a good option. Gladier will automatically determine information about the functions on the tool, and incorporate them into the flow. The usage looks like this:
from gladier import GladierBaseTool, generate_flow_definition
def ls(data):
"""Do an 'ls' on the filesystem, given a ``dir``"""
import os
the_dir = data.get("dir", "~")
return os.listdir(the_dir)
@generate_flow_definition
class FileSystemListCommand(GladierBaseTool):
"""List files on the filesystem"""
compute_functions = [ls]
The decorator @generate_flow_definition
will automatically set the flow_definition
attribute when a GladierClient includes it in tools
. The flow it generates will be
identical to the following:
{
"Comment": "List files on the filesystem",
"StartAt": "Ls",
"States": {
"Ls": {
"Comment": "Do an 'ls' on the filesystem, given a ``dir``",
"Type": "Action",
"ActionUrl": "https://compute.actions.globus.org",
"ExceptionOnActionFailure": false,
"Parameters": {
"tasks": [
{
"endpoint.$": "$.input.compute_endpoint",
"function.$": "$.input.ls_function_id",
"payload.$": "$.input"
}
]
},
"End": true,
"ResultPath": "$.Ls",
"WaitTime": 300
}
}
}
Flow Generation Modifiers
You’ll notice that flow generation makes some assumptions you might want to change. For
example, ls
is not a compute heavy task, and might be better to run on the head node.
Overriding some attributes of the flow can be done with modifiers
.
@generate_flow_definition(modifiers={
ls: {'endpoint': 'compute_endpoint'}
})
class FileSystemListCommand(GladierBaseTool):
"""List files on the filesystem"""
compute_functions = [ls]
Flow Generation on Gladier Clients
When Flow Generation is applied to Gladier Clients it does not generate the flow from scratch, but instead combines the flow on each Gladier Tool into one big flow.
Note that modifiers can also be used with Gladier Clients to customize some attributes.
@generate_flow_definition(modifiers={
'generate_metadata': {'endpoint': 'compute_endpoint'},
'publish_to_search': {'endpoint': 'compute_endpoint',
'payload': 'generate_metadata'}
})
class ProcessData(GladierBaseClient):
gladier_tools = [
'mytools.TransferData',
'mytools.processing.ProcessData',
'mytools.processing.GenerateGraphs',
'mytools.processing.GenerateMetadata',
'mytools.processing.PublishResults',
]
Modifiers
A complete reference is coming soon!