gigl.src.common.utils.dataflow#

Attributes#

MAX_WORKFLOW_RUNTIME_WALLTIME_SECONDS

logger

Functions#

get_sanitized_dataflow_job_name(name)

init_beam_pipeline_options(applied_task_identifier, ...)

Can pass in any options i.e.

Module Contents#

gigl.src.common.utils.dataflow.get_sanitized_dataflow_job_name(name)[source]#
Parameters:

name (str)

Return type:

str

gigl.src.common.utils.dataflow.init_beam_pipeline_options(applied_task_identifier, job_name_suffix, component=None, custom_worker_image_uri=None, **kwargs)[source]#

Can pass in any options i.e. init_beam_pipeline_options(num_workers=1, max_num_workers=32, …) The options passed in will override default options if we define them. For example, you can override the job_name by passing in job_name=”something”

Parameters:

  • applied_task_identifier (AppliedTaskIdentifier)

  • job_name_suffix (str) – Unique identifier for the dataflow job in relation to this task (applied_task_identifier) i.e. job_name_suffix = “inference”

  • component (Optional[gigl.src.common.constants.components.GiGLComponents])

  • custom_worker_image_uri (Optional[str])

  • kwargs (Any)

Returns:

options you can use to generate the pipeline

Return type:

PipelineOptions

gigl.src.common.utils.dataflow.MAX_WORKFLOW_RUNTIME_WALLTIME_SECONDS = 86400[source]#
gigl.src.common.utils.dataflow.logger[source]#