gigl.orchestration.kubeflow.kfp_pipeline#

Attributes#

SPECED_COMPONENTS

logger

Classes#

ExitTask

What the pipeline runs when it exits, whatever the outcome.

Functions#

generate_pipeline(common_pipeline_component_configs[, ...])

Generates a KFP pipeline definition for GiGL.

Module Contents#

class gigl.orchestration.kubeflow.kfp_pipeline.ExitTask[source]#

Bases: str, enum.Enum

What the pipeline runs when it exits, whatever the outcome.

A KFP pipeline has exactly one exit task, so this is an either/or choice made at compile time.

EMAIL: Send a Vertex AI notification email to notification_emails. CLEANUP: Cancel the Vertex AI, Dataflow, and Dataproc jobs the run launched.

Meant for CI, where nobody reads the email and orphaned jobs hold capacity.

Initialize self. See help(type(self)) for accurate signature.

CLEANUP = 'cleanup'[source]#
EMAIL = 'email'[source]#
gigl.orchestration.kubeflow.kfp_pipeline.generate_pipeline(common_pipeline_component_configs, tag=None, exit_task=ExitTask.EMAIL)[source]#

Generates a KFP pipeline definition for GiGL. :param common_pipeline_component_configs: Shared configuration between components. :type common_pipeline_component_configs: CommonPipelineComponentConfigs :param tag: Optional tag, which is provided will be used to tag the pipeline description. :type tag: Optiona[str] :param exit_task: What runs when the pipeline exits. Defaults to the email notification.

With ExitTask.CLEANUP the notification_emails parameter is accepted but unused.

Returns:

An @kfp.dsl.pipeline decorated function to generated a pipeline.

Parameters:
gigl.orchestration.kubeflow.kfp_pipeline.SPECED_COMPONENTS: Final[list[str]][source]#
gigl.orchestration.kubeflow.kfp_pipeline.logger[source]#