gigl.src.common.job_cleanup#

Cancel the GCP jobs a GiGL pipeline run launched.

GiGL components launch Vertex AI CustomJobs, Dataflow jobs, and Dataproc clusters from inside their own containers. Vertex AI does not know about those children, so cancelling the pipeline run only kills the component container. This module finds the children by the names GiGL gives them and cancels them.

Usage, to cancel a pipeline run and its children:: python -m gigl.src.common.job_cleanup –pipeline_run=projects/my-project/locations/us-central1/pipelineJobs/my-run

Attributes#

`logger`
`parser`

Functions#

`cancel_child_jobs`(job_name, resource_config_uri)	Cancel every Vertex AI CustomJob, Dataflow job, and Dataproc cluster of a GiGL job.
`cancel_pipeline_run_and_child_jobs`(pipeline_run)	Cancel a GiGL pipeline run, wait for it to stop, then cancel its child jobs.

Module Contents#

gigl.src.common.job_cleanup.cancel_child_jobs(job_name, resource_config_uri)[source]#

Cancel every Vertex AI CustomJob, Dataflow job, and Dataproc cluster of a GiGL job.

Each kind and each child is attempted even if an earlier one raised, so one failing call does not leave the other children running. Finding nothing to cancel is not an error.

Example

>>> cancel_child_jobs(
...     job_name="my_job",
...     resource_config_uri=GcsUri("gs://bucket/resource_config.yaml"),
... )

Parameters:

job_name (str) – The GiGL applied task identifier the children were named after.
resource_config_uri (gigl.common.Uri) – Resource config that gives the project and region.

Raises:

RuntimeError – If any client raised. The message names each failed step.

Return type:

None

gigl.src.common.job_cleanup.cancel_pipeline_run_and_child_jobs(pipeline_run)[source]#

Cancel a GiGL pipeline run, wait for it to stop, then cancel its child jobs.

The job name and the resource config come from the run’s job_name and resource_config_uri parameters.

The wait makes sure no component launches a new child after the cleanup.

A run that already stopped is not cancelled again, but its children are still cleaned up.

Example

>>> cancel_pipeline_run_and_child_jobs(
...     "projects/p/locations/us-central1/pipelineJobs/my-run"
... )

Parameters:: pipeline_run (str) – Full resource name of the pipeline run.
Raises:: RuntimeError – If the run does not stop in time, or a child cleanup failed.
Return type:: None

gigl.src.common.job_cleanup.logger[source]#

gigl.src.common.job_cleanup.parser[source]#