gigl.utils.host_memory

How much host memory this process may still allocate.

psutil.virtual_memory().available reads /proc/meminfo, which inside a container reports the HOST’s memory rather than the container’s limit, so a budget check trusting it alone can pass immediately before the container is killed. The cgroup knows the real limit; read both and believe the smaller.

Most callers want available_memory_bytes(). The public API:

• available_memory_bytes() – bytes this process can still allocate; the number to budget against.

• cgroup_limit_and_usage() – the binding cgroup (limit, current) in bytes, or None when unlimited.

• cgroup_memory_breakdown() – current usage split by reclaimability (anon, shmem, file, dirty, writeback).

• log_stage_memory() – log the memory position after a pipeline stage.

• describe_memory() – one line covering both views, for logs where a budget is decided.

Everything else in the module is the /proc and cgroup plumbing behind those five.

Attributes

logger

Functions

available_memory_bytes()	Bytes this process can still allocate, as the minimum of OS-free and cgroup-remaining.
cgroup_limit_and_usage()	The tightest (limit, current) in bytes over every cgroup constraining this process.
cgroup_memory_breakdown()	memory.stat split into the categories that behave differently under pressure.
describe_memory()	One line covering both views, for logging where a budget is being decided.
log_stage_memory(stage)	Log the memory position after a pipeline stage, split by reclaimability.

Module Contents

gigl.utils.host_memory.available_memory_bytes()

Bytes this process can still allocate, as the minimum of OS-free and cgroup-remaining.

Both are needed: the OS figure misses a container limit below the machine’s RAM, and the cgroup figure misses pressure from other processes on a shared host.

Return type:

int

gigl.utils.host_memory.cgroup_limit_and_usage()

The tightest (limit, current) in bytes over every cgroup constraining this process.

Every ancestor constrains the process, so the first finite limit found walking upward is not necessarily the binding one: a leaf with 63 GiB of headroom under a parent with 1 GiB is really limited to 1 GiB. All applicable levels are read and the least headroom wins.

Returns None when no limit can be read.

Return type:

Optional[tuple[int, int]]

gigl.utils.host_memory.cgroup_memory_breakdown()

memory.stat split into the categories that behave differently under pressure.

memory.current alone cannot say whether a peak is survivable: file_dirty bytes are reclaimable once written back, so the kernel throttles the writer, while the same fraction held as anon is fatal.

Returns an empty dict when no cgroup or no memory.stat can be read.

Return type:

dict[str, int]

gigl.utils.host_memory.describe_memory()

One line covering both views, for logging where a budget is being decided.

Return type:

str

gigl.utils.host_memory.log_stage_memory(stage)

Log the memory position after a pipeline stage, split by reclaimability.

Reads three small proc files, so call it at phase boundaries rather than in loops.

Parameters:

stage (str) – What just finished, e.g. "assembled node features".

Return type:

None

gigl.utils.host_memory.logger