cd.optim

class ReduceLROnPlateau(optimizer, mode='min', factor=0.1, patience=10, threshold=0.0001, threshold_mode='rel', cooldown=0, min_lr=0, eps=1e-08, warmup=1, verbose='deprecated')

Initializes the ReduceLROnPlateau object. This scheduler decreases the learning rate when a metric has stopped improving, which is commonly used to fine-tune a model in machine learning.

Notes

  • Adds the warmup option to PyTorch’s ReduceLROnPlateau.

Parameters:

  • optimizer (Optimizer) – Wrapped optimizer.

  • mode (str) – One of min or max. In min mode, the learning rate will be reduced when the quantity monitored has stopped decreasing; in max mode, it will be reduced when the quantity monitored has stopped increasing. Default: ‘min’.

  • factor (float) – Factor by which the learning rate will be reduced. new_lr = lr * factor. Default: 0.1.

  • patience (int) – Number of epochs with no improvement after which learning rate will be reduced. Default: 10.

  • threshold (float) – Threshold for measuring the new optimum, to only focus on significant changes. Default: 1e-4.

  • threshold_mode (str) – One of rel, abs. In rel mode, dynamic_threshold = best * (1 + threshold) in ‘max’ mode or best * (1 - threshold) in min mode. In abs mode, dynamic_threshold = best + threshold in max mode or best - threshold in min mode. Default: ‘rel’.

  • cooldown (int) – Number of epochs to wait before resuming normal operation after lr has been reduced. Default: 0.

  • min_lr (float or list) – A scalar or a list of scalars. A lower bound on the learning rate of all param groups or each group respectively. Default: 0.

  • eps (float) – Minimal decay applied to lr. If the difference between new and old lr is smaller than eps, the update is ignored. Default: 1e-8.

  • warmup (int) – Number of epochs to wait before initially starting normal operation. Default: 1.

  • verbose (str) – Deprecated argument. Not used. Default: “deprecated”.

get_last_lr()
step(metrics, epoch=None)
class SequentialLR(optimizer, schedulers, milestones, last_epoch=-1, verbose='deprecated')
step(metrics=None)
class WarmUp(optimizer: Optimizer, steps: int, lr_lambda: Callable[[int, int], float] | List[Callable[[int, int], float]] | None = None, last_epoch: int = -1, verbose: bool = False)

WarmUp.

Applies a scaling factor to learning rate for steps 1 to steps. Applies no changes otherwise. WarmUp can be chained or used sequentally with other schedulers.

Notes

  • WarmUp sets learning rates based on captured initial learning rates (base_lrs).

  • Changes from other schedulers applied before WarmUp will be ovewritten.

  • Chaining WarmUp is equivalent to SequentialLR if other schedule dynamically manipulates optimizer.param_groups without relying on base_lrs.

Examples

>>> # First WarmUp, then other schedule
>>> from torch.optim.lr_scheduler import SequentialLR, CosineAnnealingLR
>>> from celldetection.optim import WarmUp
>>> warmup_steps = 512
>>> scheduler = SequentialLR(optimizer, [
...     WarmUp(optimizer, warmup_steps),  # warmup for 512 steps
...     CosineAnnealingLR(optimizer, T_max=512, eta_min=0.00001),  # after 512 steps switch to cosine ann.
... ], milestones=[warmup_steps])

>>> # Chaining WarmUp and other schedule
>>> from torch.optim.lr_scheduler import ChainedScheduler, StepLR
>>> scheduler = ChainedScheduler([
...     StepLR(optimizer, 5, gamma=.99),
...     WarmUp(optimizer, 512),  # overwrites changes of previous scheduler during warmup
... ])
Parameters:

  • optimizer – Optimizer.

  • steps – Number of warmup steps.

  • lr_lambda – A function which computes a multiplicative factor given an integer parameter epoch, or a list of such functions, one for each group in optimizer.param_groups.

  • last_epoch – Last epoch.

  • verbose – If True, prints a message to stdout for each update. Default: False.

get_lr()