OptSchedule

Flexible parameter scheduler that can be implemented with proprietary and open source optimizers and algorithms.

Installation

optschedule can be installed through Python’s package installer pip. To install, run

pip install optschedule

in your terminal. Alternatively, install the package directly from GitHub

git clone -b development https://github.com/draktr/optschedule.git
cd monte
python setup.py install

Features

  • Exponential decay (gradual and staircase)

  • Cosine decay

  • Inverse time decay (gradual and staircase)

  • Polynomial decay

  • Piecewise constant decay

  • Constant schedule

  • Geometric decay

  • Arithmetic decay

  • Time decay

  • Step decay

Advantages

  • FLEXIBLE - the package is designed to be simple and compatible with existing implementations and custom algorithms

  • COMPREHENSIVE - the package contains the largest collection of schedules of any Python package. For more, feel free to raise a feature request in Issues.

  • NUMBA FRIENDLY - schedule produced by the package is compatible with Numba and will not cause any issues if the rest of the algorithm is Numba compatible. This can drastically speed up the algorithm.

Usage

Package contains Schedule class with methods that return an array of elements that is useful as a pre-defined parameter schedule (e.g. learning rate). The package can also be used for manually assigning varying weights to abstract particles. Overall, due to the general nature of the package a user might finds its own particular application.

Example: Variable Learning Rate in Gradient Descent Algorithm

from optschedule import Schedule

# Function to be minimized (objective function) $ f(x) = (x+2)^2 $
def foo(params):
    return (params[0] + 2) ** 2

# Creating learning rate schedule
schedule = Schedule(1000)
learning_rate = schedule.exponential_decay(initial_value=0.1, decay_rate=0.5)

# Array with objective value
objective = np.zeros(1000)
# Initial parameter value
param = [10]
# Difference
d = 0.01

# Gradient Descent Algorithm
for epoch, l in enumerate(learning_rate):
    objective[epoch] = foo(param)
    difference_objective = foo([param[0]+d])
    param[0] = param[0] - l*(difference_objective - objective[epoch])/d

print(f"Solution: {param[0]}")

Maintaining and Contributing

Feel free to reach out through Issues forum to add or request features. Any issues, bugs and improvement recommendations are very welcome.

Schedule

The schedule module houses Schedule class that produces sequences that can be implemented with proprietary and open source optimizers and algorithms.

raises ValueError:

Error if there is more or less than exactly one more element of values that boundaries

return:

Sequence of values with each element being a value (e.g. learning rate or difference) for each epoch

rtype:

ndarray

class optschedule.schedule.Schedule(n_steps)[source]

Bases: object

__init__(n_steps) → None[source]

Initializes Schedule Object.

Object properties:

  • n_steps - number of steps

  • steps - sequence of steps

Parameters:

n_steps (int) – Number of decay steps. Must be equal to the number of epochs of the algorithm

arithmetic_decay(initial_value, decay_rate, minimum_value)[source]

Sequence with arithmetic decay given by

\[\text{value} = \max(\text{initial_value} - \text{decay_rate} \times \text{i}, \text{minimum_value})\]
Parameters:

  • initial_value (float) – Initial value of the sequence

  • decay_rate (float) – Rate of arithmetic decay

  • minimum_value (float) – Minimum value of the sequence

Returns:

Sequence of values with each element being a value (e.g. learning rate or difference) for each epoch

Return type:

ndarray

constant(value)[source]

Constant sequence

Parameters:

value (float) – Value for each epoch

Returns:

Sequence of values with each element being a value (e.g. learning rate or difference) for each epoch

Return type:

ndarray

cosine_decay(initial_value, alpha)[source]

Sequence with cosine decay.

Parameters:

  • initial_value (float) – Initial value of the sequence

  • alpha (float) – Minimum sequence value as a fraction of initial_value

Returns:

Sequence of values with each element being a value (e.g. learning rate or difference) for each epoch

Return type:

ndarray

exponential_decay(initial_value, decay_rate, staircase=False)[source]

Sequence with exponential decay.

Parameters:

  • initial_value (float) – Initial value of the sequence

  • decay_rate (float) – Rate of decay

  • staircase (bool, optional) – If True decay the sequence at discrete intervals, defaults to False

Returns:

Sequence of values with each element being a value (e.g. learning rate or difference) for each epoch

Return type:

ndarray

geometric_decay(initial_value, decay_rate, minimum_value)[source]

Sequence with geometric decay given by

\[\text{value} = \max(\text{initial_value} \times \text{decay_rate}^{\text{i}}, \text{minimum_value})\]
Parameters:

  • initial_value (float) – Initial value of the sequence

  • decay_rate (float) – Rate of geometric decay

  • minimum_value (float) – Minimum value of the sequence

Returns:

Sequence of values with each element being a value (e.g. learning rate or difference) for each epoch

Return type:

ndarray

inverse_time_decay(initial_value, decay_rate, staircase=False)[source]

Sequence with inverse time decay

Parameters:

  • initial_value (float) – Initial value of the sequence

  • decay_rate (float) – Rate of decay

  • staircase (bool, optional) – If True decay the sequence at discrete intervals, defaults to False

Returns:

Sequence of values with each element being a value (e.g. learning rate or difference) for each epoch

Return type:

ndarray

piecewise_constant_decay(boundaries, values)[source]

Sequence with piecewise constant decay.

Parameters:

  • boundaries (list) – Boundaries of the pieces

  • values (list) – list of values in sequence in each of the pieces

Raises:

ValueError – Error if there is more or less than exactly one more element of values that boundaries

Returns:

Sequence of values with each element being a value (e.g. learning rate or difference) for each epoch

Return type:

ndarray

polynomial_decay(initial_value, end_value, power, cycle=False)[source]

Sequence with polynomial decay.

Parameters:

  • initial_value (float) – Initial value of the sequence

  • end_value (float) – The minimal end sequence value

  • power (float) – The power of the polynomial

  • cycle (bool, optional) – Whether or not it should cycle beyond self.n_steps, defaults to False

Returns:

Sequence of values with each element being a value (e.g. learning rate or difference) for each epoch

Return type:

ndarray

step_decay(initial_value, decay_value, decay_every)[source]

Sequence with step decay given by

\[\text{value} = \text{initial_value} \times \text{decay_value}^{\lfloor \frac{i}{\text{decay_every}} \rfloor}\]

where i is the current value in the sequence.

Parameters:

  • initial_value (float) – Initial value of the sequence

  • decay_value (float) – Drop size

  • decay_every (int) – Drop is performed every decay_every values

Returns:

Sequence of values with each element being a value (e.g. learning rate or difference) for each epoch

Return type:

ndarray

time_decay(initial_value, decay_rate)[source]

Sequence with time-based decay given by

\[\text{value} = \frac{\text{prev_value}}{1+\text{decay_rate} \times \text{i}}\]

where prev_value is the previous value in the sequence and decay_rate is the decay rate parameter.

Parameters:

  • initial_value (float) – Initial value of the sequence

  • decay_rate (float) – Decay rate

Returns:

Sequence of values with each element being a value (e.g. learning rate or difference) for each epoch

Return type:

ndarray

Indices and tables