actorcritic.kfac_utils

Contains utilities that concern K-FAC.

Classes

ColdStartPeriodicInvUpdateKfacOpt(…)

A modified KfacOptimizer that runs the inverse operation periodically and uses a standard SGD optimizer for a few updates in the beginning, called cold updates and cold optimizer.

class actorcritic.kfac_utils.ColdStartPeriodicInvUpdateKfacOpt(num_cold_updates, cold_optimizer, invert_every, **kwargs)

Bases: kfac.python.ops.optimizer.KfacOptimizer

A modified KfacOptimizer that runs the inverse operation periodically and uses a standard SGD optimizer for a few updates in the beginning, called cold updates and cold optimizer.

This can be used to slowly initialize the parameters in the beginning before using the heavy K-FAC optimizer. The covariances get updated every step (after the cold updates).

See also

• kfac.PeriodicInvCovUpdateKfacOpt
• The idea is taken from the original ACKTR implementation.

__init__(num_cold_updates, cold_optimizer, invert_every, **kwargs)

Parameters:

• num_cold_updates (int) – The number of cold updates in the beginning before using the actual K-FAC optimizer.
• cold_optimizer (tf.train.Optimizer) – An optimizer that is used for the cold updates.
• invert_every (int) – The inverse operation gets called every invert_every steps (after the cold updates have finished).

apply_gradients(grads_and_vars, global_step=None, name=None)

Applies gradients to variables.

Parameters:

• grads_and_vars – List of (gradient, variable) pairs.
• *args – Additional arguments for super.apply_gradients.
• **kwargs – Additional keyword arguments for super.apply_gradients.

Returns:

An Operation that applies the specified gradients.