ott.initializers.nn.initializers.MetaMLP.perturb#
- MetaMLP.perturb(name, value, collection='perturbations')#
Add an zero-value variable (‘perturbation’) to the intermediate value.
The gradient of value would be the same as the gradient of this perturbation variable. Therefore, if you define your loss function with both params and perturbations as standalone arguments, you can get the intermediate gradients of value by running jax.grad on the perturbation argument.
Note: this is an experimental API and may be tweaked later for better performance and usability. At its current stage, it creates extra dummy variables that occupies extra memory space. Use it only to debug gradients in training.
Example:
import jax import jax.numpy as jnp import flax.linen as nn class Foo(nn.Module): @nn.compact def __call__(self, x): x = nn.Dense(3)(x) x = self.perturb('dense3', x) return nn.Dense(2)(x) def loss(params, perturbations, inputs, targets): variables = {'params': params, 'perturbations': perturbations} preds = model.apply(variables, inputs) return jnp.square(preds - targets).mean() x = jnp.ones((2, 9)) y = jnp.ones((2, 2)) model = Foo() variables = model.init(jax.random.PRNGKey(0), x) intm_grads = jax.grad(loss, argnums=1)(variables['params'], variables['perturbations'], x, y) print(intm_grads['dense3']) # ==> [[-1.456924 -0.44332537 0.02422847] # [-1.456924 -0.44332537 0.02422847]]
If perturbations are not passed to apply, perturb behaves like a no-op so you can easily disable the behavior when not needed:
model.apply({'params': params, 'perturbations': perturbations}, x) # works as expected model.apply({'params': params}, x) # behaves like a no-op