WaveNet

A flow based vocoder and an autoregressive model.Published in 2016.

Link: https://arxiv.org/pdf/1609.03499.pdf

Github: https://github.com/r9y9/wavenet_vocoder

Introduction

​ WaveNet is a flow based vocoder and it is an autoregressive model which means that WaveNet takes previous output as input. Here will briefly explain what is flow.

Flow

​ Flow is a method based on probability. It uses a series of reversible function to represent a complex probability distribution with a simple distribution and a series of function.

​ Flow based on a constraint $\int_{-\infty}^{\infty}P(x)dx=1$, so we can know, for each function $y=f(x)$, there is

$p_y(y) \cdot dy=p_x(x) \cdot dx\\$

because we do not care about if $dx$ or $dy$ is positive or negative, and probability is always positive, so we have

$p_x(x)=p_y(y)\left|\frac{dy}{dx}\right|\\$

then we can transform it to log domain, so we have

$\log{p_x(x)}=\log{p_y(y)}+\log{\left|\frac{dy}{dx}\right|}\\$

Now we can consider about multivariate case, where we can alter $\frac{dy}{dx}$ with $\det{J(f(x))}$, where $J(\cdot)$ means the jacobian

matrix of a function. we can do this transformation because

The Jacobian matrix represents the differential of f at every point where f is differentiable.

​ ——Wikipedia

then we can make it a chain, assume $x_n=f_n(f_{n-1}(...f_1(x_0)))$ , then we have

$\log{p_n(x_n)}=\log{p_0(x_0)}-\sum_{i=1}^{n}\log{\left|\det{J(f_i(x))}\right|}$

so this is the flow of the probability distribution.

Autoregressive Flow

According to what I have said above, autoregressive means that it takes previous output as input. we combine autoregressive and flow and we can get autoregressive flow, which is used in WaveNet.

In autoregressive flow, we have

$z_t = x_t \cdot\sigma_t(x_{<t};\upsilon)+\mu_t(x_{<t};\upsilon)\\ x_t=\frac{z_t-\mu_t(x_{<t};\upsilon)}{\sigma_t(x_{<t};\upsilon)}$

where $\sigma_t(x_{<t};\upsilon)$ and $\mu_t(x_{<t};\upsilon)$ refer to neural network which have parameter $\upsilon$ and take input $x_{<t}$(output before time $t$).Then we can easily calculate the jacobian matrix of $z(x)$, and we can get

Note this jacobian matrix is a lower triangular matrix, so we can easily calculate the determinant of this matrix.

During train period we have

and during train period we have

we use $exp(\alpha)$ because $\alpha$ often uses as a natural logarithm in modeling.

Novelty

Dilated Causal Convolution

The main ingredient of WaveNet are causal convolutions.

By using causal convolutions, the model cannot depend on any of the future timesteps.

But compared to RNN, the reception field of causal convolution is only 5(layers+filter length-1).So in this paper, it had used dilated convolution.

A dilated convolution (also called a trous ` , or convolution with holes) is a convolution where the filter is applied over an area larger than its length by skipping input values with a certain step.

By using dilated convolution, the reception field of the model comes to 16($2^{layers}$), and it increases exponentially as layer increases.

Gated Activation Units

Same as PixelCNN.

$z=tanh(W_{f,k}\ast x)\odot \sigma(W_{g,k}\ast x)$

Conditional WaveNets

WaveNet can use a given additional input h to model distribution $p(x|\bold{h})$.

It can be used in multi-speaker condition or takes extra information as input.