{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "\n# Subtractive synthesis\n\n**Author**: [Moto Hira](moto@meta.com)_\n\nThis tutorial is the continuation of\n[Filter Design Tutorial](./filter_design_tutorial.html)_.\n\nThis tutorial shows how to perform subtractive synthesis with TorchAudio's DSP functions.\n\nSubtractive synthesis creates timbre by applying filters to source waveform.\n\n<div class=\"alert alert-danger\"><h4>Warning</h4><p>This tutorial requires prototype DSP features, which are\n   available in nightly builds.\n\n   Please refer to https://pytorch.org/get-started/locally\n   for instructions for installing a nightly build.</p></div>\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "import torch\nimport torchaudio\n\nprint(torch.__version__)\nprint(torchaudio.__version__)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Overview\n\n\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "try:\n    from torchaudio.prototype.functional import filter_waveform, frequency_impulse_response, sinc_impulse_response\nexcept ModuleNotFoundError:\n    print(\n        \"Failed to import prototype DSP features. \"\n        \"Please install torchaudio nightly builds. \"\n        \"Please refer to https://pytorch.org/get-started/locally \"\n        \"for instructions to install a nightly build.\"\n    )\n    raise\n\nimport matplotlib.pyplot as plt\nfrom IPython.display import Audio"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Filtered Noise\n\nSubtractive synthesis starts with a waveform and applies filters to\nsome frequency components.\n\nFor the first example of subtractive synthesis, we apply\ntime-varying low pass filter to white noise.\n\nFirst, we create a white noise.\n\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "SAMPLE_RATE = 16_000\nduration = 4\nnum_frames = int(duration * SAMPLE_RATE)\n\nnoise = torch.rand((num_frames,)) - 0.5"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "def plot_input():\n    fig, axes = plt.subplots(2, 1, sharex=True)\n    t = torch.linspace(0, duration, num_frames)\n    axes[0].plot(t, noise)\n    axes[0].grid(True)\n    axes[1].specgram(noise, Fs=SAMPLE_RATE)\n    Audio(noise, rate=SAMPLE_RATE)\n\n\nplot_input()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Windowed-sinc filter\n\n\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Sweeping cutoff frequency\n\nWe use :py:func:`~torchaudio.prototype.functional.sinc_impulse_response` to\ncreate series of low pass filters, while changing the  cut-off\nfrequency from zero to Nyquist frequency.\n\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "num_filters = 64 * duration\nwindow_size = 2049\n\nf_cutoff = torch.linspace(0.0, 0.8, num_filters)\nkernel = sinc_impulse_response(f_cutoff, window_size)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "To apply time-varying filter, we use\n:py:func:`~torchaudio.prototype.functional.filter_waveform`\n\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "filtered = filter_waveform(noise, kernel)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Let's look at the spectrogram of the resulting audio and listen to it.\n\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "def plot_sinc_ir(waveform, cutoff, sample_rate, vol=0.2):\n    num_frames = waveform.size(0)\n    duration = num_frames / sample_rate\n    num_cutoff = cutoff.size(0)\n    nyquist = sample_rate / 2\n\n    _, axes = plt.subplots(2, 1, sharex=True)\n    t = torch.linspace(0, duration, num_frames)\n    axes[0].plot(t, waveform)\n    axes[0].grid(True)\n    axes[1].specgram(waveform, Fs=sample_rate, scale=\"dB\")\n    t = torch.linspace(0, duration, num_cutoff)\n    axes[1].plot(t, cutoff * nyquist, color=\"gray\", linewidth=0.8, label=\"Cutoff Frequency\", linestyle=\"--\")\n    axes[1].legend(loc=\"upper center\")\n    axes[1].set_ylim([0, nyquist])\n    waveform /= waveform.abs().max()\n    return Audio(vol * waveform, rate=sample_rate, normalize=False)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "plot_sinc_ir(filtered, f_cutoff, SAMPLE_RATE)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Oscillating cutoff frequency\n\nBy oscillating the cutoff frequency, we can emulate an effect of\nLow-frequency oscillation (LFO).\n\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "PI2 = torch.pi * 2\nnum_filters = 90 * duration\n\nf_lfo = torch.linspace(0.9, 0.1, num_filters)\nf_cutoff_osci = torch.linspace(0.01, 0.03, num_filters) * torch.sin(torch.cumsum(f_lfo, dim=0))\nf_cutoff_base = torch.linspace(0.8, 0.03, num_filters) ** 1.7\nf_cutoff = f_cutoff_base + f_cutoff_osci"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "kernel = sinc_impulse_response(f_cutoff, window_size)\nfiltered = filter_waveform(noise, kernel)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "plot_sinc_ir(filtered, f_cutoff, SAMPLE_RATE)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Wah-wah effects\n\nWah-wah effects are applications of low-pass filter or band-pass filter.\nThey change the cut-off freuqnecy or Q-factor quickly.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "f_lfo = torch.linspace(0.15, 0.15, num_filters)\nf_cutoff = 0.07 + 0.06 * torch.sin(torch.cumsum(f_lfo, dim=0))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "kernel = sinc_impulse_response(f_cutoff, window_size)\nfiltered = filter_waveform(noise, kernel)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "plot_sinc_ir(filtered, f_cutoff, SAMPLE_RATE)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Arbitrary frequence response\n\nBy using\n:py:func:`~torchaudio.prototype.functinal.frequency_impulse_response`,\none can directly control the power distribution over frequency.\n\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "magnitudes = torch.sin(torch.linspace(0, 10, 64)) ** 4.0\nkernel = frequency_impulse_response(magnitudes)\nfiltered = filter_waveform(noise, kernel.unsqueeze(0))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "def plot_waveform(magnitudes, filtered, sample_rate):\n    nyquist = sample_rate / 2\n    num_samples = filtered.size(-1)\n    duration = num_samples / sample_rate\n\n    # Re-organize magnitudes for overlay\n    N = 10  # number of overlays\n    interval = torch.linspace(0.05, 0.95, N)\n    offsets = duration * interval\n    # Select N magnitudes for overlays\n    mags = torch.stack(\n        [magnitudes for _ in range(N)]\n        if magnitudes.ndim == 1\n        else [magnitudes[int(i * magnitudes.size(0))] for i in interval]\n    )\n    mag_x = offsets.unsqueeze(-1) + 0.1 * mags\n    mag_y = torch.linspace(0, nyquist, magnitudes.size(-1)).tile((N, 1))\n\n    _, ax = plt.subplots(1, 1, sharex=True)\n    ax.vlines(offsets, 0, nyquist, color=\"gray\", linestyle=\"--\", linewidth=0.8)\n    ax.plot(mag_x.T.numpy(), mag_y.T.numpy(), color=\"gray\", linewidth=0.8)\n    ax.specgram(filtered, Fs=sample_rate)\n    return Audio(filtered, rate=sample_rate)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "plot_waveform(magnitudes, filtered, SAMPLE_RATE)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "It is also possible to make a non-stationary filter.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "magnitudes = torch.stack([torch.linspace(0.0, w, 1000) for w in torch.linspace(4.0, 40.0, 250)])\nmagnitudes = torch.sin(magnitudes) ** 4.0"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "kernel = frequency_impulse_response(magnitudes)\nfiltered = filter_waveform(noise, kernel)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "plot_waveform(magnitudes, filtered, SAMPLE_RATE)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Of course it is also possible to emulate simple low pass filter.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "magnitudes = torch.concat([torch.ones((32,)), torch.zeros((32,))])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "kernel = frequency_impulse_response(magnitudes)\nfiltered = filter_waveform(noise, kernel.unsqueeze(0))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "plot_waveform(magnitudes, filtered, SAMPLE_RATE)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## References\n\n- https://en.wikipedia.org/wiki/Additive_synthesis\n- https://computermusicresource.com/Simple.bell.tutorial.html\n- https://computermusicresource.com/Definitions/additive.synthesis.html\n\n"
      ]
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.10.14"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}