![](\"https://raw.githubusercontent.com/GokuMohandas/practicalAI/master/images/rnn.png\")
\n",
+ "\n",
+ "\n"
+ ]
+ },
+ {
+ "metadata": {
+ "id": "VoMq0eFRvugb",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "# Overview"
+ ]
+ },
+ {
+ "metadata": {
+ "id": "qWro5T5qTJJL",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "* **Objective:** Process sequential data by accounting for the currend input and also what has been learned from previous inputs.\n",
+ "* **Advantages:** \n",
+ " * Account for order and previous inputs in a meaningful way.\n",
+ " * Conditioned generation for generating sequences.\n",
+ "* **Disadvantages:** \n",
+ " * Each time step's prediction depends on the previous prediction so it's difficult to parallelize RNN operations. \n",
+ " * Processing long sequences can yield memory and computation issues.\n",
+ " * Interpretability is difficult but there are few [techniques](https://arxiv.org/abs/1506.02078) that use the activations from RNNs to see what parts of the inputs are processed. \n",
+ "* **Miscellaneous:** \n",
+ " * Architectural tweaks to make RNNs faster and interpretable is an ongoing area of research."
+ ]
+ },
+ {
+ "metadata": {
+ "id": "rsHeBbehrKzl",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "![](\"https://raw.githubusercontent.com/GokuMohandas/practicalAI/master/images/rnn2.png\")
\n",
+ "\n",
+ "RNN forward pass for a single time step $X_t$:\n",
+ "\n",
+ "$h_t = tanh(W_{hh}h_{t-1} + W_{xh}X_t+b_h)$\n",
+ "\n",
+ "$y_t = W_{hy}h_t + b_y $\n",
+ "\n",
+ "$ P(y) = softmax(y_t) = \\frac{e^y}{\\sum e^y} $\n",
+ "\n",
+ "*where*:\n",
+ "* $X_t$ = input at time step t | $\\in \\mathbb{R}^{NXE}$ ($N$ is the batch size, $E$ is the embedding dim)\n",
+ "* $W_{hh}$ = hidden units weights| $\\in \\mathbb{R}^{HXH}$ ($H$ is the hidden dim)\n",
+ "* $h_{t-1}$ = previous timestep's hidden state $\\in \\mathbb{R}^{NXH}$\n",
+ "* $W_{xh}$ = input weights| $\\in \\mathbb{R}^{EXH}$\n",
+ "* $b_h$ = hidden units bias $\\in \\mathbb{R}^{HX1}$\n",
+ "* $W_{hy}$ = output weights| $\\in \\mathbb{R}^{HXC}$ ($C$ is the number of classes)\n",
+ "* $b_y$ = output bias $\\in \\mathbb{R}^{CX1}$\n",
+ "\n",
+ "You repeat this for every time step's input ($X_{t+1}, X_{t+2}, ..., X_{N})$ to the get the predicted outputs at each time step.\n",
+ "\n",
+ "**Note**: At the first time step, the previous hidden state $h_{t-1}$ can either be a zero vector (unconditioned) or initialize (conditioned). If we are conditioning the RNN, the first hidden state $h_0$ can belong to a specific condition or we can concat the specific condition to the randomly initialized hidden vectors at each time step. More on this in the subsequent notebooks on RNNs."
+ ]
+ },
+ {
+ "metadata": {
+ "id": "dIXlGMExJD6w",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "Let's see what the forward pass looks like with an RNN for a synthetic task such as processing reviews (a sequence of words) to predict the sentiment at the end of processing the review."
+ ]
+ },
+ {
+ "metadata": {
+ "id": "RcWE5cw0_cKA",
+ "colab_type": "code",
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 34
+ },
+ "outputId": "a44156b9-b43f-409c-f0ce-4a4bd871d6a0"
+ },
+ "cell_type": "code",
+ "source": [
+ "# Load PyTorch library\n",
+ "!pip3 install torch"
+ ],
+ "execution_count": 1,
+ "outputs": [
+ {
+ "output_type": "stream",
+ "text": [
+ "Requirement already satisfied: torch in /usr/local/lib/python3.6/dist-packages (1.0.0)\n"
+ ],
+ "name": "stdout"
+ }
+ ]
+ },
+ {
+ "metadata": {
+ "id": "o6eEK1wM_dXG",
+ "colab_type": "code",
+ "colab": {}
+ },
+ "cell_type": "code",
+ "source": [
+ "import torch\n",
+ "import torch.nn as nn\n",
+ "import torch.nn.functional as F"
+ ],
+ "execution_count": 0,
+ "outputs": []
+ },
+ {
+ "metadata": {
+ "id": "Qi9hIEV6COLF",
+ "colab_type": "code",
+ "colab": {}
+ },
+ "cell_type": "code",
+ "source": [
+ "batch_size = 5\n",
+ "seq_size = 10 # max length per input (masking will be used for sequences that aren't this max length)\n",
+ "x_lengths = [8, 5, 4, 10, 5] # lengths of each input sequence\n",
+ "embedding_dim = 100\n",
+ "rnn_hidden_dim = 256\n",
+ "output_dim = 4"
+ ],
+ "execution_count": 0,
+ "outputs": []
+ },
+ {
+ "metadata": {
+ "id": "bLEzfxjhB94C",
+ "colab_type": "code",
+ "outputId": "f2feefbf-8635-4b23-ef53-b5713cf2cdb2",
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 34
+ }
+ },
+ "cell_type": "code",
+ "source": [
+ "# Initialize synthetic inputs\n",
+ "x_in = torch.randn(batch_size, seq_size, embedding_dim)\n",
+ "x_lengths = torch.tensor(x_lengths)\n",
+ "print (x_in.size())"
+ ],
+ "execution_count": 4,
+ "outputs": [
+ {
+ "output_type": "stream",
+ "text": [
+ "torch.Size([5, 10, 100])\n"
+ ],
+ "name": "stdout"
+ }
+ ]
+ },
+ {
+ "metadata": {
+ "id": "dr6oLqtXB98N",
+ "colab_type": "code",
+ "outputId": "9817e88d-6e73-414a-dfa6-2386f40db0d9",
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 34
+ }
+ },
+ "cell_type": "code",
+ "source": [
+ "# Initialize hidden state\n",
+ "hidden_t = torch.zeros((batch_size, rnn_hidden_dim))\n",
+ "print (hidden_t.size())"
+ ],
+ "execution_count": 5,
+ "outputs": [
+ {
+ "output_type": "stream",
+ "text": [
+ "torch.Size([5, 256])\n"
+ ],
+ "name": "stdout"
+ }
+ ]
+ },
+ {
+ "metadata": {
+ "id": "ryZMOLLgB9-v",
+ "colab_type": "code",
+ "outputId": "14ec0a2a-bf37-4e03-b69b-099180f8f149",
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 34
+ }
+ },
+ "cell_type": "code",
+ "source": [
+ "# Initialize RNN cell\n",
+ "rnn_cell = nn.RNNCell(embedding_dim, rnn_hidden_dim)\n",
+ "print (rnn_cell)"
+ ],
+ "execution_count": 6,
+ "outputs": [
+ {
+ "output_type": "stream",
+ "text": [
+ "RNNCell(100, 256)\n"
+ ],
+ "name": "stdout"
+ }
+ ]
+ },
+ {
+ "metadata": {
+ "id": "rlbZ7ujxExXb",
+ "colab_type": "code",
+ "outputId": "6c83ba2b-94c5-4f76-c8fb-ef0c1ccdeb37",
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 34
+ }
+ },
+ "cell_type": "code",
+ "source": [
+ "# Forward pass through RNN\n",
+ "x_in = x_in.permute(1, 0, 2) # RNN needs batch_size to be at dim 1\n",
+ "\n",
+ "# Loop through the inputs time steps\n",
+ "hiddens = []\n",
+ "for t in range(seq_size):\n",
+ " hidden_t = rnn_cell(x_in[t], hidden_t)\n",
+ " hiddens.append(hidden_t)\n",
+ "hiddens = torch.stack(hiddens)\n",
+ "hiddens = hiddens.permute(1, 0, 2) # bring batch_size back to dim 0\n",
+ "print (hiddens.size())"
+ ],
+ "execution_count": 7,
+ "outputs": [
+ {
+ "output_type": "stream",
+ "text": [
+ "torch.Size([5, 10, 256])\n"
+ ],
+ "name": "stdout"
+ }
+ ]
+ },
+ {
+ "metadata": {
+ "id": "3TTL-jmg-MHa",
+ "colab_type": "code",
+ "outputId": "3fae323f-c37d-4dac-c8a8-7fea7a45c95c",
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 51
+ }
+ },
+ "cell_type": "code",
+ "source": [
+ "# We also could've used a more abstracted layer\n",
+ "x_in = torch.randn(batch_size, seq_size, embedding_dim)\n",
+ "rnn = nn.RNN(embedding_dim, rnn_hidden_dim, batch_first=True)\n",
+ "out, h_n = rnn(x_in) #h_n is the last hidden state\n",
+ "print (\"out: \", out.size())\n",
+ "print (\"h_n: \", h_n.size())"
+ ],
+ "execution_count": 8,
+ "outputs": [
+ {
+ "output_type": "stream",
+ "text": [
+ "out: torch.Size([5, 10, 256])\n",
+ "h_n: torch.Size([1, 5, 256])\n"
+ ],
+ "name": "stdout"
+ }
+ ]
+ },
+ {
+ "metadata": {
+ "id": "iAsyRNnbHwcT",
+ "colab_type": "code",
+ "colab": {}
+ },
+ "cell_type": "code",
+ "source": [
+ "def gather_last_relevant_hidden(hiddens, x_lengths):\n",
+ " x_lengths = x_lengths.long().detach().cpu().numpy() - 1\n",
+ " out = []\n",
+ " for batch_index, column_index in enumerate(x_lengths):\n",
+ " out.append(hiddens[batch_index, column_index])\n",
+ " return torch.stack(out)"
+ ],
+ "execution_count": 0,
+ "outputs": []
+ },
+ {
+ "metadata": {
+ "id": "PVhp1KLqHqpA",
+ "colab_type": "code",
+ "outputId": "d04be3ef-c2d6-48b9-f0f5-a93f619ec594",
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 34
+ }
+ },
+ "cell_type": "code",
+ "source": [
+ "# Gather the last relevant hidden state\n",
+ "z = gather_last_relevant_hidden(hiddens, x_lengths)\n",
+ "print (z.size())"
+ ],
+ "execution_count": 10,
+ "outputs": [
+ {
+ "output_type": "stream",
+ "text": [
+ "torch.Size([5, 256])\n"
+ ],
+ "name": "stdout"
+ }
+ ]
+ },
+ {
+ "metadata": {
+ "id": "yGk_iZ5cITZl",
+ "colab_type": "code",
+ "outputId": "84749ff2-1e45-4599-a38d-8c83cee116a9",
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 119
+ }
+ },
+ "cell_type": "code",
+ "source": [
+ "# Forward pass through FC layer\n",
+ "fc1 = nn.Linear(rnn_hidden_dim, output_dim)\n",
+ "y_pred = fc1(z)\n",
+ "y_pred = F.softmax(y_pred, dim=1)\n",
+ "print (y_pred.size())\n",
+ "print (y_pred)"
+ ],
+ "execution_count": 11,
+ "outputs": [
+ {
+ "output_type": "stream",
+ "text": [
+ "torch.Size([5, 4])\n",
+ "tensor([[0.3030, 0.2351, 0.2168, 0.2452],\n",
+ " [0.2614, 0.1912, 0.2617, 0.2858],\n",
+ " [0.2428, 0.2600, 0.2254, 0.2717],\n",
+ " [0.2379, 0.2226, 0.1901, 0.3494],\n",
+ " [0.2629, 0.2854, 0.2146, 0.2371]], grad_fn=![](\"https://raw.githubusercontent.com/GokuMohandas/practicalAI/master/images/seq2seq.jpeg\")
"
+ ]
+ },
+ {
+ "metadata": {
+ "id": "tnxUIEMdukYY",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "# Issues with vanilla RNNs"
+ ]
+ },
+ {
+ "metadata": {
+ "id": "uMx2s93VLUTt",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "There are several issues with the vanilla RNN that we've seen so far. \n",
+ "\n",
+ "1. When we have an input sequence that has many time steps, it becomes difficult for the model to retain information seen earlier as we process more and more of the downstream timesteps. The goals of the model is to retain the useful components in the previously seen time steps but this becomes cumbersome when we have so many time steps to process. \n",
+ "\n",
+ "2. During backpropagation, the gradient from the loss has to travel all the way back towards the first time step. If our gradient is larger than 1 (${1.01}^{1000} = 20959$) or less than 1 (${0.99}^{1000} = 4.31e-5$) and we have lot's of time steps, this can quickly spiral out of control.\n",
+ "\n",
+ "To address both these issues, the concept of gating was introduced to RNNs. Gating allows RNNs to control the information flow between each time step to optimize on the task. Selectively allowing information to pass through allows the model to process inputs with many time steps. The most common RNN gated varients are the long short term memory ([LSTM](https://pytorch.org/docs/stable/nn.html#torch.nn.LSTM)) units and gated recurrent units ([GRUs](https://pytorch.org/docs/stable/nn.html#torch.nn.GRU)). You can read more about how these units work [here](http://colah.github.io/posts/2015-08-Understanding-LSTMs/).\n",
+ "\n",
+ "![](\"https://raw.githubusercontent.com/GokuMohandas/practicalAI/master/images/gates.png\")
"
+ ]
+ },
+ {
+ "metadata": {
+ "id": "tirko0kwp-9J",
+ "colab_type": "code",
+ "colab": {}
+ },
+ "cell_type": "code",
+ "source": [
+ "# GRU in PyTorch\n",
+ "gru = nn.GRU(input_size=embedding_dim, hidden_size=rnn_hidden_dim, \n",
+ " batch_first=True)"
+ ],
+ "execution_count": 0,
+ "outputs": []
+ },
+ {
+ "metadata": {
+ "id": "UZjUhh4VBWxM",
+ "colab_type": "code",
+ "outputId": "9fe275fe-c8d9-42f0-e5d0-0295268ed83d",
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 34
+ }
+ },
+ "cell_type": "code",
+ "source": [
+ "# Initialize synthetic input\n",
+ "x_in = torch.randn(batch_size, seq_size, embedding_dim)\n",
+ "print (x_in.size())"
+ ],
+ "execution_count": 13,
+ "outputs": [
+ {
+ "output_type": "stream",
+ "text": [
+ "torch.Size([5, 10, 100])\n"
+ ],
+ "name": "stdout"
+ }
+ ]
+ },
+ {
+ "metadata": {
+ "id": "xJ_SE7AvBfa4",
+ "colab_type": "code",
+ "outputId": "b9411aaa-fab1-4104-aee7-8f9a423332ab",
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 51
+ }
+ },
+ "cell_type": "code",
+ "source": [
+ "# Forward pass\n",
+ "out, h_n = gru(x_in)\n",
+ "print (\"out:\", out.size())\n",
+ "print (\"h_n:\", h_n.size())"
+ ],
+ "execution_count": 14,
+ "outputs": [
+ {
+ "output_type": "stream",
+ "text": [
+ "out: torch.Size([5, 10, 256])\n",
+ "h_n: torch.Size([1, 5, 256])\n"
+ ],
+ "name": "stdout"
+ }
+ ]
+ },
+ {
+ "metadata": {
+ "id": "ij_GA2Rr9BbA",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "**Note**: Choosing whether to use GRU or LSTM really depends on the data and empirical performance. GRUs offer comparable performance with reduce number of parameters while LSTMs are more efficient and may make the difference in performance for your particular task."
+ ]
+ },
+ {
+ "metadata": {
+ "id": "9agJw4gwK1LC",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "# Bidirectional RNNs"
+ ]
+ },
+ {
+ "metadata": {
+ "id": "Xck0n-KpmXkV",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "There have been many advancements with RNNs ([attention](https://www.oreilly.com/ideas/interpretability-via-attentional-and-memory-based-interfaces-using-tensorflow), Quasi RNNs, etc.) that we will cover in later lessons but one of the basic and widely used ones are bidirectional RNNs (Bi-RNNs). The motivation behind bidirectional RNNs is to process an input sequence by both directions. Accounting for context from both sides can aid in performance when the entire input sequence is known at time of inference. A common application of Bi-RNNs is in translation where it's advantageous to look at an entire sentence from both sides when translating to another language (ie. Japanese → English).\n",
+ "\n",
+ "![](\"https://raw.githubusercontent.com/GokuMohandas/practicalAI/master/images/birnn.png\")
"
+ ]
+ },
+ {
+ "metadata": {
+ "id": "gSk_5XrvApCd",
+ "colab_type": "code",
+ "colab": {}
+ },
+ "cell_type": "code",
+ "source": [
+ "# BiGRU in PyTorch\n",
+ "bi_gru = nn.GRU(input_size=embedding_dim, hidden_size=rnn_hidden_dim, \n",
+ " batch_first=True, bidirectional=True)"
+ ],
+ "execution_count": 0,
+ "outputs": []
+ },
+ {
+ "metadata": {
+ "id": "Fx7-GTptBCtZ",
+ "colab_type": "code",
+ "outputId": "f0242cc5-534a-460b-ebe0-4e8c504fab22",
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 51
+ }
+ },
+ "cell_type": "code",
+ "source": [
+ "# Forward pass\n",
+ "out, h_n = bi_gru(x_in)\n",
+ "print (\"out:\", out.size()) # collection of all hidden states from the RNN for each time step\n",
+ "print (\"h_n:\", h_n.size()) # last hidden state from the RNN"
+ ],
+ "execution_count": 16,
+ "outputs": [
+ {
+ "output_type": "stream",
+ "text": [
+ "out: torch.Size([5, 10, 512])\n",
+ "h_n: torch.Size([2, 5, 256])\n"
+ ],
+ "name": "stdout"
+ }
+ ]
+ },
+ {
+ "metadata": {
+ "id": "k5lvJirLBjI6",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "Notice that the output for each sample at each timestamp has size 512 (double the hidden dim). This is because this includes both the forward and backward directions from the BiRNN. "
+ ]
+ },
+ {
+ "metadata": {
+ "id": "mJSknbofK2S9",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "# Document classification with RNNs"
+ ]
+ },
+ {
+ "metadata": {
+ "id": "JgYdEZmHlmft",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "Let's apply RNNs to the document classification task from the [emebddings notebook](https://colab.research.google.com/drive/1yDa5ZTqKVoLl-qRgH-N9xs3pdrDJ0Fb4) where we want to predict an article's category given its title."
+ ]
+ },
+ {
+ "metadata": {
+ "id": "eIvXqvPQEiDC",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "## Set up"
+ ]
+ },
+ {
+ "metadata": {
+ "id": "muTcvMynlmAu",
+ "colab_type": "code",
+ "colab": {}
+ },
+ "cell_type": "code",
+ "source": [
+ "import os\n",
+ "from argparse import Namespace\n",
+ "import collections\n",
+ "import copy\n",
+ "import json\n",
+ "import matplotlib.pyplot as plt\n",
+ "import numpy as np\n",
+ "import pandas as pd\n",
+ "import re\n",
+ "import torch"
+ ],
+ "execution_count": 0,
+ "outputs": []
+ },
+ {
+ "metadata": {
+ "id": "00ESjecep-_y",
+ "colab_type": "code",
+ "colab": {}
+ },
+ "cell_type": "code",
+ "source": [
+ "# Set Numpy and PyTorch seeds\n",
+ "def set_seeds(seed, cuda):\n",
+ " np.random.seed(seed)\n",
+ " torch.manual_seed(seed)\n",
+ " if cuda:\n",
+ " torch.cuda.manual_seed_all(seed)\n",
+ " \n",
+ "# Creating directories\n",
+ "def create_dirs(dirpath):\n",
+ " if not os.path.exists(dirpath):\n",
+ " os.makedirs(dirpath)"
+ ],
+ "execution_count": 0,
+ "outputs": []
+ },
+ {
+ "metadata": {
+ "id": "m67THDvxEl1e",
+ "colab_type": "code",
+ "outputId": "7118c77b-cbf9-4d7e-ff7a-b9dc1fb63cbb",
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 34
+ }
+ },
+ "cell_type": "code",
+ "source": [
+ "# Arguments\n",
+ "args = Namespace(\n",
+ " seed=1234,\n",
+ " cuda=True,\n",
+ " shuffle=True,\n",
+ " data_file=\"news.csv\",\n",
+ " split_data_file=\"split_news.csv\",\n",
+ " vectorizer_file=\"vectorizer.json\",\n",
+ " model_state_file=\"model.pth\",\n",
+ " save_dir=\"news\",\n",
+ " train_size=0.7,\n",
+ " val_size=0.15,\n",
+ " test_size=0.15,\n",
+ " pretrained_embeddings=None,\n",
+ " cutoff=25, # token must appear at least \n",
+ "\n",
+ "\n",
+ " \n",
+ "
\n",
+ "
"
+ ],
+ "text/plain": [
+ " category title\n",
+ "0 Business Wall St. Bears Claw Back Into the Black (Reuters)\n",
+ "1 Business Carlyle Looks Toward Commercial Aerospace (Reu...\n",
+ "2 Business Oil and Economy Cloud Stocks' Outlook (Reuters)\n",
+ "3 Business Iraq Halts Oil Exports from Main Southern Pipe...\n",
+ "4 Business Oil prices soar to all-time record, posing new..."
+ ]
+ },
+ "metadata": {
+ "tags": []
+ },
+ "execution_count": 25
+ }
+ ]
+ },
+ {
+ "metadata": {
+ "id": "ICl2MNK4EwrL",
+ "colab_type": "code",
+ "outputId": "d2073597-71e5-40b1-a845-90bf4913ea7a",
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 85
+ }
+ },
+ "cell_type": "code",
+ "source": [
+ "# Split by category\n",
+ "by_category = collections.defaultdict(list)\n",
+ "for _, row in df.iterrows():\n",
+ " by_category[row.category].append(row.to_dict())\n",
+ "for category in by_category:\n",
+ " print (\"{0}: {1}\".format(category, len(by_category[category])))"
+ ],
+ "execution_count": 26,
+ "outputs": [
+ {
+ "output_type": "stream",
+ "text": [
+ "Business: 30000\n",
+ "Sci/Tech: 30000\n",
+ "Sports: 30000\n",
+ "World: 30000\n"
+ ],
+ "name": "stdout"
+ }
+ ]
+ },
+ {
+ "metadata": {
+ "id": "76PwKQHLEww5",
+ "colab_type": "code",
+ "colab": {}
+ },
+ "cell_type": "code",
+ "source": [
+ "# Create split data\n",
+ "final_list = []\n",
+ "for _, item_list in sorted(by_category.items()):\n",
+ " if args.shuffle:\n",
+ " np.random.shuffle(item_list)\n",
+ " n = len(item_list)\n",
+ " n_train = int(args.train_size*n)\n",
+ " n_val = int(args.val_size*n)\n",
+ " n_test = int(args.test_size*n)\n",
+ "\n",
+ " # Give data point a split attribute\n",
+ " for item in item_list[:n_train]:\n",
+ " item['split'] = 'train'\n",
+ " for item in item_list[n_train:n_train+n_val]:\n",
+ " item['split'] = 'val'\n",
+ " for item in item_list[n_train+n_val:]:\n",
+ " item['split'] = 'test' \n",
+ "\n",
+ " # Add to final list\n",
+ " final_list.extend(item_list)"
+ ],
+ "execution_count": 0,
+ "outputs": []
+ },
+ {
+ "metadata": {
+ "id": "CQeS0KHOEwzm",
+ "colab_type": "code",
+ "outputId": "93c9aadb-25c4-4029-f002-8a43f3956045",
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 85
+ }
+ },
+ "cell_type": "code",
+ "source": [
+ "# df with split datasets\n",
+ "split_df = pd.DataFrame(final_list)\n",
+ "split_df[\"split\"].value_counts()"
+ ],
+ "execution_count": 28,
+ "outputs": [
+ {
+ "output_type": "execute_result",
+ "data": {
+ "text/plain": [
+ "train 84000\n",
+ "val 18000\n",
+ "test 18000\n",
+ "Name: split, dtype: int64"
+ ]
+ },
+ "metadata": {
+ "tags": []
+ },
+ "execution_count": 28
+ }
+ ]
+ },
+ {
+ "metadata": {
+ "id": "pPJDyVusEw3-",
+ "colab_type": "code",
+ "colab": {}
+ },
+ "cell_type": "code",
+ "source": [
+ "# Preprocessing\n",
+ "def preprocess_text(text):\n",
+ " text = ' '.join(word.lower() for word in text.split(\" \"))\n",
+ " text = re.sub(r\"([.,!?])\", r\" \\1 \", text)\n",
+ " text = re.sub(r\"[^a-zA-Z.,!?]+\", r\" \", text)\n",
+ " text = text.strip()\n",
+ " return text\n",
+ " \n",
+ "split_df.title = split_df.title.apply(preprocess_text)"
+ ],
+ "execution_count": 0,
+ "outputs": []
+ },
+ {
+ "metadata": {
+ "id": "IAetKendEw6b",
+ "colab_type": "code",
+ "outputId": "d5946f7e-840e-4a0b-e492-d3da68cefd44",
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 204
+ }
+ },
+ "cell_type": "code",
+ "source": [
+ "# Save to CSV\n",
+ "split_df.to_csv(args.split_data_file, index=False)\n",
+ "split_df.head()"
+ ],
+ "execution_count": 30,
+ "outputs": [
+ {
+ "output_type": "execute_result",
+ "data": {
+ "text/html": [
+ "| \n", + " | category | \n", + "title | \n", + "
|---|---|---|
| 0 | \n", + "Business | \n", + "Wall St. Bears Claw Back Into the Black (Reuters) | \n", + "
| 1 | \n", + "Business | \n", + "Carlyle Looks Toward Commercial Aerospace (Reu... | \n", + "
| 2 | \n", + "Business | \n", + "Oil and Economy Cloud Stocks' Outlook (Reuters) | \n", + "
| 3 | \n", + "Business | \n", + "Iraq Halts Oil Exports from Main Southern Pipe... | \n", + "
| 4 | \n", + "Business | \n", + "Oil prices soar to all-time record, posing new... | \n", + "
\n",
+ "\n",
+ "\n",
+ " \n",
+ "
\n",
+ "
"
+ ],
+ "text/plain": [
+ " category split title\n",
+ "0 Business train general electric posts higher rd quarter profit\n",
+ "1 Business train lilly to eliminate up to us jobs\n",
+ "2 Business train s amp p lowers america west outlook to negative\n",
+ "3 Business train does rand walk the talk on labor policy ?\n",
+ "4 Business train housekeeper advocates for changes"
+ ]
+ },
+ "metadata": {
+ "tags": []
+ },
+ "execution_count": 30
+ }
+ ]
+ },
+ {
+ "metadata": {
+ "id": "NHzGXAI3E7lF",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "## Vocabulary"
+ ]
+ },
+ {
+ "metadata": {
+ "id": "ZIRUjX0MEw88",
+ "colab_type": "code",
+ "colab": {}
+ },
+ "cell_type": "code",
+ "source": [
+ "class Vocabulary(object):\n",
+ " def __init__(self, token_to_idx=None):\n",
+ "\n",
+ " # Token to index\n",
+ " if token_to_idx is None:\n",
+ " token_to_idx = {}\n",
+ " self.token_to_idx = token_to_idx\n",
+ "\n",
+ " # Index to token\n",
+ " self.idx_to_token = {idx: token \\\n",
+ " for token, idx in self.token_to_idx.items()}\n",
+ "\n",
+ " def to_serializable(self):\n",
+ " return {'token_to_idx': self.token_to_idx}\n",
+ "\n",
+ " @classmethod\n",
+ " def from_serializable(cls, contents):\n",
+ " return cls(**contents)\n",
+ "\n",
+ " def add_token(self, token):\n",
+ " if token in self.token_to_idx:\n",
+ " index = self.token_to_idx[token]\n",
+ " else:\n",
+ " index = len(self.token_to_idx)\n",
+ " self.token_to_idx[token] = index\n",
+ " self.idx_to_token[index] = token\n",
+ " return index\n",
+ "\n",
+ " def add_tokens(self, tokens):\n",
+ " return [self.add_token[token] for token in tokens]\n",
+ "\n",
+ " def lookup_token(self, token):\n",
+ " return self.token_to_idx[token]\n",
+ "\n",
+ " def lookup_index(self, index):\n",
+ " if index not in self.idx_to_token:\n",
+ " raise KeyError(\"the index (%d) is not in the Vocabulary\" % index)\n",
+ " return self.idx_to_token[index]\n",
+ "\n",
+ " def __str__(self):\n",
+ " return \"| \n", + " | category | \n", + "split | \n", + "title | \n", + "
|---|---|---|---|
| 0 | \n", + "Business | \n", + "train | \n", + "general electric posts higher rd quarter profit | \n", + "
| 1 | \n", + "Business | \n", + "train | \n", + "lilly to eliminate up to us jobs | \n", + "
| 2 | \n", + "Business | \n", + "train | \n", + "s amp p lowers america west outlook to negative | \n", + "
| 3 | \n", + "Business | \n", + "train | \n", + "does rand walk the talk on labor policy ? | \n", + "
| 4 | \n", + "Business | \n", + "train | \n", + "housekeeper advocates for changes | \n", + "
![](\"https://raw.githubusercontent.com/GokuMohandas/practicalAI/master/images/layernorm.png\")
"
+ ]
+ },
+ {
+ "metadata": {
+ "id": "oAhAHcgZBMFe",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "The most useful location to apply layernorm will be inside the RNN on the activations before the non-linearities. However, this is a bit involved and though PyTorch has a [LayerNorm](https://pytorch.org/docs/stable/nn.html#torch.nn.LayerNorm) class, they do not have an RNN that has built in layernorm yet. You could implement the RNN yourself and manually add layernorm by following a similar setup like below.\n",
+ "\n",
+ "```python\n",
+ "# Layernorm\n",
+ "for t in range(seq_size):\n",
+ " # Normalize over hidden dim\n",
+ " layernorm = nn.LayerNorm(args.hidden_dim)\n",
+ " # Activating the module\n",
+ " a = layernorm(x)\n",
+ "```"
+ ]
+ },
+ {
+ "metadata": {
+ "id": "1YHneO3SStOp",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "# TODO"
+ ]
+ },
+ {
+ "metadata": {
+ "id": "gGHaKTe1SuEk",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "- interpretability with task to see which words were most influential"
+ ]
+ }
+ ]
+}
\ No newline at end of file
diff --git a/notebooks/14_Advanced_RNNs.ipynb b/notebooks/14_Advanced_RNNs.ipynb
new file mode 100644
index 0000000..cbc9522
--- /dev/null
+++ b/notebooks/14_Advanced_RNNs.ipynb
@@ -0,0 +1,2916 @@
+{
+ "nbformat": 4,
+ "nbformat_minor": 0,
+ "metadata": {
+ "colab": {
+ "name": "14_Advanced_RNNs",
+ "version": "0.3.2",
+ "provenance": [],
+ "collapsed_sections": [],
+ "toc_visible": true
+ },
+ "kernelspec": {
+ "name": "python3",
+ "display_name": "Python 3"
+ },
+ "accelerator": "GPU"
+ },
+ "cells": [
+ {
+ "metadata": {
+ "id": "bOChJSNXtC9g",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "# Advanced RNNs"
+ ]
+ },
+ {
+ "metadata": {
+ "id": "OLIxEDq6VhvZ",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "![](\"https://raw.githubusercontent.com/GokuMohandas/practicalAI/master/images/logo.png\"){kind=logo}
\n",
+ "\n",
+ "In this notebook we're going to cover some advanced topics related to RNNs.\n",
+ "\n",
+ "1. Conditioned hidden state\n",
+ "2. Char-level embeddings\n",
+ "3. Encoder and decoder\n",
+ "4. Attentional mechanisms\n",
+ "5. Implementation\n",
+ "\n",
+ "\n"
+ ]
+ },
+ {
+ "metadata": {
+ "id": "41r7MWJnY0m8",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "# Set up"
+ ]
+ },
+ {
+ "metadata": {
+ "id": "EJDhjHCHY0_a",
+ "colab_type": "code",
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 34
+ },
+ "outputId": "beb1c764-e47f-41a6-f8cf-e4150ee3befd"
+ },
+ "cell_type": "code",
+ "source": [
+ "# Load PyTorch library\n",
+ "!pip3 install torch"
+ ],
+ "execution_count": 1,
+ "outputs": [
+ {
+ "output_type": "stream",
+ "text": [
+ "Requirement already satisfied: torch in /usr/local/lib/python3.6/dist-packages (1.0.0)\n"
+ ],
+ "name": "stdout"
+ }
+ ]
+ },
+ {
+ "metadata": {
+ "id": "p0FbOd6IZmzX",
+ "colab_type": "code",
+ "colab": {}
+ },
+ "cell_type": "code",
+ "source": [
+ "import os\n",
+ "from argparse import Namespace\n",
+ "import collections\n",
+ "import copy\n",
+ "import json\n",
+ "import matplotlib.pyplot as plt\n",
+ "import numpy as np\n",
+ "import pandas as pd\n",
+ "import re\n",
+ "import torch"
+ ],
+ "execution_count": 0,
+ "outputs": []
+ },
+ {
+ "metadata": {
+ "id": "bOsqAo4XZpXQ",
+ "colab_type": "code",
+ "colab": {}
+ },
+ "cell_type": "code",
+ "source": [
+ "# Set Numpy and PyTorch seeds\n",
+ "def set_seeds(seed, cuda):\n",
+ " np.random.seed(seed)\n",
+ " torch.manual_seed(seed)\n",
+ " if cuda:\n",
+ " torch.cuda.manual_seed_all(seed)\n",
+ " \n",
+ "# Creating directories\n",
+ "def create_dirs(dirpath):\n",
+ " if not os.path.exists(dirpath):\n",
+ " os.makedirs(dirpath)"
+ ],
+ "execution_count": 0,
+ "outputs": []
+ },
+ {
+ "metadata": {
+ "id": "QHfvEzQ9ZweF",
+ "colab_type": "code",
+ "outputId": "a69944ff-021d-4d04-e920-cfc49112a34c",
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 34
+ }
+ },
+ "cell_type": "code",
+ "source": [
+ "# Arguments\n",
+ "args = Namespace(\n",
+ " seed=1234,\n",
+ " cuda=True,\n",
+ " batch_size=4,\n",
+ " condition_vocab_size=3, # vocabulary for condition possibilities\n",
+ " embedding_dim=100,\n",
+ " rnn_hidden_dim=100,\n",
+ " hidden_dim=100,\n",
+ " num_layers=1,\n",
+ " bidirectional=False,\n",
+ ")\n",
+ "\n",
+ "# Set seeds\n",
+ "set_seeds(seed=args.seed, cuda=args.cuda)\n",
+ "\n",
+ "# Check CUDA\n",
+ "if not torch.cuda.is_available():\n",
+ " args.cuda = False\n",
+ "args.device = torch.device(\"cuda\" if args.cuda else \"cpu\")\n",
+ "print(\"Using CUDA: {}\".format(args.cuda))"
+ ],
+ "execution_count": 5,
+ "outputs": [
+ {
+ "output_type": "stream",
+ "text": [
+ "Using CUDA: True\n"
+ ],
+ "name": "stdout"
+ }
+ ]
+ },
+ {
+ "metadata": {
+ "id": "VoMq0eFRvugb",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "# Conditioned RNNs"
+ ]
+ },
+ {
+ "metadata": {
+ "id": "ZUsj7HjBp69f",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "Conditioning an RNN is to add extra information that will be helpful towards a prediction. We can encode (embed it) this information and feed it along with the sequential input into our model. For example, suppose in our document classificaiton example in the previous notebook, we knew the publisher of each news article (NYTimes, ESPN, etc.). We could have encoded that information to help with the prediction. There are several different ways of creating a conditioned RNN.\n",
+ "\n",
+ "**Note**: If the conditioning information is novel for each input in the sequence, just concatenate it along with each time step's input."
+ ]
+ },
+ {
+ "metadata": {
+ "id": "Kc8H9JySmtLa",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "1. Make the initial hidden state the encoded information instead of using the initial zerod hidden state. Make sure that the size of the encoded information is the same as the hidden state for the RNN.\n"
+ ]
+ },
+ {
+ "metadata": {
+ "id": "pKlb9SjfpbED",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "![](\"https://raw.githubusercontent.com/GokuMohandas/practicalAI/master/images/conditioned_rnn1.png\")
"
+ ]
+ },
+ {
+ "metadata": {
+ "id": "jbrlQHx2x8Aa",
+ "colab_type": "code",
+ "colab": {}
+ },
+ "cell_type": "code",
+ "source": [
+ "import torch.nn as nn\n",
+ "import torch.nn.functional as F"
+ ],
+ "execution_count": 0,
+ "outputs": []
+ },
+ {
+ "metadata": {
+ "id": "cFoiV-fqmvRo",
+ "colab_type": "code",
+ "outputId": "9843f756-8d71-4686-b479-b521df9b6f3c",
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 34
+ }
+ },
+ "cell_type": "code",
+ "source": [
+ "# Condition\n",
+ "condition = torch.LongTensor([0, 2, 1, 2]) # batch size of 4 with a vocab size of 3\n",
+ "condition_embeddings = nn.Embedding(\n",
+ " embedding_dim=args.embedding_dim, # should be same as RNN hidden dim\n",
+ " num_embeddings=args.condition_vocab_size) # of unique conditions\n",
+ "\n",
+ "# Initialize hidden state\n",
+ "num_directions = 1\n",
+ "if args.bidirectional:\n",
+ " num_directions = 2\n",
+ " \n",
+ "# If using multiple layers and directions, the hidden state needs to match that size\n",
+ "hidden_t = condition_embeddings(condition).unsqueeze(0).repeat(\n",
+ " args.num_layers * num_directions, 1, 1).to(args.device) # initial state to RNN\n",
+ "print (hidden_t.size())\n",
+ "\n",
+ "# Feed into RNN\n",
+ "# y_out, _ = self.rnn(x_embedded, hidden_t)\n"
+ ],
+ "execution_count": 7,
+ "outputs": [
+ {
+ "output_type": "stream",
+ "text": [
+ "torch.Size([1, 4, 100])\n"
+ ],
+ "name": "stdout"
+ }
+ ]
+ },
+ {
+ "metadata": {
+ "id": "REgyaMDgmtHw",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "2. Concatenate the encoded information with the hidden state at each time step. Do not replace the hidden state because the RNN needs that to learn. "
+ ]
+ },
+ {
+ "metadata": {
+ "id": "yUIg5o-dpiZF",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "![](\"https://raw.githubusercontent.com/GokuMohandas/practicalAI/master/images/conditioned_rnn2.png\")
"
+ ]
+ },
+ {
+ "metadata": {
+ "id": "eQ-h28o-pi4X",
+ "colab_type": "code",
+ "outputId": "4143190d-c452-48cc-cc96-1a2f0f7fc5ee",
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 34
+ }
+ },
+ "cell_type": "code",
+ "source": [
+ "# Initialize hidden state\n",
+ "hidden_t = torch.zeros((args.num_layers * num_directions, args.batch_size, args.rnn_hidden_dim))\n",
+ "print (hidden_t.size())"
+ ],
+ "execution_count": 8,
+ "outputs": [
+ {
+ "output_type": "stream",
+ "text": [
+ "torch.Size([1, 4, 100])\n"
+ ],
+ "name": "stdout"
+ }
+ ]
+ },
+ {
+ "metadata": {
+ "id": "2Z6hYSIdqBQ4",
+ "colab_type": "code",
+ "colab": {}
+ },
+ "cell_type": "code",
+ "source": [
+ "def concat_condition(condition_embeddings, condition, hidden_t, num_layers, num_directions):\n",
+ " condition_t = condition_embeddings(condition).unsqueeze(0).repeat(\n",
+ " num_layers * num_directions, 1, 1)\n",
+ " hidden_t = torch.cat([hidden_t, condition_t], 2)\n",
+ " return hidden_t"
+ ],
+ "execution_count": 0,
+ "outputs": []
+ },
+ {
+ "metadata": {
+ "id": "Tjyzq_s5pixL",
+ "colab_type": "code",
+ "outputId": "f4f62742-044e-46ef-cc46-fc21a3c52c78",
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 34
+ }
+ },
+ "cell_type": "code",
+ "source": [
+ "# Loop through the inputs time steps\n",
+ "hiddens = []\n",
+ "seq_size = 1\n",
+ "for t in range(seq_size):\n",
+ " hidden_t = concat_condition(condition_embeddings, condition, hidden_t, \n",
+ " args.num_layers, num_directions).to(args.device)\n",
+ " print (hidden_t.size())\n",
+ " \n",
+ " # Feed into RNN\n",
+ " # hidden_t = rnn_cell(x_in[t], hidden_t)\n",
+ " ..."
+ ],
+ "execution_count": 10,
+ "outputs": [
+ {
+ "output_type": "stream",
+ "text": [
+ "torch.Size([1, 4, 200])\n"
+ ],
+ "name": "stdout"
+ }
+ ]
+ },
+ {
+ "metadata": {
+ "id": "A-0_81jMXg_J",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "# Char-level embeddings"
+ ]
+ },
+ {
+ "metadata": {
+ "id": "w0yUKKpq3pu_",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "Our conv operations will have inputs that are words in a sentence represented at the character level| $\\in \\mathbb{R}^{NXSXWXE}$ and outputs are embeddings for each word (based on convlutions applied at the character level.) \n",
+ "\n",
+ "**Word embeddings**: capture the temporal correlations among\n",
+ "adjacent tokens so that similar words have similar representations. Ex. \"New Jersey\" is close to \"NJ\" is close to \"Garden State\", etc.\n",
+ "\n",
+ "**Char embeddings**: create representations that map words at a character level. Ex. \"toy\" and \"toys\" will be close to each other."
+ ]
+ },
+ {
+ "metadata": {
+ "id": "-SZgVuwebm_4",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "![](\"https://raw.githubusercontent.com/GokuMohandas/practicalAI/master/images/char_embeddings.png\")
"
+ ]
+ },
+ {
+ "metadata": {
+ "id": "QOdIvz0G3O8C",
+ "colab_type": "code",
+ "colab": {}
+ },
+ "cell_type": "code",
+ "source": [
+ "# Arguments\n",
+ "args = Namespace(\n",
+ " seed=1234,\n",
+ " cuda=False,\n",
+ " shuffle=True,\n",
+ " batch_size=64,\n",
+ " vocab_size=20, # vocabulary\n",
+ " seq_size=10, # max length of each sentence\n",
+ " word_size=15, # max length of each word\n",
+ " embedding_dim=100,\n",
+ " num_filters=100, # filters per size\n",
+ ")"
+ ],
+ "execution_count": 0,
+ "outputs": []
+ },
+ {
+ "metadata": {
+ "id": "raztXIeYXYJT",
+ "colab_type": "code",
+ "colab": {}
+ },
+ "cell_type": "code",
+ "source": [
+ "class Model(nn.Module):\n",
+ " def __init__(self, embedding_dim, num_embeddings, num_input_channels, \n",
+ " num_output_channels, padding_idx):\n",
+ " super(Model, self).__init__()\n",
+ " \n",
+ " # Char-level embedding\n",
+ " self.embeddings = nn.Embedding(embedding_dim=embedding_dim,\n",
+ " num_embeddings=num_embeddings,\n",
+ " padding_idx=padding_idx)\n",
+ " \n",
+ " # Conv weights\n",
+ " self.conv = nn.ModuleList([nn.Conv1d(num_input_channels, num_output_channels, \n",
+ " kernel_size=f) for f in [2,3,4]])\n",
+ "\n",
+ " def forward(self, x, channel_first=False, apply_softmax=False):\n",
+ " \n",
+ " # x: (N, seq_len, word_len)\n",
+ " input_shape = x.size()\n",
+ " batch_size, seq_len, word_len = input_shape\n",
+ " x = x.view(-1, word_len) # (N*seq_len, word_len)\n",
+ " \n",
+ " # Embedding\n",
+ " x = self.embeddings(x) # (N*seq_len, word_len, embedding_dim)\n",
+ " \n",
+ " # Rearrange input so num_input_channels is in dim 1 (N, embedding_dim, word_len)\n",
+ " if not channel_first:\n",
+ " x = x.transpose(1, 2)\n",
+ " \n",
+ " # Convolution\n",
+ " z = [F.relu(conv(x)) for conv in self.conv]\n",
+ " \n",
+ " # Pooling\n",
+ " z = [F.max_pool1d(zz, zz.size(2)).squeeze(2) for zz in z] \n",
+ " z = [zz.view(batch_size, seq_len, -1) for zz in z] # (N, seq_len, embedding_dim)\n",
+ " \n",
+ " # Concat to get char-level embeddings\n",
+ " z = torch.cat(z, 2) # join conv outputs\n",
+ " \n",
+ " return z"
+ ],
+ "execution_count": 0,
+ "outputs": []
+ },
+ {
+ "metadata": {
+ "id": "MzHVs8Xe0Zph",
+ "colab_type": "code",
+ "outputId": "ff91c1ac-5bc4-446c-9047-8b4b58570e13",
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 34
+ }
+ },
+ "cell_type": "code",
+ "source": [
+ "# Input\n",
+ "input_size = (args.batch_size, args.seq_size, args.word_size)\n",
+ "x_in = torch.randint(low=0, high=args.vocab_size, size=input_size).long()\n",
+ "print (x_in.size())"
+ ],
+ "execution_count": 13,
+ "outputs": [
+ {
+ "output_type": "stream",
+ "text": [
+ "torch.Size([64, 10, 15])\n"
+ ],
+ "name": "stdout"
+ }
+ ]
+ },
+ {
+ "metadata": {
+ "id": "0B_Xscby2PMQ",
+ "colab_type": "code",
+ "outputId": "05b0c3ac-429f-47aa-9526-718e55dfc897",
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 153
+ }
+ },
+ "cell_type": "code",
+ "source": [
+ "# Initial char-level embedding model\n",
+ "model = Model(embedding_dim=args.embedding_dim, \n",
+ " num_embeddings=args.vocab_size, \n",
+ " num_input_channels=args.embedding_dim, \n",
+ " num_output_channels=args.num_filters,\n",
+ " padding_idx=0)\n",
+ "print (model.named_modules)"
+ ],
+ "execution_count": 14,
+ "outputs": [
+ {
+ "output_type": "stream",
+ "text": [
+ "![](\"https://raw.githubusercontent.com/GokuMohandas/practicalAI/master/images/attention1.jpg\")
"
+ ]
+ },
+ {
+ "metadata": {
+ "id": "mNkayU0rf-ua",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "Attention can sound a bit confusing so let's see what happens at each time step. At time step j, the model has processed inputs $x_0, x_1, x_2, ..., x_j$ and has generted hidden states $h_0, h_1, h_2, ..., h_j$. The idea is to use all the processed hidden states to make the prediction and not just the most recent one. There are several approaches to how we can do this.\n",
+ "\n",
+ "With **soft attention**, we learn a vector of floating points (probabilities) to multiply with the hidden states to create the context vector.\n",
+ "\n",
+ "Ex. [0.1, 0.3, 0.1, 0.4, 0.1]\n",
+ "\n",
+ "With **hard attention**, we can learn a binary vector to multiply with the hidden states to create the context vector. \n",
+ "\n",
+ "Ex. [0, 0, 0, 1, 0]"
+ ]
+ },
+ {
+ "metadata": {
+ "id": "gYSIAVQqu3Ab",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "We're going to focus on soft attention because it's more widley used and we can visualize how much of each hidden state helps with the prediction, which is great for interpretability. "
+ ]
+ },
+ {
+ "metadata": {
+ "id": "9Ch21nZNvDHO",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "![](\"https://raw.githubusercontent.com/GokuMohandas/practicalAI/master/images/attention2.jpg\")
"
+ ]
+ },
+ {
+ "metadata": {
+ "id": "o_jPXuT8xlqd",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "We're going to implement attention in the document classification task below."
+ ]
+ },
+ {
+ "metadata": {
+ "colab_type": "text",
+ "id": "0iNnQzdxnGvn"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "# Document classification with RNNs"
+ ]
+ },
+ {
+ "metadata": {
+ "colab_type": "text",
+ "id": "n38ZJoVZnGaE"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "We're going to implement the same document classification task as in the previous notebook but we're going to use an attentional interface for interpretability.\n",
+ "\n",
+ "**Why not machine translation?** Normally, machine translation is the go-to example for demonstrating attention but it's not really practical. How many situations can you think of that require a seq to generate another sequence? Instead we're going to apply attention with our document classification example to see which input tokens are more influential towards predicting the genre."
+ ]
+ },
+ {
+ "metadata": {
+ "colab_type": "text",
+ "id": "Fu7HgEqbnGFY"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "## Set up"
+ ]
+ },
+ {
+ "metadata": {
+ "colab_type": "code",
+ "id": "elL6BxtCmNGf",
+ "colab": {}
+ },
+ "cell_type": "code",
+ "source": [
+ "from argparse import Namespace\n",
+ "import collections\n",
+ "import copy\n",
+ "import json\n",
+ "import matplotlib.pyplot as plt\n",
+ "import numpy as np\n",
+ "import pandas as pd\n",
+ "import re\n",
+ "import torch"
+ ],
+ "execution_count": 0,
+ "outputs": []
+ },
+ {
+ "metadata": {
+ "id": "DCf2fLmPbKKI",
+ "colab_type": "code",
+ "colab": {}
+ },
+ "cell_type": "code",
+ "source": [
+ "def set_seeds(seed, cuda):\n",
+ " np.random.seed(seed)\n",
+ " torch.manual_seed(seed)\n",
+ " if cuda:\n",
+ " torch.cuda.manual_seed_all(seed)\n",
+ " \n",
+ "# Creating directories\n",
+ "def create_dirs(dirpath):\n",
+ " if not os.path.exists(dirpath):\n",
+ " os.makedirs(dirpath)"
+ ],
+ "execution_count": 0,
+ "outputs": []
+ },
+ {
+ "metadata": {
+ "colab_type": "code",
+ "outputId": "291c03d4-6143-4395-b5c9-ab386b061737",
+ "id": "TTwkuoZdmMlF",
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 34
+ }
+ },
+ "cell_type": "code",
+ "source": [
+ "args = Namespace(\n",
+ " seed=1234,\n",
+ " cuda=True,\n",
+ " shuffle=True,\n",
+ " data_file=\"news.csv\",\n",
+ " split_data_file=\"split_news.csv\",\n",
+ " vectorizer_file=\"vectorizer.json\",\n",
+ " model_state_file=\"model.pth\",\n",
+ " save_dir=\"news\",\n",
+ " train_size=0.7,\n",
+ " val_size=0.15,\n",
+ " test_size=0.15,\n",
+ " pretrained_embeddings=None,\n",
+ " cutoff=25,\n",
+ " num_epochs=5,\n",
+ " early_stopping_criteria=5,\n",
+ " learning_rate=1e-3,\n",
+ " batch_size=128,\n",
+ " embedding_dim=100,\n",
+ " kernels=[3,5],\n",
+ " num_filters=100,\n",
+ " rnn_hidden_dim=128,\n",
+ " hidden_dim=200,\n",
+ " num_layers=1,\n",
+ " bidirectional=False,\n",
+ " dropout_p=0.25,\n",
+ ")\n",
+ "\n",
+ "# Set seeds\n",
+ "set_seeds(seed=args.seed, cuda=args.cuda)\n",
+ "\n",
+ "# Create save dir\n",
+ "create_dirs(args.save_dir)\n",
+ "\n",
+ "# Expand filepaths\n",
+ "args.vectorizer_file = os.path.join(args.save_dir, args.vectorizer_file)\n",
+ "args.model_state_file = os.path.join(args.save_dir, args.model_state_file)\n",
+ "\n",
+ "# Check CUDA\n",
+ "if not torch.cuda.is_available():\n",
+ " args.cuda = False\n",
+ "args.device = torch.device(\"cuda\" if args.cuda else \"cpu\")\n",
+ "print(\"Using CUDA: {}\".format(args.cuda))"
+ ],
+ "execution_count": 28,
+ "outputs": [
+ {
+ "output_type": "stream",
+ "text": [
+ "Using CUDA: True\n"
+ ],
+ "name": "stdout"
+ }
+ ]
+ },
+ {
+ "metadata": {
+ "colab_type": "text",
+ "id": "xfiWhgX5mMQ5"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "## Data"
+ ]
+ },
+ {
+ "metadata": {
+ "colab_type": "code",
+ "id": "baAsxXNFmMCF",
+ "colab": {}
+ },
+ "cell_type": "code",
+ "source": [
+ "import urllib"
+ ],
+ "execution_count": 0,
+ "outputs": []
+ },
+ {
+ "metadata": {
+ "colab_type": "code",
+ "id": "3tJi_HyOmLw-",
+ "colab": {}
+ },
+ "cell_type": "code",
+ "source": [
+ "url = \"https://raw.githubusercontent.com/GokuMohandas/practicalAI/master/data/news.csv\"\n",
+ "response = urllib.request.urlopen(url)\n",
+ "html = response.read()\n",
+ "with open(args.data_file, 'wb') as fp:\n",
+ " fp.write(html)"
+ ],
+ "execution_count": 0,
+ "outputs": []
+ },
+ {
+ "metadata": {
+ "colab_type": "code",
+ "outputId": "a51463a7-f37e-41e7-aca4-74038c7c6e8e",
+ "id": "wrI_df4bmLjB",
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 204
+ }
+ },
+ "cell_type": "code",
+ "source": [
+ "df = pd.read_csv(args.data_file, header=0)\n",
+ "df.head()"
+ ],
+ "execution_count": 31,
+ "outputs": [
+ {
+ "output_type": "execute_result",
+ "data": {
+ "text/html": [
+ "\n",
+ "\n",
+ "\n",
+ " \n",
+ "
\n",
+ "
"
+ ],
+ "text/plain": [
+ " category title\n",
+ "0 Business Wall St. Bears Claw Back Into the Black (Reuters)\n",
+ "1 Business Carlyle Looks Toward Commercial Aerospace (Reu...\n",
+ "2 Business Oil and Economy Cloud Stocks' Outlook (Reuters)\n",
+ "3 Business Iraq Halts Oil Exports from Main Southern Pipe...\n",
+ "4 Business Oil prices soar to all-time record, posing new..."
+ ]
+ },
+ "metadata": {
+ "tags": []
+ },
+ "execution_count": 31
+ }
+ ]
+ },
+ {
+ "metadata": {
+ "colab_type": "code",
+ "outputId": "36145f0d-7316-4341-f270-1d8c8037c661",
+ "id": "TreK7nqEmLTN",
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 85
+ }
+ },
+ "cell_type": "code",
+ "source": [
+ "by_category = collections.defaultdict(list)\n",
+ "for _, row in df.iterrows():\n",
+ " by_category[row.category].append(row.to_dict())\n",
+ "for category in by_category:\n",
+ " print (\"{0}: {1}\".format(category, len(by_category[category])))"
+ ],
+ "execution_count": 32,
+ "outputs": [
+ {
+ "output_type": "stream",
+ "text": [
+ "Business: 30000\n",
+ "Sci/Tech: 30000\n",
+ "Sports: 30000\n",
+ "World: 30000\n"
+ ],
+ "name": "stdout"
+ }
+ ]
+ },
+ {
+ "metadata": {
+ "colab_type": "code",
+ "id": "35nb3LxLmLCA",
+ "colab": {}
+ },
+ "cell_type": "code",
+ "source": [
+ "final_list = []\n",
+ "for _, item_list in sorted(by_category.items()):\n",
+ " if args.shuffle:\n",
+ " np.random.shuffle(item_list)\n",
+ " n = len(item_list)\n",
+ " n_train = int(args.train_size*n)\n",
+ " n_val = int(args.val_size*n)\n",
+ " n_test = int(args.test_size*n)\n",
+ "\n",
+ " # Give data point a split attribute\n",
+ " for item in item_list[:n_train]:\n",
+ " item['split'] = 'train'\n",
+ " for item in item_list[n_train:n_train+n_val]:\n",
+ " item['split'] = 'val'\n",
+ " for item in item_list[n_train+n_val:]:\n",
+ " item['split'] = 'test' \n",
+ "\n",
+ " # Add to final list\n",
+ " final_list.extend(item_list)"
+ ],
+ "execution_count": 0,
+ "outputs": []
+ },
+ {
+ "metadata": {
+ "colab_type": "code",
+ "outputId": "3b188412-5c0a-4e71-ef50-20c4ba18082b",
+ "id": "Y48IvuSfmK07",
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 85
+ }
+ },
+ "cell_type": "code",
+ "source": [
+ "split_df = pd.DataFrame(final_list)\n",
+ "split_df[\"split\"].value_counts()"
+ ],
+ "execution_count": 34,
+ "outputs": [
+ {
+ "output_type": "execute_result",
+ "data": {
+ "text/plain": [
+ "train 84000\n",
+ "val 18000\n",
+ "test 18000\n",
+ "Name: split, dtype: int64"
+ ]
+ },
+ "metadata": {
+ "tags": []
+ },
+ "execution_count": 34
+ }
+ ]
+ },
+ {
+ "metadata": {
+ "colab_type": "code",
+ "id": "RWuNBxAXmKk2",
+ "colab": {}
+ },
+ "cell_type": "code",
+ "source": [
+ "def preprocess_text(text):\n",
+ " text = ' '.join(word.lower() for word in text.split(\" \"))\n",
+ " text = re.sub(r\"([.,!?])\", r\" \\1 \", text)\n",
+ " text = re.sub(r\"[^a-zA-Z.,!?]+\", r\" \", text)\n",
+ " text = text.strip()\n",
+ " return text\n",
+ " \n",
+ "split_df.title = split_df.title.apply(preprocess_text)"
+ ],
+ "execution_count": 0,
+ "outputs": []
+ },
+ {
+ "metadata": {
+ "colab_type": "code",
+ "outputId": "7bb68022-5848-44ac-f90c-7cdf6a7eb988",
+ "id": "fG9n77eLmKWB",
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 204
+ }
+ },
+ "cell_type": "code",
+ "source": [
+ "split_df.to_csv(args.split_data_file, index=False)\n",
+ "split_df.head()"
+ ],
+ "execution_count": 36,
+ "outputs": [
+ {
+ "output_type": "execute_result",
+ "data": {
+ "text/html": [
+ "| \n", + " | category | \n", + "title | \n", + "
|---|---|---|
| 0 | \n", + "Business | \n", + "Wall St. Bears Claw Back Into the Black (Reuters) | \n", + "
| 1 | \n", + "Business | \n", + "Carlyle Looks Toward Commercial Aerospace (Reu... | \n", + "
| 2 | \n", + "Business | \n", + "Oil and Economy Cloud Stocks' Outlook (Reuters) | \n", + "
| 3 | \n", + "Business | \n", + "Iraq Halts Oil Exports from Main Southern Pipe... | \n", + "
| 4 | \n", + "Business | \n", + "Oil prices soar to all-time record, posing new... | \n", + "
\n",
+ "\n",
+ "\n",
+ " \n",
+ "
\n",
+ "
"
+ ],
+ "text/plain": [
+ " category split title\n",
+ "0 Business train general electric posts higher rd quarter profit\n",
+ "1 Business train lilly to eliminate up to us jobs\n",
+ "2 Business train s amp p lowers america west outlook to negative\n",
+ "3 Business train does rand walk the talk on labor policy ?\n",
+ "4 Business train housekeeper advocates for changes"
+ ]
+ },
+ "metadata": {
+ "tags": []
+ },
+ "execution_count": 36
+ }
+ ]
+ },
+ {
+ "metadata": {
+ "colab_type": "text",
+ "id": "m-a0OpqhmKJc"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "## Vocabulary"
+ ]
+ },
+ {
+ "metadata": {
+ "colab_type": "code",
+ "id": "RUMQ_MwumJ8F",
+ "colab": {}
+ },
+ "cell_type": "code",
+ "source": [
+ "class Vocabulary(object):\n",
+ " def __init__(self, token_to_idx=None):\n",
+ "\n",
+ " # Token to index\n",
+ " if token_to_idx is None:\n",
+ " token_to_idx = {}\n",
+ " self.token_to_idx = token_to_idx\n",
+ "\n",
+ " # Index to token\n",
+ " self.idx_to_token = {idx: token \\\n",
+ " for token, idx in self.token_to_idx.items()}\n",
+ "\n",
+ " def to_serializable(self):\n",
+ " return {'token_to_idx': self.token_to_idx}\n",
+ "\n",
+ " @classmethod\n",
+ " def from_serializable(cls, contents):\n",
+ " return cls(**contents)\n",
+ "\n",
+ " def add_token(self, token):\n",
+ " if token in self.token_to_idx:\n",
+ " index = self.token_to_idx[token]\n",
+ " else:\n",
+ " index = len(self.token_to_idx)\n",
+ " self.token_to_idx[token] = index\n",
+ " self.idx_to_token[index] = token\n",
+ " return index\n",
+ "\n",
+ " def add_tokens(self, tokens):\n",
+ " return [self.add_token[token] for token in tokens]\n",
+ "\n",
+ " def lookup_token(self, token):\n",
+ " return self.token_to_idx[token]\n",
+ "\n",
+ " def lookup_index(self, index):\n",
+ " if index not in self.idx_to_token:\n",
+ " raise KeyError(\"the index (%d) is not in the Vocabulary\" % index)\n",
+ " return self.idx_to_token[index]\n",
+ "\n",
+ " def __str__(self):\n",
+ " return \"| \n", + " | category | \n", + "split | \n", + "title | \n", + "
|---|---|---|---|
| 0 | \n", + "Business | \n", + "train | \n", + "general electric posts higher rd quarter profit | \n", + "
| 1 | \n", + "Business | \n", + "train | \n", + "lilly to eliminate up to us jobs | \n", + "
| 2 | \n", + "Business | \n", + "train | \n", + "s amp p lowers america west outlook to negative | \n", + "
| 3 | \n", + "Business | \n", + "train | \n", + "does rand walk the talk on labor policy ? | \n", + "
| 4 | \n", + "Business | \n", + "train | \n", + "housekeeper advocates for changes | \n", + "
\n",
+ "\n",
+ "In this notebook we're going to cover the basics of computer vision using CNNs. So far we've explored using CNNs for text but their initial origin began with computer vision tasks.\n",
+ "\n",
+ "\n"
+ ]
+ },
+ {
+ "metadata": {
+ "id": "wKX2R_FT4hSQ",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "![](\"https://raw.githubusercontent.com/GokuMohandas/practicalAI/master/images/cnn_cv.png\")
"
+ ]
+ },
+ {
+ "metadata": {
+ "id": "zOUWqHjL6hmU",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "# Set up"
+ ]
+ },
+ {
+ "metadata": {
+ "id": "kjXAaAyx6i5W",
+ "colab_type": "code",
+ "colab": {}
+ },
+ "cell_type": "code",
+ "source": [
+ "!pip3 install torch torchvision\n",
+ "!pip install Pillow==4.0.0\n",
+ "!pip install PIL\n",
+ "!pip install image"
+ ],
+ "execution_count": 0,
+ "outputs": []
+ },
+ {
+ "metadata": {
+ "id": "vXjCadon6toa",
+ "colab_type": "code",
+ "colab": {}
+ },
+ "cell_type": "code",
+ "source": [
+ "import os\n",
+ "from argparse import Namespace\n",
+ "import collections\n",
+ "import json\n",
+ "import matplotlib.pyplot as plt\n",
+ "import numpy as np\n",
+ "import pandas as pd\n",
+ "from PIL import Image\n",
+ "import re\n",
+ "import torch"
+ ],
+ "execution_count": 0,
+ "outputs": []
+ },
+ {
+ "metadata": {
+ "id": "N518ySE16trp",
+ "colab_type": "code",
+ "colab": {}
+ },
+ "cell_type": "code",
+ "source": [
+ "# Set Numpy and PyTorch seeds\n",
+ "def set_seeds(seed, cuda):\n",
+ " np.random.seed(seed)\n",
+ " torch.manual_seed(seed)\n",
+ " if cuda:\n",
+ " torch.cuda.manual_seed_all(seed)\n",
+ " \n",
+ "# Creating directories\n",
+ "def create_dirs(dirpath):\n",
+ " if not os.path.exists(dirpath):\n",
+ " os.makedirs(dirpath)"
+ ],
+ "execution_count": 0,
+ "outputs": []
+ },
+ {
+ "metadata": {
+ "id": "KG_bcOZ58vhB",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "# Data"
+ ]
+ },
+ {
+ "metadata": {
+ "id": "PGQLzyss8wja",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "We're going to first get some data. A popular computer vision classification dataset is [CIFAR10](https://www.cs.toronto.edu/~kriz/cifar.html) which contains images from ten unique classes."
+ ]
+ },
+ {
+ "metadata": {
+ "id": "NYy0WlkB9AoK",
+ "colab_type": "code",
+ "colab": {}
+ },
+ "cell_type": "code",
+ "source": [
+ "# Don't worry we aren't using tensorflow, just using it to get some data\n",
+ "import tensorflow as tf\n",
+ "import matplotlib.pyplot as plt"
+ ],
+ "execution_count": 0,
+ "outputs": []
+ },
+ {
+ "metadata": {
+ "id": "Ka-WxeEJ8vAd",
+ "colab_type": "code",
+ "outputId": "ec24e935-0562-4c55-c2ab-2bfe59a49128",
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 85
+ }
+ },
+ "cell_type": "code",
+ "source": [
+ "# Load data and combine\n",
+ "(x_train, y_train), (x_test, y_test) = tf.keras.datasets.cifar10.load_data()\n",
+ "X = np.vstack([x_train, x_test])\n",
+ "y = np.vstack([y_train, y_test]).squeeze(1)\n",
+ "print (\"x:\", X.shape)\n",
+ "print (\"y:\", y.shape)"
+ ],
+ "execution_count": 8,
+ "outputs": [
+ {
+ "output_type": "stream",
+ "text": [
+ "Downloading data from https://www.cs.toronto.edu/~kriz/cifar-10-python.tar.gz\n",
+ "170500096/170498071 [==============================] - 50s 0us/step\n",
+ "x: (60000, 32, 32, 3)\n",
+ "y: (60000,)\n"
+ ],
+ "name": "stdout"
+ }
+ ]
+ },
+ {
+ "metadata": {
+ "id": "0YIiwLWcBH07",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "Each image has length 32, width 32 and three color channels (RGB). We are going to save these images in a directory. Each image will have it's own directory (name will be the class)."
+ ]
+ },
+ {
+ "metadata": {
+ "colab_type": "code",
+ "id": "xWqzC-M1NCzx",
+ "colab": {}
+ },
+ "cell_type": "code",
+ "source": [
+ "!rm -rf cifar10_data"
+ ],
+ "execution_count": 0,
+ "outputs": []
+ },
+ {
+ "metadata": {
+ "id": "AdZjOciC-Bzm",
+ "colab_type": "code",
+ "colab": {}
+ },
+ "cell_type": "code",
+ "source": [
+ "# Classes\n",
+ "classes = {0: 'plane', 1: 'car', 2: 'bird', 3: 'cat', 4: 'deer', 5: 'dog', \n",
+ " 6: 'frog', 7: 'horse', 8: 'ship', 9: 'truck'}"
+ ],
+ "execution_count": 0,
+ "outputs": []
+ },
+ {
+ "metadata": {
+ "id": "DbNtoIxD8dxc",
+ "colab_type": "code",
+ "colab": {}
+ },
+ "cell_type": "code",
+ "source": [
+ "# Create image directories\n",
+ "data_dir = \"cifar10_data\"\n",
+ "os.mkdir(data_dir)\n",
+ "for _class in classes.values():\n",
+ " os.mkdir(os.path.join(data_dir, _class))"
+ ],
+ "execution_count": 0,
+ "outputs": []
+ },
+ {
+ "metadata": {
+ "id": "wf5EY4Ey8kFq",
+ "colab_type": "code",
+ "colab": {}
+ },
+ "cell_type": "code",
+ "source": [
+ "# Save images for each class\n",
+ "for i, (image, label) in enumerate(zip(X, y)):\n",
+ " _class = classes[label]\n",
+ " im = Image.fromarray(image)\n",
+ " im.save(os.path.join(data_dir, _class, \"{0:02d}.png\".format(i)))"
+ ],
+ "execution_count": 0,
+ "outputs": []
+ },
+ {
+ "metadata": {
+ "id": "PrD2oUFu_KVF",
+ "colab_type": "code",
+ "outputId": "e336f8c8-5cf5-4235-ec10-68761b6c725c",
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 101
+ }
+ },
+ "cell_type": "code",
+ "source": [
+ "# Visualize some samples\n",
+ "num_samples = len(classes)\n",
+ "for i, _class in enumerate(classes.values()): \n",
+ " for file in os.listdir(os.path.join(data_dir, _class)):\n",
+ " if file.endswith(\".png\"):\n",
+ " plt.subplot(1, num_samples, i+1)\n",
+ " plt.title(\"{0}\".format(_class))\n",
+ " img = Image.open(os.path.join(data_dir, _class, file))\n",
+ " plt.imshow(img)\n",
+ " plt.axis(\"off\")\n",
+ " break"
+ ],
+ "execution_count": 14,
+ "outputs": [
+ {
+ "output_type": "display_data",
+ "data": {
+ "image/png": "iVBORw0KGgoAAAANSUhEUgAAAd8AAABUCAYAAADDAD33AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAIABJREFUeJzsvXm4XVV9//9aa+29z3zumHtv5puE\nTAQpiFgHMCQINwFRDGgtBae29qnap0i1aMXqY621Po/a1mprbX9Pa60jUpywSouggiAQ5iGEhISb\n5CY3ufOZ9rDW+v2x9jlJlNxL/cKNbc/7efLknnP22eez1/6sz/z5bGGttbTRRhtttNFGG/MGebIJ\naKONNtpoo43/a2gr3zbaaKONNtqYZ7SVbxtttNFGG23MM9rKt4022mijjTbmGW3l20YbbbTRRhvz\njLbybaONNtpoo415xq+c8n3ve9/LZz/72ZNNxv963HXXXVxwwQW/8P4nPvEJvvzlLz+rc1xwwQXc\nddddzzVpzzmiKOLGG288qTTcc889bN68+aTS8P+C/wn0v/vd72bjxo38+Mc/PtmknBD79u3j1FNP\nPdlkPOeY7bq++MUv8ld/9VfzTNFRfO1rX3tOzvNc3zvvOTtTG/8r8Ed/9Ecnm4TnHI8++ig33ngj\nl1566ckmpY3nEd/97nf5/ve/z7Jly042KW0cgyuvvPKk/bbWmo9//OO8/vWvP2k0nAgnzfO96667\nuOSSS/jYxz7G0NAQmzdv5v777z/umPvuu49t27axZcsWLrroIu644w7AWSDnnHMOX/jCF7jkkks4\n99xzuemmmwCw1vK3f/u3DA0NsWnTJj7ykY+gtZ6Xa7rxxhsZGhpiaGiI97znPURRxNe//nW2bt3K\nhRdeyG/91m+xf/9+AG644Qbe+c538qY3vYmPf/zj80LfM+Ev//IvGRoaYsuWLWzfvv24yMPmzZtb\na3ngwAEefvhhLr74YoaGhvjoRz960miGZ7/WR44c4Z3vfCf3338/V1xxxbzS+NnPfpaNGzdy6aWX\ntng3iiI+8pGPtHj+7//+71vHP/nkk1x55ZUMDQ1xySWX8NBDDwFur7zhDW/gD//wD+fVOHom+sMw\n5E//9E8ZGhpi69atfOxjH2vtrx//+Mds3LiRrVu38tWvfpUXvvCF7Nu3b15oveqqqzDG8Nu//dtc\nfvnlfOpTn2Lr1q1s376dyclJ/vAP/5ChoSEuuugi/uEf/qH1vRtuuIGXv/zlvPrVr+aGG25g7dq1\n80Lv9ddfzyWXXMLGjRv5zne+gzGGT33qU2zZsoUtW7bw3ve+l1qt1rq2Y6/nZz/7Ga997Wu56KKL\n2Lp1K9/73vcAmJ6e5j3veQ9DQ0Ocf/75fOMb33heaE+ShPe///0MDQ1xwQUX8M53vpNKpfKM1wXw\n6U9/mve///2Akymf//zn2bZtGy95yUued4/4LW95CzMzM2zZsoVNmzYdt45XXXUV3/zmN1vHHvv6\nRz/6UUvW/d7v/R6Tk5O/cO53v/vd/Nmf/dkvT5w9Sbjzzjvt+vXr7Xe/+11rrbVf+9rX7Gte8xp7\n7bXX2s985jPWWmtf9apX2e985zvWWmv//d//3b7yla+01lo7PDxsTz31VPuv//qv1lprb7rpJnvB\nBRe0jrv44ovt9PS0jePYvu1tb2sd93xieHjYvuQlL7EHDx60xhj7jne8w37uc5+zp512mh0ZGbHW\nWvve977X/smf/Im11tpvfOMb9owzzrBPPfXU807bM6G5/s31/epXv/oL679p0yZ73XXXtb5z2WWX\n2a985SvWWrfm69ats3feeee80/7LrPWb3vSmeaVx586d9uyzz7aHDx+2SZLYt7/97XbTpk32b//2\nb+2b3vQmG4ahrVar9tJLL7W33HKL1VrbCy+80H7ta1+z1lp7zz332HPOOcfGcWzvvPNO+4IXvMDe\ncccdJ53+z33uc/Z3f/d3bRzHtl6v28suu8zeeOONNkkS+7KXvczeeuut1lprP/axj9l169bZ4eHh\neaN5zZo1dmRkxF555ZX2rW99q9VaW2ut/cAHPmA/8IEPWGutnZiYsOedd569++677cTEhD399NPt\njh07rNbavutd77Jr1qx5XmkcHh62a9eutV/60pestdZ+73vfs+eff779zne+Yy+99FJbrVZtkiT2\n93//91v78OevZ9u2bfauu+6y1lr71FNP2WuuucZaa+373vc++8d//MdWa23Hxsbsxo0b7Y4dO57z\na/jhD39o3/jGN1pjjDXG2E996lP2y1/+8jNel7XW/s3f/E1rL27atMm+/e1vt0mS2CNHjtizzz7b\nPvbYY885jU0MDw/b9evXW2t/cR2vvPJKe+ONN7aObb6uVqv2xS9+cWvtPvKRj9gPfehDx52ruQ+S\nJPmlaTupOd98Ps/WrVsBuPDCC3nssceo1+utz2+88cbW52eddRbDw8Otz5IkYdu2bQBs2LCBAwcO\nAPDDH/6Qyy67jFKphOd5vO51r+MHP/jB834tt99+O2eeeSb9/f0IIfjEJz7BW9/6Vu69914GBgYA\neNGLXnTcNQwODjI4OPi803YiZDKZ1vpu3bqVxx57jDAMjzvmvPPOA5zH89BDD3HRRRcBsGXLFnK5\n3LzS28Qvs9bzjbvvvpuzzz6b3t5elFK8+tWvBhx/XnHFFQRBQD6f5zWveQ0/+MEP2L17N2NjY1x+\n+eWA4/fu7m7uu+8+ALLZLC996UtPOv233norr3/96/E8j2w2yyWXXMLtt9/Onj17iKKIjRs3Akc9\n0ZOFjRs3IqUTb7fddlsr6tHZ2ckFF1zA7bffzgMPPMDg4CBr1qxBSslv/uZvzgtt1tpWCuTUU0/l\n4MGD3HrrrVx66aXk83mUUmzbto3bb7/9Ga+np6eHG2+8kV27djE4OMgnPvEJwPHWG9/4RqSUdHd3\nc8EFFzwvsq+7u5tdu3Zx8803U6/XufrqqznnnHOe8bqeCZdeeilKKXp6ejjrrLPYvn37c07jiXDs\nOp4I27dvZ2BggDVr1gDwnve8h/e9732tz2+99VZuuukmPvnJT6KU+qVpOak533K5jBCi9TfAzMxM\n6/Nvf/vbfOELX6BarWKMwR4zhlopRT6fB0BK2droMzMz/NM//RNf/epXARfz7+7uft6vZWJionUN\n4BSb1pq/+Zu/4ZZbbkFrTbVaZcWKFa1jOjo6nne6ZkNnZ2eLEYvFIuBCV8eiSWMz7NI8Tghx3PXO\nJ36ZtZ5vTE1NUSqVWq+P5e+/+Iu/4JOf/CTgwtCnn34609PTNBqNljEEUKlUmJycpFwuzzuvnIj+\n8fHx42jp6OhgbGyMqamp4+5JX1/f/BH7DDiWxvHx8eNoK5fLjI6OMj09fdxx/f3980KbUqpluDZl\n14nW9djXTXz0ox/l7/7u73jLW95CNpvlmmuuYcuWLczMzHD11Ve3FEIYhmzZsuU5p//000/nuuuu\n41//9V+59tpr2bx5M7/zO7/zjNf1TPj56/x5mfN84tnso5+XL0EQtP42xvD+97+fFStWUCgU/p9o\nOanK99g4+tTUFHB0cQ4dOsR1113H17/+ddavX8+ePXsYGhqa85x9fX1s3rx53pP8XV1dLS8FnOC8\n+eabueWWW/jiF79Id3c3X/va1/j2t789r3TNhuaaw1GleyLmbL5fqVQolUoYY477/nzif8Jal8vl\n4wzJiYkJwPHnW9/6VjZt2nTc8fv27aNQKPAf//Efv3Cuk1FRfiL6e3t7j9u3k5OT9Pb2UiwWWzlK\ngCNHjswfsXOgSfOiRYuAE9M8Ojp6skg84bqe6NgPfOADfOADH+AnP/kJf/AHf8C5555LX18fn/nM\nZ1oe2/OJZm56cnKSP/mTP+Ef//Efn/V3m7wE7jpPlhPy8wZCU551dXUdR2O9Xj9O1n3pS1/ive99\nL//yL//Cm9/85l/+93/pbz4HaDQa/Od//icA3//+9znttNPIZDKAs1bz+TwrV64kSZKWJ1utVmc9\n5/nnn883v/nNVvj6K1/5Cv/+7//+PF6Fw8aNG9m+fTv79u3DWssHP/hBDh06xOLFi+nu7mZiYoLv\nfe97c9I/n2g0Gtx8882AW/8XvOAFx1l5xyKbzbJu3brW8d/97nd/IUQ9X/jvrrXneVQqleMiJ883\nzjzzTO69917Gx8fRWvOtb30LcPz59a9/Ha011lo++9nP8qMf/YjFixczMDDQUr7j4+Ncc801xymH\n+cSJ6D/vvPO4/vrr0VpTq9X45je/ycaNGxkcHCRJkpah8OUvf7kV1TrZOO+881ryY3x8nJtvvpnz\nzjuPDRs2sGPHDvbu3Ysxhuuvv/6k0vitb32Ler1OkiRcf/31rRD+sYjjmKuuuqplKGzYsAHP85BS\nsnnzZr7yla8ALi330Y9+lEceeeQ5p/Ub3/gGn/nMZwAXPVu5cuV/617fdNNNGGM4cuQI27dv50Uv\netFzTmMTvu9jjGkVhB2LBQsW8PjjjwOuuHfPnj2AS/kcPnyYBx98EHCFh83rlVKyfPly/uIv/oK/\n+7u/Y/fu3b80bSdV+S5evJh7772XoaEhPve5z/HBD36w9dm6det4xStewdDQEL/xG7/B5s2bOeOM\nM7jqqqtmPecrX/lKNm3axGtf+1q2bNnCLbfcwjnnnPN8XwoDAwN8+MMf5k1velPLQ7/kkkuYnJzk\nggsu4I/+6I+4+uqrOXjwIB/72Meed3qeDVauXMl9993Hli1b+Od//mf+9E//dNbjP/ShD/H5z3+e\noaEhHnzwQVatWjVPlB6P/+5an3XWWYyOjnLuuefOW+X7+vXrecMb3sBrX/tatm3bxgtf+EIArrji\nChYtWsTFF1/Mli1b2LVrF2eddRZCCD75yU/yb//2b2zZsoUrr7ySl770pa3UynzjRPRfddVVDAwM\ncPHFF3PZZZdx3nnnsXXrVoIg4EMf+hDve9/7eM1rXsOKFSuQUv5KKOCrr76a6enp1rq+7W1v4/TT\nT6evr49rrrmGN77xjbzuda/jrLPOOmk0btmyhVe84hVs27aNV73qVQwMDPDGN77xF47zfZ/LL7+c\nN7/5zVx00UVcddVVXHfddeRyOa6++mpmZmYYGhri4osvxhjzvFRvn3/++TzyyCNceOGFbN26lSef\nfJK3vOUtz/r7q1ev5vLLL+fiiy/mqquuYvXq1c85jU0sWLCAs846i02bNh0XLQNXCX3rrbeydetW\nbrzxRl7+8pcDkMvl+PSnP92qHN+xYwfvete7jvvu4OAg73jHO7j22mt/aZki7Hy6A8fgrrvu4rrr\nrmt5Um200cb/HtRqNc4880zuueee43LHv2qw1rYMhJ07d3LFFVdw9913n2Sq/vdi8+bNfPzjH39e\nvd3/KfiVm3DVRhtt/M/EZZdd1uq3v+mmm1i1atWvtOJNkoRzzz2XBx54AHA0n3HGGSeZqjb+r6A9\n4aqNNtp4TvC+972PD3/4w/z1X/81hULhVya9ciJ4nscHP/hBrr32Wqy1LFiwgD//8z8/2WS18X8E\nJy3s3EYbbbTRRhv/V9EOO7fRRhtttNHGPKOtfNtoo4022mhjnjEvOd9/ueEr1OsV6nXXs6jjhCRO\niOMYAGMStE7cZ2n/IwgSLVpN0FprjDEIYbHWtno2jTF4+CjPQ2NBCQjchBclJBkrsEKglQDpqhoF\nEs8L+PNrrp2V7isv3oQvFSZ2tEVJjPA9jNbEcUxiDMJzfbG5bBkhBdbW6S56xNr91iO79lNLPKSE\nro4izRh/FIYkiUEnhjAMEeLotQprsBiMBPeOs5GkVBhjePSxHSek+bP/30vcOdIKzuZaNds+hBQI\njrZ/JFhiDNaYFm3WWowxxERoqzHpWrs1twgrkOnBltYfKKUQSoFUqPT3JQIJXPPme2dd609+9F1I\nBMWsa60xRqPDBCV98qUCvX09FNMe8LhSRUcNpLAIpWhY91uhCIiNQOgQJWPEdNrb10iIcllCIZFI\nsBbVGjFnscYSxZpEa4TneCexhtho3vHu2Qenu2pZ97exhqnpmGrF0qhbKpWEsOGa8+O4RpDN06hb\nwrrEWNeDrM0M5cJCLAGFEmSzbktOjmvQHkImCDlMV3eWjs6lABTLHvm8h0Cm+6F5LYK5OnsevPM2\napUGC5a6J/8E+Rw3fPELfOcbX+WUVat40cvPI0rnDhzY+ThP/PQ2TFxn4aoVnPpi14qhhUepWKRQ\nKBBHMSZdfyNiYhuhtUcum8NOTHBzWoA1sH45ow8/gYnA7ywyPuGmN73yVa+hZ/EKfuOtb5uV7gvP\nX0+lApNTrrc81hatIY4TlAfK85FeLl0FH60tSSPEVx6ldFpRvlxkZU8RtGHnwYNo6c7l2YSucjc1\nrQmThP4FXZRVA4DVCzt5bM8IM7WIUqlILXJtJaOTNRJ8fnb7z05I8+d/+CSaDCLdP0JYjr1BQliw\nx7epWAsYSWLcdxIr0BaE1UjBca1bibUYODooQqT7FAPaoIQkUApfNn/f8cfbzp+9teeG2/ZSDxsk\niZN7sdFoY5BARlmq9ah1z0GgjUZJgzd9mFA4eWhyZYTRCOHoU6mqkUqSYEEoPCnJ+j6+56VnEhhr\nEVKQUR6k70spUUKxbfPyE9J8z/YHuOfOn/Cj2/4LgKf37qajs8yqlSvpHxhgYnSEmf2u1SgXjLJk\n9QVM20G8fJnly5YA8Gsb1lEsl53s1hppnH5K0vsisWhxrPQ8Kl+1cTKjKSu1MVgLLz/7zBPSPC/K\nt9Fo/MJABscIAq01SaKBowK++S9JDGEYtd4HkJLjZnN6nocnBAiD7ymsFBibKjEsBolVEitFi8+V\ndMJ9TghJrE2TNIRQSCtBCDK+gjCEOKVbaWQ2wAqFn8ki0tMbrTEJIAWHDx85qmCFQAiJNYC1WHFU\ncGYCHyEFodbHKV/P857FvFyBtUfXy/0nWv8LK44XAIBnJVbI5sE41S9ABCiOMpTBrYVT3/K43wGQ\nRuIJhUIi05Gn0hNINXdZge9nkBaUcizpKQ8rAzDufpcKRfK+D4CWCtsIsDpGSIFMmkaLR0YKlIgQ\nCOqpcedLhfQChFIYna5oykPGGIw1KCXxvEzLQFNGI1ODcDYkiUan54wNDO9tcOigplKzWN1g+Cmn\nfMYOHWTZqjNJjKJnwVq8TJh+fwqtJVL2MV2BKHZK2RoBRuLJIj+946vkgjwvednbASh0KBYtyVIq\nQK4gKBT8lBqDMSClz4ngK4UUkiTdVx093SxbuYplgysQUhD4CpFe9oKOTp5Uisp0SCbIk/PTWd7K\np5jJ40sPY5LW+mubMDkzSSW2IBUTw/sZGXeTguT+PBPTVXq7F6ATjUzXuVgoMDd3wMHDDbT2qYfu\n6CjRKKmQ0kMpCVJhU4XQiELiKMIXEik9gkzKNyaBRkTR9/CSkGrkjLNs3md85jCJFqA8uspLWNzZ\nBUBfV5aHdj2N50uSJMJPedD3JcyxF4UwCHRLWGsgsWCNk29Cypbh1NzXzr61x62JtRaEAUHrXNY6\nJSuE22PWgi/dObLSEiifQHl4UhAoJ4yUbMqB2RHHMWEck6Qy0mhngCfGoAzEWhMn7jMrBNYYRBRB\nlDBdcVO6Do4/ztT0OGecupbFAwNUKs4IHR6ZpNDZTy5XRAPaaHydKlkhsThHIUkSSN/3PZ9gjhht\nd1cnWy56FatPcYbFZz79CfY8tYvq1DSDy5cxODjIwcjd04OjNUQpptAbMzV2gHuGnwCgOvIELz7r\nLHoGlmD9PI3ECTFrmutukMZgjrocLUdFG4sxlqYLY9wNmhXzonyb3mzT09WxxhpDHMfOatAaY456\nvqSWkbUWpY5aWEopPM9rWWRNGBNjBdhYglKt2aYSgRaGJF0In1QoWTBJPCfdsTFgLCb9PYFEAVq7\n5ZcoZLrh4zBGZAKE7yGUj68ct+TzeeqTNayRLQXoaLZIYdHa4HkeQRCQyTirsaOUY6ZSJao3aNQb\nx2yYo2t4IjSVYdNCPtYTEuKo0dOEh6BpxLYUtjEYKZBCoq3FCHP0c2vdymrPMVj6XSmdMeFZgTIg\nVJMOg/CehfL1AnylUMLdO6O1u9+BB8aSRBEmva9RHJORHiZJEAb81Dhpfm5iQ9ioE1VdpCUICiiL\nE9ZCYKxo7QstBFYpZwgmuhlqQAqPjDf79qhHlpkJQ2XGne3wkSqT45MIq/CET2QbHNi3B4Adj9zC\n6OEn6OtfxYoVpxBpN84zjkM0dWJzGG0snn/U+EiQICQH9+1HqiynneF+pzJjmDwSks8Lgoxk0dJ0\nPndZUp1OGFw1i/L1fISpEbeMYUlX3wBnnH02ykQoDwo5N7PWq3dgPZ/pMKGjq58FC9xDKxILSjl+\n9vwMBseTk6OHePThB7lvxy6OTFWI63W8lKeUVDTqIXmd4FnIZrLufaWe1eSxWuijVIAMvPR8Gmks\nRscY6wS38tLrjgzSU/R095D1A/LpPPJ61GBkYhKh66iMhwnduaxVSJHgedCI6kyMH+GUPudlWa0p\nZ7JMzdTxSJApf67o60UF2VlpDgKPBAX66P5RpMattQg0Mt2LSnGcYmxyqEQjnS+LRXDUSQElLEpa\npJAgwE8NmqwSBJ7EAwQG0fQEOKocZkMUJ8RJQpIOjzBGkxiNTRKM56KSsXb3PKxNcWRsnLGD44Th\nBDY16g6OT+H5MWevX0ZPyUdaF7WKogayXsXzMwghSKwmTn9HSgHWORmJlMimPAJaobYTwGpLgmX5\nipUAvP4NV3LTd77FkzseZ8fjT7D/wDBZz9FQ7h4kTAxBZZRAKgqpTTmy+1HuOryTVWvWsfy0s8n2\nLE7XOiDRxjkkWjjj6JjIq7VpNBCb3iNnw8+10u2cbxtttNFGG23MM+bF8zXGHpPLTfO3aQ63ZTmk\nFqDneWn+0yID0SJRG+PygFof5/kKIZAkCCkBkXq1qcUkgEBhLNjEItLcrUU/q1BoI05Qx3iKUihC\nbUFIFyKxSSt8qTXEiUZKSyNKaLqE5XIZbRWT01XCRsQxgaNWyEJKiVJe64lBxVIWg6UeJWhtj/Fe\nLXON67M2gWNoFgjnuQoLwtnCzfAcAqxMAIMywdG1FglaRMRCYFEI4zwKZQ2SGIFFSxBatvLHnvRd\nJCA2YDVJam0nwriQ2Rzw/YBcJtMycE2SOOve8/ClpFGp0Ujns8ZxRMYqfGuwJsbPOtM1k8syNVOl\nXq0hMASpd2USg5KSTOCBkGhDK68pTcqfiUEEHl7qPQscL86G0YOa+pRh5KDzsCvTMUHmCEZLpC0D\ngnUb3AMU9u15iMcfuIPSr+fI+sZ52YA1MVFYB+ETx5CkPJrN5hDC4nszhGFINpchk3XzypPYECWa\naEpi8Tk84S7Gy0TouMHgqtNOSLOyIKzFpr+vE0Oxs5t8uUyGGGsTJifHAbj77p+x88B+/MBjKozR\naTg7SmIa1Rr79+3n4OFRosR50ZVDBwgrMyRRxKGDo9Rjw5L+HgAGugY4KI8wMj5O2RMEWRflqVar\ndJaf+QECx8KqDEYezbf5QqDjBKstBoOSR2Oy2XwW31eUykWs1ijP7VGFoqYyhFGDYiFLd8YN9I8q\nk3SVcyxY1Mf+kcMYHVOpuHvqNTS9hTL95R5KhSz5ohsaUupcQGaO0Z9CGIRNWpGlrCecLDp2PzfT\nUIC1xskzpZDNhJNNkNZghMLY46NWvhT4qdgTQiLTnK+wGmEjZDPK1hR1gl/IMT8TYmOIE+f9ggsN\nG6PRSUJiLXnPw6Se78z0NEIKFi5diDKddBbc+lQaIYk1TEWKOx/aS5SmJqqJQcYhcRp5FEK0IpVK\nqZY8MVIizVEZLed6cp+VWJLW+vz6y17BKWvW86P/upnbfvifHBzZS5RzXrlBku8okgkU09UG4+lz\nAPq7OqjUQ564/2EmDxxg5boNACxYfSZeoYPQCrRx2YZWmuCYWhn3R7MGxhwfdnwGzIvyTZIkDTE3\nhbBtKdxmEYAQTSWmW4raCicY4Wgu2Ojj84zNc1gDiTZk8xkyadFOYi1T9Sqe53HK8pV0FdxmU8pS\nD48+seVEqNTrSEuLiZUXEGsXWBBSIrH4KVNERkC9QVYqLIJGwwmkIPApFvJMTMygtcVLC3qaBVZS\nusfK5XLa5a4A5UmkFNQbDUC2QlMIM3fOV5iUAdOCK7d4WAxa2FQpNkPSCqQrpBAaRFokphEkgBGa\nVgIQsDKL0b3u+3IaKS0KF8oJZBahJZqQMImoasfQsY7x7bNQvl6AlD4iDdH5vodULuDmIfCkSmsD\nnCGWoAi8gEa1xlT6AIVMMaYSagrd/SgSDo+5XE7eCkoSgmwGiyQ2ohXejo2gEScQuPBdS/kaS3mO\n6Uz7925HSUk9dnku61tq0ShG17FkiIzG73J8tvbXB6lUnkYzw/jM3VjtDIbEVDEcQMuIUNRaQs1r\nBFgt8UTMVO1JZKGbQ5V/ASBM6q540SislURxPV2XOjqp8xq+dGKirUXi8nQA9XqNcncPKshQmZhg\nZGSE+7Y/DMDh0VFWnraOfftHuPPBh9g36a6lUq8Rhg327nmaalSn2OcKmnL1Oi859TTWlrrYNXwI\noyPqqTGxeGAJkzrkqSeeRhYL9OQc3yilWg9TmQ2xiRBG48vUQEw0URzieQI/44HQ2NTgE9JDKMn0\nzDRoQ5B1RliiNVEgyHTlkQgWl1wB25H9mpWndBERky9mEIkA4QyNrq4+lgzkCHyPnK/w0wePRFog\n/dk1grIJPgkylUeH9+8myHZSzndgdYhWCl81DXuDp1yYWVjTLD1AILBINK7+41jlm+pdJ1etbtm4\n0lqsBC1ciFiZlE5haBWjzIJGEtGII6JUQVqtwbqcZqg1PmDSeohMoYOs8sCGqBjIOL5WUYWnntzL\n8jUvoFKvteYfZzK5VsoRjn+ykDHOSBbWYpVqheGFECg7uyIzVoM4Gg6O44Surm5e/dptnPaCF3Db\nLbdwzz0/BWBqcpLazCS1yjTLV51CvvkkvbEx+gtdrFw1yOSBXWz/0W0ALHtqmIUvPItgySoEJbAR\nTSUrMSRWo43AaItu1cdw1M86AeZF+Sol8X2fpHkzpYQmPwiXx2jmfI/NSVrsccZD82Yc6yk33zfG\nUih0smjJIMP7RwAIjebJp4a0iPTGAAAgAElEQVTJeQGve9XlLOxzz+vUpsEdP71tbrqDDDbRNOqu\n8jEJExLr8pEIge+plsLEKkxoEDJDXMy0vPPA95nRNZIkIQgyLU+5mT/V1infWmp9AXieol6vEYUh\nSmZoJiJ//pnGJ8IzHmMhEQlagEo92YzN4E0W0OMCkYAwbk0zQYZM4BN5dWxuGptzRTNJUCX2qkAW\nz1qU9VBNDrMCYwSxgBnboKKdlyq8BDFHvgacIaCUR/PZ1J5UCCnRNkFqjRKCXKoMRaiIqjFaQCaf\nZ7LilEJtaoJsZz+l/qWMje5n59P7AVjR20NPxqe7twdtJWFiMakXV40M8UwFk0YVyulGzHg+hdSj\nPhGOHHkYTwVI5SxqYRpEjSmMiAjjKmFSo25cbtfkJ+ldmWfPU49TftywZJkbeK+8IomdoRKOgoqQ\nvhNS9cRQaVRoTM8wXR2lZ5XgcD2trLUB2XwRJTwa4TTCc7wmbQxmriIxV4YYh47f6tNTdC1ZRhgb\n7rznAXY+/ig2rVhdtvZUGnGdp/aPMTm5h5899CjQ9IQgm83wa2edTkOlhlYSMVWL2Tc6QTU2oDwm\nUw/yrsceRvuCSAZEwpIrOoVUKJfwieagGaSMWbxwQRo9gpHhQ0RRRLHcSbFUZHxyEi8t1kvihETH\neEgCz0On3l4UR1iZsLS/g97OMrUjTh5li7BkZTf7Rg7T0V2mI99LHLo9Nz7ZQHR5WGUp5XItJZJg\nUXNEdOzkISYODrOgZwEAd1z/ZboXrmLt6g2u1iWbRXnufIGUFDI+Od+j4IFOnZHIKiINWgbOqDi2\nPkNKpJJI5XZh04FRQiGSBJ00wGp0amhqS4vvZ0M9jgjjiLgZKdQGV25piZMYzx7NB8faVWzruIaI\nExqprIobMVLXiMIZhBBkU6VskwZaJDRiVxQpxNEuFE9KAqlIlERKiafcd4xNC0VngZCWVkmKI5ok\ndrSsWrOOxUuXs27DqQD88Obvsfepnewb3sf0zAwrVp0CgDQJu/fvYenK5fSvP4Mjw08CsOvA00xO\nTrBk9QG61p2GV+4hNM2qZotGYK3GWNOqwTMI5BxJ3flRvlLh+37L87XaFUAdW/l7tDq3mWQ//nUT\nP18w5N6TFAoF+hcu5r7tD/L0yEEAch1logRMvcHuJ54in4YhH3xoOzueeHxOuivVmCgMU+YDISHI\n+AR+BiUdg0QNp5iNSUiSCJMTgEHRLLLQ1Gp1kjghm/FaHoczIBRWgFLOcrWkGzuJicMGvhRYJYji\nZoWucqXac8Ax4TG/AxhhSIRGaoWcdrfdHJJElYCezsUUfNkqAKiHMH6kTjHjE5QKVAInKMPyJLZj\nnESNgcliTAbScnyTVidW4jpTuk6SKgTPizFqbmtbKlfAFQRNj8hzLQx4SKGoxQlxyhOZbB5fJET1\nCsJGqYUPT+7eiw0meWF5EflCB6s3nO7uW6NCplig3NmJFR7K84hTBbNvvEISQWgScp5HR7cz0PJK\ntormToQkqiBUDt0KlUsy5PCzeUyuTCOq0BDuenoXdDO4cCU3f/kOJkeqrDrT8Y2Xr2EtFCjhKdkK\nYQkEsYg4tGsMY3ez4ew1DK5xD6hPEvBkgEk0liK2WaRmIpI5KrStMcQ6brVRxJUZ8kGGWMMjO3az\navkq+ha5NqQf33k3Dz74YCs10tp2wkNIA0KSkT6BcApxvzbc+/AODo9XMMpDmaRlhN724H10dnYg\nvAwTlQojo26P3nnPPSxZ0MlFV/zOrHT3DXTw8nNezI5H3L6tT1c4Mh4TBAFr16/mwYceYnzCGTpS\nBkjhu6IeaBUyjk6MMZD3WdrhUy4LHt3rjPSlq7qpi2m8jECTgIBDo+55xDaXhwyUvAyLOvppNFIj\nObbI/OyKLHPoYfbd8j12Tznj1QwPoyf2sn/4bgI/S3n5GlS3M/as55FksyS+j8xJopSpKqFlpqFp\nmABSA7WJxLr2OKkUnpKQ8oHyAnK+RJnEGWOphz5TC0HlgDnmV9um/E29OGFIEuMiaMZQD21rbxiT\nYIWiEGSIdUI1clGoOIzJ5AIatSlibfFTGeJnfKQngRhrXdFeU54nRqGTGKkMUnn42tGdtWbOyvKm\nY9LSF0a3XsdxDMLywrNfDMCyZYPc87M7ueuOnzA6OsKO1Kgsl4oo3+P2O37EujVrWbXCFd0VBhZz\n6JFHmLj9Nhbu28nSDWeTW7wCgEaugIkk0hpnANEM/cNcru/8VDsLV4lmmrJMHrOYabuKlc2+VIPW\n2oVlj2nzcUrEVQgjj+YxrIFI+sxM1Zmq7uXw4QmEdZc1OnKEyalJFvX2cu+991Cput7Cp/fteVbt\nDRMTU0RhdPRYE5PNBGR8n1w2RxjFTE07ZhPCki9kEV6ANZZ8mtMKY1fZaKxBCoE+JleAlNi0itul\nq91nmWyGnq5uqjMhUVoZCeBJr9UzOBeONVoMoLEuHDWVJTzgSu5zlSJdnR2obEKko1buZe/hQzz8\n2F7W9neycsUpjE+7c+2tVfH6esktqJDvHiMTVPFEM3yfJ6HADJaGLxBeGj724mOqLU8M5XkkWmOa\n5qKnkMb1E1uhCSMXCgMoK49yIYcvYyqVCtWGuwf1KGFBdyeVsTFEKcfi5W6DRI0K5d4BsoUyvu+j\nMMQp62cqESrIIpMQ3/fIpWFFGzYQco5q5/ok47U95AKX1+zuXES90aBRDZHKI1Al1zIEeNqifMtp\np6/lnrsfZPFBdw9WnrbQeWzWpWfC1NsQVpDN+RzYdZhCd46+FSV0kHqIKvU+ZYy1ohVNkcKg9Nz8\nYbSmaZaHYUgU1hno6+Pii19FT1cHX/ySe/bt/fffj0yNzOMEW/P6Gw0OPPU065Y4z+7Jap1aLaRc\nLFMOJLXqFJmsUy6VWo0w0RQCQRxFTE46Rfnkrt14dumcNBcKZQ4eGqXJSbliiXzDoGxAfbJOyc8S\n+Y4PZuoz+LkOPBJyyqe3y4XFhw/tQwR5tM0ipEJmnFIsdZbZP3KQhb09LMj5BI0xlg06PtARjE5M\nYXJlRqb3cWTURVPy2W66s32z0pyjQlfWUi669VnVm8P6inBmP2ImovrYCLbb8U7X4CCdxYVYzyOM\nDJOT6cPbhUdfsYN6UiOMEpqWXhBksNYQm5ioETunpKWYFVb6NBKDlB6xTXO3YZVAGeCyWek+cGCC\nelhpGXLaJATCIoFGlKCEpZRzsmKiGpIYQ5SBsFFnKo3grerrpFBaTP+iRSRa03zq3p6xKUanKgga\nYBUuBek+k17Aqv4i/eUcCMFUGimdmq636jdmgz3G9W2lKVtK+WjKrqevnwsvejXr1m/g9h/9kB0P\nuwdrTE2MITxBtlFj+/1VDh1xBthpp6ymc+1aDg/vZXRkGjt6GwtWO8NNnrqOTKfjA21N6/ddVfyv\ngPK1yhVBxc3Yvo2xrvkV0epeSZWvATcHwaKEbPXsNiEwaXuSe62UTzW2TE3X8UWCQLHjUWfJLF62\nnMP7RmhMTLB82UJ279npfl8c/b3ZIC0EntfqCbb4mMQZEqFIqDVijHSejRCW2PoYkUH5AYGf5lUV\nLOjtYXyqhspksc0ihjhOw+pHma95qb4fUCgU8P2AMNYtj6MpBJ8NjvZCu15nJRReZEimOukrO8u3\n0J8nrh3AC2vkMopq2pqTVYIN61fT291LXBbUlHtgdJzZR90YqlMaX9QoFUOKabrOU0W0J0lsgLEa\nKVzOW+s60bNoqe7u7qaRRhHA9clJKVFIEqMRUtJsza7VamSUpVjIkg26GT3kjKqujhK9PR1MT4wy\nNhqxeOkiAAYG+sl19LqUQKOOIG4ZMfm4SrcIqYoGZT9PJs1Vh1EdoWYPO9cqMdVajN+ZNuMnFSQJ\nWhuqlWm8IEMtdJ8FgcBaw/LBU3j4gd3sudcJ8eVLFnJo/zjV6ZByqUTPWhdatzkNGHY/OsbyDQN4\nOY8wSotULGASYu0KTLy0GkVKgZwj1iWl866b3G90THV6gsULB5ieXM63v/Ut7n/wIfc7aZTJDbc5\nJh10jICbnpiEUjrEIpunGml6uzrpLJeoNsrkCq5t6fDYJKPjE2R8RW+xxGmnuWKWFaedTT4z914c\nHR1nenoamxZTHjkygZIZOrq6OTQySi6TZ/UqZ9AcnppA+jmiyRo9nZ3o1GjTScxMtcHOnQdZe0ov\nvZ3NgqsavcU+qIcs7O6gXFJMjLq0Saanl/27Rqk2DHv2H2JyzD3EfkEPRGL2tr8pAxWZ5bRT3bU+\nsv0nTE80sNMTdHkWpS1R1Qn4Q5WDTOzvxuYKBOVO/KzbWEZA2BgDJDrReOlQn0bVugIlASoJyXgK\nZZvN9T5RBPUGTFcipqdcTUKgQsq5uXvX73hwJyZuHB18REI5JyhlfJSQhLF7DTBVrTNZi1jUmWFy\npsLpqwYB2HbhuSzoKpPLFYmjkCh2m/euR5/kX77zQ8arJpX9thW5yfgeKswzbAyepwhT9TQTZvCy\nc9cFgBsi5P5qFrBJwIBxw5bA5YeFgFWrV7N06RKeeNTJw9tu+S+e2PEo1ekZwkaDRto2NXHkEAML\nFzM4uIrOJVmqe3dTecKFpGuVKXpWrmL50lPwg8wxbWXM2dbVbjVqo4022mijjXnG/Hi+FhfmSS10\n53m6/IEVzvtrFu5Y46rdmuMKE51OuDJpFlh4aGuclQE0woQk0uSzeTIqw4//81bGx1zIpqu7l66u\nTl72srPp6enESGfJGeyz8nzzmSAtsEhbbaTCGEtiNBo3Ki2rmi1SCmMTPGkplspk0yKjmekKGd9z\neSdPEkXNkW0JSko8TxDHpP0HTavNTebxPB/fHh0190z57mdCc5gGzbMJCTrAq2bJRSUC69anVjmC\nJUZ6UK3UW8NEikFAR76TRRvW8HTjQYKefQD0yINU6jHTFRgPDRVj6Mk6+60QKHyvijINZFIjSlwI\nMBQhFW/uaudCoUCxWKSehq2q1SpCuJGYrhVLYZOm5WqI4pDEDygViqxf64qXhBxmanKUKHFjJ5M0\n0uLni2iVJYwth/fuZebgHuJaWqQVxYQobManmslTabj3M7kiKjN7NevY6CGEzDKJ+46JDUq4QqRs\nJkNsLLWaiyaMjVXo7RkgE+Q5/cwXcN9298D227++HVUS5BcW8Yyhq7lHEo+ZpyzTk1XOXn8KUa1B\nNR1VKYRHwS+SkwXipEGEixgo5c3JH57vo9L1BIh1wszkBB3lEsP79nHbT35KlFaVH+v1SqXoSMc0\n1mo1wjRKMV2tUk/XOQiy5IKIjo4yidVIPyCK3P7t7+2iXpkkalTxcj6rVrlhCKe/8EWYpM5ciEJN\nEietKvVctoSX8ZBZwYLOPuIwankaCxYsomE0vpH42YCpSRdeDiTEM3X2HzlEPq5y+batrfMfHD3M\n4UMjjExNkJRKrFjhinMmq4ZaY5jKyBR9Czro6V+fXvdhSsHs/PHAjic4MDLGqkXOu8XXyKCbiZm9\nwBSeJyiUnAjO6Sr1AxX2DI8SdHawbLUrAvJKJbRSKM/j0Mih1kQsz8uAsUS6gdU1MkajUl8qloJc\nvgjZDqZnYqKa24vVeAbTEcy51pXmxCbbjFQa6tWESiNkqe8xHceEaQW11paqlURJjeUDnVx0nsur\nLunvJpPJUCiUiKNMq8PpvLPP5J7Hn+Q/730CT3kYaWmNxTSaRw7UCI0hCAKaVbnZbJkgmT181hw/\n3JycJkU6VEcKd3qrW3UwxuhWTVEun+OsX3cjeZcNruTOO37Cz+64jYMjw0RpF4OOasxMzzA1OcHy\nUwYZWLkcNen4v1qvMrz9fg7uH2XVmrV0pcV11qbFaLNgnpSvTSf8Hm2zcbrGtNoDbEqoSSxaW4JM\nBqRE22ZoR7n+VS9LqZynUXNhzcbMJArJ3j1Pc98926lMVQgCF+o6MnqIvv5OFvR1YKV2M0Vxyv/n\nw9nPhEI2QxRFrX5Pz/MRUtKII/x8lp6uLkppbjebCbA2oW+gj/6+BeSaRQ6Tk0xXY4pdvRilmEpz\nOTOVCnEYMjU5QRiFSKkQ6frEsXZ9ztL1nYq03ac5o3k2nGhiUNJQRPsyBAdrTM84wS+yHkfqMcr3\nCYgopMKk3NFFFCYM73yIWv4IouDyLSLoQhMS2ZBQ18CAn1Z8ahHhi2kaGmaqIZVKWiEeaeZolwXc\nCNJMJtNqOwnDMG1vMCjlYVSCbm4sKUiMoR5G5IKAUsmFD5cu0YR799NIGhQKJXJpTyZeQD02bjpZ\nHLFn52PomSPpekmCQolqElNXWdS023D5zl56Fs+eiywVS0xVZxgfc9+JqhG9vZ2MjR1GSEmh0Ikv\n3MVPNyKSqEE1SegsFrHpWNKRp8c5/zd/nQUre0hEHSPTVqPxInfecDdrThvklS+7AqkFTx90LUBd\n5X76SsspZjpphFXu2/99AA7Vd+KJ2YuABOD7fircwEYhjdoM2XyJBx98mInJSdeCBrTGfYGrV0hz\nPW7coUBYS4Tl8TQUSyGgN1tkamaCySQhHxSIKk7xLeop8cK1gxwan6IsQzo7nCK3UiHV3CFFayS5\nXB6VGidBMUNsG1iRID1JOB1SSPvkw8QgPOju7aQcZBnZfwCAYqDISEGx2MmCYpGDT7p0ik4SIm0o\nySyd+RI6CWkOdo2Jefjhh9BeJ4sXL2PZcteTnMsrojmmza1YsZ7RXYeo1N25iqUBqmGElYJisYhU\nHqTyLQ41VlsyNiKoTDG968n0OouIbBYR+GTihDjN4eQLJRrTM8QzU1RrFToyiq5SOicg45PzJUGx\nxMDSXhDdAExPT9E3MHueGsCTFpRBNQcCaoGHBybikAnRUqNSg+sFK5YwsKBEiSnOPuMsNqx2RlWl\nMoPyXBGlNqYV+s8IxdDLX8Rdj+5iotbA91SrlTKxFit9crlCKvfSFk9h8eZwlqI4AmuOTkT0JEp6\nSCFb06Za8w1SGaqsaFanAtA/0M+rXnM5Lzj9DG7/0S3ce7drTZqYOEzYCKnVK4yNHaZ/8WIWLVwI\nQK0RUS51MT01xvZ7fsrAQicz+hcuITa/AsoXIAzjVp5Wm2YrkRvA4FqF3GfGGDwVoFSGmVoNpBMS\nAuGUsfQ5PD5Do+asZWHgwPBe7v7pT6lXndAtFBwT6qhO4HdhjHtAQJIKpthozLPwILNBgNUaP/US\nyqUC0lcEhTxrNqxjw4YNFFMhlssFSEna0K/w083rGYNQWaYaIY0kbgmw6ekZJieneOKJJ7ntth+j\nE9sSelqbVPkqLLrV35oJ5vZsHKeJY6qdwSKI61AfbaCmLKVsyuxSE9UalLoLFIIM06kQHR+bprO7\nk8aBA8iOHJWyy31mV/v4/jCZfEiHChA6cfNygUockeiYemxp1CCupxsnypAL5mazOI5bxT2Qeu5O\nxruheNa28pTWWuqxxuJjp+tk/XSmbb5IuauTKjPkymWCnKM7yOTwlEcU1Tk8epCwUSGT5uviWh1M\niK8CenvLmJpTpIdnppiaHuO0C19/YqL9AsVSjrjuKncnxifQOiKfzxPHEfXaEcppPrSYKzAxNoqS\neYyNWz2ZSwdXQC3H9N4QkTcEaUvGT3/wELGv+Z3feCtL9SL8oIt8odPdt0qdbFJAZhTRuGBN2Xkb\nY7V9REmD2RAnCYajRlw+myWyCbt37+KBh5q53uY6H13vJEmopENOgsDH8xRJ5Aa67J9wQzl6ZJmF\nA4uoRDXqlSpxKNCp15Vf0kt/fzfj1QYL+he1qsqVEK1iy9ngywAbC1Rz/CYKL5ulv6+HjmyRydHD\nxGklcmdXByKwdBUKFP2AjjQ/mSQx3eUM3QWfjIgIZxy/53IZFvT0UO7oZWJsFAKf0QmnsDt6+ynk\nfAhK5PO9jB5xkQzLAZLG7F7kzsceQ2DJpD3NI2Mhjz/9NHa6Tk+xg0IuQ3XSnU9pjac8iqUsDZ1Q\ni91aTxyZRvoBvvLwpEcmrRouFItoGYFIqFSrSBtQzjv50YmlOxAcGhlmZHonHYucokgMRJ09c661\njmNsfbo12MJYi5YCa2OMEQhiTlnozrPt5eso51ydx0BfN/W0biSONIGfuL7y5oNtcD3Daxf28soX\nrmb7Y7sZmwlpRM6RkkBiE2jEaF+lVdGQzcLRat1nRhSGYHVL+eZEFis0VtijPbfNYqi0xcnNZZHH\n1TJIJVi7Zg3Lli5mzXoX5fivW25m966d1OtVokZIpVJj9JDb81IFLFiwiIUL+4jjhJ073WyBQ6OH\n01Gol5+Q5nlRvp7nMzi4mH3DrkKsWqthXOwZi0Zb0wqn+F6Wzs5ewlgTzdRbQkIphTFQqYeEYUSS\nFp+MPD1MfWoMHdV50dlnkskW2LN7DwBn/NoZnLJ+JcIzWARGNzvXVSs8MRtyQYCOIrKpN9bd1UGx\no0DDJKxes5JFy5ZwYMR5UEfGp4niOr7vsaBYppgWXC3o72NmpsHDD95PtVFj2VJnGVWqVfbs3o3v\nB7zsZS/lp3f8jDgtStDaIIRCJyb929HzbAquwtbY77QPzSYoSlSOJPi1CZYuWcqOp53QqYYRndkS\nOSy+UJQ7XMHKU3v3UOzMcfrpy7j//gPkEhdKSUaOIPvLKL+EshqSOtI6i9a3mkxsUDpBCoNMIwKe\nlyXvzd1b2KjXj+vfNsa0hps0K+KbA1cQThhgJMSiVQ0ulMbP5SkQEBRLqDQ1YXQCWtGoVxkdG6Xe\nqFGvOoURNepk4yKZIE9tahLfc16+9Hyq47MrsieffIJCrpNas+KdiImJiLARUi53c+TwQVdZDJTL\nXdQbdQ4eGiZfyGJSj6enp4es7YCZGFn3eOAB10ojozy//5a34E2E3P/ID/FzZaLE0XZo5GkW9HTQ\n17+Y4acP0BCOzql8HZ2rzEpznLj+aJkOe/GxYGLuvvtu9u07kBo9vxgVck9uSYt1hEJJiSYdnJbe\nlsZMlUqjQrmjjNg/SmhiN4EJWLpiJZGuMxMnNFSRbMl5kEqHrWrc2eBbQTlfZPGAUyRCWBIZkZUe\nWSl5yRmnoTzHL2PjoxhiujOWxQMdRHXHf5XKFMWcoBhoOnIZcovc7N7xyTEmp4ZZsXIpT+05zFNP\nP8WyFYMA9C8d4IxTT2HX0zWsjsiVnHEUxxmOjI7PvtaVwwwu7qany+2FJ/dE9K1azoH6NE/sq9LT\nq/FS/shmLP2dHQgvojo50WpZsTbts/3/2XuvZ8mOO8/vkyePLV91veu+tw26G44E6Lkct+NnR2Y2\npFCstNoJ6Q/Q/ClShJ4VelKEdnZDOzszwSWHMySHHIIWhrCNBtr39absqeMz9ZBZ1U3F8l48IfSA\nfEADje66WXlOZv7M10gDdBY2EQijCE2dtKhATEhLxVls2h+hX3G1e4XFpQ1Gb37I7fceALB9/RrS\nOx9ECIYu48RHiMBc9NKLjBphVeIK8FyFTs179tH9u9zc3qFei8jzjEbDnCEayMuM49NDBqcn7O4/\nBmBv/5DpNGG4u8dypBlOnzJaHPFMQsZTumleZHSb558hSpVkaYKcUX0qU1pu1Bv4vm+7jM/oRyDm\nGfev0FyrkrzQOK7Hq1/+OgCbO1d5/ec/4cc//B6Hu0+IJ9N5NXRppcXR4QFnp8dsbG3OBZ4m8ZAq\nPr+6+qlcvstLS2xfvs5bb74HQFRzQfgolRuuF3peUhRIJmnOeJKghTvXFcsrRZpm5EqTJikjCwMf\nnBwTiII//oPf5sr1G5yc9bl53fAUF7pdcjET7ZbGdgYjBnAhAxpTpvAdl17Nli9VyWA0Ik5yPOES\nj6c82DWZkpHkK1hfWaTrF/N5h0HIG299wH/4D3+N53t86Utfsp+t+acfvka92+Zf/av/jvFkyN2P\nHpr5FU2KomI0jcFxCH3bVxaSi/wgyrL8FVqI0hU6zQlGLlHpcrh3MBegT0djZJ7hl2NUvcH+oe2N\neT77+32ubl/CKRS+PRuPDgRZ0GNSy4lViVOF1B2zvlEkaHgVURnji4zCllvzymFcXWxikVgu9Fxq\nzpUI6aGFg1Zq3tMxa1dauzGB9MJ5eafMC0ohkYFLWGvgWgeaLEsR2ggElFXFYDgit9maUiVRCbVQ\no9UzYgDSRV9QL/ddODx8yNqKCajOTo6QrsNgeIYS8MGdj+m2TQtkfX2NNE0Yjc548nDAdGoOryxP\nyZOKwPd57623KTMz5z//N/+GTnOR1976JuPpGccnp7ieKRm22h0ePHnAMJ6ys/MC3/7Hvwdg3JtQ\nXz1/w5dKGRMS+9/CcQj9gCdPHpNlKfKcAG8udKAkGm2wGwJmeonTtOKjR7vcei5isdchTXMWu+aS\nncZj0mTEUtOn7pSEPKW/ZJ/AYezS+jqXNi/PxfaLImGaK65vXuUbX/o8G8stdh8bNsNPfvpDsmlF\nI6hgekynZoKwThgwHp4wSTJU4lDZZzPKhohQsryxQHDbp9Xt8PLnjRXcwcEhNy6vs73V4gdv3MZp\nmOpDu96kU2ucO+cvv/o8/cMJuT3kr3/uRZa2LnN8Y5tv/uXf8P79Q7ZXzdlSbwYEtTpK+fjOlDSf\nPQNJpTSxKmjUXRyLJcnyHK09eo0WD6sjpCOJhyYIfFJp1sYTlhou1y+t8vpDU9F4dNBn6/mLKTu+\nW/HlV56n2THf9duvf4CjUhAOUoIrCh7s28w8njKJMxZbNR4e7BFYet5wMmJhqYvveQxOjtjbOwTg\nO//wM3pLPRQOC0srdDevMMkN7Uwr4whbC00FzJm3P/SF/fV4MiZLpvPAcTToU5UlG5ubeL0uqKdW\nqkbc4ymm4dl2iipNy085Ym6SsLS8wh/+0Z9wZWeH7/39d3jnzbcY9s257/oeGxuXGY5G3L59m96i\nqQh0ej2i2vmBzmdo58/GZ+Oz8dn4bHw2PuXxqWS+xycn/N23/5FHj0wfZWm5y6VLa7jCpawKKiHm\n5Hk0TMcxZQXC9RmMZghPE4VkWY7Qmh1rgJz0T9he67Bz5TKFyml3agRzZaIEVWmyAvIKSj2TdlRz\nm7zzxsl4RCQdPNuzmY/OmoQAACAASURBVOZTnhyfsLy4TqfZJZ3E1Cwqob2ySrddo1UPaPnQiszS\nhkHAk70DJnFGWGiePDa9glu3buF6EXmpOBue8t/8t/8V/+f/8X+ZHywdSgcqqQxf2AbBWZJSpOeX\ny+dZ76xCKwRFWhKfFbS8Bo7QlJnpyzRrdaosRUc1JkXJ0qrpwxWFYjCKOTgucAMfnVgQkhsxPo0J\nfEEpoSicuVhF5oDEVDIUkjgx5ejJuDTk8wtGlRcIpZ8aaCuNClxKAXmWk6eZKXWDKX0Jje9ohFBo\nm8m7bkgdqLke7XaHmu35WolbdA5KCQaDmGps+kxB6FFph6LSDOMp48RqcofhPHP+dePqtVscvfbj\nOWl2cWmJo8OHCCGYTAfg5Lz//gMAnjx5TLvdxZUuaSaYOfodHRzSqLVBVTx4/IQXnzcZ19HpEXEm\nODiZ8NFHtxGOR7Nu3t9eu8Pe8SmTOGV96xo3bxlk7nElyHT/3DkHUYiKx0wsCtuNQqTkKcfaAqng\naXY8G46eWXU6KEw/zQqd2z8vOR2l3H24y40rWwSOAcsAvPHTX3D96gbPX1ohHZzxwes/AuDaSy9T\n613ch4xcl7g/nGc2W1vLlKpD0p+yd+8B1cDl+Piu+Y46Zn2pjdIlp/19ImkAed1Oi2awwOnJmREk\nsW2Jlc1NBqMhk2nM1vYOQaM7zzzf/uVtrq9f5ca1Nd79+CMcC4gTpaIZdc+d89JSh2RS8WRg3ttL\nO1u4VKxtLfG1f/4b/O2//xty28sMaw2mcUKaa5JcMUwtUj8IkA4IYdC62rI/pvEE4QZIoSh0iS49\nelbDOk9y+v0+ssjISxds7/T23YdsPnfjwrWuB5p/+ce/h56as/dH795mOs1AeIiqMCAxaUrp/Til\nlAa4+d0332fX9jwbtZCv/bOvsLq0SHdxhUUrRPHWm7s8eHSfF156kbQ/5NXPNZhpg4ymGRJNPTDY\nFluoNOIhF1RHdp88RhUFvj0/JqMhQRjQXehSL+oIx5u3sRzHGMJoa/gz05kuq8p4BwBa6HlxVCvT\nBNi5doOFxVUWOgt897sG5Hiwt89kknDz+RdwXMmTXcPfPx30WegtnTvnT+XyHQ5HPHr4mCwxC/jw\n0T6jyYSdy5uEnmvoRDNBe+GiHWX8P9PUoqSNv+lwOKRRr1GhOTo0/ePr16+wulAny1IKDWfDIant\nfWxtrJNVgsrxUDjzB+hKOQdBnDeihRaqLMjrlv6RB6xvXeb61essLS1weLDH1SVT4++06jTrAb4n\nKMuKRmTLO1oxHJyysb5ikdPmBVhc7LBz6RLK9ZBK8vkXXuR//vP/AYBBOqFSJcl0yjSOefTIUH0e\n3n/A6fH5h6sZ4mkjDlDS4azMcPOc1W4LrJBEu1GnanbZixOEEggLYlvtdbm02eP241MWmprnNw1a\n8mRaIX0FPRiJlAElZTm7+DyUK4izkuNJzHRsN0sm+AR2vlRlaVS/nqJ8qLSmUFAWma1umnchqEW0\nGg2ENKVpZctTjuujhUC4Es8Rc4SlEIKyLMimMWmaEccJM+WPeruF9AKSJKXIs6e90Kl/oeC/dAI2\nNi6xt2fexcubl1hdvcTjR7sMJye023WyiSndBWGdIDAoYyElp7bsfbC3S56kXNrcoNtb5N6jRwCE\ntYBO5zJ7R4eMJim3bt0iszSocTZl77TPlTDkpP+EVstcXkW8ABd4zAopcTyXZObaoyvGx30+/vju\nfK1modK8/27BbmL2TqkST2q80KOqKjIrfFEJI+AxmYw53ntIrxnSaZmy6qXVHttry6wvdikCl/Gp\nCUIffeRx9XMXyB0CL+wsczacoO2h/9ytm8Rxxb0PP+CtN3/K1pqHUkZspdl0qTU0SZLS6vkox0Q6\nB2f7XNq4TnFwgkRx45q5iPb7B8SjhHiYcrR3iB/WGPfNWt+8/iJ1v82b794hr3yGpzM3HskkPf9C\nUCqlu9JkuG/WJ00rQpGgqpwrV7ZY2VgmHpo5V8plmsYkWcmj05hRZj673ZbUApdIStKswtE2anMz\nfCkIpIOWklGm6NbNXuzVJeloSBZKBkk+b8sMRmNu3/nowrWu+3D73Z/w8J65SCZ5TqWg6Uy5ub5A\nVqY8tPt7tddluVXnR+99iCoVwiY4WxsbbK2v4UrB6sISDXvT/Is/+AZ/+w8FL794i8AP2NlYJrO9\n5Z999JCaJxhnGbUgMFrlwLhwSJPzwW39k2PG/RGNmmklpNMpUaNGkVdMkwztaXzPfsZMclcbP/Wn\netDMa8HimX/O/q0sKxrNJn/4p//l3NHqP33zb9jfO2A4jlldX8e3AXs8njAanE+h+1Qu3zDwWOh0\nObZI0nGSkhyOGI/vcnV7k3q9jmN7hIXtSSWpAWVJZxbJxLz501/gORW9XpvLNvON6iHDaUplzder\nCqqZ0osMcR2jBOMKMdfgdaREfgKxf60UW9uXWLK2aL4bkacFL7xwg04rwlc9PDWDwyvQCTotUaVm\nYiUPhydH/MHvfJ0/+P3fRjhi7ooiHYevfOnzeEGEIEenZ7z6kvlOmcoJIo/A99HCRWP7kNWEvd3H\n585ZiV/NWJTWyMhh+foy6ccJg8mEYmyew95oyvb2NXouPD4eEzbNRZG7AQrB0WRM5HvU7HEc5SNW\nPB+CGoHIiNoSixFDeCWVkxuOaGWsv81iczFCm6c637P+i1IKnWUIYYIl1wtwbEWjs7BIGNVJ0hSl\n1RwNnicj454lHCPp+YwebFkWFNMxw36fSgs8z1ysrh9SVhVpai9fiz2ovBwu0Em+d+8jlpaWmNVt\njs/2WV/b4sr159ndu0uRDFi1Zh7DyZR+/4StrUusrW1Rn1VThkMEDienI+7df8iMU/7irZdIk5x6\nu8vVmk+rWWeoLKVJ5YS1Oo1Wl4cP7+F6BivQaLo0uuc7MVUolleWOSxMFSrPc4aDIcPhCBA0GnWm\n1iVKaQt8s9rryn5Pz/NY7japuQ7SgTtPzAUyLhVaCNaWety4vMDVy+usLJnov39yhiscpPRwag0c\ny03ee3iX0XTEH/73f3HuvNe6Ll//2tfIrVb22bjg4OA+gj4bGzVatZTS0rfabZ/h6AjPd+m0axQW\naDk6OKPV6VGvn9KIPB7ee2Sf2yG1Rot0bLAcUgs2rL51WXm8/uYDXn9vj+MYcuviFYR1Ti8wRgs9\nj7KAZs38/EAWiDJDKIV0NUtrPT60ScRkWlITiqzMeTLMmFjPy1jHLLQilPaMxaoy66Zdj44vkb5P\nqR0enw7wHQuQkh4tHLKyJAVaXbOv/UHB3sHR+ZMG6s2IH/zs5wz75j34ra98jVbdw5kccanb5HA4\nYCE15/KXXv4cbQ+2em02vvAK42uGanTjuSu8+OJNdKVohnV8e44vr13hc196hUtb20RBAFVB455N\nLs5G1DxBM3ShgtAe4wk+o+L8q+rk+JjTwxOWegZj4AjjYlbkOZPxGNloEVmjFAPi1QhlbQzF04xY\naYWaVw+f6isgwBMGD9Fotfj9P/4TAKZJzI/+6QeMxhN2Hz8mtEDTza1Nkvz/B4ArUZX82Z/+KalF\nf/3lX/0VTw52GSYxb4+G3Lp1i4alB+VlQVbmKAQSl8y6r9x+9x2m8ZilTo2r21sEgTmkqjLDkxIv\n8KnKioVOh3B1FQDP94nTxMikmQ47ALpUz9gb/vrRCUIW6w0atqwpfZ9mGBHJkuOHH6HzHJRZQuko\nVJkiKHGlR2D/jgBWek1qrSau688RdkVRoKkI66X130xRdmOFFDi5QBUS7TWplMngmlFByz1/xz/F\nOc9XHxxBe3GBbhmR3HtM3QLIjo7OePu9d7h0+RJLCx32Tg3woaoqNjY36bYiKKccn5os7ejshNZa\nj7ZsgoyoCf3UVECU5KoCT1M2FHZvUpZ8orLzTMxhJsSvteH2GSESF9DzS3Y8SZjmxrAizxIKK+RQ\n5CllWaJwfqWclOc5RVGQxwPOjg9xXJ9G00TIlYYknqAqjesHcz6h77n4/gUIS13x+PEjRkNTnhuO\n+uRFxfrGKqsbq5wcOOQWFdr1JEqFxNMJR0fvo21UX5Uljw9HHB4e02773Lpp6A3S8fC9iFZ7mVbk\nMB6N6PdN1UOmFc16k/XNNe7e+xBlEeeVB8PqfIT2xtYlShSjvplzkmcEQUCzXgM00nHmHGBHunOR\njLIs0TYYWew2ub65TMdzWFnsENTN4fmzDx5TaY2ocnypqUdPdZXzyZh2q8vKpctUWcHgwFw6H9+/\nx/0nj86dM8D16zusby7xxi8N1/nx430aTsrGWkEzyAlkxSSzIMNpyUp3gWk64dLaJsruUV/XiEcx\no3HM2vIOP/3JawC89MpLfOFLX+Lv/+77LK6u0uwu88HHJuv7+N4Bu3sVQX0NJx0TW1vFMtd4F52e\nFchC0vVMlaFunA9RWiFdwfrWOm/9/G0ADo4HXFrwqZRmXEJiOf9qauzrpI4IHJ9ZXcJTFWWRcTQ+\nY2F1hUejkru2KjYtQgLfQ3gx0dIyDSvkUavHjG1gdd5o+Q6Lm5eIm+Y82JIJr75wA50tMJ3EeLUu\nl60X8j//yhcpyoxL6+v0egv4NiN0HRfPc3Bw0Yg5Ojifjrl+/RqhX2MSj6lKxe983pTv17od4smA\ndk1ydDLi8fExAEFUI46zc+c8Go04Oz01FzrQbrXQWpFOp0wmY6puTr1mqkJBEJjgHo1wnlrGVkpR\nlYYeJcSzUpWzHp5J3PI05eFdU0FYXl7iG7/5G4xHYx4+fMCjRyYQnsYTltY2zp3zp3L5OrqiUQ95\n541fAPAn3/g6D3cf8Z0f/gAvauE4cm43mGcZlVb4QYBQ8Oj+fQAG/VPqUchSt81Cu4HWM39IDykF\nqiypJFSqII1L+1ke2hG4roMWzjyLfvaQP29Enkc+nnBsFXIq4bG1ucpi5OAlfVztoKV5oJ60mqFU\nCF3i2kNLui4iySn1mKJ4ikIuy5JpOka4iqqocHRBaVWhdFFSZDm5rHH7KOO9j8xB9cLVBVYWI658\n4dfPeWbR+CuAVUeRMiH0IlpLq5Qjs4HXdxx2D0/54NEeS0srLC+YTVBpydLSOtNkShkXfHRokbnC\nwwWWQ4HrBJRSznWbSxRFKcGdErYUqm77rdQRFwg/gLnwi9KofoHJroQwvRlVKXSpGNr+/8nREWEt\nIs0ypnH8lDamFWWlSNPcGAbYZ5BlGZWq0NmY0ekJUoEXWUGGbEReaWr1Fp12GzlDH1TlvF/560Yc\nx2RJOld72ly/TFmW5EWfy5cvs7SxwqOPHwDw/i9/QVUmnJwMOdgf0LIiE6eDAiVcmg1YXlphZ9so\nG33w4bssLW2wsLBMd2EDP6xz545BrZZZn8+/eoNKpVRFyoL105W1HM35VYawvUQaDwnszTFVBb1W\nkxefu8a773/EZDTBmVHDhMmitNa4EjoN80x/44uvcn1zCZmO8F1IbEXi4eNDpoUmKRQfPDnEcSVr\nXZP51qI6tcUlOitbiEqwuHoZgCJo8tqPXzt3zgCajCob0D804hPF8JTr11epeYpQSqpC0aiZn+VI\njyBwmY6GNLwavQVDT+o11vjJ678ky3KStGTzslnrovJBRiwsL1FoyY9//h6/eNsIcIxijSNXaVqf\n7qZFTuf5lOACP99h/wRddaj7tl0iCyprNSddwa3nrnD7yjYARw8e4DsRuaqYZDnYs0pLjzhTnDkp\noGlYHWwvVIzihLDZ5Bu/+Q38xTt86z99B4BsVNA4m4IruHJ9G31kqhz1yJlbPJ43rm6t0ynbeDdN\nEDYdjri/d0Q9cKByaXc71GwAf9o/xS9K+nuPqUcuRWk5u45PWRj98bIo51aXo/GQJM+plGI8mTBN\np6SWNjoYjTjrD0iKnHGW8qBvEi/f8ehd0E556aWX8YRkNLDVoTyl1og4PjxgNB6Tb6QsLJrzzZFP\nlQKNIYX5DEeBK59RGpz/GZiJDlRFwWvf+RZ7Nni8/uLLHPeH3L7zJlEUUre8/ie7e6Tl+UnHZ2jn\nz8Zn47Px2fhsfDY+5fGpZL6+6/Dv//L/5h++/UMAvvr8Lf6Xv/gLbj1/izfu3GGapmTWsF5Ih9AP\nqVQFlaZnnUeq9TXu3blD7bKRbpz184psSpJlSGn0aqUQuL7NuqSHFoJSK4qqpKhMVCYc51f6gb9u\niMBlOBlRZdZ71fGJA4FMN1HpkCTJKG105DqQpWPKMqOswBMmUvN9D7dl3JjyOH8qHwQoXaKEIk81\nQivKwkSlpQItJHuTPn/1s/s82DfR3OOHXX7rN185d85KGdK6nhefBVqmiMYJg70pm40tPGEiwEGS\n4rV7+G5AqSpaVh1mYeUS9UaPpZUNHj5KGdpscKoL1nurFDJDVakp76pZyUagtEckVmj5l+lGxlGo\n66+S9M8vhYIpw0sp5zxfx3FQCCrhopVGlQWHexZJeHyI6xmOntLQ6hpAWFQ3/Og8zymybK6rnKWp\nUa8ppyhVUa+10FY5LS8UrhdS6yzTWlygTEyWX6YxZX4+YGL38WMCz5vrDcejIVeubjNJznjjR7/g\n8o3rbO2Ydaiqz/Gzf/weeRJz7eY2Spvscv/4HssrTV566Spb69u8/IrhgR9/55vcfv+nLK9dYnm5\nwah/yoH1wN28tM6o/4j7H+/RH5/yaNf8/M2XlqjVz0fg5klC3D+bf09fGmzeb//GP+Px3iE/f/v9\nueJQkWdGnm8G+7S/7h+fcXljmas7V6DKCNcNVqHRXSetXJzIR4iSays91izgKisK/KUVKi2RgGcz\np+c/90Wi1sVo58HgBE8mqNT0l7dX61AmtBa6DM/OKFUx505vXbrMWneZMi9J44wnE/Pe3L+/y2A4\nYvPSJfygxumZwU8cDVLw36HbXeDJfp8He/05f/9kmFGLNGp3l8AXLC2YeS8shmyunD/vLM+ZJn1a\nkbUnrFJcLzDSqUDo1/niK8bx6MfDMxqtiP5wxAs3r8z/zODkjDybMiohSybUbbbtSpe1netsP/8c\nrV6Pm1e3ea1uzsrD4YR7+wNWF2q0FhZZXTMVoNFoyNHpyYVr/eKlDnUnpNU2726By3g8YBqPSZKU\nOJ0wsUjo/uF9nosH6Ef3eOdwj77FUozTlHg6ZZqkTNOUJDdnQFIaa9A8LxnFJSXlPMMsSkEtCul0\nGmS6IrUw6EoGDL3zM99Go0m32+Vgz7RAxpOKldUl9nYnoCFf6j11p9NPub3P6uULi3DWVtdaPIO+\nmvV9iyxh98P3yWwL7O6dj/l33/w7Hj9+yJ/92X/BDLrdH084PDo9d86fyuWrypJer4uYSX+5Hv/P\nv/23fPn3f48/+aM/5q//9m+YqeLNzKIlBgD9/M3rAFRXd3jpxnV6DZc4HoK9EHzfxXMlURCCpaUo\n64nqaEFeFijHGDu4z0oXyvNLRgBePcIXDunY/L3pOCEeTTi894CoHKGLbC7B5nmSXClK6UPYJtMW\nmCEkQZaQ908pphnCop0d1yFXhRWR8IzgvVWfEULgeA0ePdyllA3qViEnxuXNO4fnr7V49uI1wwG8\noMC9XPDowW3IzNyS0mVwfEgtqOgsLhpRD+Ds0V32RxO8yKHy64wz0/tZvbbC4uVFpvou6BJRBpCb\n8q1X1WgHNboLyzSCBTyLSs0nCYeHH1y41jMDhafgLKNDo5A4QpMmI07t5bP/5BGtusfC0gL1dne+\nsWPhIKWP6zgIz2NqA6x0OjUa0SqlUW/QbLbna1QpjRf6NHqL9JbXOTs0Jbo0Teco3l83JpMYwtq8\n0JtlU959t8+tW9doe3UOH+1xPDHlqcXlda6/+CI/+/EP8PyAJDHvaK/rc+P6Gle3N4lCn8SaJ7Q7\nNaqqQ6ZiDk8+Yvfjj7m0ZYKMVsOhSE/RZckg7jMTiGqdNRj1zxc0OXjwIZP+CUViepeeFBSF5tq1\na/xPf/4/cuPnb/DWL015++joiNFoRJIkZGXF4Zm5EP7x56/T7Da5dHWH7Y1lHNvPe/7lBiU+ldBo\nXRFoRVDYXp3jUHgRSlU4zyiZacdhe3v73DkDXLu6Qzw64PKWoawIVZEpyTgPqLwOUc3l7MT02372\nzpu8tPU8tajG7Q/v8GTP9A6vXLnBl7/yRX74Tz9i0lw05vJAWmoWJgVvv/dz6u010lwyGJt3qlKC\nJDkl9AXbaxvcuGZK2O12g7ZF1v66kSuPWiOk3TIXx2SUMpgOybMM13XJ0gzPUgFcKWg3GwgBv/k7\nv0u7bS7So70DDvYPmMQpRZ7TaZn9trW+xtrOCoQuWVGy2GmzaEUxnpxMkGGNsF5Dq5TNTYOBGZyc\ncnhwfOFa/91Pf0bXifm9L/8OAPVGl06tRrseUlbK9EDtWer2T/B/8j38/T0Wb32OwNJrGumE8SRg\nOEnIRwMyW47OpUMlA1Il6U8KalFIzfqSqrREa0EYBpRFgSNnVq6StDq/SPujH/yQPEsYxyYAk45k\nOBgzHg9ZXFqgVg/mMsGlgjRJyYoSqdVcNcz3Q1wtyKsCpMCbmeYISSkVuipxS0XDcRgOzMV6Z/eY\n2x98iOtLo5xog4/9o1PCCzyIP5XLN01zokYTYaHedx48xHuyy7de+zH/8l//a/7+W9/mxk0D+2/3\nukSNOq7v4QlN3VJ24mJMr9ummPZxHGfuNZmmGUKDqhKMOliJpe8ROIaRKB2XUlXzrNNxHKR78eWr\ntYMbhFg5VSrHR0uYDFNct0To3OjbAmXhkjgBZ5Xg7GzC8chEmGk84sZane1mgHRCk2oAbiPEdcCV\nAcIL0L5v5F0wHLNJ5uB3Ur74hWv8+BemV56pksP++YCrqirRml/RSBZCooRG9go6kUdxbBWJyjEr\nukV8OsbxG3z+5ecA+Mlrr3N68JCoUUMIyYrZ01zZqNN0IkbxKr4TEog2zZbZbM2wS5HmFEXMwfAx\nZ8Nj+3xGjIf7F671s7rOAEVRAhXSqUA6aCEI6ibrcIIaSoArPRa6XdMjA+7tHlIqB1dBkedk9vfz\nosRxHMrcQVUCP6zAUjaU7yBCDzcKwfUpLVL+5HSEU53fH1Ma4jRFWbxCkRe4Liz0Fhn3h4yKCest\nkxUenDwgakvaC+ucnozm6P5uq0O73mQ6HDE4PWIcm0DHrXm0vC61VoO9w/s8fHiHb/zuH5nvIxzG\n4xO069Grrc4PqfXeOvkFCO2z/cdUZT4/WBzhIBwfpOLq9iV2ti/zx7//ewCc9vvs7+/z8ccf8+57\n788BMJWueP2d9+h022xf/a+JLDBNVR6e8KhEieN4OJVAzJ6pcEBLpCMMVcmbUdRckvTiysje7i79\n04ecHprM5mtf/gJv3L7P2d4pX3r1C7iuxrG0nUIPGUxytAzoraxyYMFlUavGYHSGIzW7+3s8emKQ\nv63FDfqjhJN+RqYznuweoaqZKbqm5lV87dUX+fyta/jyKQc6DM4/Ph23iR8pKkvtQwnyLKMWRbie\nRxzHTGdZh+dz9+Ee3YUWfgidngnEu+1tXnjxGlo4JFlGaUGEge9Toci0IisKms2Q5VUTnJUfPmCa\nK477Ez788A5XXjRUrsVui079Yorl9x+PqQ0fkz/5dwC8tLlMtLyCV2ugXYkMA+OaBPiqYLef8HAY\ns+IJFpbteVC0WcgLJpOMhcmE/b45D48GpwzSKZPpENdTdDo+rnU9k1RIAU5ZojKFw4xC6D19j37N\nGI6GhKHP6oYBOelCk6QZG5tbhDWfaRLzzptvAbC5c41aI6IqS4SqUKU5Jx7ffchkNODq9StEUlJY\nzv9wNCUdDZhMx4yzjCKOCWzmvNqocWtrlQdHh5wdn9Cum55vWDo0o/Pn/OmgnaVPq9NkyWqp3nv/\njolcy4L//X/93/BqIU+emE3VW1hAV+aFEhIGZ+ah5UlGGseoKsFxoJw39jHuFRi0muM4hLYMqKsK\nB2NV5bnuPCOulLoAlmLGaDLh5PiEmiWvN7sdGlFIe3UNHR8iZTi38fLCgOEo586DJ5RBnamN1PYP\nT5DFmK/92e/hug6FBfHkKJK0YJo6xGlB/2TM0ak5PAbjEcf9mFp3na0rnTlCuioV+gKBcdfKdYqn\nAD3QisqBioIghMUtq0vaCsgPPe68qzkl46sW/PG7UY3Xv/t9ismUoBFBzWyObDyEwQZbvc8bZyoK\nEss9vb3/IafDI/AzvJpCWqtB6ZZ0PoGgyUzqbdYOEEJYGbgKrQzYwbN8wFwJdJqRJhlZEtNuW8m/\nVoOzQWwMx7XhHgNEymjTVn7ANJ5wNhxQtyL0jnRRSMaTMVL4OBboEoYRZXI+wlI4HhXVUyemNKfZ\najIaDsmqiskkY2SR4kLlLPaWKLVkOOjj2uBvfXmBmucyGvQRkUOAtVQ87KMrReR51P0aiyurHJ8Z\n4F9CTpXnjAc5ruvQaJp18UOfxbB37py1NkHojBtdKW2t2BxcXVEUOauLJuu6tLHKSzev842vfJGz\nfp/9M/NdhuMRke+ytbFmHGhmB6ProSqQlGitcFwf7DNQSiErgSo1QjBHomul5kjY88bDB4+IJ0dc\n29kGTIsBWbC42mBtfZnvfPsfOLXZyM6VHZaWLyHdiu5im/aKoZ/4QcRoPGZldYmV5QgnMAfl/umE\ng5MBYWOBBw/3OTw4mQPSvMDl+Z1Vbl3ewM2T+bkRhHVcfX4AXyhjGYoy71Gz1abRbVBVijzPabgu\nrnVgu/58wc9/8GO8pCDNUuLEVKHyaUq9XqNWD8mKEWlqLVZlAylchHbQWlGKis1La/Z7SnIt2D8a\nMc0/pmtRt64raNt35byx2A7ppx7fOzOl9zcGx3ScD1iSkrbvE4UhUTBzc4uI1tf5pVRcvXOXmeBm\n4UjKosLRJc0ix5MmyOg1HRLf47KocyRTsnxCMrLVmrJE+S7JBPpZQOmYAESi4QJq6NaVbYQQ873o\nVJo8y7h6/Tp+GHA2OOb+hwah3Gt1qNVWGR3tkyXxvHL2xvd/zHTvmPzVz+O44ikIuCwoswTlKCai\nguXe3DrQPZ3wQMBVpQAAHzdJREFUte3r7PiS2pP7SM+s7yuNJiI7nwf+qVy+3YVFksxl87LhgH34\n/l0Cz8PzPMLAZTQezb1ky7KEXFDkKUjNro22VxaXUKqk0iaTmZWnkeDIgDIvjVmDkE91GlSF40oq\npdBlNYeUa60Rn6DnO5iMEcpYYgGUjsNCs46qN2m2Q1ypcexl6NciDg/eZpxX/NbvfgPf9rS+881v\nErgV+9OCwWTEobWfOz7rc3w8ZH+3z8lgSKH13P8xKzUVgp1rsD+YMhlbj1nXp/TOz2wcAP2M7682\nokACgcRHakllS4F1ryDcdrl19TLjpMYbB0bUv3aW0G4H5E5KKQRxaqsPe1PWthLq4ZjheEQmR/Rz\n0zdL5Ri55uL5nikf22cgC4nvXCzmPkOfP2ukYY2NqKqK6XRKal1rpONQTs3FOxn157ShrdUlBIIi\nU6YNYPtmldJGRMQR+JVjEdLm8Gg1IsCjyDJyb0pof77vuwxOzqdl1Oo18jKZW7mhMsoyQ+mQLC1o\ntZtz9xNZKaQOcV2fUZwjpdnYl9aaOCQstiKk55DZC/9od5/VtSVG/RGBFyLcGu9/aE0XIseUGRfW\nGadDUpvhP3q0z+rC4rlzdqSPdkDLmW9wia4KpIOhUrhizltWZYYnXRq1kFZjg23Lra+qClcKdFWa\nSov9bKMLraAq8QMfJTSpzSqKoiD0GvjSp7DiJ/YvofTFobDnSpZXFljdNBdMgeDq9ZsoR1DmKZ1G\ng4mlfHVqHT748EOSPEE5JTs2qLx6dYdXX/0CK4v7HO6foYR5b+7tvc37b98mzXIO90dI4c0DkOeu\nrvHC1ct4uoJC41vhfF8GlNX5Z0joaUINQtvepc7RqiIvcsqyxPcCwobZW+vrHVrtEEdLnNLlseW+\nOg4U7RaHhyVVpWhZT2VROUyzhDCq4QuBKx3W1wynvNcIef7VF4nyhOL0hMhiYIqaord+cX89ENBr\n1lEWYTxMS3JPMZIVOitID/bQicWnjGLIFeO84pfvPean7X8CIKqFRFEErja9VnsgpGXGNK8MS8R1\nCKJg3gLyIx/t1slKiZDB3J9Za3Wh/Xqt0WA6TRAzxcGOsSUcJykqnuL5AaVFek8O9ql3I975+c94\ntP9kJtDGgnbYunmdwoE8z8iVFVTRFaoocZICbzwhjUeMhuZMHp4MqNKcpUITH8UcV6YtqIKAwj+/\nLfGpXL6VFlQCmh3rN9msE3keusqZxkNWVxbZWJ25nKQ4RU6gKyb9Ee++ZXhwna9/HelqfOmS55Vx\n+sBI3aELI6LhGH5nYg+jMPCQWqCFNF6OM9k8pU0Z+oKRZQWB6zHbY+PxmAdPNL5WFPGQUf+M1Moo\nKkeyd2ScLQql6DXMwvtByHvvv8P3fvQzBtPp/KCplAThkGYZlRKEtcb8xakcH9Dce7RLeHJAYLOU\nwHXRF8xbCkEp1VycxdECR7k4VYvhnoJT2LLi8BlniG5FLAY8ORxw/2NT8iwPRlxrNWjJhNGkop+b\nbKoKKu4+vsNJorh3dMzCepulDfM9Q69BKStybSD7s0BHOM5c3u688f8V2Sgt7UhLkw2X5dNSZb1R\nJ85GJMmUeOQxbZl5txaXaNZCjqcjI44+t1W07GehTFZQeEztJRcFNRr1Oi4OeRrPA60sS0wGfc64\ncfMyb77xE8ZjEzisba0yGhyRpRln8Qg/9Vi9bA7EoK7RRcXaepvRqMfCYt3Ouc5xluLlBY4q8Gwp\nc5rliEAiXEGWxgxGKaUl7V/dWaMsE4qqj1MIGm0T6BVZzuHZ+ZgAzwsQ1izdrI0wYhpaIBxpWjIz\nPIQQaFXiOoYeM+M7SoEpfypl3I2qWf98ZgvpGxF7VSHUDJDo4kmJg+BXbXDFhR7VAC+/fItCn3H3\nkaEavfT5r7LZ3aTV7DAdHLOx1mNp0axDPI3JiylnowleFCEs+LFKKzxZo9tZ5GS/z/vvvg/AJNWk\nyuPu3iN04XNpcZHf/boBvq00KwJXoZRDWpQkNhsqi5Ts2SDiPzNUPqUQgiwxQaMXuuBAWZRG7Q6X\nYmZzF9QJGx2mwwmFhs7i0uwRoKmgLPA9ibImJXFcoHEo8hxHCARP++itZp3Pfe4mjGPe/cExnq0+\nLC4tzqVyzxtxIfDdOhrLRdcp4yynVJ5xl1tYwHfMJR7kBdPjAfnxCfsSMsf8LF8L3DRHaUVVQWLP\nyf2TY5K8oNtdoNPtcKWzRM0C0hwEd+/vkSpFbSWag5pk6OD45+/FB/fuMxrGeLa1ubG5whtvvM7x\n8Rl5lvP1b3yV3shcvuP9A+obXc4OD0nznMDuufxswORkhEhy9GhCYTnReTGmikumcUkmBJkXUNjA\nJG80EcsNnKhOUItoW+9z6btI9QkSpc/GZ+Oz8dn4bHw2Phuf3vhUMt+z0zMqt4ayvY/LmyuM+wPK\nUrD9wg22tzefGqVXOU5pIN+DaYyygKYknrKwUMcX4ClJaaNtXSmjEOVIHMdFWwACGDx03fVR1hNX\n29JHVVXzCOm8URWKaZrgzuoSUlMVOY/uP2QyGFIV2bwH4/gBQsL6zg737t7jo9smQp+MJhweDhgl\nFVo28Gw25eIgXUGzpXCkh+N4uBYpJsMI13ORPoShR2h7Yq4WlBcAatoHK2RBQaFnfRSHPIFCNckO\nFcFRRWzt3No7KywtRKy5Et/PGQnz+0felFNZ0mlGZHHBKDW/ryrF4fGA7naT5bUI5WTEs/CtAmzW\nqITCETPAF3jeJ5OXFFbGcPbfWil0ZWgBWmvqDVM56fV6VNMhWTZmNB4TWREUGYak6RStjdTozCN6\nJhmttERJj1q9QaJNFBwnOa2WIJtOifPpHIg0jccGzXzOWFtd5HG7TlVZjIHOqTcaRt3mUkQ+1OS2\nn5VNMzzfodFr88pXLxNYMEnUrOE1muRxjBrGrNi+XVzcp77UJPAbpIOYMGszs6PuNNoUpeTkeMJS\n7xrRTL7QFwhxgX2j1uZ9thBpU2kQ5EVJEAS4nkdp2xLSKl4pu4Az0ItwBEIJHM8zgjW2vK91aZGj\nArTAky4zoehKV6iyonympw+WYvYJer7xNOXDj99hPDFtm1brYxy3xsrSMh4ljx/dJbF7ce/giK3t\nbW6tXmXv8JTf+Po3ABieHfHj137ErRu36HbbjMa2hz6tGPT7TCcJ3VbAzvYKtchqzaPx/Ii0qJBI\ntK1CJemv0gb/c+Nvf/AWAcUc+OdHAVpXZs2FoBUG+PYzSiGZVqbPGJDTnKE8qZDSIcYlSXO0zZcE\nAq1NhShNU7LMp28FJsJawPrqAlOhcUOfwrZ0Wq0GzoWyXCBcSZE9NaKRXkBeKiZ5jutqarUQz2IW\nFpcCwobLsZPjLy1Q69mqWpaTTmKSaUqaVfNWRrMe4LmSKkk4K1JEmRParDbOcvaPT9FeSJALag1T\n+vcV4Jwv1PPX//E/kueanm27fPGrr/DL995Buj6tZov+YETT9mBvv/Mu04WQ45NTcs8jiqwH7/Ep\n+r17hEmGo6q5mljitejffJ7g2iUafoSvi/n9UfMbZFIQDE8pB6cMdg1bYjwcMJqcD479VC7fIi/Z\n398ltbzLMJAsXl5nc2sdP5DEcTw3HXeloZY4aBa6TdaWDW+xHrq4wpSdpQ+ynLkcWe4u4HoeSjiG\nIwzMhBajyFAcKivDpxEI5xM47aQFZZ6DXegg9BBI4qxkkkPgNZA126sWEqELnjx5wvJiD2VLJsOT\nPvVah7Dh4HgSKZ9KdjtSEdQMSKUqFb4tZbiRh+P7hPU6rh8wsZuqKtN5CenXjf3vTmitNljfMdq0\ntU7HcCyFIHge9v0PSC1w5tL2l6hqkrsP7vH4yYTSwvR7rZC1jTbtdsTR5BRpzciLrKR/VHG8X9Js\nK3I3ZcqMC+siKx+JUYypZlQS4XwS62TyPP+Vkudc1N+CsJRSuPb/NRoNxvUWo2xKWigODg0u4OOH\nuwzilI2tHbrd7vxSnH2eQpMXBUIk80shT3OS6RTt5GTJkNKCL1SlyPOLfIglZVbR65ly53A8Iqq1\nqEURouZQX40oDsxnpLnDKJ5w9YUtDg4f4c6ct0rY7Paoum2SdkqrZ0r8L3+hCSLh9GSPeqtHvaqT\nxWb/eHgIt8bOzia91hp7j9+3T6DGYu/8nm9ZVr9C6ZrxHKWUzyDObanYHrDPciLB9ONnbYJnudlC\nGBnBmffys16pM+11ExQ9VXpzzvEPfnYcHZ8RhHXjxQ2cHB3S6rT51rf+kaVOh8lwgmd7bGvra1y7\nuk08Lfj+d/+Jex+ZQLjKc6ZxTJJMWF1b4ctffhWA5sMzZHRMt91m5/Iql1d6LCzYdooSaOEgVImq\nCppNcyFM05RJfD4a/q3bj3BUQalnLRgBmBaKEAJX6Xn7TAvNTtPhpbUeo1FKoczB3ek0iaKAKPRp\nNFwjf4gBjeaFuchnz+3IepwvLC1QDz2IXILoKd7C8zyyiwzBgem0QOVPn12uckAgfR9cTeZIQntW\nTSpFph2KZpvC8xlbjrhSIMMamXBIRMbUgl2VVghdUGRWxe/sbN4iq5Q5R3UlCEuBymZYmxJ5QVBZ\n6QqkN6e9ecKhVW/Q6nVYXF0mrUrG9kx+sPcQ/XAN4fsMh2MSayjTnCZEZWYMHYSgmtERRUl49QpX\nXr7G9LVfkBwfkA/NmXw2ThhIQTEZUJY5hU06qsDnRJ4fnH06PN8KxoMR6yum/7VaD6kFLr7v0u+f\n4VDNdTQdAVqVaF2x2Gny1S8amLzSFXk2RUcNtFZI+RSYU1aF1YI2L5ijLJhEK9I0NYeNo+fUXqU1\nVXk+khXAE+BJ9yk4qygZxRNUUVKPIjwpUTNDCC3wHI84nvDBO2/TqD8VuF9cWsJxNUpnuLb/GUU+\njusgHJhOUzzXn8tfBpFHicYJHfwoAnshlGnJgtXK/XXj8heusrnRoWt7iqUnoFCEZUXqZmixyMAE\nZzw6fsIPv7/HO+8eUQ+brKxaA4mgYFRN0SohqxeUjnk5deWSTCQHT3LkikPZ0qT+zIrOIag0Pg4O\net5TrMqKUXFxf312YM8OeN83Oti6UlaH9ekQwsENIoQXkhYp+0dGROGkP6S7tEKtFhkj65m3g73U\nteOQ5QUTOULbyklRlmRFhicLdFWQzgwx+mNUdT6aFe1y5cpNtDSHcNQLoJQMTnMYFrj1iNjSlWrN\nFs16m/27B1Qoauvm+YjE5c57d9l57iqttWUmY6u5nMY0ohDP8XGAmic5tZWJeDolkD6+r0nGByzZ\nALVQJZN4eP6UtZ4f/rO1mf06qzx4M6Sv61JW5Vx8fnZJzn51XWP9NtOCnl3kQoi5nvazlDfP8wy1\nKEnmB7sRs7/48n28e0i7I3nllS8DcGXnCu++90sa9RApBUEYcu05Y62Y5YppPOWjO/e4ur2NZ/eo\nBuq1Jq1mGz+I2NgwALIPHhyxvbnG5c1tnru2wuRsl3rNBJzZqGQ4GiF9D4TDwIIfcRz84HwgYdPT\nRIFPYKVMz0ZTsizHdb15kDMDWbqqpObB4uIif/uDX4LFf3Q6TVqtJovtgHarQb1en699GHrUIp9u\nt8c0HvPECkwsr67hOiXCddi+eYvFpWU7ZYH/CTLfUZyiy2J+0ZdUIBQChasgF5Brs+/rUuAoSRIE\nJvC2F6Z0HDwp0VHN0M5mAL+xQ6UytKuROsQJntKIRKlRpUI6AUEY4oemEqiFpMjPr/hVpUI74qkx\nS17hKMHe7h4ngz47V69QWOOYoypFvP8hQmvGaUZqkeUL0zEdaapkldCkkXkGWlcU7/2SPGrQ/8Xr\njPsH6KZ59mf1gCpoUHY38TtNIhuwpi40Ljj2PpXLd7HXY5oWZFaBpl7zSJMRQntIpyJX5bxEKFwH\nVVUUZYYK3DlWR2mF8CR5muC67nzDuo5RKRI4hr4g5dMyljJlrclkQhQFuM7TsvMn4Rottuo4pVHH\nAhCehyuhFXiEgY8QgswCIFzXw5EOvoSFdo3I8r2GcYaqFK2aBzqcBxlhPQIBSZoQRiFhVJ8DwpxK\ngYRSOuCouUdlr9NitXs+gu7qF65STI55cvgAMEorJ6djNBG516JKfUYHpsyifU3QucH1V24iihyN\nAS75CwkL12rUGoq8GDB4bC5fN+iQlwXDk4rmro/v15hio3+3QJUCjYuuzDMFU37SF2aQoKoCIZ4B\nZ+nKgtOEcRlBM4c8CoHne/hhyMlhnzgxn99qdVheXKbZMtqzM4oWQpiL3JG4nkLgzvnZSZKgKInj\nMZPh2RwEURQV0+n5/NOT049ZWGzQXt4B4N7999BCsbTWY3R2gpqWtLrm4B1PY3pRm3h8ghd4DCxa\nMhQRg+GQwXvvcP3/be9Meuw6zjP81HCmO3bfHkhRJGVZ1ixZVuwggRFkk2wTOCvnF+T3JRv/gKwC\nBwgUwLFoayapVo/s6Q5nqCGLqntIImC3FsFd1bPhom+Dp+85p76qb3jfD9+miylKrQSL5TUqq5hu\n7/Ds7BAZyyZKWzpnMK5BqoKd3Qfhmn3Xjza9iheDLDz3f35R5ET2Y0iu/x3ZG1zEbucs61+hl2fK\ng2661vr/zG6vT8N5nvf+wUopslsMLACu5w37d/dRItrP/cdnNN2STz/9a7YnW3z7zVNmuyGYFoMJ\n9eIKZwqm0xn/+fv/BuBf/+13/Paf/5G/+MUv+eqrJ5zFyYOt0QiZb/HZZ48Q9Qm/+uTtXpDBZoLt\n2TZZljNfLnsjkXIwoL1lU/nb3/wd0teU0Wzg5KLm8dNDFvM5Osu4c2cHGdOx9dkl9ekBW/f2+Xm1\nzWUseQjg5PSELx+vMNYxjqUXISXKG3amI9556028acP7A3zw4Zt4Z2hVwXu//hsKwkHj6PB7quGP\nKLeJoIXv5DqjoUCGjmUvwobVxnXPaciiEp21z5u+hDV4qcLvZvQe63VjMbkPdqAIXCb763adQzmP\n1gW+VDCI5TZVYG7ubcNpSZYJdB6drbbGvPXmGyyvr7gz3eKN3Tt0MSM62tulqIaUVclYq35stTg5\npTu5wNJhvcdHF6TKS5ZPvuH8w19R/vpjnn3WcP/nQWlwW5Z8axzz5RwrHevps7pZcn2L9/pGgu/Z\nyRHYjkEe01nW4XRwJMqVoqHra0HOhFy7lhLTteSxE61rGnRW0K5WiBfSVutUV2csXdtgHL1t39qC\nrm1bnDdMi3UdRfwoV6PRoEJ70c+BySynyDRFrshVmKdt4w0VUuKdo6pK3nvnTXbuBGm204s511cX\n3Nuf0TarPu3TWkPXdUy3dyjKEmNsn1atsiBUcLlcoIoJW+NwsqloeHj35lGB05On/PnrP/P0KATS\n7540nJ81/OKDd7l7d4TvLCru2Pf39pjtTiiLnFwPuYhp54PrA+rzJYYLtscNFzrK4RUaM3Qsliua\nJx3lbo6IogNGWKz1tMJj8L3qkrSSTN8+aiQIs9frbm5nu6Bw5SXGWUys+wIIJRiNBjg75frqKrzg\nwKAoqYoBSuYUxeClACOlBCVwGKpqiJ2G9O58fo30jrzKaRc5q0UcZxIaf0uqy9QLfCZ6j9UHdx/y\nxed/oBoM2L0/xbWyfwbVRFBNHPmypb3oGMTMyOnpOW+9/x5ee04ODvtZ+J3tGZcnh9SrJbXt6Lxh\nfz8oFe3u7dI0DVVZ4YHDkyAWsbu1S1XcbCmY51nY2fepZPq0c58+XqcMfLDjfPGkDOHU5WOWQir1\n3Iw8dqSvMw0vBl4pJcaY/jNlnJ13MW16G3Xr+f7pMzTh3Tat4bunP3D6rOY3//BLdP4af/oyjL1t\n7YzpFuAZYlrF0+9DWeKTT/+KxcLxp0ePuZyvGA3Du/TWT7ZwaO7t3WOQOzI6bExRznZHeOfRWrO7\nu9NPUnTGcD2/WX70Z/fv0NWX+PjdzUYF7z2YMp/PWa1WTCZbNE14Uc5zwezd15jNJjx4d9Jneup6\nRdsZLhaGs4urPt2fZRnXV3MkmrOTc44OjnnwMKgBVuMZP5xeI/WIsqw4Oogd8FlBe4tSFIBSBda2\nrM8wXinQDucMzhqs9b0lq2ss0jmUC/7ZRb6e/80QmUIgkUKgYxZJ6AKXOZTWIRulJSL+R8IRnLMc\nGEVvnlMqgahu3qD9/ccfkTlY61oMv3vC28bjRE5+do09fsZ5vA+v/ewNfJZRdy12uaKJTmCruqaQ\nilZJcJ5ufX+FYNAazj7/nNf/8h20l6gu/OwNPeD3jx7x6Nkxo0yzHy09p6MBXx7cPHmQup0TiUQi\nkdgwGzn5KunIlGW9d9FKYtugduOiAER/4vGOMs+oly1N63CxzpVlOU3dIYTomzfgxZOviyk0+lNt\n6H72FEWBdYZV1HLN8uxWpSiA8XQKnaGLOzCV50zGY4S3aAnWWQaxhtJ1XdjVa83e/j7vf/QxAGeX\n1zz+5ium4yFCyf5U7mRU/RGSsgjKS/N59H7dnoaT72LBsvHMJqGJpnArdsY3z+mZ1ZL92U6/q//p\nwxEH3z7lo5+UvPmg5PzkjDoOmw/KI0b6islgi665ZBhTxeMq45uzhkvXMJlkvH4v1j6Ol+zPQOd7\nPHMLXG0o2ucdxdbbKFfIc2m4TP8oBaOgskQvfm6tACTWBwUmH/+FcM+LqmQqpjR1zcXa59YL8ryI\n37F46fNSKow3fbf4WgmnKHPa1RIpJVVVUcdU89qO8CaeXR6TZwZ5Gk4Bu7OHbE93WHRLWEIu87W4\nEaP9KdkwJxcDhplieytkM7ZmW0z3Z2yPRqwGFWex+WM5b8mLCUJ4yqrkg/c/wsTT6mpxjvCepq5R\nWvV6totmwe7oZmMFh4gn1uc131wIhFjX3T352nRcBJnPXIe58xc1w50QKKWRWiNjfV/6oKGupML5\n2FS4VmezFmNd7DoPn4O1iP3tNaDdvTuUmSDT0T7RGR4++BhV5PzPHw9ZLGueHIZU/tdP5mHOvjMc\nH/7ABx98AkA5niFdx+PvjpmvOpSODY55QdMtGA1HFLJAuYyrq3BabjsL1jAcjMI1x2vNlCTPb36u\nRdcyKfN1bx9XlwvKvGA8mcGEYPYSxYK2Ht6hqAqKrKBQVZ+1aZolO6MpkxFsDfP+GSiLAsQOSEnT\nWI7vb/fll7PTC67nV+zt5IwLzTHPn/lenOgGwnIne6tOrxQy13gv8U7ijUDH+zqtNDmeernE+Kzv\nF6hKRVkUOAfGOmwXvoViDKXOn0vgKolYGyg4h/A+1I6FDg1egMhz/C2havDoa1TdImKW8tA6zKhA\njAuGRU5lFVeX4Wd1nrHEIzKJr2tqE965QWe4qyUtweiGOFe/EIpOCo6Pn5B3HzK59x5fnoeSxb+f\nH/Do8Jg5jnm96meTTV71UwCvYjPBV3ky5XBNWPSta5GA8AKsRwkJPJebkwIynWG9RcTFWEsQUiNE\nNDted2QqjXVQlgWrzuJwL3nmrhs9lC5wMbhIkfUGBzdRtzXCOLK4UAupaLu1Lm54ONebBq0lZRTZ\nPzo5Yxl1RC8ur1lcnvG9BF2UFDHdpouMtjNBXF1C19RB1QvQZYHQYZG6uG6o8tBI8frOmCO34m//\n6dXXXE0LcqdRV9fxmlvu3XmDB/fuI6zBXp1x/05IbQrTYm2LKFe0XQ1l+E4G+YSiKfBWoDPN/bdC\nh+fswYypuEBRcqWmPLYHNGHdp80FVhgQYTxlXdv2MejdhjEmbqheUCGTCo96qSYJccLDBwGT2c5e\nn3Z2XcdoMu1TpfIFWRznHJ1tMaZFeNWr2gwGFbnqsFcrjDF9133XdfQfegWTyYjFakm9iM+AHzMc\nlIzGFYvrc7ru+eJ6eXmOP7dY21GNtnBxMZrujlheHlGfXbB3bxfifauXVzRNw9aool2sMMDJdZBP\nLIsB29MZy6tjfjg7pNShDvj07JQvhONfbrpoqRBC9n+nUBqlY0OVMeAcPo51OBGeRecdpjOIKBMq\npEStnV6kQufrkRSHtS7orstQ07NxI+yQQaO3d0mKARvBS910r2A82kJ6S72Ki7gu0TojyysOfriK\nXexRBnY8Is8LLq+WSJ8xHIZGm7pzYByrhaXtPKO4uK/mNa1pyaRkMJjQ1AYdTU4GgxzftUg811cX\n/cakGgz7sbRXkekcrVw/3pPlI5yxnJ8HfXoPVFHhKtMFrnHYlaCugmY9gJUZK+Np2pZMlRRZNHlZ\nLLC2YzwZBslIaXDxQDEopxwcH+PqJd/98TO62BxVqCnqR6jNZVVFkENZK08Vsf/LgrDgLUV8T4fK\n4bqWXGW0zpAXUSFOSfI8p8gKGtPRxOd9JTK6vI2zf/TSsQA6vuNSZ0ipyOLmSJD3/savojk6YoCg\njWtBoxX5pETcG+NXHc0Xx2QirOOn80uMlphW4U2Lj2nn2jgOVclwa4SqBujozWsm26jZPvfGJTs7\nu+SLOX/4MryL/3V6is0Uuc7oWngWA+7q8Bhvbu4JEP7H9PknEolEIpH4fyPVfBOJRCKR2DAp+CYS\niUQisWFS8E0kEolEYsOk4JtIJBKJxIZJwTeRSCQSiQ2Tgm8ikUgkEhsmBd9EIpFIJDZMCr6JRCKR\nSGyYFHwTiUQikdgwKfgmEolEIrFhUvBNJBKJRGLDpOCbSCQSicSGScE3kUgkEokNk4JvIpFIJBIb\nJgXfRCKRSCQ2TAq+iUQikUhsmBR8E4lEIpHYMCn4JhKJRCKxYVLwTSQSiURiw6Tgm0gkEonEhknB\nN5FIJBKJDZOCbyKRSCQSGyYF30QikUgkNsz/Ag2iN32AKflBAAAAAElFTkSuQmCC\n",
+ "text/plain": [
+ "\n",
+ "\n",
+ "\n",
+ " \n",
+ "
\n",
+ "
"
+ ],
+ "text/plain": [
+ " category image\n",
+ "0 plane [[[160.0, 173.0, 167.0], [151.0, 164.0, 155.0]...\n",
+ "1 plane [[[190.0, 228.0, 243.0], [188.0, 223.0, 238.0]...\n",
+ "2 plane [[[255.0, 255.0, 255.0], [253.0, 254.0, 251.0]...\n",
+ "3 plane [[[193.0, 216.0, 227.0], [191.0, 213.0, 225.0]...\n",
+ "4 plane [[[234.0, 234.0, 234.0], [231.0, 231.0, 231.0]..."
+ ]
+ },
+ "metadata": {
+ "tags": []
+ },
+ "execution_count": 20
+ }
+ ]
+ },
+ {
+ "metadata": {
+ "id": "GXtRpahp5V6p",
+ "colab_type": "code",
+ "outputId": "6ea08026-d651-4a75-d40f-94086fb211ce",
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 187
+ }
+ },
+ "cell_type": "code",
+ "source": [
+ "by_category = collections.defaultdict(list)\n",
+ "for _, row in df.iterrows():\n",
+ " by_category[row.category].append(row.to_dict())\n",
+ "for category in by_category:\n",
+ " print (\"{0}: {1}\".format(category, len(by_category[category])))"
+ ],
+ "execution_count": 21,
+ "outputs": [
+ {
+ "output_type": "stream",
+ "text": [
+ "plane: 6000\n",
+ "car: 6000\n",
+ "bird: 6000\n",
+ "cat: 6000\n",
+ "deer: 6000\n",
+ "dog: 6000\n",
+ "frog: 6000\n",
+ "horse: 6000\n",
+ "ship: 6000\n",
+ "truck: 6000\n"
+ ],
+ "name": "stdout"
+ }
+ ]
+ },
+ {
+ "metadata": {
+ "id": "AYVNBhLgt-38",
+ "colab_type": "code",
+ "colab": {}
+ },
+ "cell_type": "code",
+ "source": [
+ "final_list = []\n",
+ "for _, item_list in sorted(by_category.items()):\n",
+ " if args.shuffle:\n",
+ " np.random.shuffle(item_list)\n",
+ " n = len(item_list)\n",
+ " n_train = int(args.train_size*n)\n",
+ " n_val = int(args.val_size*n)\n",
+ " n_test = int(args.test_size*n)\n",
+ "\n",
+ " # Give data point a split attribute\n",
+ " for item in item_list[:n_train]:\n",
+ " item['split'] = 'train'\n",
+ " for item in item_list[n_train:n_train+n_val]:\n",
+ " item['split'] = 'val'\n",
+ " for item in item_list[n_train+n_val:]:\n",
+ " item['split'] = 'test' \n",
+ "\n",
+ " # Add to final list\n",
+ " final_list.extend(item_list)"
+ ],
+ "execution_count": 0,
+ "outputs": []
+ },
+ {
+ "metadata": {
+ "id": "o8GNPotNt-6X",
+ "colab_type": "code",
+ "outputId": "162a2ddb-db83-4708-b48d-ecbf1d61dd4f",
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 85
+ }
+ },
+ "cell_type": "code",
+ "source": [
+ "split_df = pd.DataFrame(final_list)\n",
+ "split_df[\"split\"].value_counts()"
+ ],
+ "execution_count": 23,
+ "outputs": [
+ {
+ "output_type": "execute_result",
+ "data": {
+ "text/plain": [
+ "train 42000\n",
+ "test 9000\n",
+ "val 9000\n",
+ "Name: split, dtype: int64"
+ ]
+ },
+ "metadata": {
+ "tags": []
+ },
+ "execution_count": 23
+ }
+ ]
+ },
+ {
+ "metadata": {
+ "id": "cLdJQPBmX0yJ",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "### Vocabulary"
+ ]
+ },
+ {
+ "metadata": {
+ "id": "EB-kpxhct-_S",
+ "colab_type": "code",
+ "colab": {}
+ },
+ "cell_type": "code",
+ "source": [
+ "class Vocabulary(object):\n",
+ " def __init__(self, token_to_idx=None):\n",
+ "\n",
+ " # Token to index\n",
+ " if token_to_idx is None:\n",
+ " token_to_idx = {}\n",
+ " self.token_to_idx = token_to_idx\n",
+ "\n",
+ " # Index to token\n",
+ " self.idx_to_token = {idx: token \\\n",
+ " for token, idx in self.token_to_idx.items()}\n",
+ "\n",
+ " def to_serializable(self):\n",
+ " return {'token_to_idx': self.token_to_idx}\n",
+ "\n",
+ " @classmethod\n",
+ " def from_serializable(cls, contents):\n",
+ " return cls(**contents)\n",
+ "\n",
+ " def add_token(self, token):\n",
+ " if token in self.token_to_idx:\n",
+ " index = self.token_to_idx[token]\n",
+ " else:\n",
+ " index = len(self.token_to_idx)\n",
+ " self.token_to_idx[token] = index\n",
+ " self.idx_to_token[index] = token\n",
+ " return index\n",
+ "\n",
+ " def add_tokens(self, tokens):\n",
+ " return [self.add_token[token] for token in tokens]\n",
+ "\n",
+ " def lookup_token(self, token):\n",
+ " return self.token_to_idx[token]\n",
+ "\n",
+ " def lookup_index(self, index):\n",
+ " if index not in self.idx_to_token:\n",
+ " raise KeyError(\"the index (%d) is not in the Vocabulary\" % index)\n",
+ " return self.idx_to_token[index]\n",
+ "\n",
+ " def __str__(self):\n",
+ " return \"| \n", + " | category | \n", + "image | \n", + "
|---|---|---|
| 0 | \n", + "plane | \n", + "[[[160.0, 173.0, 167.0], [151.0, 164.0, 155.0]... | \n", + "
| 1 | \n", + "plane | \n", + "[[[190.0, 228.0, 243.0], [188.0, 223.0, 238.0]... | \n", + "
| 2 | \n", + "plane | \n", + "[[[255.0, 255.0, 255.0], [253.0, 254.0, 251.0]... | \n", + "
| 3 | \n", + "plane | \n", + "[[[193.0, 216.0, 227.0], [191.0, 213.0, 225.0]... | \n", + "
| 4 | \n", + "plane | \n", + "[[[234.0, 234.0, 234.0], [231.0, 231.0, 231.0]... | \n", + "