Noggin

Noggin is a simple Python tool for ‘live’ logging and plotting measurements during an experiment. Although Noggin can be used in a general context, it is designed around the train/test and batch/epoch paradigm for training a machine learning model.

Noggin’s primary features are its abilities to:

• Log batch-level and epoch-level measurements by name
• Seamlessly update a ‘live’ plot of your measurements, embedded within a Jupyter notebook
• Organize your measurements into a data set of arrays with labeled axes, via xarray
• Save and load your measurements & live-plot session: resume your experiment later without a hitch

A Simple Example of Using Your Noggin

Here is a sneak peak of what it looks like to use Noggin to record and plot data during an experiment. The following code is meant to be run in a Jupyter notebook.

%matplotlib notebook
import numpy as np
from noggin import create_plot
metrics = ["accuracy", "loss"]
plotter, fig, ax = create_plot(metrics)

for i, x in enumerate(np.linspace(0, 10, 100)):
    # record and plot batch-level metrics
    x += np.random.rand(1)*5
    batch_metrics = {"accuracy": x**2, "loss": 1/x**.5}
    plotter.set_train_batch(batch_metrics, batch_size=1, plot=True)

    # record training epoch
    if i%10 == 0 and i > 0:
        plotter.set_train_epoch()

        # cue test-evaluation of model
        for x in np.linspace(0, 10, 5):
            x += (np.random.rand(1) - 0.5)*5
            test_metrics = {"accuracy": x**2}
            plotter.set_test_batch(test_metrics, batch_size=1)
        plotter.set_test_epoch()
plotter.plot()  # ensures final data gets plotted

Noggin

Noggin is a simple Python tool for ‘live’ logging and plotting measurements during experiments. Although Noggin can be used in a general context, it is designed around the train/test and batch/epoch paradigm for training a machine learning model.

Noggin’s primary features are its abilities to:

• Log batch-level and epoch-level measurements by name
• Seamlessly update a ‘live’ plot of your measurements, embedded within a Jupyter notebook
• Organize your measurements into a data set of arrays with labeled axes, via xarray
• Save and load your measurements & live-plot session: resume your experiment later without a hitch

A Simple Example of Using Your Noggin

Here is a sneak peak of what it looks like to use Noggin to record and plot data during an experiment. The following code is meant to be run in a Jupyter notebook.

%matplotlib notebook
import numpy as np
from noggin import create_plot
metrics = ["accuracy", "loss"]
plotter, fig, ax = create_plot(metrics)

for i, x in enumerate(np.linspace(0, 10, 100)):
    # record and plot batch-level metrics
    x += np.random.rand(1)*5
    batch_metrics = {"accuracy": x**2, "loss": 1/x**.5}
    plotter.set_train_batch(batch_metrics, batch_size=1, plot=True)

    # record training epoch
    if i%10 == 0 and i > 0:
        plotter.set_train_epoch()

        # cue test-evaluation of model
        for x in np.linspace(0, 10, 5):
            x += (np.random.rand(1) - 0.5)*5
            test_metrics = {"accuracy": x**2}
            plotter.set_test_batch(test_metrics, batch_size=1)
        plotter.set_test_epoch()
plotter.plot()  # ensures final data gets plotted

Installing Noggin

Noggin requires: numpy, matplotlib, and xarray. You can install Noggin using pip:

pip install noggin

You can instead install Noggin from its source code. Clone this repository and navigate to the Noggin directory, then run:

python setup.py install

A Typical Workflow Using Noggin

Here, we will create a simple mock-up of an experiment in which we use Noggin to record and plot our measurements. We will exercise the critical features that this library provides us with. The following demo is intended to be conducted in a Jupyter notebook.

Please note that, if you don’t need to visualize your data as you collect it, you can use LiveLogger to record your measurements in a nearly-identical manner.

Recording and Plotting Data During an Experiment

To begin, let’s make up some functions to represent a data loader and a model that we are training:

"""
Defining mock data-loader and model-training functions for this simple demo.
The details here are not important, other than the fact that
`training_loop` returns a tuple of two floats.
"""

from time import sleep
from typing import Tuple

import numpy as np
np.random.seed(0)

def batch_loader(num_batches):
    """Simulates loading batches of data of varying sizes"""
    for i in np.linspace(-10, 10, num_batches):
        batch_size = np.random.randint(1, 10)
        yield batch_size * [i]

def training_loop(batch) -> Tuple[float, float]:
    """Simulates data processing Takes ~10ms to process a batch.
    Returns a 'loss' and 'accuracy'"""
    sleep(0.01)
    x = np.mean(batch)
    x += np.random.rand(1)*5 # add some noise
    return np.exp(-x / 5), 1 / (1 + np.exp(-x))

Noggin’s operation is centered around metrics: the various measurements that we want to record and visualize. Here, we will be interested in measuring the accuracy of our model - on both training data and validation data - along with the training loss. In general, we can work with any variety and number of metrics in Noggin; a metric boils down to being any scalar value.

Let’s create a live-plot for these two metrics; the resulting empty plot pane will automatically update as we proceed to make measurements during our experiment. In order to permit live-plotting in a Jupyter notebook, we need to enable the appropriate plotting backend: this is done by invoking the ‘cell-magic’ %matplotlib notebook (Note: one typically has to run this command twice before it will take effect - this seems to be a minor bug in Jupyter).

%matplotlib notebook
from noggin import create_plot

metrics = ['loss', 'accuracy']
plotter, fig, axes = create_plot(metrics)

There are two sets of axes in this figure, one for each of the metrics that we passed to create_plot(). Regarding the objects that this returned:

• plotter is an instance of LivePlot; it will be responsible for logging and plotting our measurements.
• fig and axes are the standard matplotlib figure and axes objects that are produced when one invokes matplotlib.pyplot.subplots; these can be used to affect and save the plot as you would with any matplotlib plot.

Without further ado, let’s run our mock-experiment.

We will be passing batches of training data to our model-training function, recording the training-loss (i.e. the training objective) and the accuracy of our model. An ‘epoch’ will represent one hundred training batches. Here we will plot the average model accuracy for that epoch. We will also, at each epoch, measure the accuracy of our model on a set of test data. This is the standard affair for training a machine learning model.

# logging and plotting measurements during an experiment
for nbatch, batch in enumerate(batch_loader(1000)):
    loss, train_accuracy = training_loop(batch)
    recorded_metrics = dict(loss=loss, accuracy=train_accuracy)
    plotter.set_train_batch(recorded_metrics,
                            batch_size=len(batch))
    if (nbatch + 1) % 100 == 0:
        # record epoch-level statistics
        for test_cnt in range(10):
            # Measure model-accuracy on a validation set
            _, test_accuracy = training_loop(batch)
            plotter.set_test_batch(dict(accuracy=test_accuracy),
                                   batch_size=len(batch))
        plotter.set_train_epoch()
        plotter.set_test_epoch()
# make sure any "straggler" data gets plotted
plotter.plot()

As this experiment runs our plot pane will draw batch-level data with thin, semi-transparent lines. The epoch-level data will appear in bold, with each marker indicated. The most-recent epoch value for a metric will be recorded in the plot’s legend. Please note that the x-axis, the number of batch iterations, is indicated using scientific notation. Once the experiment is complete our plot will look as follows:

There are a number of ways that you can customize your live plot; these are detailed elsewhere in the Noggin documentation. You can control:

• the figure-size of the plot and axis-grid layout for your metrics
• the plot colors across metrics and train/test splits
• the rate at which the plot is updated
• the maximum number of batches to be included in the plot
• whether or not you want to plot the batch-level data at all

Accessing Your Data

There are two ways to access the data that you recorded during your experiment: via xarray datasets or via dictionaries. It is recommended that you make keen use of the xarrays and their ability to handle data-alignment, missing data, and many other features.

via xarray Datasets

The metrics that we recorded during our experiment are recorded as so-called ‘data-variables’ in an xarray dataset, which can be accessed via to_xarray(). And iteration-count serves as the coordinate that uniquely indexes these metrics.

# accessing train-metrics as an xarray dataset
>>> train_batch, train_epoch = plotter.to_xarray('train')
>>> train_batch
<xarray.Dataset>
Dimensions:     (iterations: 1000)
Coordinates:
  * iterations  (iterations) int32 1 2 3 4 5 6 7 ... 995 996 997 998 999 1000
Data variables:
    loss        (iterations) float64 3.176 3.154 3.842 ... 0.1056 0.06601 0.1135
    accuracy    (iterations) float64 0.003083 0.003193 0.001193 ... 1.0 1.0 1.0

>>> train_epoch
<xarray.Dataset>
Dimensions:     (iterations: 10)
Coordinates:
  * iterations  (iterations) int32 100 200 300 400 500 600 700 800 900 1000
Data variables:
loss        (iterations) float64 3.825 2.526 1.764 ... 0.2331 0.1495 0.09778
accuracy    (iterations) float64 0.00388 0.02844 0.1339 ... 0.9998 1.0

Each metric can be easily accessed as an attribute of this dataset; this returns an individual xarray DataArray for that metric:

  # accessing the data array for 'accuracy'
  >>> train_batch.accuracy  # or `train_batch['accuracy']
  <xarray.DataArray 'accuracy' (iterations: 1000)>
  array([0.003083, 0.003193, 0.001193, ..., 0.999987, 0.999999, 0.999981])
  Coordinates:
* iterations  (iterations) int32 1 2 3 4 5 6 7 ... 995 996 997 998 999 1000

xarray’s data structures are powerful and highly-convenient. They provide a natural means for aligning batch-level and epoch-level measurements using iteration count. Furthermore, they handle missing data gracefully.

Towards this end, if you run multiple iterations of an experiment, then you can use concat_experiments() to combine your data sets along a new ‘experiments’ axis. This will gracefully accommodate combining experiments that were run for differing numbers of iterations, and will permit you to seamlessly compute statistics across them.

via Dictionaries

You can access your recorded metrics as dictionaries via train_metrics() and test_metrics().

The structure of the resulting dictionary is:

'<metric-name>' -> {"batch_data":   array,
                    "epoch_data":   array,
                    "epoch_domain": array,
                    ...}

>>> plotter.train_metrics['accuracy']['batch_data']
array([3.08328619e-03, 3.19260208e-03, ..., 9.99981201e-01])

Saving and Resuming Your Experiment

Instances of Noggin’s LivePlot and LiveLogger classes can both be converted to dictionaries, which can then be “pickled” - saving them for later use.

Let’s convert plotter to a dictionary using to_dict() and save it:

# converting `plotter` to a dictionary and pickling it
import pickle

with open('plotter.pkl', 'wb') as f:
    pickle.dump(plotter.to_dict(), f, protocol=-1)

We can now easily load our pickled plotter and recreate our plot as we left it, via from_dict()

# loading the pickled plotter and recreating the plot
from noggin import LivePlot

with open('plotter.pkl', 'rb') as f:
    loaded_dict = pickle.load(f)
    loaded_plotter = LivePlot.from_dict(loaded_dict)

fig, ax = loaded_plotter.plot_objects
loaded_plotter.plot()

We can now resume recording measurements in our experiment just as we were doing earlier; our metrics will be logged and plotted just as before!

Logging Data with Noggin

Noggin’s core role in an experiment is to log your measurements and store them in an organized, accessible manner. LiveLogger is responsible for facilitating this. This class is meant to serve as a drop-in replacement for LivePlot, and is useful for running experiments where you do not need a live visualization of your data.

Batch-level measurements can be logged for both train and test splits of data, and an epoch can be marked in order to compute the mean-value of each metric over that epoch.

Let’s record measurements for two batches of data and mark an epoch. Note that we need to provide the logger the measurements by name, via a dictionary.

>>> from noggin import LiveLogger
>>> logger = LiveLogger()
>>> logger.set_train_batch(dict(metric_a=2., metric_b=1.), batch_size=10)
>>> logger.set_train_batch(dict(metric_a=0., metric_b=2.), batch_size=4)
>>> logger.set_train_epoch()  # compute the mean statistics
>>> logger
LiveLogger(metric_a, metric_b)
number of training batches set: 2
number of training epochs set: 1
number of testing batches set: 0
number of testing epochs set: 0

Accessing our logged batch-level and epoch-level data works the same way as when working with an instance of LivePlot:

# accessing the logged data as xarrays.
>>> batch_array, epoch_array = logger.to_xarray("train")
>>> batch_array
<xarray.Dataset>
Dimensions:     (iterations: 2)
Coordinates:
  * iterations  (iterations) int32 1 2
Data variables:
    metric_a    (iterations) float64 2.0 0.0
    metric_b    (iterations) float64 1.0 2.0
>>> epoch_array
<xarray.Dataset>
Dimensions:     (iterations: 1)
Coordinates:
  * iterations  (iterations) int32 2
Data variables:
    metric_a    (iterations) float64 1.429
    metric_b    (iterations) float64 1.286)

Note that the epoch-level measurements are aligned with the batch-level measurements along the ‘iterations’ coordinate-axis. (E.g. we can see that our first epoch was recorded at batch-iteration 2).

Additionally, saving and loading your logger is as simple as converting your logger to a dictionary, and pickling it:

import pickle

# converting `logger` to a dictionary and pickling it
with open('logger.pkl', 'wb') as f:
    pickle.dump(logger.to_dict(), f, protocol=-1)

# loading the logger
with open('logger.pkl', 'rb') as f:
    loaded_dict = pickle.load(f)
    loaded_logger = LiveLogger.from_dict(loaded_dict)

Converting a Logger to a Plotter

It is easy to visualize your logged data and to convert your logger to an instance of LivePlot, thanks to plot_logger():

from noggin import plot_logger
plotter, fig, ax = plot_logger(logger)
plotter.show()

This gives us access to the matplotlib figure and axes objects for our plot, and plotter is the instance of LivePlot that stores our logged data. plotter.max_fraction_spent_plotting will be 0 by default, but you can increase this value and proceed to use plotter to visualize your measurements in realtime.

Documentation for Noggin

Documentation for noggin.logger

LiveMetric(name)

Holds the relevant data for a train/test metric for live plotting.

LiveLogger(*args, **kwargs)

Logs batch-level and epoch-level summary statistics of the training and testing metrics of a model during a session.

Documentation for noggin.plotter

LivePlot(metrics, Sequence[str], Dict[str, …)

Records and plots batch-level and epoch-level summary statistics of the training and testing metrics of a model during a session.

Documentation for noggin.xarray

metrics_to_xarrays(metrics, Dict[str, …)	Given noggin metrics, returns xarray datasets for the batch-level and epoch-level metrics, respectively.
concat_experiments(*exps)	Concatenates xarray data sets from a sequence of experiments.

Documentation for noggin.utils

create_plot(metrics, Sequence[str], …)	Create matplotlib figure/axes, and a live-plotter, which publishes “live” training/testing metric data, at a batch and epoch level, to the figure.
plot_logger(logger, plot_batches, …)	Plots the data recorded by a LiveLogger instance.
save_metrics(path, pathlib.Path], liveplot, …)	Save live-plot metrics to a numpy zipped-archive (.npz).
load_metrics(path, pathlib.Path])	Load noggin metrics from a numpy archive.

Changelog

This is a record of all past noggin releases and what went into them, in reverse chronological order. All previous releases should still be available on pip.

0.10.1 - 2019-07-21

Fixes bug which last_n_batches was specified for a LivePlot instance, and a metric’s test-epochs were being plotted, but its train-epochs were not. In this scenario, noggin was not properly tracking the batch-iterations associated with the plotted epochs, and all of the test-epochs were being plotted.

0.10.0 - 2019-06-15

Normalizes the interfaces of LiveLogger and LivePlot so that they can be used as drop-in replacements for each other more seamlessly.

This is an API-breaking update for LivePlot, as it renames the methods plot_train_epoch and plot_test_epoch to set_train_epoch and set_test_epoch, respectively. As stated above, this is to match the interface of LiveLogger.

0.9.1 - 2019-06-06

Adds plot_logger(), which provides a convenient means for plotting the data stored by a LiveLogger, and to convert it into an instance of LivePlot

LivePlot not longer warns about a bad matplotlib backend if max_fraction_spent_plotting is set to 0.

0.9.0 - 2019-05-27

This is the first public release of noggin on pypi.

Indices and tables

• Index
• Module Index
• Search Page