Valohai's Jupyter Notebook Extension

Juha Kiili / May 28, 2019

Valohai is a deep learning platform that helps you execute on-demand experiments in the cloud with full version control. Jupyter Notebook is a popular IDE for the data scientist. It is especially suited for early data exploration and prototyping.

The data scientists tend to waste a lot of time just waiting for the training results when the optimal way is to work asynchronously and iterate with different approaches in parallel. To read more about asynchronous workflows in deep learning, go to Asynchronous Workflow in Data Science blog.

We at Valohai are developing an extension for Jupyter Notebooks, which is specially optimized to provide full version control for deep learning experiments, and a smooth asynchronous workflow without additional technical hassle.

This tutorial describes what the Valohai + Jupyter Notebook combination can do, and is divided into four steps:

  1. Starting a new experiment,
  2. Monitoring experiments,
  3. Checking experiment results, and
  4. Version controlling your trainings.

Let’s take a look at how the extension works.

1. Starting a new experiment

Run deep learning experiments on cloud Starting a new cloud experiment

When we create a new experiment, the Notebook is uploaded to the cloud where new server instance goes up. Once it has downloaded the training data, it executes all the cells in the notebook from top to bottom. A small rectangle appears to top-right corner to signify an experiment running in the cloud.

2. Monitoring experiments

Monitoring deep learning experiments from Notebook with Valohai extension Monitoring experiment status and logs

Here we have three experiments running. We can monitor the status and log outputs by hovering the experiment gizmo. Blue color means that the experiment is currently running. Green is for finished experiments. If an exception has occurred or the user manually stops the experiment, the gizmo turns red, and the server is immediately shut down in the cloud.

3. Experiment results

image2-1 Downloading finished notebook from the cloud

Once the experiment is finished, we can click the Notebook button and download the resulting notebook back to another browser tab. If we like the results, we can continue working on the notebook right away, perhaps make a few changes and start another experiment in the cloud!

4. Version control

Versioned Notebook experiments on Valohai UI Versioned experiments in Valohai machine learning platform

The experiments are version controlled and fully reproducible in the Valohai machine learning platform. Training data, notebook, docker image, hyperparameters, cloud server type, cost, username, and notes are all stored to keep your work safe and reproducible for years to come.

Sign up for the Early Access Program to try Valohai's Jupyter Notebook extension in practice!