Jupyter notebook [jupyter.org] is an interactive shell in a web browser that can display output, graphics, and tables inline to make research and development faster, easier, and more reproducibility. It is primarily used for the Python language but is also used by the Julia community. This approach will let you use jupyter notebooks with the GPUs from a cluster.

Here are the scripts I use to connect to a machine via slurm and have the ports configured so I call access the notebook via http://localhost:8889 where the code is executing on a GPU in a cluster.

cluster.head.node The DNS address of your clusters entry machine. This should be a machine where you can submit jobs. Also, this guide will use the port number
8889. You should replace this with a number between 1024–49151 so it does not conflict with other users.

You can configure your laptop to forward ports automatically so you can just type “ssh cluster.head.node” To have this setup you need to modify your local ssh config as follows:

==~/.ssh/config========
Host cluster.head.node
User username 
LocalForward 8889 127.0.0.1:8889
====================

Then you need to launch a job on your clusters head node so the jupyter command will run on a compute node. This example uses slurm:

$sbatch --gres=gpu jupyter.sh

The “jupyter.sh” script will launch a reverse tunnel to connect back to cluster.head.node from whatever node you landed on. It also sets the JUPYTER_RUNTIME_DIR to deal with a permissions issue.

== jupyter.sh ==========
#!/bin/sh
ssh -N -R 8889:localhost:8889 cluster.head.node &
export JUPYTER_RUNTIME_DIR=/tmp/jupyterchangeme
jupyter notebook --port=8889 --no-browser
====================

Everyone should have a unique port and tmp folder when using this config.

If reverse port forwarding is blocked

Sometimes nodes are not able to ssh into a head node (like due to an authentication issue). In that case you can start a screen/tmux session on the head node and then ssh into the execution node with port forwarding. As long as the screen/tmux session is still running you can disconnect and connect to the head node and the port will stay connected.

Look up the node that is running the job using squeue -u $USER and then ssh into that hostname:

ssh -L 8889:localhost:8889 nodehostname

Setting a password

If the webpage doesn’t prompt you to mset a password then follow these instructions. More info here

Generate the hash of your password like this in a python shell:

from notebook.auth import passwd
passwd()

Then take that hash and set it as follows in this file: ~/.jupyter/jupyter_notebook_config.py

c.NotebookApp.password = u'sha1:67c9e60bb8b6:9ffede0825894254b2e042ea597d771089e11aed'

 

Common fixes

Error when having a too recent version of ipython:

$ jupyter-kernelspec list
[ListKernelSpecs] WARNING | Native kernel (python2) is not available
[ListKernelSpecs] WARNING | Native kernel (python2) is not available
No kernels available

Install an earlier version of ipython:

$pip install --user --upgrade ipython==4.1.1

Also to fix some issue with NFS that will happen after you switch between machines set these config options

In ~/.jupyter/jupyter_notebook_config.py

c.NotebookNotary.db_file = ':memory:'

In ~/.ipython/profile_default/ipython_config.py

c.HistoryAccessor.enabled = False
c.HistoryAccessor.hist_file = ':memory:'