Jupyter notebook and HPC systems

Jupyter notebook [jupyter.org] is an interactive shell in a web browser that can display output, graphics, and tables inline to make research and development faster, easier, and more reproducibility. It is primarily used for the Python language but is also used by the Julia community.

Here are the scripts I use to connect to a machine via slurm and have the ports configured so I call access the notebook via http://localhost:8889 where the code is executing on a GPU in a cluster.

cluster.head.node The DNS address of your clusters entry machine. Where you can submit jobs.
8889 Replace this with a number between 1024–49151.

I have my laptop configured to forward ports automatically so I just need to type “ssh cluster.head.node” To have this setup you need to modify your local ssh config as follows:

==~/.ssh/config========
Host cluster.head.node
User username 
LocalForward 8889 127.0.0.1:8889
====================

Once on cluster.head.node go into a screen [gnu.org] (so the laptop can disconnect and reconnect). Then launch your job. This example uses slurm. You want an interactive session so we add “–pty $SHELL”

==slurm.sh========
$srun --gres=gpu -C"gpu6gb" --pty $SHELL
====================

Then run “sh jupyter.sh” which will launch a reverse tunnel to connect back to cluster.head.node from whatever node you landed on. It also sets the JUPYTER_RUNTIME_DIR to deal with a permissions issue.

== jupyter.sh ==========
ssh -N -R 8889:localhost:8889 cluster.head.node &
export JUPYTER_RUNTIME_DIR=/tmp/jupyterchangeme
THEANO_FLAGS="device=cuda0,floatX=float32" jupyter notebook --port=8889
====================

Everyone should have a unique port and tmp folder when using this config.

Also to fix some issue with NFS that will happen after you switch between machines set these config options

In .jupyter/jupyter_notebook_config.py

c.NotebookNotary.db_file = ':memory:'

In .ipython/profile_default/ipython_config.py

c.HistoryAccessor.enabled = False
c.HistoryAccessor.hist_file = ':memory:'

You should set a password for your jupyter notebook using this setting. https://jupyter-notebook.readthedocs.io/en/latest/public_server.html

c.NotebookApp.password = u'sha1:67c9e60bb8b6:9ffede0825894254b2e042ea597d771089e11aed'

Common fixes

Error when having a too recent version of ipython:

$ jupyter-kernelspec list
[ListKernelSpecs] WARNING | Native kernel (python2) is not available
[ListKernelSpecs] WARNING | Native kernel (python2) is not available
No kernels available

Install an earlier version of ipython:

$pip install --user --upgrade ipython==4.1.1