Jupyter notebook [jupyter.org] is an interactive shell in a web browser that can display output, graphics, and tables inline to make research and development faster, easier, and more reproducibility. It is primarily used for the Python language but is also used by the Julia community.
Here are the scripts I use to connect to a machine via slurm and have the ports configured so I call access the notebook via http://localhost:8889 where the code is executing on a GPU in a cluster.
cluster.head.node The DNS address of your clusters entry machine. Where you can submit jobs.
8889 Replace this with a number between 1024–49151.
I have my laptop configured to forward ports automatically so I just need to type “ssh cluster.head.node” To have this setup you need to modify your local ssh config as follows:
==~/.ssh/config======== Host cluster.head.node User username LocalForward 8889 127.0.0.1:8889 ====================
Then you need to launch a job on your clusters head node. This example uses slurm.
==slurm.sh======== $sbatch --gres=gpu jupyter.sh ====================
The “jupyter.sh” will launch a reverse tunnel to connect back to cluster.head.node from whatever node you landed on. It also sets the JUPYTER_RUNTIME_DIR to deal with a permissions issue.
== jupyter.sh ========== #!/bin/sh ssh -N -R 8889:localhost:8889 cluster.head.node & export JUPYTER_RUNTIME_DIR=/tmp/jupyterchangeme THEANO_FLAGS="device=cuda0,floatX=float32" jupyter notebook --port=8889 ====================
Everyone should have a unique port and tmp folder when using this config.
Also to fix some issue with NFS that will happen after you switch between machines set these config options
c.NotebookNotary.db_file = ':memory:'
c.HistoryAccessor.enabled = False c.HistoryAccessor.hist_file = ':memory:'
You should set a password for your jupyter notebook using this setting. https://jupyter-notebook.readthedocs.io/en/latest/public_server.html
c.NotebookApp.password = u'sha1:67c9e60bb8b6:9ffede0825894254b2e042ea597d771089e11aed'
Error when having a too recent version of ipython:
$ jupyter-kernelspec list [ListKernelSpecs] WARNING | Native kernel (python2) is not available [ListKernelSpecs] WARNING | Native kernel (python2) is not available No kernels available
Install an earlier version of ipython:
$pip install --user --upgrade ipython==4.1.1