{"id":1372,"date":"2019-08-16T13:28:01","date_gmt":"2019-08-16T17:28:01","guid":{"rendered":"http:\/\/josephpcohen.com\/w\/?p=1372"},"modified":"2022-04-10T23:01:49","modified_gmt":"2022-04-11T03:01:49","slug":"dealing-with-internet-restricted-compute-nodes-in-a-cluster","status":"publish","type":"post","link":"https:\/\/josephpcohen.com\/w\/dealing-with-internet-restricted-compute-nodes-in-a-cluster\/","title":{"rendered":"Dealing with internet restricted compute nodes in a cluster"},"content":{"rendered":"

There are some clusters, such as Beluga in the Compute Canada\/Calcul Qu\u00e9bec system of computers, which restrict access to the internet on compute nodes. One justification can be to prevent GPU nodes from participating in bitcoin mining pools. However, in the world of python and deep learning most scripts need to fetch data as part of the primary script so this make everything more difficult for researchers. It makes more sense to just ban someone caught doing something bad from all compute clusters as a better deterrent instead of making like difficult for the valid users of a cluster.<\/p>\n

Anyway, here is a way to bypass this restriction in an easy way. We will use proxychains<\/a> which will populate the LD_PRELOAD variable and override the network system calls to instead go into a proxy. This way all commands which use the internet go into the proxy without any change my the program that needs internet access. This can be run directly with python to give any script internet access.<\/p>\n

First install proxychains-ng in a folder like ~\/software\/proxychains-ng<\/code> you can install it like this:<\/p>\n

$ .\/configure –prefix=~
\n$ make
\n$ mkdir ~\/lib
\n$ cp libproxychains4.so ~\/lib\/<\/p><\/blockquote>\n

Then set up your your path and set the path to the config file:<\/p>\n

$ PATH=~\/software\/proxychains-ng:$PATH
\n$ export PROXYCHAINS_CONF_FILE=~\/software\/proxychains-ng\/proxychains.conf<\/p><\/blockquote>\n

Second write this script as something like run-with-proxy.sh<\/code>:<\/p>\n

$ ssh -N -D 9050 beluga1 &
\n$ proxychains4 -q $@<\/p><\/blockquote>\n

Note: the port 9050 is used because it is the default in the proxychains config<\/p>\n

Then you can proxy any command with:<\/p>\n

$ run-with-proxy.sh python script.py<\/p><\/blockquote>\n

You can run with bash to have all shell commands have internet:<\/p>\n

$ run-with-proxy.sh bash<\/p><\/blockquote>\n

More reading:<\/h3>\n

https:\/\/stackoverflow.com\/questions\/31639742\/how-to-pass-all-pythons-traffics-through-a-http-proxy<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"

There are some clusters, such as Beluga in the Compute Canada\/Calcul Qu\u00e9bec system of computers, which restrict access to the internet on compute nodes. One […]<\/a><\/p>\n<\/div>","protected":false},"author":1,"featured_media":1377,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":[],"categories":[13],"tags":[],"yoast_head":"\nDealing with internet restricted compute nodes in a cluster - Joseph Paul Cohen PhD<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/josephpcohen.com\/w\/dealing-with-internet-restricted-compute-nodes-in-a-cluster\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Dealing with internet restricted compute nodes in a cluster - Joseph Paul Cohen PhD\" \/>\n<meta property=\"og:description\" content=\"There are some clusters, such as Beluga in the Compute Canada\/Calcul Qu\u00e9bec system of computers, which restrict access to the internet on compute nodes. One [...]\" \/>\n<meta property=\"og:url\" content=\"https:\/\/josephpcohen.com\/w\/dealing-with-internet-restricted-compute-nodes-in-a-cluster\/\" \/>\n<meta property=\"og:site_name\" content=\"Joseph Paul Cohen PhD\" \/>\n<meta property=\"article:published_time\" content=\"2019-08-16T17:28:01+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2022-04-11T03:01:49+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/josephpcohen.com\/w\/wp-content\/uploads\/internet-censorship-800x600.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"800\" \/>\n\t<meta property=\"og:image:height\" content=\"600\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"Joseph Paul Cohen\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Joseph Paul Cohen\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"1 minute\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/josephpcohen.com\/w\/dealing-with-internet-restricted-compute-nodes-in-a-cluster\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/josephpcohen.com\/w\/dealing-with-internet-restricted-compute-nodes-in-a-cluster\/\"},\"author\":{\"name\":\"Joseph Paul Cohen\",\"@id\":\"https:\/\/josephpcohen.com\/w\/#\/schema\/person\/e25d0d5746952220f35d182ca7aa8684\"},\"headline\":\"Dealing with internet restricted compute nodes in a cluster\",\"datePublished\":\"2019-08-16T17:28:01+00:00\",\"dateModified\":\"2022-04-11T03:01:49+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/josephpcohen.com\/w\/dealing-with-internet-restricted-compute-nodes-in-a-cluster\/\"},\"wordCount\":291,\"publisher\":{\"@id\":\"https:\/\/josephpcohen.com\/w\/#\/schema\/person\/e25d0d5746952220f35d182ca7aa8684\"},\"articleSection\":[\"References\"],\"inLanguage\":\"en-US\"},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/josephpcohen.com\/w\/dealing-with-internet-restricted-compute-nodes-in-a-cluster\/\",\"url\":\"https:\/\/josephpcohen.com\/w\/dealing-with-internet-restricted-compute-nodes-in-a-cluster\/\",\"name\":\"Dealing with internet restricted compute nodes in a cluster - Joseph Paul Cohen PhD\",\"isPartOf\":{\"@id\":\"https:\/\/josephpcohen.com\/w\/#website\"},\"datePublished\":\"2019-08-16T17:28:01+00:00\",\"dateModified\":\"2022-04-11T03:01:49+00:00\",\"breadcrumb\":{\"@id\":\"https:\/\/josephpcohen.com\/w\/dealing-with-internet-restricted-compute-nodes-in-a-cluster\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/josephpcohen.com\/w\/dealing-with-internet-restricted-compute-nodes-in-a-cluster\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/josephpcohen.com\/w\/dealing-with-internet-restricted-compute-nodes-in-a-cluster\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/josephpcohen.com\/w\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Dealing with internet restricted compute nodes in a cluster\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/josephpcohen.com\/w\/#website\",\"url\":\"https:\/\/josephpcohen.com\/w\/\",\"name\":\"Joseph Paul Cohen PhD\",\"description\":\"\",\"publisher\":{\"@id\":\"https:\/\/josephpcohen.com\/w\/#\/schema\/person\/e25d0d5746952220f35d182ca7aa8684\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/josephpcohen.com\/w\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-US\"},{\"@type\":[\"Person\",\"Organization\"],\"@id\":\"https:\/\/josephpcohen.com\/w\/#\/schema\/person\/e25d0d5746952220f35d182ca7aa8684\",\"name\":\"Joseph Paul Cohen\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/josephpcohen.com\/w\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/a810b57939e75247f570c9094e7bd16e?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/a810b57939e75247f570c9094e7bd16e?s=96&d=mm&r=g\",\"caption\":\"Joseph Paul Cohen\"},\"logo\":{\"@id\":\"https:\/\/josephpcohen.com\/w\/#\/schema\/person\/image\/\"}}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Dealing with internet restricted compute nodes in a cluster - Joseph Paul Cohen PhD","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/josephpcohen.com\/w\/dealing-with-internet-restricted-compute-nodes-in-a-cluster\/","og_locale":"en_US","og_type":"article","og_title":"Dealing with internet restricted compute nodes in a cluster - Joseph Paul Cohen PhD","og_description":"There are some clusters, such as Beluga in the Compute Canada\/Calcul Qu\u00e9bec system of computers, which restrict access to the internet on compute nodes. One [...]","og_url":"https:\/\/josephpcohen.com\/w\/dealing-with-internet-restricted-compute-nodes-in-a-cluster\/","og_site_name":"Joseph Paul Cohen PhD","article_published_time":"2019-08-16T17:28:01+00:00","article_modified_time":"2022-04-11T03:01:49+00:00","og_image":[{"width":800,"height":600,"url":"https:\/\/josephpcohen.com\/w\/wp-content\/uploads\/internet-censorship-800x600.jpg","type":"image\/jpeg"}],"author":"Joseph Paul Cohen","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Joseph Paul Cohen","Est. reading time":"1 minute"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/josephpcohen.com\/w\/dealing-with-internet-restricted-compute-nodes-in-a-cluster\/#article","isPartOf":{"@id":"https:\/\/josephpcohen.com\/w\/dealing-with-internet-restricted-compute-nodes-in-a-cluster\/"},"author":{"name":"Joseph Paul Cohen","@id":"https:\/\/josephpcohen.com\/w\/#\/schema\/person\/e25d0d5746952220f35d182ca7aa8684"},"headline":"Dealing with internet restricted compute nodes in a cluster","datePublished":"2019-08-16T17:28:01+00:00","dateModified":"2022-04-11T03:01:49+00:00","mainEntityOfPage":{"@id":"https:\/\/josephpcohen.com\/w\/dealing-with-internet-restricted-compute-nodes-in-a-cluster\/"},"wordCount":291,"publisher":{"@id":"https:\/\/josephpcohen.com\/w\/#\/schema\/person\/e25d0d5746952220f35d182ca7aa8684"},"articleSection":["References"],"inLanguage":"en-US"},{"@type":"WebPage","@id":"https:\/\/josephpcohen.com\/w\/dealing-with-internet-restricted-compute-nodes-in-a-cluster\/","url":"https:\/\/josephpcohen.com\/w\/dealing-with-internet-restricted-compute-nodes-in-a-cluster\/","name":"Dealing with internet restricted compute nodes in a cluster - Joseph Paul Cohen PhD","isPartOf":{"@id":"https:\/\/josephpcohen.com\/w\/#website"},"datePublished":"2019-08-16T17:28:01+00:00","dateModified":"2022-04-11T03:01:49+00:00","breadcrumb":{"@id":"https:\/\/josephpcohen.com\/w\/dealing-with-internet-restricted-compute-nodes-in-a-cluster\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/josephpcohen.com\/w\/dealing-with-internet-restricted-compute-nodes-in-a-cluster\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/josephpcohen.com\/w\/dealing-with-internet-restricted-compute-nodes-in-a-cluster\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/josephpcohen.com\/w\/"},{"@type":"ListItem","position":2,"name":"Dealing with internet restricted compute nodes in a cluster"}]},{"@type":"WebSite","@id":"https:\/\/josephpcohen.com\/w\/#website","url":"https:\/\/josephpcohen.com\/w\/","name":"Joseph Paul Cohen PhD","description":"","publisher":{"@id":"https:\/\/josephpcohen.com\/w\/#\/schema\/person\/e25d0d5746952220f35d182ca7aa8684"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/josephpcohen.com\/w\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-US"},{"@type":["Person","Organization"],"@id":"https:\/\/josephpcohen.com\/w\/#\/schema\/person\/e25d0d5746952220f35d182ca7aa8684","name":"Joseph Paul Cohen","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/josephpcohen.com\/w\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/a810b57939e75247f570c9094e7bd16e?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/a810b57939e75247f570c9094e7bd16e?s=96&d=mm&r=g","caption":"Joseph Paul Cohen"},"logo":{"@id":"https:\/\/josephpcohen.com\/w\/#\/schema\/person\/image\/"}}]}},"_links":{"self":[{"href":"https:\/\/josephpcohen.com\/w\/wp-json\/wp\/v2\/posts\/1372"}],"collection":[{"href":"https:\/\/josephpcohen.com\/w\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/josephpcohen.com\/w\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/josephpcohen.com\/w\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/josephpcohen.com\/w\/wp-json\/wp\/v2\/comments?post=1372"}],"version-history":[{"count":10,"href":"https:\/\/josephpcohen.com\/w\/wp-json\/wp\/v2\/posts\/1372\/revisions"}],"predecessor-version":[{"id":1683,"href":"https:\/\/josephpcohen.com\/w\/wp-json\/wp\/v2\/posts\/1372\/revisions\/1683"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/josephpcohen.com\/w\/wp-json\/wp\/v2\/media\/1377"}],"wp:attachment":[{"href":"https:\/\/josephpcohen.com\/w\/wp-json\/wp\/v2\/media?parent=1372"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/josephpcohen.com\/w\/wp-json\/wp\/v2\/categories?post=1372"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/josephpcohen.com\/w\/wp-json\/wp\/v2\/tags?post=1372"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}