1. if you want to run jobs for weeks or which require many cores (>10) for multiple days, please contact [it-support@idiv.de](mailto:it-support@idiv.de) beforehand. At this point you should consider to move to the HPC cluster.
1. if you want to run jobs for weeks or which require many cores (>10) for
multiple days, please contact
[it-support@idiv.de](mailto:it-support@idiv.de) beforehand. At this point
you should consider to move to the HPC cluster.
## ssh access
You can only reach the server from the internal network, i.e. if you are connected via VPN or cable at iDiv. Note that the internal name is rstudio1 instead of rstudio!
You can only reach the server from the internal network, i.e. if you are
connected via VPN or cable at iDiv. Note that the internal name is rstudio1
instead of rstudio!
### Linux
From a terminal run
...
...
@@ -35,60 +40,116 @@ Currently there are three important places for you on the server:
3. /home/$USER/winhome a mounting point for your network home directory
### Home directory
When you log in either via https or ssh this is usually the first place where you end up. Your home directory can be abbreviated by "~/". There is not a lot of space here, so do not place any larger data here. RStudio has the ugly not configurable habit to write quite some temporary data here (e.g. cached plots and suspended sessions), so please make sure that you clean up occasionally. It is a good place for your scripts.
RStudio stores several files like suspended sessions, graph history etc. in `~/.rstudio`. This folder is purged every 90 days to remove "forgotten" suspended sessions and reduce wasted space. R itself stores workspace objects in `.RData` of your current working directory if you end your session or call `save.image()` without further arguments. Please ensure that you do not save large `.RData` files in your home directory.
When you log in either via https or ssh this is usually the first place where
you end up. Your home directory can be abbreviated by "~/". There is not a lot
of space here, so do not place any larger data here. RStudio has the ugly not
configurable habit to write quite some temporary data here (e.g. cached plots
and suspended sessions), so please make sure that you clean up occasionally. It
is a good place for your scripts.
RStudio stores several files like suspended sessions, graph history etc. in
`~/.rstudio`. This folder is purged every 90 days to remove "forgotten"
suspended sessions and reduce wasted space. R itself stores workspace objects
in `.RData` of your current working directory if you end your session or call
`save.image()` without further arguments. Please ensure that you do not save
large `.RData` files in your home directory.
### /data
You can create your own directories here and place data inside. By default others can read the directories you created here, but not write in them. If you need any special permission let us know via it-support@idiv.de. To directly jump to another directory select the *Files* pane and click on the three dots (…) located on the right hand side of the current file path. Now enter the path you want to browse, e.g. `/data/`. Additionally there should be a symbolic link in your home directory called data, which points to the /data directory.
You can create your own directories here and place data inside. By default
others can read the directories you created here, but not write in them. If you
need any special permission let us know via it-support@idiv.de. To directly
jump to another directory select the *Files* pane and click on the three dots
(…) located on the right hand side of the current file path. Now enter the path
you want to browse, e.g. `/data/`. Additionally there should be a symbolic link
in your home directory called data, which points to the /data directory.
### iDiv group shares
If you are part of an iDiv group you can access your group share on the rstudio server at `/data/GROUPNAME-group-share/`. If your group share is not available yet please contact [it-support@idiv.de](mailto:it-support@idiv.de).
If you are part of an iDiv group you can access your group share on the rstudio
server at `/data/GROUPNAME-group-share/`. If your group share is not available
This is about \\idiv.de\public\homes. To make it available on the rstudio server you need to get active. Log in via ssh or enter the pane labeled terminal in RStudio web IDE and run
This is about \\idiv.de\public\homes. To make it available on the rstudio
server you need to get active. Log in via ssh or enter the pane labeled
terminal in RStudio web IDE and run
```
su - $USER
```
You will be asked to enter your password again and afterwards your network home will be available at `~/winhome`. However, it will only stay connected until you log out or quit your current session. If you have any longer running jobs you might want to consider either moving stuff from the ~/winhome to /data or open a [tmux](tmux.md) session from which you detach (Ctrl+b, d) before you disconnect.
You will be asked to enter your password again and afterwards your network home
will be available at `~/winhome`. However, it will only stay connected until
you log out or quit your current session. If you have any longer running jobs
you might want to consider either moving stuff from the ~/winhome to /data or
open a [tmux](tmux.md) session from which you detach (Ctrl+b, d) before you
disconnect.
## R and data
### compression
Please note that R is able to work with compressed files. This is especially useful if you are the typical csv/txt file user. Those files usually contain highly redundant data. Therefore compression can be very effective, e.g. the file which triggered me to write this was a txt file of 4 GB the gz compressed file had 98 MB. Many tools to read or write (e.g. save, save.image, read.table, fread from data.table) allow transparent use of compressed files, i.e. you just specify the compressed file instead of the uncompressed file.
Please note that R is able to work with compressed files. This is especially
useful if you are the typical csv/txt file user. Those files usually contain
highly redundant data. Therefore compression can be very effective, e.g. the
file which triggered me to write this was a txt file of 4 GB the gz compressed
file had 98 MB. Many tools to read or write (e.g. save, save.image, read.table,
fread from data.table) allow transparent use of compressed files, i.e. you just
specify the compressed file instead of the uncompressed file.
## Data transfer
You can use either use the web IDE to upload and export files.
### Web IDE
In the *Files* pane click the *Upload* button to upload files to the current directory. For multiple files check the displayed *TIP* in the upload window. For big files please see below.
In the *Files* pane click the *Upload* button to upload files to the current
directory. For multiple files check the displayed *TIP* in the upload window.
For big files please see below.
If you want to download/export files, select the checkbox for each file or directory and click *More -> Export*. If you selected multiple files a zip file will be downloaded.
If you want to download/export files, select the checkbox for each file or
directory and click *More -> Export*. If you selected multiple files a zip file
will be downloaded.
### Tools
Other file transfer tools are often more reliable and faster than the web IDE. Short instructions are available for several tools like [scp](scp.md), [rsync](rsync.md), [filezilla](filezilla.md).
Other file transfer tools are often more reliable and faster than the web IDE.
Short instructions are available for several tools like [scp](scp.md),
[rsync](rsync.md), [filezilla](filezilla.md).
## Code organisation
It is strongly recommended to use the version control system [git](https://git-scm.com) to track changes in your code. It also helps you to distribute your code, be it for yourself (local machine, rstudio server, HPC cluster, …) or with others (e.g. via [github](https://github.com) or the [iDiv gitlab](https://git.idiv.de)). Ensure that you only track your code and not your data or results!
It is strongly recommended to use the version control system
[git](https://git-scm.com) to track changes in your code. It also helps you to
distribute your code, be it for yourself (local machine, rstudio server, HPC
cluster, …) or with others (e.g. via [github](https://github.com) or the [iDiv
gitlab](https://git.idiv.de)). Ensure that you only track your code and not
your data or results!
If you want your code to run in differenct environments (e.g. local machine, rstudio server or HPC cluster) be sure to separate environment specific code from your buisness logic. Recommendations and templates can be found in the [EVE HPC Cluster wiki](https://wiki.ufz.de/eve/index.php/R_Advanced_-_Tips_%26_Tricks#Interactive_and_Command_Line).
If you want your code to run in differenct environments (e.g. local machine,
rstudio server or HPC cluster) be sure to separate environment specific code
from your buisness logic. Recommendations and templates can be found in the
Many R packages are installed already via the system's package management and many more are available via the same track. This is the preferred way to install R packages, as it avoids duplicate installations per user, removes the need for manual package updates and helps that underlying libraries and packages are compatible with each other. Currently there are >3500 packages installed—check with `installed.packages()` for more details.
Of course you can still install packages via R's own functions (install.packages, devtools, …) if there is the need for it. Long term please drop a note to it-suppot@idiv.de so that we can install the package via the system.
Many R packages are installed already via the system's package management and
many more are available via the same track. This is the preferred way to
install R packages, as it avoids duplicate installations per user, removes the
need for manual package updates and helps that underlying libraries and
packages are compatible with each other. Currently there are >3500 packages
installed—check with `installed.packages()` for more details.
Of course you can still install packages via R's own functions
(install.packages, devtools, …) if there is the need for it. Long term please
drop a note to it-suppot@idiv.de so that we can install the package via the
system.
-[https://github.com/eddelbuettel/rcppapt](https://github.com/eddelbuettel/rcppapt) not yet implemented:
## Other servers with R
There are some other servers which have R and RStudio (desktop version) installed, but support for R on those is limited. This means you might frequently encounter outdated versions of R and associated packages.
There are some other servers which have R and RStudio (desktop version)
installed, but support for R on those is limited. This means you might
frequently encounter outdated versions of R and associated packages.
- idivgis01.idiv.de Biocon windows 2012r2 terminal server with gpu but restricted access
- idivts6.idiv.de Biocon windows 2012r2 terminal server but restricted access
