filezilla.md

+# filezilla
+TODO

rstudio1.md

 # Short Notes documenting the RStudio server at iDiv
 
-## Access Web-IDE of RStudio
-1. [https://rstudio.idiv.de](https://rstudio.idiv.de)
+Subscribe to updates of this documentation via
+[RSS](https://git.idiv.de/sc/RStudio-Docu/-/commits/master?format=atom).
+
+[[_TOC_]]
+
+## Access Web IDE of RStudio
+
+1. check if you have been given access from [GSU](https://www.idiv.de/?id=532)
+   (i.e. if you are the group g_r_users)
+1. go to https://rstudio.idiv.de
 1. use your iDiv username and password to login
+1. please remember to **end your session** when you have finished your current
+   calculations
+    - interactively via the red button in the upper right
+    - in your longer running scripts use a statement like this after your
+        computations. You might want to replace `save.image()` with `save()` if
+        you are only interested in some result data and not the whole
+        workspace.
+      ```R
+      # above some fancy computations
+      save.image('/data/myproject/myproject.RData')
+      quit(save = 'no')
+      ```
+1. if you want to run jobs for weeks or which require many cores (>10) for
+   multiple days, please contact
+   [it-support@idiv.de](mailto:it-support@idiv.de?subject=heavy%20rstudio%20use)
+   beforehand. At this point you should consider to move to the HPC cluster.
 
 
 ## ssh access
-You can only reach the server from the internal network, i.e. if you are connected via VPN or cable at iDiv. Note that the internal name is rstudio1 instead of rstudio!
 
-### Linux
-From a terminal run
+You can only reach the server from the internal network, i.e. if you are
+connected via VPN or cable at iDiv. Note that the internal name is rstudio1
+instead of rstudio!
+
+From a terminal/command prompt run (replace `ab12cdef` with your iDiv user name)
 
 ```bash
-ssh rstudio1.idiv.de
+ssh ab12cdef@rstudio1.idiv.de
 ```
 
+This even works on current windwos systems.
+
+
 ## Data organisation
+
 Currently there are three important places for you on the server:
 
-1. /homes/$USER your home directory 
-2. /data place any larger files you want to work with here
-3. /home/$USER/winhome a mounting point for your network home directory
+1. `/homes/$USER` your home directory
+2. `/data place` any larger files you want to work with here
+3. `/home/$USER/winhome` a mounting point for your network home directory
+
 
 ### Home directory
-When you log in either via https or ssh this is usually the first place where you end up. Your home directory can be abbreviated by "~/". There is not a lot of space here, so do not place any larger data here. RStudio has the ugly not configurable habit to write quite some temporary data here (e.g. cached plots and suspended sessions), so please make sure that you clean up occasionally. It is a good place for your scripts.
+
+When you log in either via https or ssh this is usually the first place where
+you end up. Your home directory can be abbreviated by "~/". There is not a lot
+of space here, so do not place any larger data here. RStudio has the ugly not
+configurable habit to write quite some temporary data here (e.g. cached plots
+and suspended sessions), so please make sure that you clean up occasionally.
+
+The home directory is a good place for your scripts, which you hopefully have
+in a git repository already.
+
+RStudio stores several files like suspended sessions, graph history etc. in
+`~/.local/share/rstudio`. This folder is purged every 90 days to remove "forgotten"
+suspended sessions and reduce wasted space. R itself stores workspace objects
+in `.RData` of your current working directory if you end your session or call
+`save.image()` without further arguments. Please ensure that you do not save
+large `.RData` files in your home directory.
+
 
 ### /data
-You can create your own directories here and place data inside. By default others can read the directories you created here, but not write in them. If you need any special permission let us know via it-support@idiv.de.
 
-## network home
-This is about \\idiv.de\public\homes. To make it available on the rstudio server you need to get active. Log in via ssh and run
+You can create your own directories here and place data inside. By default
+others can read the directories you created here, but not write in them. If you
+need any special permission let us know via
+[it-support@idiv.de](mailto:it-support@idiv.de?subject=rstudio%20file%20permissions).
+To directly jump to another directory select the *Files* pane and click on the
+three dots (…) located on
+the right hand side of the current file path. Now enter the path you want to
+browse, e.g. `/data/`. Additionally there should be a symbolic link in your
+home directory called data, which points to the /data directory.
+
+
+### iDiv group shares
+
+If you are part of an iDiv group you can access your group share on the rstudio
+server at `/data/GROUPNAME-group-share/`. If your group share is not available
+yet please contact
+[it-support@idiv.de](mailto:it-support@idiv.de?subject=rstudio%20group-share).
+
+
+## Network home
+
+This is about \\idiv.de\public\homes. To make it available on the rstudio
+server you need to get active. Log in via ssh or enter the pane labeled
+terminal in RStudio web IDE and run
+
 ```
 su - $USER
 ```
-You will be asked to enter your password again and afterwards your network home will be available at `~/winhome`. However, it will only stay mounted until you log out. If you have any longer running jobs you might want to consider either moving stuff from the ~/winhome to /data or open a [tmux](tmux.md) session from which you detach (Ctrl+b, d) before you disconnect.
+
+You will be asked to enter your password again and afterwards your network home
+will be available at `~/winhome`. However, it will only stay connected until
+you log out or quit your current session. If you have any longer running jobs
+you might want to consider either moving stuff from the ~/winhome to /data or
+open a [tmux](tmux.md) session from which you detach (Ctrl+b, d) before you
+disconnect.
+
+
+## R and data
+
+### sessions and R jobs
+
+When you have logged in a R session is started or resumed. Sometimes a R job
+can make your session unresponsive. In this case first check if you can
+stop your currenlty running R job by pressing the red stop sign in the top
+right corner of the console. This sign is only visible if you are executing
+code. If you can not interact with the web interface anymore, you can still end
+your session via *ssh*:
+
+```bash
+ssh ab12cdef@rstudio1.idiv.de 'kill-my-rstudio-session'
+```
+If you can not access the machine via ssh please [ask the
+GSU](mailto:it-support@idiv.de?subject=kill%20my%20rstudio%20session) to kill
+your session. Do not forget to include your username.
+
+If you have problems to resume a old session and get the message `Error occurred during transmission` you might need to remove the old session data, which is located in the `~/.local/share/rstudio` folder. Usually it is sufficient to remove the sessions folder. The command via `ssh`:
+
+```bash
+ssh ab12cdef@rstudio1.idiv.de 'rm -r ~/.local/share/rstudio/sessions/'
+```
+
+If you can not access the machine via ssh please [ask the
+GSU](mailto:it-support@idiv.de?subject=remove%20my%20old%20rstudio%20session) to remove
+your old session. Do not forget to include your username.
+
+
+### compression
+
+Please note that R is able to work with compressed files. This is especially
+useful if you are the typical csv/txt file user. Those files usually contain
+highly redundant data. Therefore compression can be very effective, e.g. the
+file which triggered me to write this was a txt file of 4 GB the gz compressed
+file had 98 MB. Many tools to read or write (e.g. save, save.image, read.table,
+fread from data.table) allow transparent use of compressed files, i.e. you just
+specify the compressed file instead of the uncompressed file.
+
+
+## Data transfer
+
+You can use use the web IDE to upload and export files, use ssh to transfer
+files (see tools) or download files to the server using the command line or R
+language.
+
+
+### Web IDE
+
+In the *Files* pane click the *Upload* button to upload files to the current
+directory. For multiple files check the displayed *TIP* in the upload window.
+For big files please see below.
+
+If you want to download/export files, select the checkbox for each file or
+directory and click *More -> Export*. If you selected multiple files a zip file
+will be downloaded.
+
+Avoid the web IDE for large files, resuming of a transfer is usually not
+possible.
+
+
+### Tools
+
+Other file transfer tools are often more reliable and faster than the web IDE.
+Short instructions are available for several tools like [scp](scp.md),
+[rsync](rsync.md), [filezilla](filezilla.md).
+
+
+## Code organisation
+
+It is strongly recommended to use the version control system
+[git](https://git-scm.com) to track changes in your code. It also helps you to
+distribute your code, be it for yourself  (local machine, rstudio server, HPC
+cluster, …) or with others (e.g. via [github](https://github.com) or the [iDiv
+gitlab](https://git.idiv.de)). Ensure that you only track your code and not
+your data or results!
+
+If you want your code to run in differenct environments (e.g. local machine,
+rstudio server or HPC cluster) be sure to separate environment specific code
+from your buisness logic, some [recommendations and templates](https://gist.github.com/wookietreiber/0ff145270c75f11e956284412b3847b8#file-readme-md).
 
 
 ## Packages
-Many R packages are installed already via the system's package management and many more are available via the same track. This is the preferred way to install R packages, as it avoids duplicate installations per user, removes the need for manual package updates and helps that underlying libraries and packages are compatible with each other.
 
-Of course you can still install packages via R's own functions (install.packages, devtools, …) if there is the need for it. Long term please drop a note to it-suppot@idiv.de so that we can install the package via the system.
+Many R packages are installed already via the system's package management and
+many more are available via the same track. This is the preferred way to
+install R packages, as it avoids duplicate installations per user, removes the
+need for manual package updates and helps that underlying libraries and
+packages are compatible with each other. Currently there are >3500 packages
+installed—check with `installed.packages()` for more details.
+
+Of course you can still install packages via R's own functions
+(install.packages, devtools, …) if there is the need for it. Long term please
+drop a note to
+[it-suppot@idiv.de](mailto:it-suppot@idiv.de?subject=rstudio%20packages) so
+that we can install the package via the system.
+
+- https://github.com/eddelbuettel/rcppapt (not yet implemented)
+
+### ignore system packages
+
+If you do not want to use the system packages for some reason you need to
+change the path where R is looking for packages. This must be done at session
+startup and is done by defining environment variables. For rstudio to use the
+environment variables specify them in a file called `.Renviron`. This file can
+either be placed in your home directory or in your working directory. If it is
+in a working directory and rstudio did not start with that working directory,
+change to it and restart R.
+
+Example:
+
+1. Check your current pathes with `.libPaths()`
+1. create directory `~/testsession`
+1. create file `.Renviron` in that directory with the content
+   ```
+   R_LIBS_SITE=/usr/local/lib/R/site-library/
+   ```
+1. login to https://rstudio.idiv.de
+1. `Session -> Set Working Directory -> Choose Directory` and select the
+   `~/testsession` directory
+1. `Session -> Restart R`
+1. Ensure `.libPaths()` contains only the pathes you expect
+1. Now install the packages you require.
+
+[R Startup Documentation](https://stat.ethz.ch/R-manual/R-devel/library/base/html/Startup.html)
+
+
+## Other servers with R
+
+There are some other servers which have R and RStudio (desktop version)
+installed, but support for R on those is limited. This means you might
+frequently encounter outdated versions of R and associated packages.
+
+Please see
+[Terminalservers](https://git.idiv.de/gsu/gsu-user-wiki/-/wikis/Terminal-Servers)
+for an overiew.
+
+
+### External resources available to you
+
+Be aware that we do not control external resources and that you can connect
+your iDiv network shares directly.
+
+
+### Scientific Computing at the University of Leipzig
 
-- https://github.com/eddelbuettel/rcppapt <- not yet implemente: 
+The UL department [Scientific
+Computing](https://www.urz.uni-leipzig.de/de/fue/sc/) makes a set of [rstudio
+servers](https://www.urz.uni-leipzig.de/de/fue/sc/rstudio/) available as well.
+If you do not have an UL scientific computing account you can simply [register
+one](https://register.sc.uni-leipzig.de/).

rsync.md

+# rsync
+
+rsync syncronizes files. It allows to continue started transfers and incremental syncronisation. It is available on linux and mac. On windows it is available if you followed the instructions for the [Windows Subsystem for Linux](wsl.md). The general syntax of rsync is
+
+```
+rsync [options] source user@destination
+```
+
+## Examples
+
+The user ab12cdef transfers the local file mydata.csv to a data project folder on rstudio1, display the progress and continue unfinished transfers (-P is the shorthand for --partial --progress).
+
+```
+rsync -P mydata.csv ab12cdef@rstudio1.idiv.de:/data/myproject/
+```
+
+
+Recursive (-r) transfer of the local myproject directory to a data project folder on rstudio1:
+
+```
+rsync -rP myproject ab12cdef@rstudio1.idiv.de:/data/
+```
+
+
+The following command does exactly the same. Note that with a trailing `/` on the source no additional directory is created, without it is always created!
+```
+rsync -avhP myproject/ ab12cdef@rstudio1.idiv.de:/data/myproject
+```
+
+If you are unsure about pathes and options, specify the `-n` option for dry run. Add `-v` for verbosity.

scp.md

+# scp
+
+Secure copy (scp) is available on linux and mac. On windows it is available on newer Windows 10 installations and if you followed the instructions for the [Windows Subsystem for Linux](wsl.md). The general syntax of scp (secure copy) is
+
+```
+scp [options] source user@destination
+
+```
+
+## Examples
+
+The user ab12cdef transfers the local file mydata.csv to a data project folder on rstudio1:
+
+```
+scp mydata.csv ab12cdef@rstudio1.idiv.de:/data/myproject/
+```
+
+Recursively (-r) transfer the myproject data directory to a data project folder on rstudio1:
+
+```
+scp -r myproject ab12cdef@rstudio1.idiv.de:/data/myproject/
+```

tmux.md

@@ -18,6 +18,7 @@ Simply invoke `tmux` to start a new tmux session. You will notice a new (customi
 | "           | splits the current pane into two horizontal panes                                     |
 | cursor keys | switches between panes                                                                |
 | !           | break out current pane to new window                                                  |
+| :           | enter tmux command                                                                    |
 
 ## Walkthrough: detaching and attaching
 
@@ -30,3 +31,24 @@ echo "I am a plain terminal"
 tmux ls
 tmux a -t mysession
 ```
+
+## Enter tmux commands
+
+tmux has a command interface as well. You can enter it by pressing prefix followed by `:`. You will see a input line at the bottom of your tmux session.
+
+###  Walkthrough: synchronize the input to multiple panes
+
+```bash
+tmux
+# split pane vertically
+Ctrl+b, %
+# syncronize-panes
+Ctrl+b, :
+setw synchronize-panes
+echo hello
+# de-syncronize-panes
+Ctrl+b, :
+setw synchronize-panes
+```
+
+**Hint** If you keep `Ctrl` pressed you can issue `Ctrl-b,s` to switch synchronisation state as well.

wsl.md

+# Microsoft-Windows-Subsystem-Linux (WSL)
+
+Follow the steps at https://docs.microsoft.com/en-us/windows/wsl/install-win10 to
+1. *Install the Windows Subsystem for Linux*
+2. Install Ubuntu during *Install your Linux Distribution of Choice*
+  - Section *Fall Creators Update and later: Install from the Microsoft Store*
+
+From the start menu you can now launch Ubuntu and get a bash with many useful tools like:
+
+- ssh
+- scp
+- rsync
+- git
+- tmux
+- vim
+
+You can also install additional software inside the WSL using apt. GUI Applications require more work and are not covered here (Google for 'XMing and WSL').
+
+## apt commands
+
+```
+sudo apt update # update the list of packages and package versions
+sudo apt full-upgrade # upgrade all packages to the latest avaialbe version
+sudo apt install htop # installs htop to monitor processes
+```
+