Laura Dietz

• Home

Howto use Linux shell tools

In this class, you will write software that you will be unable to run on your laptop or PC. This is why we provide you with shell access to a large “number crunching” computer.

I suggest that you develop code on your local PC, test it with a little bit of data to make sure it runs. Then you commit your changes to a GitHub repository. Then you ssh into the server, check out your code from GitHub, compile it, and run it.

For those who are struggling with man pages: have a look at https://tldr.sh/

Nohub / screen / tmux

Now, here is the problem: As soon as you loose Internet access to the number cruncher, your process dies. Here is the solution:

1. You ssh into the server
2. You start a screen session by typing screen (alternatively you can use tmux or nohub). You are now “within a screen session”.
3. You start running your code. (I highly recommend to redirect stdout and stderr into a file.)
4. Now you hit “Ctrl-a d” to detach the screen session. This means, you have left the screen session, but whatever you were starting within, continues to run.
5. You can drop your internet connection – stuff in the screen session keeps on running

If you want to check in on your program’s progress, you can re-attach your screen session as follows

1. You ssh into the server
2. You reattach the session with screen -rd (with -d, any shell who was already attach gets automatically detached)

Screen and tmux also have a couple of other features, such as different tabs. I recommend learning more about this tool, because it is really useful.

Bash Tools

Here a list of other bash tools that I regularly use

Basic Shell

• top or htop: for seeing which processes are running and how much memory or CPU they are using
• rsync -a (or scp) copy files over ssh
• ls -lh directory listing with dates, owners, and softlink targets
• tree directory listing of full subtree
• less read a text file page by page
• find . -iname "*.java" lists all files in this subdirectory that match the file name pattern *.java
• killall $processname kills all processes with this name, e.g. all java programs. Check process name with top or ps uax
• history lists the last commands you entered
• tar -czvf archive.tar.gz $file1 $file2 $dir3... create a tar.gz archive containing files or directories. (Alternative tar -cJvf creates a tar.xz archive which takes longer to compress, but yields smaller file

Software development

• git use GitHub
• maven for dependency management of java programs
• anaconda or pyenv for installing python libraries locally
• make or nix to manage pipelines of intermediate steps and data in a reproducible way.

File magic

• ln -s create a softlink to a file — useful if you have different experimentation directories, which all needs the same (large) file.
• cut -f1 $file (or awk) prints the first column of a tab separated file.
• sed -e "s/$search/$replace/g $file > $outfile search and replace in file, then write to $outfile
• sort -u $file sort lines in a file and remove duplicates
• grep -e "$pattern" $file find patterns in file.

SSH / Mosh

To connect to the server, you can use ssh. However, if your Internet connection is unstable or you want to suspend your laptop, use mosh instead.

I also recommend learning how to write bash scripts to avoid having to memorize long command lines.

SSH Port tunnel

As you can only connect to c01 through agate, it is painful to transfer files. Here is a trick: Create an SSH Tunnel! Keep this one running in the background:

ssh -L $fancyPort:c01.cs.unh.edu:22 agate.cs.unh.edu

Now you can scp/rsync to agate.cs.unh.edu:$fancyPort and files will automatically land on c01. (in fact, you are rsyncing to c01, not agate. You can also use this connection for ssh connections. Remember: it is just a tunnel!

Powered by Hakyll