Introduction to UNIX and the Command Line Interface

John Brunelle

ComputeFest, January 14, 2014

About us

FAS Research Computing

Research Computing: http://rc.fas.harvard.edu/

What is UNIX?

GNU provides free software that runs on top of the base Linux operating system

Organizations like RedHat, Ubuntu, etc. configure and add things their own way and release GNU/Linux distributions

UNIX characteristics

"UNIX is basically a simple operating system, but you have to be a genius to understand the simplicity." -- Dennis Richie, one of the developers of C and UNIX

Command Line Interface (CLI)

"UNIX was not designed to stop its users from doing stupid things, as that would also stop them from doing clever things." -- Doug Gwyn

Accessing Odyssey

Start a terminal -- Mac

On a Mac open Terminal.app

Applications | Utilities | Terminal

 

Start a terminal -- Mac (2)

You will probably want to login to Odyssey for today's workshop:

ssh myusername@login.rc.fas.harvard.edu

Type yes at this prompt:

The authenticity of host 'login.rc.fas.harvard.edu (140.247.139.206)' can't be established.
ECDSA key fingerprint is c0:7b:40:5f:c9:d4:97:6f:33:27:76:8f:5e:b9:25:92.
Are you sure you want to continue connecting (yes/no)? yes

Enter your Odyssey account password:

Password: 

And then six digits displayed in JAuth or Google Authenticator app, no spaces:

Verification Code:

Start a terminal -- Windows (PuTTY)

Let's get started by opening up a terminal connected to a UNIX-like OS

We can use PuTTY (putty.exe), a free ssh client

Download putty.exe, run it, and fill in the Host Name

Start a terminal -- Windows (PuTTY) (2)

Accept any warnings:

Enter your Password, and then Verification code -- the six digits displayed in JAuth or Google Authenticator app, no spaces:

Start a terminal -- Windows (SecureCRT)

You can alternatively use SecureCRT, but you must make the following change:

The terminal / shell

We'll be typing commands at a prompt that looks something like this (on a Mac):

MyComputer:~/Desktop MyName$

or this (on the Odyssey cluster):

[cfest350@rclogin03 ~]$

The thing we're interacting with is called a terminal

The program that's running is called a shell

Shell

Note: there are different families of shells; this is bash (from the sh family) as opposed to tcsh (from the csh family)

Shell commands

Shell commands are entered by typing text after the prompt, followed by the key:

[cfest350@rclogin03 ~]$ whoami
cfest350
[cfest350@rclogin03 ~]$ hostname
rclogin03.rc.fas.harvard.edu
[cfest350@rclogin03 ~]$ pwd
/n/home00/cfest350

The last command, pwd, meaning print working directory, shows us our current location in the filesystem

Shell commands (2)

The filesystem

Just like what you're probably used to, data is kept in files, files are kept in folders, and files and folders can be put in other folders, creating a hierarchy

Terminology and notation:

The filesystem (2)

Terminology and notation (contd.):

More on filesystems and storage

One, big virtual filesystem

But the distinctions between filesystems matter:

When a process interacts with a filesystem, we say it's doing i/o (input/output)

memory is not storage

Example files

Before we move on, let's download some example files so that we have some common material to work on

Make sure you're in your home directory:

[cfest350@rclogin03 ~]$ cd

Download the file examples.tar.gz (\ is the line-continuation character so this fits on the slide, and > is the line-continuation prompt):

[cfest350@rclogin03 ~]$ wget \
> https://software.rc.fas.harvard.edu/training/examples.tar.gz

This is a zipped, tarball archive; we can unpack it like so:

[cfest350@rclogin03 ~]$ tar xvf examples.tar.gz

Options and arguments

[cfest350@rclogin03 ~]$ ls -l ~/examples/
total 363
-rw-r--r-- 1 cfest350 computefest_group     4 Jul 12  2009 aaa
-rw-r--r-- 1 cfest350 computefest_group    71 Jul 12  2009 answers.out
-rw-r--r-- 1 cfest350 computefest_group     4 Jul 12  2009 bbb
...

Option formats

[cfest350@rclogin03 ~]$ ls -alFtr
...
total 428
drwxr-xr-x   2 cfest350 computefest_group     0 Jan 14 06:38 myfolder/
drwxr-xr-x   2 cfest350 computefest_group     0 Nov 11  2010 .gnome2/
-rw-r--r--   1 cfest350 computefest_group    18 Feb 21  2013 .bash_logout
-rw-------   1 cfest350 computefest_group    50 Jan 14 06:03 .Xauthority

Man pages

Shell commands are documented in man pages

Browse the manual for any COMMAND by issuing the command man *COMMAND*, e.g.:

[cfest350@rclogin03 ~]$ man ls

The terminal will fill up with the documentation about the command

Use the apropos command to search find man pages related to keywords

CLI shortcuts and tricks

Ctrl-c

Exercises

  1. Change directories to /tmp and then change back to your home directory (try using an absolute path, a relative path, ~, etc.)
  2. Are there any other users with home directories alongside yours?
  3. List the contents of / and of some of the directories one below that
  4. Run the command ~/examples/bin/ticktock, and kill it once you've had enough

More on next page...

Advanced exercises

  1. List the contents of ~/examples, ordered by time
  2. Execute the command echo 'Hello, World!' twice, followed by the command echo 'Hello, Odyssey!', without typing the word Hello more than once in total (hint: use up-arrow)
  3. Type the absolute path of your home directory using the minimum number or keystrokes (hint: use tab-completion)
  4. Find all the commands that begin with the letter q

Break

Making a place to work today

If you're logged into Odyssey and you know where your group's data directory is, cd into your group's directory now

And let's make a directory to play around in:

[cfest350@rclogin03 ~]$ mkdir workshop
[cfest350@rclogin03 ~]$ cd workshop

Copying files

copy files with the command cp:

[cfest350@rclogin03 workshop]$ cp ~/examples/aaa .
[cfest350@rclogin03 workshop]$ cp ~/examples/bbb ~/examples/ccc .
[cfest350@rclogin03 workshop]$ cp aaa zzz

If last thing is a directory, it copies the things into that directory

Copying files (2)

Moving files

move files with the command mv:

[cfest350@rclogin03 workshop]$ mkdir subdir/
[cfest350@rclogin03 workshop]$ mv aaa bbb subdir/

The same command is used to rename files, too:

[cfest350@rclogin03 workshop]$ mv zzz yyy

Deleting files

remove files and directories with rm and rmdir:

[cfest350@rclogin03 workshop]$ rm subdir/aaa
[cfest350@rclogin03 workshop]$ rmdir subdir
rmdir: subdir: Directory not empty
[cfest350@rclogin03 workshop]$ cp -r subdir subdir2
[cfest350@rclogin03 workshop]$ rm subdir/bbb
[cfest350@rclogin03 workshop]$ rmdir subdir
[cfest350@rclogin03 workshop]$ rm -fr subdir2

Deleting files with rm and rmdir PERMANENTLY DELETES THEM !!!

File attributes

ls -l shows file attributes:

-rw-r--r--   1  cfest350   computefest_group   124   Jun  3 15:26  .bashrc
    |            |       |       |         |           |
permissions     owner  group    size  modifiedtime  filename

File permissions


Use the chmod command to change the mode bits:

For example, create a new file with the touch command:

[cfest350@rclogin03 workshop]$ touch myfile
[cfest350@rclogin03 workshop]$ ls -l myfile
-rw-r--r-- 1 cfest350 computefest_group 0 Jan 14 06:50 myfile

Give your group permission to write to it:

[cfest350@rclogin03 workshop]$ chmod g+w myfile
[cfest350@rclogin03 workshop]$ ls -l myfile
-rw-rw-r-- 1 cfest350 computefest_group 0 Jan 14 06:50 myfile

File permissions (2)

Take away read permission (and write and execute) for everyone else:

[cfest350@rclogin03 workshop]$ chmod o-rwx myfile
[cfest350@rclogin03 workshop]$ ls -l myfile
-rw-rw---- 1 cfest350 computefest_group 0 Jan 14 06:50 myfile

You may be thinking that the letters and positions are redundant, and that's pretty much correct

Thus, you'll see suggestions like chmod 755 to set the mode bits exactly to -rwxr-xr-x (think binary and octal representations of the mode bits)

Changing other file attributes

The touch command is used to change a file's modified time (it will create the file if it doesn't already exist)

The chown and chgrp commands are used to change the owner and group to which a file belongs

Hidden files

The first option described in the ls man page, -a, causes ls to show all files, including those beginning with the . character

[cfest350@rclogin03 ~]$ ls -a
.              .bashrc          .lesshst             .viminfo
..             .gconf           .lsbatch             workshop
.bash_history  .gconfd          .matlab              .Xauthority
...

Filenames beginning with . are Linux's version of hidden files

More on text input

The shell splits the command line on whitespace (spaces, tabs, etc), and many characters have special meaning

Surround text with single quotes, or use the \ escape character, in order to treat whitespace as regular text:

[cfest350@rclogin03 workshop]$ echo  foo      bar
foo bar
[cfest350@rclogin03 workshop]$ echo 'foo      bar'
foo      bar
[cfest350@rclogin03 workshop]$ echo  foo\ \ \ \ \ \ bar
foo      bar

Double quotes work similarly, but will expand variables and a subset of special characters:

[cfest350@rclogin03 workshop]$ echo "my home directory is $HOME"
my home directory is /n/home00/cfest350

More CLI shortcuts and tricks

Use globbing to refer to multiple filenames at once using wildcards

Use brace expansion to automatically type out sequences (some of these only work in bash)

Exercises

  1. Copy the file ~/examples/gpl-3.0.txt to your home directory
  2. Rename it to myfile
  3. Determine its size and when it was last modified
  4. Make it readable by only you
  5. Make a directory named stuff and copy all the files from ~/examples/stuff/ into there
  6. Move all the files in that directory that begin with the letter z up one directory
  7. delete all the files in that directory that end with the number 3

Break

Reading and writing files

Dump the contents of a file to the terminal using cat:

[cfest350@rclogin03 workshop]$ cat ~/examples/gpl-3.0.txt

Use a pager like less to pace the output:

[cfest350@rclogin03 workshop]$ less ~/examples/gpl-3.0.txt
[cfest350@rclogin03 workshop]$ cat ~/examples/gpl-3.0.txt | less

Use the head and tail commands to look at just the start and end of files (tail -f is especially helpful)

reading and writing files (2)

nano is a simple text editor:

[cfest350@rclogin03 workshop]$ nano myfile
^X
Save modified buffer (ANSWERING "No" WILL DESTROY CHANGES) ? y
File Name to Write: myfile

vim and emacs are some fancier editors

I/O redirection

Shell programs usually do i/o with the terminal, using three streams:

  • input:
    stdin
  • output:
    stdout
    stderr

Interactively, you rarely notice there's separate stdout and stderr, but the distinction is helpful when scripting

I/O redirection (2)

Shell commands can be redirected to write to files instead of the screen, or read from files instead of the keyboard

To any command...

In any of the above, replace > with >> in order to append to the files instead of overwriting them

I/O redirection examples

[cfest350@rclogin03 workshop]$ date > date.out
[cfest350@rclogin03 workshop]$ cat date.out 
Sun Jul 12 12:57:02 EDT 2009
[cfest350@rclogin03 workshop]$ zzzzzz > my.out 2> my.err
[cfest350@rclogin03 workshop]$ cat my.out 
[cfest350@rclogin03 workshop]$ cat my.err 
-bash: zzzzzz: command not found

Piping I/O between programs

The output of one command can be fed as input to the next by putting a pipe character, |, between the commands:

[cfest350@rclogin03 workshop]$ ls -a ~ | grep bash
.bash_history
.bash_logout
.bash_profile
.bashrc

(grep is a command that searches for strings of characters)

[cfest350@rclogin03 workshop]$ ls -a ~ | wc -l
19

(wc is a command that counts lines of text (-l), numbers of characters, etc.)

Named pipes and subshells (advanced)

There's also a syntax to create named pipes, which let you use the output of one command as an argument to another command that usually expects a filename:

[cfest350@rclogin03 workshop]$ diff <(ssh odyssey cat ~/myfile) ~/myfile
1c1
< result = 0.56578
---
> result = 0.56413

(diff is a command that shows the lines that differ between text files)

Or you can use the output of a command itself directly on the command line:

[cfest350@rclogin03 workshop]$ cp myfile myfile.backup.$(date +%Y-%m-%dT%H:%M:%S)
[cfest350@rclogin03 workshop]$ ls myfile*
myfile  myfile.backup.2009-11-15T12:20:58

More shell commands

Exercises

  1. Save the output of ls /bin in a file by issuing one command
  2. List the last 5 files in /bin by combining the ls and tail commands with a pipe
  3. Count the number of lines that contain the word free in ~/examples/gpl-3.0.txt
  4. Determine exactly how files out.1 and out.2 (in ~/examples) differ
  5. Find all the files in ~/examples that begin with the letter a
    (hint: use find ~/examples -name X where X specifies what type of names you're looking for; find understands globbing, too, but it has to be passed as a quoted string so the shell doesn't un-glob it)

Break

The shell environment

There are many configurable things about how your shell works, what software is available to you, the default behavior of commands, etc.

The PATH

For example, commands are files in the filesystem, too, and the shell needs to know where to find them; the PATH environment variable controls this

[cfest350@rclogin03 ~]$ echo $PATH
/usr/local/bin:/usr/local/openssh-6.3p1/sbin:/usr/local/openssh-6.3p1/bin:/
usr/lib64/qt-3.3/bin:/lsf/7.0/linux2.6-glibc2.3-x86_64/etc:/lsf/7.0/linux2.
6-glibc2.3-x86_64/bin:/usr/local/bin:/bin:/usr/bin:/n/home00/cfest350/bin

The LD_LIBRARY_PATH is something similar, used by programs that share subroutines with other programs and therefore need to know where to find the shared subroutines

The umask

The default permissions that new files have are controlled by the umask

You can both see the current value and set it with the umask command, either in octal or symbolic form

umask (octal) umask (symbolic) permissions of new files / directories
0022 u=rwx,g=rx,o=rx -rw-r--r-- / drwxr-xr-x
0077 u=rwx,g=,o= -rw-r--r-- / drwx------
0007 u=rwx,g=rwx,o= -rw-rw---- / drwxrwx---
0002 u=rwx,g=rwx,o=rx -rw-rw-r-- / drwxrwxr-x

The umask (2)

Here is an example of setting the umask so that others in your group can write to files and directories that you create:

[cfest350@rclogin03 workshop]$ umask
0022
[cfest350@rclogin03 workshop]$ touch foo
[cfest350@rclogin03 workshop]$ ls -l foo
-rw-r--r-- 1 cfest350 computefest_group 0 Nov 15 18:03 foo
[cfest350@rclogin03 workshop]$ umask 0002
[cfest350@rclogin03 workshop]$ touch bar
[cfest350@rclogin03 workshop]$ ls -l bar
-rw-rw-r-- 1 cfest350 computefest_group 0 Nov 15 18:04 bar
[cfest350@rclogin03 workshop]$ umask 0022

~/.bashrc

Writing scripts

You can collect commands together in a script

Let's create an cd into a new directory:

[cfest350@rclogin03 workshop]$ cd
[cfest350@rclogin03 ~]$ mkdir myscripts
[cfest350@rclogin03 ~]$ cd myscripts
[cfest350@rclogin03 myscripts]$

Use nano to make a text file named colors with the following contents:

[cfest350@rclogin03 myscripts]$ cat colors
#!/usr/bin/env bash
for c in red blue green; do
    echo $c
done

Writing scripts (2)

The current directory is not in your PATH, so the shell cannot find the command by itself:

[cfest350@rclogin03 myscripts]$ colors
-bash: colors: command not found

So you must specify its full path:

[cfest350@rclogin03 myscripts]$ ./colors
-bash: ./colors: Permission denied

And, lastly, give yourself execute permission so that it's allowed to run:

[cfest350@rclogin03 myscripts]$ chmod u+x colors
[cfest350@rclogin03 myscripts]$ ./colors
red
green
blue

Adding a directory to the PATH

You can add the directory containing your script to your PATH:

[cfest350@rclogin03 myscripts]$ export PATH="~/myscripts:$PATH"

Or, if the command echo $0 outputs tcsh, use this:

[cfest350@rclogin03 myscripts]$ setenv PATH "~/myscripts:$PATH"

And then it can be run from anywhere:

[cfest350@rclogin03 myscripts]$ cd
[cfest350@rclogin03 ~]$ colors
red
green
blue

Building and using other software

It usually goes something like this:

Get and unpack the software:

wget http//somewhere/something.tar.gz
tar xvf something.tar.gz
cd something

Configure it to install somewhere in your home directory:

./configure --prefix ~/mysoftware

And compile and install it:

make
make install

System resources

Process control

We already saw how Ctrl-c will terminate a running process

Technically, it sends the process a SIGINT signal, to interrupt it

There are other signals we can send, too

Exercises

  1. Create a script that lists the contents of the current working directory in long format
  2. Create an alias that lists the contents of the current working directory in long format
  3. Add the ~/examples/bin to your PATH and make sure you can execute the ticktock while in various different directories
  4. Look at the contents of your ~/.bashrc file and ponder what it does
  5. Add an alias to your ~/.bashrc file named cdexamples that will cd to ~/examples no matter where you are
  6. Run ~/examples/bin/ticktock again, and practice pausing and resuming it
  7. Run the command sleep 999 in one terminal, and use the kill command in another terminal in order to kill it (hint: use ps to find its process id (PID)

Break

Getting help

We're here to help you with all computing aspects of your research

Just ask, we're happy to help!

Thanks

And don't forget to exit:

[cfest350@rclogin03 workshop]$ exit
exit