Sections marked with green color describe output relevant for your learning diary and count for points towards completing the course.
The paja instructors check your exercises (in the report) on the spot.
Sharpening your tools
A quick refresher on moving around the command shell is provided by Ubuntu:
The manual pages are your best friend not only on this course, but during many other periods of life in general.
Try it by executing the command man followed by the program you want to read the manual of:
man man -> manual of the man command
The solutions are meant to be iterative in the sense that a student builds up on the previous program code to achieve more and more difficult tasks.
Open a terminal to your home directory, and create a
LinuxFundamentals2015 folder using the terminal. Within that directory create a
Put in your report, the command lines for creating the directories
alias command can make life simple by abbreviating very long command line argument lists into shorter commands.
Mask the standard
ls command by creating your own
alias, using your favourite command line arguments for
Put in your report
- the alias of your favorite command line for
- an alias
cmanwhich uses the Chromium browser to read man pages.
Making things permanent is necessary as no one wants to recreate their favourite aliases or other tweaks each time they open a terminal.
There are many ways to achieve it. The three main possibilities are the files
.bash_profile for now.
For more information about the different possibilities, try :
We will soon figure out the difference between login and non-login shells.
Put in your report, a copy of your ~/.bash_profile and make a copy to this week's home folder
Remote shells with SSH
ssh is a very useful utility. One of its uses is to start remote sessions to hosts running a variant of the ssh daemon.
One can use ssh either with passwords, private and public keys, or a combination thereof. The last variant is the most useful.
Look at the list of Ukko computing cluster nodes: http://www.cs.helsinki.fi/en/compfac/high-performance-cluster-ukko
Choose your favourite node from the list and log on to it with
Put in your report, a copy of the alias command to log on to your favourite Ukko node with ssh
Remote invocations with SSH
Instead of starting a login shell,
ssh can be used to just input one or more commands and return the output.
ls on your favourite Ukko node without starting a login shell.
Put in your report the command line for invoking
ls without starting a login shell
Is the output in the same format as on your own host?
Private and public keys
By now, it should be getting tedious to input your passwords to
ssh on every login. Let's set up private and public keys.
Note: on department, ssh logins to for example shell and users (.cs.helsinki.fi) work without asking password via KERBEROS. However, UKKO nodes do ask password, and we'll be using UKKO so let's fix this.
ssh-keygen will create your own set of keys in the
~/.ssh subdirectory. They are typically named
id_rsa.pub. You will need to do this only once.
Remember to protect your private key with a passphrase that is at least as strong as your department login's password, because this key is equivalent to your login and password.
The contents of the client computer's
~/.ssh/id_rsa.pub go into the server computer's
~/.ssh/authorized_keys. Do not add extra line breaks when inserting the key!
There are many guides on the net which can be read for hints:
After this, you can do
ssh -i ~/.ssh/id_rsa ukko-node.hpc.cs.helsinki.fi. If your private key had no password,
ssh will not ask for it.
To be honest, the
-i ~/.ssh/id_rsa part is redundant, since that is the default name for the private keyfile anyhow. You can omit it in the future.
Hint: be very careful with the permissions of the
.ssh directory and
id_rsa file. See the FILES section of
man ssh for details.
What about the passwords?
Using a password for the private keyfile is crucial. However, a user does not wish to be prompted for the private key's password on every ssh login. This would be exactly as tedious as using passwords for the login in the first place!
The solution is to set up a SSH agent that handles your private keys. It will ask for your password only once, and then remember it for the duration of your session.
If you did not add a key to your
~/.ssh/id_rsa in the last step, you can use
ssh-keygen -p -f ~/.ssh/id_rsa to add (or change) the passphrase. You can also check the contents of
id_rsa to see if it's encrypted.
ssh-agent must be run to manage your private keys. It's output is a list of shell variables to be set. By default, Ubuntu runs
ssh-agent for you, so you don't have to worry about this.
ssh-add must be run to add your private key from
ssh-keygen to the store managed by
ssh-agent. Hint: if you chose the default file names
ssh-add will discover them automatically.
ssh-add -llists keys managed by ssh-agent.
Now paste your private key's password into... Just kidding! You should never do that. Never, ever, and then some more.
Put in your report, the sequence of commands that need to be entered to get a password-prompt-less login to Ukko
Ubuntu handles the ssh-agent key store very nicely. Ubuntu integrates stored ssh keys into it's keyring. Open the GUI program "Passwords and Keys" and find your added ssh key from there.
Log out and back in to the machine you are on. Connect to Ukko. Keyring should (graphically) ask you to unlock the saved key, asking the passphrase for it. If you want, you can change the key to be automatically unlocked on login, which causes it to be never asked again.
The right file, at the right time
It's now time to split your
alias invocations into their own file. This file is named
~/.bash_aliases by convention.
Move all lines beginning with
alias from your
~/.bash_profile files into
~/.bash_aliases. This file should not begin with a she-bang line, or
What we now want to do is to invoke the
alias commands on every login and non-login shell session. For this to happen we must:
~/.bashrcfor *non-*login shells
~/.bash_profile and insert the above snippet in order to call
Make sure to update your report with the added lines.
The key difference between login and non-login shells can be demonstrated using
You can try out the difference by removing the snippet invoking
~/.bash_aliases from one of the bash configuration files and then invoking
ssh for a login or directly by using the
Hint: when you call a command like
ssh ukko160.hpc.cs.helsinki.fi, the host operating system actually invokes
ssh ukko160.hpc.cs.helsinki.fi "bash -" if your default shell is Bash. Check
echo $SHELL on the target host to verify this.
Present an example where the output of
ls differs depending on how you have invoked
Shared home directories
By now, you might have noticed that in the Department's Linux system, the home directories are shared. Technically, it's done through the Network File System protocol, or NFS for short. However, Ukko nodes have their own shared directories, accessible from the Department's Linux system at /cs/work/home/$USER/
Go on, try it out. We know you want to. Create a file called
hostname.txt in your subdir by doing
hostname > ~/LinuxFundamentals2015/Week1/hostname.txt. Copy the file to /cs/work/home/$USER/. Then login to your favorite Ukko node and use
cat to print the contents of that file.
Do this with
ssh. Show in your output that
ssh does not ask for a password.
Can you both produce and print the contents in one non-login
ssh session? Hint: commands are separated with semicolons (;).
This is tremendously helpful, since it makes copying files between hosts mostly redundant. In many cases, things are not this simple.
Between hosts which do not share their home directories, the best way to copy files is through
rsync. It is a very powerful utility, specially for backups, since it can detect changes between a source and a destination file, and then send only the changed portions.
In the Dept. network, we need some content to actually see the speed improvement. Let's copy a sizeable chunk of files.
The Exactum-kamera keeps watch of your Dept's home building. It saves one image every hour, on the hour. These archives go back many, many years, and can be browsed if one knows where to look. I'll tell you, but let's keep this our secret:
http://www.cs.helsinki.fi/u/tkt_cam/2015/09/30 is the web URL for Wednesday, September 30th, 2015. You could go there and click on each link to view the pictures, or you could just copy the files over the NFS system. Two alternatives:
cp --archive /home/tktl-csfs/home/tkt_cam/public_html/2015/09/30 ~/LinuxFundamentals/Week1/Wednesday.2015.09.30
rsync --archive /home/tktl-csfs/home/tkt_cam/public_html/2015/09/30/ ~/LinuxFundamentals/Week1/Wednesday.2015.09.30
Please be very thorough with those commands.
Experiment by running
rsync on either a previously empty subdir
Wednesday.2015.09.30 and one previously populated with
cp. Show how
rsync's output changes. Explain what's going on. Try adding the
--stats argument to
Time and Date
The commands from the previous exercise are fine, but very static. They are good for that one day only.
man date and experiment with its format parameters. For example, try
Print today's date in the format equivalent to 'Wednesday.2015.09.30'. Show the command and it's output.
You can use backquotes to evaluate a command into a string
Create a script
rsync-todays-Exactum-cam.sh that uses
echo to print a
rsync command with the correct paths for the current date. Paste the code into your diary.