UNIX Lectures.pdf

Lecture 1 - Basic UNIX Commands The general syntax of a UNIX command: command [options] [arguments] The UNIX filesystem has three categories of access: user: user owning the file group: a group of users linked to the file; may or may not include the owner others: any other users Permi...

Lecture 1 - Basic UNIX Commands The general syntax of a UNIX command: command [options] [arguments] The UNIX filesystem has three categories of access: user: user owning the file group: a group of users linked to the file; may or may not include the owner others: any other users Permissions: each file can be (r)eadable, (w)ritable, e(x)ecutable. Permissions are specified for each access category for a total of 3 x 3 = 9 combinations Permissions are often expressed as octal numbers: 4 readable, 2 writable, 1 executable, 5 readable + executable, 6 readable + writable, 0 nothing, 7 all, For files, the permissions mean: r: the file can be read w: the file can be modified/overwritten x: the file can be executed (like a command) For directories, the permissions mean: r: you can list the files in that directory w: you can create or remove a file in that directory x: you can move to that directory Information about commands chown and chgrp for changing owner and group are in the Notes Page Lecture 2 - Scripting 1. Shell and Environment Variables Ls / “/” is an argument Shell variables Set Unset Environment variables Env | more Env | less 2. Configuration Files The shell reads configuration files when it is started. For bash, these files include ~/.profile, ~/.bash_profile, ~/.bash_login, ~/.bash_logout, ~/.bashrc (this last not by the login shell) 3. Bash as a programming environment Some of its features (bash): shell variables command substitution evaluation of arithmetic expressions globbing: words containing special characters *, ?, and [ are regarded as patterns and replaced by matching file names → redirection, piping, functions, built-in commands, control structures and much more 4. Examples Edit text file display.sh as: 1. # This script displays the date, time, username and 2. # current directory.  3. echo "Date and time is:" 4. date 5. echo 6. echo -e "Your username is: `whoami` \n" 7. echo -e "Your current directory is: \c" 8. pwd Run./display.sh : Date and time is: Wed Mar 8 00:01:38 EST 2006 Your username is: massimo Your current directory is: /home/massimo/USP Notes: line 3: echo displays its string/argument and goes to a new line line 4: date displays its output and goes to a new line line 5: echo with no arguments displays an empty line and goes to a new line line 6: echo with –e option interprets special parts of its string/argument; \n tells to add an additional new line line 6: pay attention!: the ` ` delimiters tell the shell to execute command whoami BEFORE the execution of the shell script, capture its output and paste it into the shell script as if you had typed it yourselves. This is called command substitution (it has two different syntaxes) line 7: \c tells echo not to go to a new line Shell scripts: for and if Bash can use control structures such as for (to implement loops) and if (for conditional execution) The syntax of for is: ○ for variable in list of words; ○ do ○ # variable is assigned with a word in each iteration ○ # all your commands go here ○ done The syntax of if is: ○ if condition; # the condition can use either [ ] or test ○ then # commands go here, followed by ; ○ elif # commands can go here, if needed ○ else # commands can go here, if needed ○ fi Example example.sh: ○ #!/bin/bash ○ # Simple bash example for shell variables, for and if ○ echo ○ LIST=`ls` ○ echo $LIST ○ echo ○ for FILENAME in $LIST;  ○ do ○ echo $FILENAME ○ if test $FILENAME == "food2.txt"; # or if [... ] ○ then ○ echo "Food list found!"; ○ fi ○ done Notes: line 4: stores the output of ls into shell variable LIST line 7: for FILENAME in $LIST; (do...; done): ○ scans $LIST for words and assigns one at a time to FILENAME line 10: command test returns true (== 0) or false (!= 0) based on its test outcome; ○ if (...; then...; fi): ○ takes the value returned by test as its argument; if 0, executes then Lecture 3 - Piping and Redirections STDIN, STDOUT, STDERR Command input and output are considered filters Standard input (or STDIN, file descriptor: 0) from the keyboard Standard output (or STDOUT, file descriptor: 1) to the screen Standard error (or STDERR , file descriptor: 2), also to the screen, but with a special meaning: it conveys error messages Piping Piping is the process of redirecting the standard output of one command to the standard input of another one A pipeline is the sequence of one or more commands separated by the character | (pipe, or vertical bar) ○ Example: ○ ls -l | wc -l ○ tells you how many files and directories are in the current directory (NB: plus one, since ls –l produces an extra line) Example with more than two commands: pick the third file in the long listing of the current folder, in appearance order: ○ ls -l | head -4 | tail -1 Count the words in a file: ○ cat food2.txt | wc -w Be aware that it’s only the standard output to be redirected, not the standard error Each command in a pipeline is executed as a separate process. As such, the two commands proceed in parallel: you don’t have to wait for the first to complete for the second to start Redirection Standard input, output and error can also be redirected to files Examples: ls –l > l.txt ○ saves the file listing in long format to file l.txt; if file exists, overwrites it  cat < l.txt ○ redirects the input of cat to come from file l.txt ls –l >> l.txt ○ appends the file listing in long format to existing file l.txt; if file doesn’t exist, creates it Commands and STDIN Many UNIX commands such as cat, head, wc, sort and others expect to be called with at least one argument – a file to read from If they are called without any argument, they read from STDIN (until ^D (end of file) is entered) Example: ○ -bash-4.3$ sort ○ beta ○ alpha ○ (^D) ○ Output: ○ alpha ○ beta Notice the difference between: wc –w food2.txt ○ Possible output: ○ 6 food2.txt ○ the command outputs the file name since it knows it (it is passed as an argument) wc –w < food2.txt ○ Possible output: ○ 6 ○ the standard input is redirected from file food2.txt without the command knowing about it (0 arguments) Predictably: ○ ls -l food2.txt massimo > l.txt ○ outputs this on your screen: ○ ls: massimo: No such file or directory ○ if file massimo does not exist: the standard error is not redirected to l.txt > l.txt ○ easy way to create an empty file (!) Advanced redirection The previous examples can be re-written equivalently with explicit file descriptors: ls –l 1> l.txt ○ standard output (1) to txt cat 0< l.txt ○ redirects the input of cat to come from file l.txt A way to redirect the standard error: ls non_existing.txt 2> l.txt A way to redirect the standard output and error: cat l.txt non_existing.txt > l.txt 2>&1 & is needed, otherwise bash will think it’s a file name Globbing (pathname expansion) Bash scans each word in the command line for the “wildcard” characters *, ?, and [ If it finds one of these characters, it regards the word as a pattern, and replaces it by an alphabetically sorted list of the file names matching the pattern ○ * Matches any string, including the null string ○ ? Matches any single character ○ [...] Matches any one of the enclosed characters Examples Ex 1: ls *.txt ○ lists all the files whose name ends with.txt Ex 2: cat * ○ displays the content of all files Ex 3: ls *.?? ○ lists all the files whose name ends with a. followed by any two characters (example: display.sh) Ex 4: ls food[1-3].txt ○ lists all the files whose name is any of food1.txt, food2.txt, food3.txt Special parameters The shell has several special parameters. These parameters are automatically assigned and can only be read. To read their value, use the $ sign in front of their name: ○ * expands to the positional parameters with which the script (or a new shell) are called (i.e. the script’s arguments) as a single string. The separator (default: a space) can be chosen ○ 1 - 9 each expands to one positional parameter (more than nine is possible) ○ @ expands to the positional parameters, as separate strings ○ # expands to the number of positional parameters ○ ? expands to the exit status of the most recently executed command ○ $ expands to the process ID of the script (or shell) ○ 0 expands to the name of the script (or shell) Arithmetic expansion Operator $(( )) evaluates an arithmetic expression and replaces it with its value Example: ○ echo $((3 * 21 + 2)) outputs ○ 65 It may be nested. Try: echo $(($((3 * 21)) + 100)) Quoting Quoting is a mechanism to remove the special meaning of certain characters or words to the shell Characters such as | & ; ( ) < > space tab ! and a few others must be quoted to remove their special meaning and use them literally Three quoting mechanisms exist: the escape character, single quotes, and double quotes The escape character (\ aka backslash) preserves the literal value of the next character ○ Examples: \|, \&, \\  Single quotes preserves the literal value of each character within the quotes. Cannot be nested ○ Example: 'this is the backslash \' Double quotes are similar, but some characters ($, ` and \) retain their special value ○ Example: "The value of variable A is: $A" Examples 1. What happens here? (easy) VAR="Hello World" In example 1, the double quotes remove the separator meaning from the space character. The string is seen as a single string and assigned to VAR. Without the quotes, only Hello is assigned to VAR and World is mistaken as a command, generating a “command not found error” 2. What happens here? (medium) echo \\'\' In example 2, the first backslash preserves the literal meaning of the next one, which is simply printed to the screen. In addition, the single quotes preserve the literal meaning of the last backslash which is also printed to the screen. The overall output is: ○ \\ 3. What happens here? (hard) echo "\" In example 3, the double quotes cannot remove the special meaning from backslash! Therefore, the first " opens a double quote pair; however, the next \" has literal value and does not close it: the shell stays waiting for another " to close the double quote pair. If entered, the overall output is just the literal double quote: ○ " Appendix: some UNIX filters Cut cut [options] [file(s)] ○ Cut columns out of input (“vertical” cutting, opposed to the “horizontal” cutting of head/tail) ○ Options ○ -c list characters to cut (example: -c 1-72) ○ -f list fields to cut (not universally supported) list can be specified as: 1,5 (one and five); or 1,10-12 (1 and 10 to 12); etc ○ -d field separator ○ Can accept input from STDIN instead of files awk grep grep (global / regular expression / print) ○ grep [options] regex [file(s)] ○ Selected options: ○ -c print only matching lines ○ -i ignore uppercase/lowercase differences ○ -h don't show file names ○ -l only give file names when match is found ○ -n display line numbers ○ -v print lines except those that match sed sort uniq  join paste split tr Lecture 4 - Grep & RegexGrep Regular Expressions Regular expressions are strings made of ordinary and special characters A regular expression is a compact way to represent one or more strings of ordinary characters. First, intuitive examples: 1. Regex p[aui]nt represents strings pant, punt, pint 2. Regex help.. represents any strings of 6 characters beginning with “help” (like help48, helper etc) 3. Regex \d\d represents any strings of 2 digits Grep Unix command grep searches its argument files for lines containing at least one match to a given regular expression (or pattern, for short). By default, grep prints the matching lines to the screen Example: ○ grep foo file.txt ○ prints all the lines that contain a string matching expression "foo" in file "file.txt"; e.g.: ○ There were tons of food at the party ○ I hurt my left foot playing soccer Command Usage Description Options Example grep grep [options] earch for -c – Print only grep -i "pattern" regex [file(s)] patterns in files matching lines. file.txt using regular - -i – Ignore case. expressions. - -h – Don’t show file names. - -l – Only list file names where matches are found. - -n – Display line numbers. - -v – Invert the match, show non-matching lines. Grep Output The default output from grep can be changed Two options:  ○ -c prints the count of the matching lines for each argument file ○ -l prints the name of each argument file containing matching lines Example with option -l: ○ grep –l foo file1 file2 file3 ○ file1 ○ file3 ○ → file1 and file3 contain a match, file2 doesn’t Grep and Regex The first argument of command grep is a regular expression As such, it can contain special characters such as $,*,.,\... always enclose the regular expression between single quotes to prevent the shell from changing it!!! Example: ○ grep –E 'f?' f2 as you expect, will search for lines matching expression f? in file f2. Without quotes, the shell will replace f? with a filename before running the command! Examples cat file1 ○ big ○ bad bug ○ bag ○ Bigger ○ boogy ○ \bad grep -E b.g file1 → The. (period) character matches any single character ○ big ○ bad bug ○ bag Repetition operators How many times? ○ ? The preceding item must be there 0 times or 1 time ○ * 0 or more times ○ regex.* matches 0 or as many characters ○ + 1 or more times ○ regex.+ matches 1 or as many characters ○ {n} n times ○ {n,m} at least n times, but not more than m times Operators () and | ( ) can be used to group a sequence of characters as if they were one Operator I means “this or that” (‘or’ operator) grep -E 'ad|i' file1 → Lines matching 'ad' or 'i' ○ big ○ bad bug ○ Bigger ○ \bad  grep -E ' (ad|i) ' file1 → Lines matching 'ad' followed by a space, or 'i' followed by a space ○ bad bug Bracket expressions A bracket expression is a list of characters enclosed between [ and ] (“square brackets”). It matches any single character in that list For example, regular expression matches any single digit If there is a caret ^ in front of the list, then it matches any character not in the list; for example, [^A-Z] matches any character that is not an uppercase letter Examples -E = range expression grep -E '[aez6]' file1 ○ bad bug ○ bag ○ Bigger ○ \bad grep -E '[a-c]' file1 ○ big ○ bad bug ○ bag ○ boogy ○ \bad grep -E '[^a-zA-Z]' file1 ○ bad bug ○ \bad Anchors: ^ and $ Anchors force the match to occur in certain positions of the input line the caret ^ and the dollar sign $ force the match to take place at the beginning and end of a line (or just before the newline at the end of the line) Moreover, \b forces the match to take place at the edge of a word (only works in some versions!) Examples grep -E '^b' file1 → only lines starting with b match ○ big ○ bad bug ○ bag ○ boogy grep -E 'g$' file1 → only lines ending with g match ○ big ○ bad bug ○ bag grep -E 'u\b' file1 → only lines with a word ending in u grep -E 'u\b' file1 ○ Prints no lines because u doesn’t appear at the end of a word grep -E '^[^a-z]' file1 → Lines matching any character that is not a lowercase letter at the beginning of the line. Please note the different use of the caret! ○ Bigger ○ \bad Grep's exit status Normally, grep's exit status is 0 if any matching lines are found, 1 if not, and 2 if an error occurred So, one could write a simple script like this: ○ #!/usr/bin/bash ○ grep $1 $2 >/dev/null 2>&1 ← suppresses both STDOUT and STDERR ○ STATUS=$? ○ if [ $STATUS -eq 0 ] ​← checks if the exit status was 0 ○ then echo "Found!" ○ elif [ $STATUS -eq 1 ] ← checks if instead the exit status was 1 ○ then echo "Not found!" ○ elif [ $STATUS -eq 2 ] ← checks if instead the exit status was 2 ○ then echo "Error!" ○ fi Matching multiple regex’s If you want to find lines matching multiple regex’s, you can cascade grep commands: ○ grep 'this' file1 | grep 'that' If you want to find files matched by multiple regex’s, you can cascade grep commands using xargs: ○ grep –l 'this' * | xargs grep –l 'that' xargs is a general-purpose Unix command that converts its standard input into arguments for the following command A realistic example: validating email addresses As a final, realistic example, let’s aim to design a regular expression that matches valid email addresses This is a rather complicated problem that can have increasingly complex solutions An intermediate solution: grep -E '^[^@]+@[^@]+\.[^@]+$' (how does it work? Comments in the Note Page) Bash’ main features Bash has many features and a comprehensive description is offered by its man page Main features relevant to scripting: parameters (variables are just parameters denoted by names) metacharacters with special meaning, such as | and & quoting, to remove the special meaning from the metacharacters expansions (see details Appendix B) functions (see details in Appendix C) compound commands such as: for, select, case, if, while, until, { list; }