Unix All Notes PDF

1. CSC 209 UNIX Tools Hierarchical Diagram Showing Folders and Files Created in Asgn 1 / home jdoe1 1 2 3 4 csc209...

1. CSC 209 UNIX Tools Hierarchical Diagram Showing Folders and Files Created in Asgn 1 / home jdoe1 1 2 3 4 csc209 public_html asgn01 asgn02 csc209 6 5 Files Created: 1-4: Hidden files –.cshrc.login.bashrc.bash_logout 5: index.html file, an html file that should be properly edited 6: hello.bash, a Bash script file 1  1. Login and Bash Initialization Sign on as a new user – the user needs Net-ID (userid, loginid, username) and password Verify password and establish user’s individual and group identity Establish login (home) directory and shell Announce presence of email (traditional) Adapt to characteristics of the terminal Publish message of the day (motd) (traditional) Start command interpreter (shell) o Perform user-specified initialization (code) o Shell will place a prompt and wait for the user to type a command 2  Password File Historically /etc/passwd file specifies all the required information to go through the login o Each line in /etc/password file represents one user o Each line contains seven pieces of information separated by colon (:) ▪ Userid ▪ Encrypted password hash ▪ UserID number (uid) ▪ GroupID number (gid) ▪ Description ▪ Login (home) directory ▪ Login shell For security reasons, modern systems use /etc/shadow file or other mechanisms 3  Shell Initialization Files At Brockport the C Shell is assigned as the default login shell for all users o At start up a C Shell executes.cshrc and.login initialization files o In Assignment 1, we set up the.login file to switch to bash Initialization files for Bash o At login, executes /etc/profile and then executes ~/.bash_profile o Since a non-login bash uses ~/.bashrc as the initialization file, ~/.bash_profile often invokes ~/.bashrc o It is convenient to place user-specific umask, aliases, etc., in ~/.bashrc At logout, a login bash executes ~/.bash_logout 4  Sample ~/.bashrc File #.bashrc # Source global definitions if [ -f /etc/bashrc ]; then. /etc/bashrc fi # User specific aliases and functions alias cls='clear' alias h='history' alias lo='logout' # Control history parameters HISTSIZE=60 HISTFILESIZE=50 # Control default access permission umask 077 # Protect files in redirection set -o noclobber # Set shell prompt PS1='[\!] [\u@\h:\w]\$ ' 5  1. What is an Operating System? General Purpose Electronic Digital Computers Hardware, Software Application software o Word processing, spreadsheet, database management systems, graphics applications, … System software o Operating system, compilers, assembler, editors, debuggers, … Operating System (OS) o System software in-charge of the entire computer system o Examples ▪ MS-DOS, Windows 11, macOS 14, UNIX, Linux, Android, iOS o Goals ▪ Providing a convenient working environment for all users – programmers as well as end users (ability to create and organize files, manipulate them in various ways) Command Line Interface (CLI) or Graphical User Interface (GUI) ▪ Efficient management of the resources 6  Brief History of UNIX MULTICS project at MIT, Professor F. Corbato, early sixties UNIX project at Bell Laboratories (AT&T), Ken Thompson and Dennis Ritchie, 1969 o UNIX System Laboratory (USL) UNIX extensions and communication protocols (TCP/IP) at University of California, Berkeley, eighties o Berkeley Software Distribution (BSD) Major flavors o AT&T System 5 Release 4 (SVR4) ▪ USL (AT&T) → Novell → Santa Cruz Operations (SCO) o BSD 4.4 Products from vendors o Sun OS and Solaris from Sun, AIX from IBM, HP-UX from HP, AUX from Apple Linux for PCs, Linus Torvalds, 1992 GNU/Linux (open source and free operating system) o Linux distributions – Red Hat Enterprise Linux, CentOS, Fedora, Ubuntu, … o Graphical user interfaces – GNOME, KDE desktops o Linux distribution – Linux kernel, GNU tools, desktop managers, installation and package management systems 7  UNIX/Linux Linux is a multi-user, time-sharing, multi-tasking operating systems Linux OS = Kernel + Shell Kernel is the real core of the operating system, doing most of the work Shell is the user interface or the command interpreter Bourne Shell, C shell, Korn Shell, TENEX C shell, Bourne- again Shell (Bash) Features Hierarchical file structure with mountable files I/O devices supported as special files Simple user interface, can be easily augmented System designed for programmers – flexible and powerful Portable, written mostly in C Wide free distribution of source code of early version First multi-platform operating system Linux is case sensitive 8  Simple Linux Commands date Print date and time who Report who is logged in whoami Display user id logout Log out pwd Print working directory mkdir d Make directory d rmdir d Remove directory d cd d Change to directory d ls d List entries in directory d chmod mode d Change protection mode of d Protection mode may be expressed in octal form; view the protection mode Protection Mode r w x - - x - - - Binary 1 1 1 0 0 1 0 0 0 Octal 7 1 0 9 https://www.geeksforgeeks.org/permissions-in-linux/ Permissions in Linux (modified by TM) Difficulty Level : Easy Last Updated : 09 Jul, 2021 Linux is a multi-user operating system, so it has security to prevent people from accessing each other’s confidential files. Introduction When you execute an “ls” command, you are not given any information about the security of the files, because by default “ls” only lists the names of files. You can get more information by using an “option” with the “ls” command. All options start with a ‘-‘. For example, to execute “ls” with the “long listing” option, you would type ls -l When you do so, each file will be listed on a separate line in long format. There is an example in the window below. There’s a lot of information in those lines. 10 1a. The first character will almost always be either a ‘-‘, which means it’s an ordinary file, or a ‘d’, which means it’s a directory (a special kind of file). 1b. The next nine characters (rw-r–r–) show the security – the access permissions. 2. The next column shows the number of links to the file. 3. The next column shows the owner of the file. My userID is “aditya314”. 4. The next column shows the group owner of the file. In my case I want to give the “aditya314” group of people special access to these files. 5. The next column shows the size of the file in bytes. 6. The next column shows the date and time the file was last modified. 7. And, of course, the final column gives the filename 11  2. Access Permissions (modes) for Files Each user on the Linux system has a unique identity; each user is authenticated at the time of login Users are grouped o Groups were disjoint in early days; not so in recent implementations Output of an ls command with long option drwx--x--x 2 tmullins home 4096 Jan 27 17:54. drwxr-xr-x 23 tmullins home 4096 Jan 26 11:14.. -rw------- 1 tmullins home 23 Jan 26 12:38 test Access permissions are specified as three sets of three permissions (read, write, execute) by the owner of the file Nine bits of information o rwx for user (owner) of the file o rwx for group members, other than the owner o rwx for all others, other than group members and the owner Who (Access Level) User/Owner Group Other Protection Mode r w x - - x - - - Binary 1 1 1 0 0 1 0 0 0 Octal 7 1 0 In the above, the owner can read, write, and execute; group members can execute; others have no permission For a directory, execute permission indicates that you can reach that directory and traverse through it 12  What does all this mean? Ordinary file Permission on the file (Note additional permissions on parent directories may be needed for some commands) Action How Permission Needed Display cat, more, head, tail, od read Rewrite cp, redirection write Execute the By typing its filename or execute file pathname Edit the file pico, nano, emacs read, write Copy the file cp read Move the file mv Delete the file rm Link to the file ln 13 Directory file Permission on the directory (Note additional permissions on parent directories may be needed for some commands) Action How Permission Needed Move to a cd execute directory Traverse cd execute through dir List the ls read directory Long list the ls –l read, execute directory Create new cp, mv, redirection write and entry execute Delete existing rm , mv write and entry execute Delete the rmdir directory 14  Default Access Permissions Default access permissions for files is 666 and directories 777 unless a umask is specified umask (user mask) o Can be checked with the command umask o Specifies the permissions that are taken away We have set the umask in.bashrc file as 077 o As a result the default access permissions for files we create is 600 and directories 700 How to change mode? chmod mode filename... Absolute specification of mode in octal chmod 751 file1 Relative specification of mode chmod ug+x file1 chmod go-r file1 15  Superuser (root) The system administrator is known as superuser or root o Note the use of the word root here (do not confuse with root directory) Access permission restrictions do not apply to superuser o He/she can access any file/directory o Create/delete file/directory o Change ownership, group ownership 16  2. Format of Linux Commands command_name [option] … [argument] … Options are single character long and preceded by hyphen [arg] indicates that it is not necessary to specify an argument … indicates more than one of the preceding is possible Examples cmd_name options arguments wc -l –w file1 file2 file3 wc -lw file1 file2 file3 echo -n Hello, World! echo -n "Hello, World!" cat file1 cp ~tmullins/.login./.login pwd 17 Handling errors while typing commands cntl-h or Backspace to delete previous character cntl-w to delete the previous word cntl-u to delete the entire command Use of other control characters cntl-c to abort or interrupt the execution cntl-z to temporarily suspend execution cntl-d to terminate keyboard input On-line help man [section] cmd_name_for_help_needed man 1 cat man man 18  Hierarchical File System Linux organizes the files and directories created by various users in a hierarchical structure (tree) Towards the end of the semester, we will see that the structure is an acyclic directed graph, not necessarily a tree Ordinary files and directory files Top-level directory is called the root and is denoted by / Parent-child relationship Login or home directory for each user cd command to move around Current working directory pwd command to determine current working directory Notations:. current directory.. parent directory ~ login (home) directory (in csh, tcsh, bash) mkdir, rmdir commands to create and remove directories ls command for directory listing hidden files file name begins with a period.login.cshrc.bash_profile.bashrc 19  Absolute and Relative Path Names Many Linux commands require naming files or directories as one or more arguments A file or a directory can be specified by giving its absolute path or its relative path An absolute path specifies how to reach the file or directory of interest starting from the root (/) of the file system o Remember that ~ is simply a notation for the login (home) directory; starting from ~ is also considered an absolute path A relative path specifies how to reach the file or directory of interest starting from the current working directory An absolute path specification starts with / or ~ A relative path specification does not start with either / or ~ Suppose I wish to refer to.login file under my login directory: Absolute path: (also called a full pathname) /home/tmullins/.login or ~/.login Suppose I am currently in asgn02, which is under csc209, which in turn is under my login directory: Relative path:../../.login 20  File Manipulation Commands Concatenate files and output to standard output (normally screen) cat file1 file2 file3 Specification of each file may involve relative or absolute path cat../../.bashrc ~/.bash_logout Display a file on standard output (normally screen) cat /etc/passwd cat –n../asgn01/hello.bash Octal dump of a file showing all characters (including control characters) od file1 file2 Each byte is displayed as three octal digits (little endian on our machine) Use option –b or –c to make better sense od –c file1 The left column indicates byte offset in octal. The last line indicates file size (number of bytes) in octal 21  For a large file, display one screen at a time more file1 Press enter (return) for one more line Press spacebar for one more screen Type q to quit (exit) An alternative is the less command Display top n lines head -20 file1 Display bottom n lines tail -20 file1 Compare files and show location of first difference cmp file1 file2 Compare files and show all differences diff file1 file2 Check the calendar, here for all of one year cal 2001 22  File System Commands Edit a file (there are a number of editors – pico, nano, emacs, …) pico file1 Copy file1 as file2 cp file1 file2 Note file2 will be overwritten if it exists already Use option –i to check interactively, if necessary. Copy file1 into a directory dir1 cp file1 dir1 Move file1 into a directory dir1 mv file directory Rename (and move) file1 as file2 mv file1 file2 Remove files (very dangerous – files are lost forever) rm file1 file2 rm -i filenames 23  Directory Listing List specific files (whose location may be specified with absolute or relative path) ls file1 file2 file3 List all non-hidden entries in dir1 ls dir1 If no directory is specified, current working directory is assumed List all hidden, non-hidden entries in dir1 ls –a dir1 List in long format all non-hidden entries in dir1 ls -l dir1 List in long format the directory dir1 alone, and not the entries in dir1 ls –ld dir1 Note the difference between the last two commands 24  Directory Manipulation Commands Create a new directory mkdir dir1 Remove a directory rmdir dir1 A non-empty directory will not be deleted Recursive removal of a directory (removes the directory and everything below – very dangerous) rm –r dir1 Command to move to dir2 cd dir2 Print current working directory pwd Notation. Current working directory.. Parent directory ~ Login (home) directory 25  3. Bash Special Characters I/O redirection > < & | Pipe | & Variable substitution $ : [ ] History substitution ! : ^ / File name expansion * ? [ ] { } ~ Command termination \n ; & Command output substitution backquote Word separation blank Quoting \ ' " 26  3. Shell Substitutions and Quoting Shell is a command interpreter (user interface) Performs input processing in this order o History substitution o Alias substitution o Variable value substitution o Command output substitution o File name substitutions Three ways of quoting to avoid some of the substitutions o Backslash o Single quote o Double quote 27  History Substitution Substituting previously issued shell commands Shell variables HISTSIZE and HISTFILESIZE, usually set in.bashrc file play an important role HISTSIZE=60 HISTFILESIZE=50 There should be no spaces around equal to symbol HISTSIZE is the number of commands kept track of during a session. HISTFILESIZE is the number of commands saved in the history file at logout History is saved in ~/.bash_history file history command can be used to view the list of previous commands Each command in the history has a history event number History expansion is activated by the use of ! (bang) character General format event [:word designator] [:modifier... ] 28  Common Use Specification Meaning !n The event with the sequence number n !-n The nth previous event !! The last event (same as !-1) ^str1^str2 The last event, with the string str1 replaced by str2 !-n:s/bb/dd/ The nth previous event and with the string str1 replaced by str2 29  [tmullins@holly2:~]$ cd csc209/lecture [tmullins@holly2:~/csc209/lecture]$ pwd /home/tmullins/csc209/lecture [tmullins@holly2:~/csc209/lecture]$ echo red green blue red green blue [tmullins@holly2:~/csc209/lecture]$ !51 cd csc209/lecture bash: cd: csc209/lecture: No such file or directory [tmullins@holly2:~/csc209/lecture]$ !-2 echo red green blue red green blue [tmullins@holly2:~/csc209/lecture]$ !! echo red green blue red green blue [tmullins@holly2:~/csc209/lecture]$ ^blue^violet echo red green violet red green violet [tmullins@holly2:~/csc209/lecture]$ !56:s/red/orange/ echo orange green blue orange green blue [tmullins@holly2:~/csc209/lecture]$ 30  Alias Substitution Shorten the name, rename, or redefine existing commands alias cls='clear' alias h='history' alias lo='logout' alias rm='rm -i' There should be no spaces around equal to symbol Find aliases alias displays all aliases alias lo displays specific alias Undo aliases unalias h Aliases are usually provided in ~/.bashrc file Potential problems Alias string involving history substitution Repeated substitutions leading to loops 31  Shell Variable Value Substitution Shell variables o No need to declare variables – all are string type ▪ Arithmetic can be done when needed o Predefined o User defined Shell variable name preceded by $ indicates its value firstname=Jane lastname=Doe echo $firstname echo "My name is $firstname $lastname" echo "My userid is $USER" Display all current variable values set Remove a specific variable unset firstname 32  Command Output Substitution The command is enclosed in backquotes dir=`pwd` The pwd command is executed and the resulting output replaces the right hand side of the assignment dir=`pwd` cd / ls –l cd $dir The sequence of commands will allow switching to root, do a directory listing in long format and return back to the earlier directory 33  File Name Expansion Shell wildcard symbol * matches 0 or more characters in the filename (except. at the beginning of the name) ls -l data* will perform long listing of all files in the current working directory whose name begins with data Suppose data1, data3, and test2 are three files in the current working directory The shell replaces data* with data1 and data3 as if the user typed ls -l data1 data3 Most dangerous command rm * will delete all non-hidden files in the current working directory Shell symbol ? matches exactly one character Quote the filename if you do not desire filename wildcard substitution 34  Special Characters and Quoting Three ways quoting Backslash, single quote, double quote Backslash Quotes or escapes the next character, i.e., the character after \ loses its special meaning to the shell; i.e., preserves its literal value Using \ at the end of the line continues the shell command to the next line echo * will print all non-hidden entries in the current working directory; same effect as the ls command echo \* will print the asterisk on the screen echo Mon Tue two arguments echo Mon\ Tue one argument Single quote No substitution is allowed echo '!$*' Double quote history substitution, shell variable value substitution, and command output substitutions are allowed 35  4. Redirection and Pipe Standard Files Standard input o Normally bound to the keyboard o File Descriptor Number 0 Standard output o Normally bound to the monitor screen o File Descriptor Number 1 Standard error o Normally bound to the monitor screen o File Descriptor Number 2 Consider the following situation: file1 contains one line: Hello Tim file2 does not exist file3 contains one line: Hello Sally cat file1 file2 file3 Hello Tim cat: file2: No such file or directory Hello Sally The first and third lines are meant to go to standard output, the second line is meant to go to standard error 36  Input, Output Redirection and Pipes Input redirection o Take standard input from a file instead of keyboard o Denoted by < symbol wc < data01 Output redirection o Send standard output to a file instead of monitor o Denoted by > symbol wc -l data01 > line Note the file named line will get wiped out first, unless the noclobber option is set Output append o Append standard output to a file o Denoted by >> symbol wc -l data02 >> line Redirect standard error cat file1 file2 file3 2> error o Output redirection can be written as 1> Redirect both standard input and standard output sort < infile > outfile 37  Redirect standard output and standard error to same file cat file1 file2 file3 > file4 2>&1 Temporarily override noclobber option cat file1 file2 file3 >| file4 Pipe o Connect the standard output of a command to the standard input of another command o Denoted by | symbol cat file1 file2 file3 | wc -l Here the output of the cat command is connected through the pipe to the input of the wc command Concatenate the contents of the three files file1, file2, and file3 and feed the output to the wc command as input. The resulting output is the total number of lines in the three files. 38 5. Linux Filenames & Shell Substitutions Linux considers a file to be merely a sequence of bytes In early UNIX versions, filenames were at most 14 characters o Long names (up to 255 characters) are now permitted o Filename extensions are not required in Linux Some applications may assume a line structure for a file o End of line is marked by the newline (\n) character o Linux does not use any special character to mark the end of the file Some applications may require filenames to begin or end in a particular way o For example the C compiler expects the program source file name to end with.c Shell wildcard symbol * matches 0 or more characters in the filename (except. at beginning of name - hidden file) ls -l data* will perform long listing of all files in the current working directory whose name begins with data Suppose data1, data3, and test2 are three files in the current working directory The shell replaces data* with data1 and data3 as if the user typed ls -l data1 data3 39  Most dangerous command rm * will delete all non-hidden files in the current working directory Shell symbol ? matches exactly one character Quote the filename if you do not desire filename wildcard substitution Three forms of quoting o Use of backslash – quotes only the next character o Use double quote – quotes all characters inside the pair of double quotes; prevents file name expansion, but allows shell variable value substitution, history substitution and command output substitution o Use single quote – quotes all characters inside the pair of single quotes; prevents all substitutions 40  6. Linux directories, inodes, links Each Linux file system has a data structure named inode o An array of inodes, each with a unique number For each file or directory, there is a unique inode in which all the properties of the file or directory are maintained Linux directory has a sequence of entries o Each entry contains a file or directory name and its associated inode number What properties are maintained in the inode? o type (ordinary file, directory, symbolic link file, …) o access permissions o link (reference, pointer) count o ownership (uid), group (gid) o size in bytes o time of last modification of file (mtime) and (ls –l file) o time of last access to file (atime) and (ls –ul file) o time of last modification to inode (ctime) and (ls –cl file) o location on disk Shell commands ls -i filename inode number ls -l filename file modification time ls -ul filename file access (use) time ls -cl filename inode change time 41  Linux Links How can the number of links (references, pointers) to a file be more than 1? o Refer to the same file from more than one directory, or may be from the same directory with two different names o The file system is no longer a tree Hard link ln oldpathname newpathname New entry for the same file – same inode number o increment link (reference) count o allowed only within the same file system o issues ▪ incorrect disk usage statistics ▪ deleting files – a file is physically deleted only when the link count reduces to zero ▪ What permissions are needed to link to a file – none Another user may place a link preventing you from physically deleting your file Solution in modern systems o Symbolic link ln -s oldpathname newpathname New entry for a new symbolic link file that contains the path to the linked file o does not increment the link count to the linked file o allowed over other file systems as well 42 43  6. Shell Working and Process Control Display prompt Receive and parse command Perform input processing o history, alias, variable value, command output, file name wildcard substitutions Search for command o PATH specifies the order in which directories must be searched for a command, if only name is given Carry out the command o Built-in command – shell has necessary code, e.g., cd, exit, logout, … o External command – shell creates a child process, e.g., ls, cp, rm, …, and other user executables When to display next prompt? o Foreground job ▪ Wait till command is done o Background job indicated by & at the end ▪ No need to wait for command to be completed 44  Process Control Commands Process – Program in execution o Each process has a unique PID Use ps command to identify processes ps Processes from this terminal ps –u jdoe1 Processes from all terminals To terminate a process, use kill command kill 1235 Terminate process with this PID kill -9 1235 Sure way to Terminate Foreground and background jobs o A foreground job can take keyboard input and output on terminal o At anytime you can have one foreground and several background jobs o Each job may involve one or more processes Use & at the end of the command line to set up background job sort sortedfile & sleep 1200 & A foreground process can be suspended and placed in background by typing control-Z 45  The command jobs will list all background jobs – suspended or executing Use fg command to move a background job to the foreground fg %2 46  Ext2-inode.svg By timtjtim - Own work, CC BY-SA 4.0, https://commons.wikimedia.org/w/index.php?curid=75836001 Other links for further glimpse into Linux File Systems: https://linuxhandbook.com/tag/explain/ Everything You Need to Know About inodes in Linux What is inode in Linux? What is it used for? Why is it important and how to check inode in Linux? This guide explains all the important aspects of inodes. Apr 20, 2022 — Eric Simard https://www.geeksforgeeks.org/inode-in-operating-system/ Inode in Operating System Last Updated : 28 Jan, 2024 https://www.howtogeek.com/465350/everything-you-ever-wanted-to-know-about-inodes-on-linux/ By Dave McKay Published Jan 21, 2020 47  7. Locating a File in the File System Locate one or more files in the file system and perform some operation o Use find command Typical form find directory criterion action o Inspect the directory and all its subdirectories o Select all entries (files and directories) that meet the criterion; if no criterion is specified, select all entries o Perform the required action; if no action is specified, print the path name from the directory Command Directory Criterion Action find ~/csc209 -name hello -print find ~/csc209/asgn05 -links +2 -delete 48  Selection criterion o May involve name, type, permission, number of links, user (owner), group, size, mtime, atime, ctime, … o Selection criterion can involve logical conditions o If the criterion uses shell special characters, it must be quoted (use single quote or \ as appropriate) o For numerical parameters ▪ +2 means greater than 2 ▪ 2 means exactly 2 ▪ -2 means less than 2 Examples find. Print the path name of all entries in the current working directory and below. Since no criterion is specified all entries are selected; since no action is specified, path name is printed find. -name '*.c' -print Print the path name of all entries whose name ends with.c in the current working directory and below. Here * is shell file name wildcard symbol, which is a special character. Hence quote the criterion. It could have been done as \*.c [it perhaps works without treating * as special character – something has changed in OS environment] 49 find ~ \(-name core -o -name a.out\) -exec rm \{\} \; Execute the action rm on all entries, i.e., delete all entries in the login directory (~) and below whose name is either core or a.out in those directories. The command is supposed to look like find ~ (-name core -o -name a.out) -exec rm {} ; The exec requires a semicolon at the end. Further, the notation {} allows the action to be performed on the selected entry (, ), {, }, ; are all special characters. Hence we need to quote them with backslash. Also the action -exec rm {} ; can be replaced by -delete Thus, we could have written the above as find ~ \(-name core -o -name a.out\) -delete 50 find ~ -exec chmod go-rwx \{\} \; Change permissions for all entries by removing read, write and execute for all group members and others in the login directory and below. chmod go-rwx file1 removes read, write, and execute for group members and others for the specific file file1. find ~ -size -200 -atime +365 -print Print the path name for all entries of size less than 200 KB and whose access time is more than 365 days ago in the login directory and below. Observe that placing two criteria one after another implied and of those two criteria. 51  7. Regular Expressions Compact way to describe a set of strings based on common characteristics Consider the following English language verbs: sing, sting, spring, … They all begin with the letter s and end with the three character sequence ing. But 0 or more characters may be present in between. Identify more verbs that fit this pattern The concept is similar to shell file name wildcards, but we use a different set of meta characters 52  For simplicity, assume the ASCII Character set as the alphabet Meta characters o Characters that have special meaning in regular expressions. * + ? ( ) [ ] ^ $ | \ o If any of the meta characters appear in a regular expression, quote it using backslash (\) The regular expression wildcard symbol is period (.), and not asterisk (*) o It stands for any single character in the alphabet other than the newline (\n) character The asterisk (*) indicates 0 or more repetitions of the preceding. For example, the pattern s.*ing stands for all strings that begin with the letter s and end with the three character sequence ing, but has 0 or more characters present in between The pattern abc*d stands for abd, abcd, abccd, abcccd, abccccd, … Sometimes, the repetition applies over more than one character; use parentheses The pattern a(bc)*d stands for ad, abcd, abcbcd, abcbcbcd, … 53  Some meta characters o Use of special characters (different from shell wildcards). any single char except \n ( ) enclose a regular expression (re) * repeat 0 or more times the preceding re + repeat 1 or more times the preceding re ? repeat 0 or 1 time the preceding re [ ] indicates one of the enclosed char [^ ] not one of the enclosed char ^ beginning of a line $ end of a line | or Note The regular expression a+ stands for one or more repetitions of a, i.e, a, aa, aaa, … The regular expression a* stands for 0 or more repetitions of a, i.e., null string, a, aa, aaa, … The regular expression a+ is equivalent to aa* 54 Use of Regular Expressions in grep Display all lines in the file data, with line numbers that has a word that begins with s and ends with ing, and has 0 or more characters in between. Do the search for the pattern case insensitive. grep –inw 's.*ing' data Display all lines with line numbers that end with a question mark (?) grep -n '\?$' data Display all lines with line numbers that begin with the character t, case insensitive grep –in '^t' data 55  8. Stream Editor (sed) Useful for repetitive editing tasks Editing commands may be provided in command line or in a separate file sed will read the data file line-by-line, apply all applicable editing commands, and send the resulting edited line to standard output sed does not modify the original data file itself. An edited file may be obtained by redirecting the standard output Edit a file named chapter1, inspect the first 10 lines of the file, line by line, look for strings of the form tree and replace every occurrence in each line by forest, and redirect the output to a file named chapter1Revised sed '1,10 s/tree/forest/g' chapter1 > chapter1Revised Note that the instruction for editing is within single quotes to prevent the shell from misunderstanding any of the special characters 56  Each editing instruction has two components: address action and parameters, if any 1,10 s/tree/forest/g /^t/ d Address component o Allows selecting specific lines on which editing is done o Could be a specific line number or a range of line numbers (written in the form 1,10 and not 1-10) o Could be in the form of a regular expression in between two forward slashes, so that all lines matching the pattern are selected (e.g., /unix/ ) o Could simply say $ indicating the last line o If no address is specified all lines are selected for editing Examples of common actions and associated parameters d delete selected lines (no parameters) s/re/string/ substitute first occurrence in the line of re by string (no parameters) s/re/string/g substitute every occurrence in the line of re by string (no parameters) 57  It is possible to give a sequence of editing instructions o Type all the editing commands into a file, say called sedProgFile o Do not put quotes around the editing commands o Do not write the sed word (the command) in the file o Run the sed command as follows: sed –f sedProgFile data sed will process the file line by line and carry out the required editing instructions, as follows : o Check if the first editing instruction applies to this line based on the address information. If that instruction applies, do the required editing o Check if the next editing instruction applies to this line and carry out the instruction if so, and so on. o In other words, carry out the applicable editing instructions one by one, depending on the order in which they are present in the file. o Output the final edited form of the line o Pick up the next line and run through the editing instructions one by one, and so on. 58  9. Bash Scripts Shell is more than an interactive command interpreter. It implements a programming language o variables, expressions, assignment, control structures, … We do not compile code, each line of code is interpreted, i.e., analyzed and executed o Even syntax errors in code will surface only when that line is encountered during execution Mechanics o Enter all the code into a file; e.g., gcd o Make the file executable – chmod 700 gcd o Type the name of the file to execute it (type the absolute or relative path name); e.g.,./gcd pico gcd create bash script file... enter code, save and exit chmod 700 gcd make the file executable./gcd execute bash script file Sample Code 59  Sample Bash Script Code #!/bin/bash # Invoke as./gcd # # The program determines the GCD # of two positive integers # # Display purpose echo –n "The program determines the GCD" echo " of two positive integers" # Prompt and read the two integers echo -n "Enter the 1st positive integer: "; read a echo -n "Enter the 2nd positive integer: "; read b # # Use Euclid's algorithm # As long as the integers do not match, # replace the larger by their difference # while [[ $a -ne $b ]] do if [[ $a -gt $b ]] then a=$(( $a - $b )) else b=$(( $b - $a )) fi done # When the two values match, that is the GCD gcd=$a # Display result echo "The GCD of the given integers is: $gcd" # exit 0 # 60  Bash Script Basics Environment variables o SHELL, USER, HOME, PATH, etc., User-defined variables o No need to declare (type-less language) o All variables are string valued; but can participate in integer arithmetic, as necessary conversion will be done when needed o Expressions may involve arithmetic, relational and logical operators There should be no spaces around “equal to” symbol Use single quotes around the value if desired name=John name='John' Need $ in front of the shell variable name to access its value echo $name x=3 # or x='3' echo $x # prints 3 x=$(( x + 1 )) # increments x echo $x # prints 4 y='5' # or y=5 x=$(( x + y )) # changes x echo $x # prints 9 61  Test Expressions String Conditions Form of expression True if [[ var ]] var is defined and is not null [[ -z str ]] str is zero length [[ str1 == str2 ]] str1 and str2 are equal [[ str1 != str2 ]] str1 and str2 are not equal [[ str1 < str2 ]] str1 sorts before str2 [[ str1 > str2 ]] str1 sorts after str2 [[ str1 =~ pattern ]] str1 matches the pattern Numerical conditions Form of expression True if [[ arg1 -eq arg2 ]] arg1 is equal to arg2 [[ arg1 -ne arg2 ]] arg1 is not equal to arg2 [[ arg1 -le arg2 ]] arg1 = arg2 [[ arg1 -gt arg2 ]] arg1 > arg2 62  File queries o On any directory entry (an ordinary file or a directory) Form of expression True if entry -e entry exists -d entry Is a directory -f entry Is an ordinary file -r entry Is readable by the user -w entry Is writable by the user -x entry Is executable by the user -o entry Is owned by the user -z entry Is of zero size -s entry Is of non-zero size Logical operators o Expressions may use && for logical AND || for logical OR ! for logical NOT 63  Multivalued Variable Array (also called list in scripting languages) o There is no maximum limit to the size of an array, nor any requirement that components be indexed or assigned contiguously o First component is indexed with the number 0 color=(red green blue yellow) There are no commas separating the values Individual components are accessed as ${color}, …, ${color} o Note the use of curly braces ${color[@]} refers to the complete list ${#color[@]} gives the number of components Size of the array (list) can change by adding new components color=orange echo ${color[@]} # prints the list echo ${#color[@]} # prints 5 64  Command Line Arguments Consider a script program (command) invoked as follows:./addsize file1 file2 file3 The above script program is named addsize It is said to have three command line arguments In Bash, the positional parameters are used to access them $0 stands for./addsize $1 stands for file1 $2 stands for file2 $3 stands for file3 $@ stands for file1 file2 file3 $# stands for 3, number of command line arguments 65  Bash Control Structures if test-expr then commandlist1 fi if test-expr then commandlist1 else commandlist2 fi if test-expr1 then commandlist1 elif test-expr2 then commandlist2 else commandlist3 fi case str in pattern1) commandlist1 ;; pattern2) commandlist2 ;;... esac 66 while test-expr do commandlist done for var in wordlist do commandlist done 67  Examples #!/bin/bash # Test string matching echo -n "Please enter a pattern: "; read pattern echo -n "Please enter string: "; read str if [[ $str =~ $pattern ]] then echo "String $str matches the pattern $pattern" else echo "String $str does not match the pattern $pattern" fi # #!/bin/bash # Print greeting echo –n "Please enter your first name: "; read name echo –n "Please enter a positive integer count: "; read count i=0 while [[ $i –lt $count ]] do echo "Hello, $name!" i=$(( $i + 1 )) done # 68  10. Sample Bash Script - 1 #!/bin/bash # # Program prints terms in the sequence: 1 2 4 8... # User controls the number of terms printed # # Display purpose, prompt and read an integer echo 'This program prints terms in the "Powers of 2" sequence' echo -n "Number of terms to be printed (enter a non-negative integer): " read limit # If limit is negative report error; else compute and print term(s) if [[ $limit -lt 0 ]] then echo "Sorry, you must enter a non-negative integer!" else termSeq=1 count=0 while [[ $count –lt $limit ]] do echo $termSeq count=$(( $count + 1 )) termSeq=$(( $termSeq * 2 )) done fi # exit 0 # 69  Sample Bash Script #!/bin/bash # # Add sizes of named files in current working directory, if they exist # # usage:./addsize file1... # # If no command line arguments, report error and exit if [[ $# -eq 0 ]] then echo "usage: $0 file1... " exit 1 fi # # Process command line arguments, one by one, determine size and add total=0 for item in $@ do if [[ -f $item ]] then # Determine file size size=`ls -l $item | awk '{print $5}'` total=$(( $total + $size )) else echo "File $item not found" fi done # Display result echo "Total size of valid files: $total" # exit 0 # 70  Sample Bash Script #!/bin/bash # The program will check whether or not a named directory contains # a named file and report. Invoke as:./search directoryname filename # if [[ $# -ne 2 ]] then echo "usage: $0 directoryname filename" exit 1 fi # if [[ !(-d $1) ]] then echo "$0: $1 is not a directory" exit 2 fi # if [[ !(-r $1 && -x $1) ]] then echo "$0: cannot open directory $1" exit 3 fi # if [[ -f "$1/$2" ]] then echo "Found file $2 in directory $1" else echo "Did not find file $2 in directory $1" fi # exit 0 # 71  10. Sample Bash Script -2 #!/bin/bash # The program will remove the named file/directory from # the current working directory and place it in ~/.trash. # Invoke as # #./delete filename # # Check for argument if [[ $# -ne 1 ]] then echo "usage: $0 filename" exit 1 fi # Check if the named entry exists in the cwd if [[ !(-e $1) ]] then echo "$0: $1 does not exist" exit 2 fi # Check if ~/.trash exists; create one if necessary if [[ !(-d ~/.trash) ]] then mkdir ~/.trash fi 72 # Check if ~/.trash contains an entry with the given name if [[ -e ~/.trash/$1 ]] then echo -n "$0: $1 exists in.trash. Overwrite? (y/n) " read resp if [[ $resp == "y" ]] then rm -rf ~/.trash/$1 else echo "$0: $1 is not deleted" exit 3 fi fi # Move the entry from the cwd to ~/.trash mv $1 ~/.trash exit 0 # 73