3.0-LinuxDevicesFiles.pdf

Linux Operating System Files and Devices Operating Systems What is an OS? • A software suite installed on a computing system to: – Efficiently manage resources on the computing system – Expose a streamlined Application Program Interface (API) • Developing system software (such as device drivers) •...

Linux Operating System Files and Devices Operating Systems What is an OS? • A software suite installed on a computing system to: – Efficiently manage resources on the computing system – Expose a streamlined Application Program Interface (API) • Developing system software (such as device drivers) • For developing software applications – Provide end-user interfaces – Typically a Graphical User Interface (GUI) • Monitor, control, and effectively use the computing system Multi-facetted view of OS Different users of a computing system use different facets of an operating system Administrator • Configuration • Security • Monitoring • Management • Failure recovery Hardware Machine & Peripherals Users • Email / Browsing • Word processing • Gaming • Control applications • Privacy Programmers • API for application development • Generic & programming-language specific •http://gcc.gnu.org/onlinedocs/libstdc++/index.html • Controlling and coordinating devices • Inter-application communications Device Driver • Device driver is an OS-specific software module – Developed to interface devices to OS – Drivers implement specific OS API calls • The driver uses kernel functionality to interact with hardware Device Driver Kernel API  User Application Kernel Code  User Application System Call User Application Device Driver Kernel API Modern desktop versions • Modern Linux distributions consists of – The Linux kernel • • • • Provides basic management of hardware devices A low-level API for use by device drivers A collection of standard device drivers Basic infrastructure for security and privacy – GNU Tools: The • GCC: The core is GNU Compiler collection – The Linux kernel is compiled using GCC • The bash shell and other shells are from GNU • Many default tools and editors (such as emacs) are from GNU – Many software systems have been developed • Includes graphical desktop systems such as: Gnome, KDE • Browsers, email clients • Data bases GNU Tools GNU is a recursive acronym for “GNU's Not Unix” Aim at developing a complete Unix-like operating system which is free for copying and modification Companies make their money by maintaining and distributing the software, e.g. optimally packaging the software with different tools (Redhat, Slackware, Mandrake, SuSE, etc) Stallman built the first free GNU C Compiler in 1991. However, GNU does not include an OS kernel. Established in 1984 by Richard Stallman, who believes that software should be free from restrictions against copying or modification in order to make better and efficient computer programs Linux kernel • Linux was developed – • The kernel has evolved over the years into a sophisticated and complex software system – – • Predominantly in C and exposes a C API Uses assembly to handle interfacing with various CPUs/architectures. Includes a collection of device drivers – – • Developed In Sept 1991, Linus Torvalds For many hardware devices Drivers for many file systems Linux is a multiuser, multitasking OS Running the Kernel • The Linux kernel is a standard C program • It is no different than the custom OS we have been developing in class – It needs to be compiled into a raw binary to run • Process of building a Linux kernel using an existing Linux system is straightforward – Download and un-package Linux kernel source from https://www.kernel.org/ • It is typically distributed in a zip file format – The kernel includes a Makefile for building it • From the top-level kernel source directory issue: $ make menuconfig $ make • The binary kernel image is built in directory name arch/<arch>/ boot/bzImage. Example arch/x86_64/boot/bzImage – The binary kernel can be run on a machine or VM • Can be run using qemu just like any other custom OS Linux Kernel Booting • The kernel does not contain any user software – No shell or any other tools • Consequently, the kernel expects an initial read-only disk image at startup – This RAM disk image is called initrd. – Contains core shell and other utilities for mounting disks/file systems – Optionally starting up graphical desktop etc. – Most Linux distributions essentially customize initrd and its operations. Devices in the Linux Kernel • Devices in Linux are exposed as files – Starting with the root path / – The hardware devices are exposed as file entries • In the /dev directory – Raw access to HDD: /dev/sda, /dev/sdb, /dev/sdc etc. – Managed devices: /dev/video0 • Each entry in the /dev directory is tied to a device driver – Associations are made using a pair of numbers, such as: 249:0 – The numbers are often called major and minor numbers • Devices and drivers are broadly classified into 2 categories – Character devices: Read or write 1 character at a time • Keyboard, mice, serial port etc. – Block devices • Read blocks of data: HDD, USB etc. – Network devices The /proc file system • Used to exposes kernel data – In the form of files and folders • These files are not physically present on disk – Practically all information about the system is exposed in /proc • Writing to selected files updates parameters in the kernel – Each program has its own directory in the /proc system • Can be read by the owner of the process Linux VFS • Linux internally uses a Virtual File System (VFS) – Provides consistent interface to different file systems • It is a layer of abstraction – VFS is used by various subsystems in the kernel and by kernel modules • The role of the VFS is to: – Keep track of available file system types. – Associate (and disassociate) devices with instances of the appropriate file system. – Do any reasonable generic processing for operations involving files. – When file system-specific operations become necessary, vector them to the file system in charge of the file, directory, or the inode in question. Overview of VFS inode vs struct inode • The index node or inode is data structure on disk used by Linux file systems (like ext3) – It stores metadata about a file, directory, or device • Synonymous to root directory entry in FAT-12 – On file systems you can use the stat command to view inode information $ stat /proc/cpuinfo/ • The struct inode is an internal data structure maintained by the kernel – Used by VFS for interfacing with the file system driver – Used to interface various devices and kernel modules • Many discussions simply use the term inode – You have disambiguate it depending on the context File Permissions • Files are fundamental to operation on Linux – Permissions to access them are controlled by the system’s permission management layer • On Linux & Unix machines permissions are organized into 3 categories: – User: The user who created an entry – Group: The set of users belonging to the user’s groups. • System administrator creates groups and adds users to various groups. • Setting group privileges permits other users in a group to access and use shared resources such as files & IPC entries – Others: Rest of the users of the system Understanding Permissions • Most Linux & Unix commands display permissions as either numbers or letters for user, group, & others in the following forms: 1. As a set of at least 9 letters letters • 3 chars for user, group, & others: rwxrwxrwx • Where r is for read permission, w is for write permission, and x is for execute permission. If a permission is not available then a - is displayed – Example: rw-r--r--, user can read & write, group and others can only read 2. As octal 3 octal digits • Each digit for user, group, & others in this order • Octal numbers have 3 bits, each bit corresponds to: – 001: (Lowest-bit) Execute privilege – 010: (Middle bit) Write Access – 100: (Highest-bit) Read Access • Accordingly, rw-r--r-- maps to 0644. Changing permissions • File permissions can be changed using the chmod command • r=read, w=write, x=execute – Give user rwx, group rw, and others rw permissions $ chmod 755 test.cpp – Make file private to just the user $ chmod 700 test.cpp – Alternative syntax (u=user, g=group, o=others) $ chmod u+rw-x,o-rwx,g+rwx test.cpp Loadable Kernel Modules • Additional modules can be dynamically added or removed from a running kernel • Convenient for handling plug-n-play devices • Eases management of features in the kernel – insmod: Insert a new module into the kernel – rmmod: Remove a module from the kernel – lsmod: List currently loaded modules. • Advantages of modules – There is no necessity to rebuild the kernel, when a new kernel option is added – Modules help find system problems (if system problem caused a module just don't load it) – Modules save memory – Modules are much easier to maintain and debug – Modules once loaded are inasmuch fast as kernel A simple loadable kernel module #include <linux/module.h> #include <linux/kernel.h> int init_module( void ) { printk(KERN_INFO "Module initialized.\n"); return 0; // Zero indicates “Initialized successfully” } void cleanup_module( void ) { printk(KERN_INFO "Module finalized. Goodbye\n"); printk( “<1>Goodbye everyone\n” ); } MODULE_LICENSE(“GPL”); module_init(init_module); module_exit(cleanup_module); ifneq ($(KERNELRELEASE),) obj-m := my_module.o else KERNELDIR = /usr/src/kernels/3.9.11-200 PWD := $(shell pwd) default: $(MAKE) -C $(KERNELDIR) M=$(PWD) modules endif A simple loadable kernel module (in file named my_module.c) along with a simple Makefile to build the module for kernel version 3.9.11-200. The kernel source is assumed to be in the default location at /usr/src/ kernels/3.9.11-200. API for a character device module • A character device module must provide functions for various operations via a file_operations structure during initialization – Initialization also obtains major and minor device numbers struct file_operations fops = { owner: read: write: open: release: }; THIS_MODULE, read_char, write_char, open, release // // // // // Defined by kernel Function to read a char Function to write a char Open a given file Close a given file Initializing a character device module • A character device module must provide functions for various operations via a file_operations structure: // Kernel structure that identifies this module struct cdev *kernel_cdev = NULL; Get’s int major_num = -1; device number from kernel. 0 is first minor device number 1 is number of devices to create Last parameter is name of device int init_module(void) { int ret; dev_t dev_no, dev; // Allocate kernel memory for character device kernel_cdev = cdev_alloc(); kernel_cdev->ops = &fops; // See previous slide kernel_cdev->owner = THIS_MODULE; ret = alloc_chrdev_region(&dev_no , 0, 1, ”cse381"); if (ret < 0) { printk("Major number allocation is failed\n"); return ret; } major_num = MAJOR(dev_no); printk (KERN_INFO" The major number for your device is %d\n", Major); return 0; // success } Handing open and close of device • A simple implementation for opening and closing the device – Currently they do nothing based on the assumption of a single device int open(struct inode *inode, struct file *fileInfo) { printk(KERN_INFO "Inside open \n"); // Optionally use inode and fileInfo to setup some // internal data structures and buffers. return 0; } int release(struct inode *inode, struct file *fileInfo) { // Optionally use inode and fileInfo to wind-up data // structures for the corresponding file. printk (KERN_INFO "Inside close \n"); return 0; } Reading & Writing in a Module // Read utmost count characters into buffer for given fileInfo // starting at offset *offp. Return number of characters actually read. // Return 0 to indicate end-of-file. ssize_t read_char(struct file *fileInfo, char *buff, size_t count, loff_t *offp) { printk(KERN_INFO "Inside read \n"); // Copy from kernel to user space ret = copy_to_user(buff, "A", 1); return 1; } // write utmost count characters from buffer for given fileInfo // logically at offset *offp. Return actual number of bytes written. // If return value < count implies error ssize_t write_char(struct file *filp, const char *buff, size_t count, loff_t *offp) { printk(KERN_INFO "Inside write \n"); // Copy from user to kernel space. char data = buff[0]; return count; } Clean up and running void cleanup_module(void) { printk(KERN_INFO " Inside cleanup_module\n"); cdev_del(kernel_cdev); unregister_chrdev_region(major_num, 1); } $ $ # # # $ $ Assumptions: 1. You have super user (su) privileges 2. The module gets a major device number of 249 make su insmod simple_mod mknod /dev/simple 249 0 exit echo “Hello” > /dev/simple cat /dev/simple Links to files and devices • Links are cross references to an existing file – Avoid copying files – Links can be created in same directory or different directories • Conveniently access the files • To most programs linking is invisible and they treat links the same as original files. – Links take up only minimal disk space • Just enough to create a directory entry for the file • There are two types of links – Links (aka Hard Links) • Data associated with files persists until last linked file is deleted – Symbolic Links (aka Soft Links or Symlinks) • If source file is deleted the symlinks are orphaned Links vs. Symlinks in Linux • In Linux links are created using the ln command – Hard link: ln original.txt copy.txt – Soft link: ln –s original.txt soft_copy.txt $ ls -lh total 304K -rw-r--r--. 2 user user 193K copy.txt -rw-r--r--. 2 user user 193K original.txt lrwxrwxrwx. 1 user user 12 soft_copy.txt -> original.txt The number of links (aka hard links) to each file. View of a soft link in the file system Links in Windows • Starting with Windows Vista NTFS supports linking – Links are created using the mklink utility • Windows supports three types of links – Hard links • Restricted to the same volume – Soft links or symlinks • Can go across volumes – Junctions • Specifically for directories • Honors reparse points in windows Problems with kernel modules • Kernel modules essentially integrate into Linux when they are loaded – They pretty much become part of the OS – They have to rely only on services provided by the OS] • Cannot use standard C/C++ APIs – Standard APIs come from glibc which is not available in kernel space. • Require a whole new set of APIs – Kernel APIs are peculiar – Changes in kernel API will require module redevelopment • Development and management is tedious – Require super user (su) privileges to manage • It is an issue in multiuser machines and on shared networks of machines • Not customizable on a “per user” basis – Module crashes are often fatal to the whole system • Kernel can recover from some issues – You will see a “Kernel Oops” or “Kernel panic” messages with various details in /var/log/messages file. User-space drivers • Current Linux trend is to migrate modules into userspace – The modules run as regular user-programs rather than as part of the kernel – Use special system calls and support from kernel • Advantages – Run on a per user basis – Safer and more secure • Crash of the module does not affect rest of the system – Easier to develop using standard APIs and any available library • Theoretically can be in any programming language • Disadvantage – Each user has to run the module to use a device – Movement of large volumes of data can be a bit slow File System in User Space (FUSE) • FUSE is an excellent example of user-mode driver – FUSE permits users to setup and use custom file systems • The files appear the same as normal modules – Uses a special kernel module to facilitate operations Overview of FUSE operations • A full FUSE driver involves over 40 methods – See http://fuse.sourceforge.net/doxygen/ structfuse__operations.html • However, 4 methods are the bare minimum: struct fuse_operations { int(*getattr)(const char *path, struct stat *stbuf) int (*readdir)(const char *path, void *buf, fuse_fill_dir_t filler, off_t offset, struct fuse_file_info *fi) int (*open)(const char *path, struct fuse_file_info *fi) int (*read)(const char *path, char *buf, size_t size, off_t offset, struct fuse_file_info *fi) }; • See simple example at: http://fuse.sourceforge.net/helloworld.html Read files in a directory static int fuse_readdir(const char *path, void *buf, fuse_fill_dir_t filler, off_t offset, struct fuse_file_info *fi) { // Verify that directory in "path" is valid. Usually check for just // "/" (root) if (strcmp(path, "/") != 0) { return -ENOENT; } // Add all entries to the list of files in root directory by calling // the filler method. File properties are set to NULL so FUSE will use // the fuse_getattr() function to get file attributes struct stat *properties = NULL; off_t ignored = 0; filler(buf, ".", properties, ignored); // Default entry for PWD filler(buf, "..", properties, ignored); // Default entry for parent // Add more entries.. return 0; // success } Get file attributes int fuase_getattr(const char *path, struct stat *stbuf) { // Place file attributes in the supplied stat buffer int dir = ...; // figure out if path is a file or directory stbuf->st_mode = (dir ? (S_IFDIR | 0755) : (S_IFREG | 0444); stbuf->st_nlink = 1; stbuf->st_size = fileSize; stbuf->st_uid = getuid(); // user id stbuf->st_gid = getuid(); // group id stbuf->st_atime = // Last access time in millis stbuf->st_mtime = // Modification time in millis stbuf->st_ctime = // creation time in millis return 0; // success } Example of open and read static int fuse_open(const char *path, struct fuse_file_info *fi) { // Logically opens file "path" for reading in fuse_read. // Steps: // 1. Check if file name in "path" is valid (return -ENOENT if not). // 2. Ensure read-only access (if writing return -EACCESS for error) if ((fi->flags & 3) != O_RDONLY) { return -EACCES; } // 3. Return 0 for success. } static int fuse_read(const char *path, char *buf, size_t size, off_t offset, struct fuse_file_info *fi) { // Read size bytes of data from offset in file "path" into buffer buf. // Steps: // 1. Check if file name in "path" is valid (return -ENOENT if not). // 2. Read data from the file into a temporary buffer // 3. Copy necessary data from temporary buffer into buf. // 4. Return 0 for success. }