IPython Magic Commands PDF

Types of Magic Commands There are two types of magic commands −  Line magics  Cell magics Line Magics They are similar to command line calls. They start with % character. Rest of the line is its argument passed without parentheses or quotes. Line magics can be used as expression and th...

Types of Magic Commands There are two types of magic commands −  Line magics  Cell magics Line Magics They are similar to command line calls. They start with % character. Rest of the line is its argument passed without parentheses or quotes. Line magics can be used as expression and their return value can be assigned to variable. Cell Magics They have %% character prefix. Unlike line magic functions, they can operate on multiple lines below their call. They can in fact make arbitrary modifications to the input they receive, which need not even be a valid Python code at all. They receive the whole block as a single string. To know more about magic functions, the built-in magics and their docstrings, use the magic command. Information of a specific magic function is obtained by %magicfunction? Command. Let us now describe some of the built-in line and cell magic commands. Built-in line magics %autocall [mode] This magic function makes a function automatically callable without having to use parentheses. It takes three possible mode parameters: 0 (off), 1 (smart) is default or 2 (always on). %automagic Magic functions are callable without having to type the initial % if set to 1. Without arguments it toggles on/off. To deactivate, set to 0. The following example shows a magic function %pwd (displays present working directory) being called without leading % when %automagic set to 1 %cd This line magic changes the current directory. This command automatically maintains an internal list of directories you visit during your IPython session, in the variable _dh. You can also do ‘cd -’ to see directory history conveniently. Usage The %cd command can be used in the following ways −  %cd − Changes current working directory to  %cd.. − Changes current directory to parent directory  %cd − changes to last visited directory. %dhist This magic command prints all directories you have visited in current session. Every time %cd command is used, this list is updated in _dh variable. %edit This magic command calls upon the default text editor of current operating system (Notepad for Windows) for editing a Python script. The script is executed as the editor is closed. %env This magic command will list all environment variables. It also reads value of particular variable or set the value of environment variable. Usage The %cd command can be used in the following ways −  %env − Lists all environment variables  %env var − Gets value for var  %env var val − Sets value for var %gui [GUINAME] When used without argument this command enables or disables IPython GUI event loop integration. With GUINAME argument, this magic replaces the default GUI toolkits by the specified one. Sr.No. Command & Description %gui wx 1 enable wxPython event loop integration %gui qt4|qt 2 enable PyQt4 event loop integration %gui qt5 3 enable PyQt5 event loop integration %gui gtk 4 enable PyGTK event loop integration %gui gtk3 5 enable Gtk3 event loop integration %gui tk 6 enable Tk event loop integration %gui osx 7 enable Cocoa event loop integration  8 (requires %matplotlib 1.1) %gui 9 disable all event loop integration %lsmagic Displays all magic functions currently available %matplotlib This function activates matplotlib interactive support during an IPython session. However, it does not import matplotlib library. The matplotlib default GUI toolkit is TkAgg. But you can explicitly request a different GUI backend. You can see a list of the available backends as shown − In : %matplotlib --list Available matplotlib backends: ['osx', 'qt4', 'qt5', 'gtk3', 'notebook', 'wx', 'qt', 'nbagg','gtk', 'tk', 'inline'] The IPython session shown here plots a sine wave using qt toolkit − While using Jupyter notebook, %matplotlib inline directive displays plot output in the browser only. %notebook This function converts current IPython history into an IPython notebook file with ipynb extension. The input cells in previous example are saved as sine.ipynb %notebook sine.ipynb %pinfo This function is similar to object introspection ? character. To obtain information about an object, use the following command − %pinfo object This is synonymous to object? or ?object. %precision This magic function restricts a floating point result to specified digits after decimal. %pwd This magic function returns the present working directory. %pylab This function populates current IPython session with matplotlib, and numpy libraries. %recall When executed without any parameter, this function executes previous command. Note that in %recall n, number in front of it is input cell number. Hence the command in the nth cell is recalled. You can recall commands in section of cells by using command such as %recall 1-4. Current input cell is populated with recalled cell and the cursor blinks till the enter key is pressed. %run This command runs a Python script from within IPython shell. %time This command displays time required by IPython environment to execute a Python expression. %timeit This function also displays time required by IPython environment to execute a Python expression. Time execution of a Python statement or expression uses the timeit module. This function can be used both as a line and cell magic as explained here −  In line mode you can time a single-line.  In cell mode, the statement in the first line is used as setup code and the body of the cell is timed. The cell body has access to any variables created in the setup code. %who This line magic prints all interactive variables, with some minimal formatting. If any arguments are given, only variables whose type matches one of these are printed. IPython Custom Line Magic function IPython’s core library contains register_line_magic decorator. A user defined function is converted into a line magic function using this decorator. Print Page 1. Line Magic Commands 2. Cell Magic Commands Line Magic Commands Line magic commands are used to provide a special functionality to a single line of code, line magic commands begin with ‘%’ followed by the line command. To check the available magic commands run ‘%lsmagic’ this command will show all the magic commands, as of now there are 97 line magic commands which are as follows: %alias, %alias_magic, %autoawait, %autocall, %automagic, %autosave, %bookmark, %cat, %cd, %clear, %colors, %conda, %config, %connect_info, %cp, %debug, %dhist, %dirs, %doctest_mode, %ed, %edit, %env, %gui, %hist, %history, %killbgscripts, %ldir, %less, %lf, %lk, %ll, %load, %load_ext, %loadpy, %logoff, %logon, %logstart, %logstate, %logstop, %ls, %lsmagic, %lx, %macro, %magic, %man, %matplotlib, %mkdir, %more, %mv, %notebook, %page, %pastebin, %pdb, %pdef, %pdoc, %pfile, %pinfo, %pinfo2, %pip, %popd, %pprint, %precision, %prun, %psearch, %psource, %pushd, %pwd, %pycat, %pylab, %qtconsole, %quickref, %recall, %rehashx, %reload_ext, %rep, %rerun, %reset, %reset_selective, %rm, %rmdir, %run, %save, %sc, %set_env, %store, %sx, %system, %tb, %time, %timeit, %unalias, %unload_ext, %who, %who_ls, %whos, %xdel, %xmode Since it is not possible to cover all the 97 line magic commands we will only cover the most widely and helpful commands in this article, we will look some line magic commands along with the short description followed by example of some line magic commands: Line Magic Command Description %alias Define an alias for a system command. %alias_magic Create an alias for an existing line or cell magic Make magic functions callable without having to type %automagic the initial % %cd Change the current working directory %debug Activate the interactive debugger %dhist Print your history of visited directories %dirs Return the current directory stack %env Get, set, or list environment variables. %gui Enable or disable IPython GUI event loop integration %load Load code into the current frontend Line Magic Command Description %load_ext Load an IPython extension by its module name %lsmagic List All Available Magic Commands Define a macro for future re-execution. It accepts %macro ranges of history, filenames or string objects %magic Print Information About Magic Commands System %matplotlib Set up matplotlib to work interactively. %notebook Export and convert IPython notebooks Print (or run through pager) the file where an object is %pfile defined %pip Run the pip package manager within the current kernel %prun Run a statement through the python code profiler %pwd Return the current working directory path %reload_ext Reload an IPython extension by its module name Resets the namespace by removing all names defined %reset by the user %run Run the named file inside IPython as a program %save Save a set of lines or a macro to a given filename %set_env Set environment variables %system Shell execute – run shell command and capture output  Line Magic Command Description Measures time execution of a Python statement or %time expression %timeit Measures Execution Time of Line %unalias Remove an alias print all interactive variables, with some minimal %who formatting %who_ls Return a sorted list of all interactive variables Like %who, but gives some extra information about %whos each variable Delete a variable, trying to clear it from anywhere that %xdel IPython’s machinery has references to it. %xmode Switch modes for the exception handlers Example of Line Magic Functions 1. %load  Load the code from external file and insert in into the cell  Syntax: %load file_name.py Example: Python3 %load fibo.py 2. %run  Run the named file inside IPython as a program  Syntax: %run file_name.py Example: Python3  %run tricks.py Output: 3. %time  Measure the execution time of a single Python statement or expression  Syntax: %time some_funtion() Example: Python3 def fibonacci(n): if n == 0 or n == 1: return n return fibonacci(n-1)+fibonacci(n-2) %time fibonacci(10) Output: CPU times: total: user 20 ns, sys:4 ns, total: 24 ns Wall time: 26.2 ns 55 4. %timeit  Measure the execution time of a single Python statement or expression by performing executions multiple time to get more accurate results.  Syntax: %timeit some_function() Example: Python3 def fibonacci(n): if n == 0 or n == 1: return n return fibonacci(n-1)+fibonacci(n-2) %timeit fibonacci(10) Output: 14.7 µs ± 869 ns per loop (mean ± std. dev. of 7 runs, 100,000 loops each) 5. %cd  Change the current working directory  Syntax: %cd directory/sub_dir Example: Python3 %cd Form_data_visualization Output: Cell Magic Commands Cell Magic functions are special commands that allow the user to modify the behavior of a code cell explicitly. Cell Magic functions have the prefix ‘%%’ followed by the command name. Cell magic functions serve various tasks and customizations which we discuss thoroughly further in this article. As per the official python documentation there are 16 cell magic functions available now which are described below, Cell Magic Command Description %%bash Run cells with bash in a subprocess  Cell Magic Command Description run the cell, capturing stdout, stderr, and IPython’s %%capture rich display() calls %%html Render the cell as a block of HTML %%javascript or Run the cell block of Javascript code %%js %%latex Render the cell as a block of LaTeX %%markdown Render the cell as Markdown text block %%perl Run cells with perl in a subprocess %%pypy Run cells with pypy in a subprocess %%python Run cells with python in a subprocess %%python2 Run cells with python2 in a subprocess %%python3 Run cells with python3 in a subprocess %%ruby Run cells with ruby in a subprocess %%script Run a cell via a shell command %%sh Run cells with sh in a subprocess %%svg Render the cell as an SVG literal %%writefile Write the contents of the cell to a file Since, its not possible to explain each cell magic command with example we will only cover some helpful and widely used cell magic commands. Example of Cell Magic Functions 1. %%bash Allows user to run Bash shell commands within the cell. Example: Python3 %%bash ls -l Output Screenshot: 2. %%html This cell magic function renders contents of code cell as html script Example: Python3 %%html  Hello World Output Screenshot: 3. %%javascript or %%js Allows you to execute JavaScript code within the cell Example: Python3 %%javascript alert("This is a JavaScript alert!") Output Screenshot: 4. %%latex Renders LaTeX equations and expressions in the cell. Python3 %%latex $e^{i\pi} + 1 = 0$ Output Screenshot: 5. %%markdown or %%md Renders the content of the cell as Markdown Example: Python3 %%markdown # This is a Markdown Heading * This is a bullet point Output Screenshot: 1. Line Magic Commands 2. Cell Magic Commands Line Magic Commands Line magic commands are used to provide a special functionality to a single line of code, line magic commands begin with ‘%’ followed by the line command. To check the available magic commands run ‘%lsmagic’ this command will show all the magic commands, as of now there are 97 line magic commands which are as follows: %alias, %alias_magic, %autoawait, %autocall, %automagic, %autosave, %bookmark, %cat, %cd, %clear, %colors, %conda, %config, %connect_info, %cp, %debug, %dhist, %dirs, %doctest_mode, %ed, %edit, %env, %gui, %hist, %history, %killbgscripts, %ldir, %less, %lf, %lk, %ll, %load, %load_ext, %loadpy, %logoff, %logon, %logstart, %logstate, %logstop, %ls, %lsmagic, %lx, %macro, %magic, %man, %matplotlib, %mkdir, %more, %mv, %notebook, %page, %pastebin, %pdb, %pdef, %pdoc, %pfile, %pinfo, %pinfo2, %pip, %popd, %pprint, %precision, %prun, %psearch, %psource, %pushd, %pwd, %pycat, %pylab, %qtconsole, %quickref, %recall, %rehashx, %reload_ext, %rep, %rerun, %reset, %reset_selective, %rm, %rmdir, %run, %save, %sc, %set_env, %store, %sx, %system, %tb, %time, %timeit, %unalias, %unload_ext, %who, %who_ls, %whos, %xdel, %xmode Since it is not possible to cover all the 97 line magic commands we will only cover the most widely and helpful commands in this article, we will look some line magic commands along with the short description followed by example of some line magic commands: Line Magic Command Description %alias Define an alias for a system command. %alias_magic Create an alias for an existing line or cell magic Make magic functions callable without having to type %automagic the initial % %cd Change the current working directory %debug Activate the interactive debugger %dhist Print your history of visited directories %dirs Return the current directory stack %env Get, set, or list environment variables. %gui Enable or disable IPython GUI event loop integration %load Load code into the current frontend %load_ext Load an IPython extension by its module name %lsmagic List All Available Magic Commands Define a macro for future re-execution. It accepts %macro ranges of history, filenames or string objects %magic Print Information About Magic Commands System Line Magic Command Description %matplotlib Set up matplotlib to work interactively. %notebook Export and convert IPython notebooks Print (or run through pager) the file where an object is %pfile defined %pip Run the pip package manager within the current kernel %prun Run a statement through the python code profiler %pwd Return the current working directory path %reload_ext Reload an IPython extension by its module name Resets the namespace by removing all names defined %reset by the user %run Run the named file inside IPython as a program %save Save a set of lines or a macro to a given filename %set_env Set environment variables %system Shell execute – run shell command and capture output Measures time execution of a Python statement or %time expression %timeit Measures Execution Time of Line %unalias Remove an alias print all interactive variables, with some minimal %who formatting  Line Magic Command Description %who_ls Return a sorted list of all interactive variables Like %who, but gives some extra information about %whos each variable Delete a variable, trying to clear it from anywhere that %xdel IPython’s machinery has references to it. %xmode Switch modes for the exception handlers Example of Line Magic Functions 1. %load  Load the code from external file and insert in into the cell  Syntax: %load file_name.py Example: Python3 %load fibo.py 2. %run  Run the named file inside IPython as a program  Syntax: %run file_name.py Example: Python3 %run tricks.py Output: 3. %time  Measure the execution time of a single Python statement or expression  Syntax: %time some_funtion() Example: Python3 def fibonacci(n): if n == 0 or n == 1: return n return fibonacci(n-1)+fibonacci(n-2) %time fibonacci(10) Output: CPU times: total: user 20 ns, sys:4 ns, total: 24 ns Wall time: 26.2 ns 55 4. %timeit  Measure the execution time of a single Python statement or expression by performing executions multiple time to get more accurate results.  Syntax: %timeit some_function() Example: Python3  def fibonacci(n): if n == 0 or n == 1: return n return fibonacci(n-1)+fibonacci(n-2) %timeit fibonacci(10) Output: 14.7 µs ± 869 ns per loop (mean ± std. dev. of 7 runs, 100,000 loops each) 5. %cd  Change the current working directory  Syntax: %cd directory/sub_dir Example: Python3 %cd Form_data_visualization Output: Cell Magic Commands Cell Magic functions are special commands that allow the user to modify the behavior of a code cell explicitly. Cell Magic functions have the prefix ‘%%’ followed by the command name. Cell magic functions serve various tasks and customizations which we discuss thoroughly further in this article. As per the official python documentation there are 16 cell magic functions available now which are described below, Cell Magic Command Description %%bash Run cells with bash in a subprocess run the cell, capturing stdout, stderr, and IPython’s %%capture rich display() calls  Cell Magic Command Description %%html Render the cell as a block of HTML %%javascript or Run the cell block of Javascript code %%js %%latex Render the cell as a block of LaTeX %%markdown Render the cell as Markdown text block %%perl Run cells with perl in a subprocess %%pypy Run cells with pypy in a subprocess %%python Run cells with python in a subprocess %%python2 Run cells with python2 in a subprocess %%python3 Run cells with python3 in a subprocess %%ruby Run cells with ruby in a subprocess %%script Run a cell via a shell command %%sh Run cells with sh in a subprocess %%svg Render the cell as an SVG literal %%writefile Write the contents of the cell to a file Since, its not possible to explain each cell magic command with example we will only cover some helpful and widely used cell magic commands. Example of Cell Magic Functions 1. %%bash Allows user to run Bash shell commands within the cell. Example: Python3  %%bash ls -l Output Screenshot: 2. %%html This cell magic function renders contents of code cell as html script Example: Python3 %%html Hello World Output Screenshot: 3. %%javascript or %%js Allows you to execute JavaScript code within the cell Example: Python3 %%javascript alert("This is a JavaScript alert!") Output Screenshot: 4. %%latex Renders LaTeX equations and expressions in the cell. Python3 %%latex $e^{i\pi} + 1 = 0$ Output Screenshot:  5. %%markdown or %%md Renders the content of the cell as Markdown Example: Python3 %%markdown # This is a Markdown Heading * This is a bullet point Output Screenshot: Input and Output in Python Last Updated : 19 Mar, 2024  Understanding input and output operations is fundamental to Python programming. With the print() function, you can display output in various formats, while the input() function enables interaction with users by gathering input during program execution. Python Basic Input and Output In this introductory guide, we’ll explore the essentials of Python’s basic input and output functionalities, empowering you to efficiently interact with users and display information. Print Output in Python At its core, printing output in Python is straightforward, thanks to the print() function. This function allows us to display text, variables, and expressions on the console. Let’s begin with the basic usage of the print() function: How to Print Output in Python In this example, “Hello, World!” is a string literal enclosed within double quotes. When executed, this statement will output the text to the console. Python3 print("Hello, World!") Ipython And Shell Commands– It is possible to adapt IPython for system shell usage. In the past, IPython shipped a special ‘sh’ profile for this purpose, but it had been quarantined since 0.11 release, and in 1.0 it was removed altogether. Nevertheless, much of this section relies on machinery which does not require a custom profile. You can set up your own ‘sh’ profile to be different from the default profile such that:  Prompt shows the current directory (see Prompt customization)  Make system commands directly available (in alias table) by running the %rehashx magic. If you install new programs along your PATH, you might want to run %rehashx to update the alias table  turn %autocall to full mode Environment variables Rather than manipulating os.environ directly, you may like to use the magic %env command. With no arguments, this displays all environment variables and values. To get the value of a specific variable, use %env var. To set the value of a specific variable, use %env foo bar, %env foo=bar. By default values are considered to be strings so quoting them is unnecessary. However, python variables are expanded as usual in the magic command, so %env foo=$bar means “set the environment variable foo to the value of the python variable bar”. Aliases Once you run %rehashx, all of your $PATH has been loaded as IPython aliases, so you should be able to type any normal system command and have it executed. See %alias? and %unalias? for details on the alias facilities. See also %rehashx? for details on the mechanism used to load $PATH. Directory management Since each command passed by ipython to the underlying system is executed in a subshell which exits immediately, you can NOT use !cd to navigate the filesystem. IPython provides its own builtin %cd magic command to move in the filesystem (the % is not required with automagic on). It also maintains a list of visited directories (use %dhist to see it) and allows direct switching to any of them. Type cd? for more details. %pushd, %popd and %dirs are provided for directory stack handling. Prompt customization Here are some prompt configurations you can try out interactively by using the %config magic: %config PromptManager.in_template = r'{color.LightGreen}\u@\h{color.LightBlue}[{color.LightCyan}\Y1{color.LightBlue }]{color.Green}|\#> ' %config PromptManager.in2_template = r'{color.Green}|{color.LightGreen}\D{color.Green}> ' %config PromptManager.out_template = r' ' You can change the prompt configuration to your liking permanently by editing ipython_config.py: c.PromptManager.in_template = r'{color.LightGreen}\u@\h{color.LightBlue}[{color.LightCyan}\Y1{color.LightBlue }]{color.Green}|\#> ' c.PromptManager.in2_template = r'{color.Green}|{color.LightGreen}\D{color.Green}> ' c.PromptManager.out_template = r' ' Read more about the configuration system for details on how to find ipython_config.py. String lists String lists (IPython.utils.text.SList) are handy way to process output from system commands. They are produced by var = !cmd syntax. First, we acquire the output of ‘ls -l’: [Q:doc/examples]|2> lines = !ls -l == ['total 23', '-rw-rw-rw- 1 ville None 1163 Sep 30 2006 example-demo.py', '-rw-rw-rw- 1 ville None 1927 Sep 30 2006 example-embed-short.py', '-rwxrwxrwx 1 ville None 4606 Sep 1 17:15 example-embed.py', '-rwxrwxrwx 1 ville None 1017 Sep 30 2006 example-gnuplot.py', '-rwxrwxrwx 1 ville None 339 Jun 11 18:01 extension.py', '-rwxrwxrwx 1 ville None 113 Dec 20 2006 seteditor.py', '-rwxrwxrwx 1 ville None 245 Dec 12 2006 seteditor.pyc'] Now, let’s take a look at the contents of ‘lines’ (the first number is the list element number): [Q:doc/examples]|3> lines SList (.p,.n,.l,.s,.grep(),.fields() available). Value: 0: total 23 1: -rw-rw-rw- 1 ville None 1163 Sep 30 2006 example-demo.py 2: -rw-rw-rw- 1 ville None 1927 Sep 30 2006 example-embed-short.py 3: -rwxrwxrwx 1 ville None 4606 Sep 1 17:15 example-embed.py 4: -rwxrwxrwx 1 ville None 1017 Sep 30 2006 example-gnuplot.py 5: -rwxrwxrwx 1 ville None 339 Jun 11 18:01 extension.py 6: -rwxrwxrwx 1 ville None 113 Dec 20 2006 seteditor.py 7: -rwxrwxrwx 1 ville None 245 Dec 12 2006 seteditor.pyc Now, let’s filter out the ‘embed’ lines: [Q:doc/examples]|4> l2 = lines.grep('embed',prune=1) [Q:doc/examples]|5> l2 SList (.p,.n,.l,.s,.grep(),.fields() available). Value: 0: total 23 1: -rw-rw-rw- 1 ville None 1163 Sep 30 2006 example-demo.py 2: -rwxrwxrwx 1 ville None 1017 Sep 30 2006 example-gnuplot.py 3: -rwxrwxrwx 1 ville None 339 Jun 11 18:01 extension.py 4: -rwxrwxrwx 1 ville None 113 Dec 20 2006 seteditor.py 5: -rwxrwxrwx 1 ville None 245 Dec 12 2006 seteditor.pyc Now, we want strings having just file names and permissions: [Q:doc/examples]|6> l2.fields(8,0) SList (.p,.n,.l,.s,.grep(),.fields() available). Value: 0: total 1: example-demo.py -rw-rw-rw- 2: example-gnuplot.py -rwxrwxrwx 3: extension.py -rwxrwxrwx 4: seteditor.py -rwxrwxrwx 5: seteditor.pyc -rwxrwxrwx Note how the line with ‘total’ does not raise IndexError. If you want to split these (yielding lists), call fields() without arguments: [Q:doc/examples]|7> _.fields() [['total'], ['example-demo.py', '-rw-rw-rw-'], ['example-gnuplot.py', '-rwxrwxrwx'], ['extension.py', '-rwxrwxrwx'], ['seteditor.py', '-rwxrwxrwx'], ['seteditor.pyc', '-rwxrwxrwx']] If you want to pass these separated with spaces to a command (typical for lists if files), use the.s property: [Q:doc/examples]|13> files = l2.fields(8).s [Q:doc/examples]|14> files 'example-demo.py example-gnuplot.py extension.py seteditor.py seteditor.pyc' [Q:doc/examples]|15> ls $files example-demo.py example-gnuplot.py extension.py seteditor.py seteditor.pyc SLists are inherited from normal python lists, so every list method is available: [Q:doc/examples]|21> lines.append('hey') Real world example: remove all files outside version control First, capture output of “hg status”: [Q:/ipython]|28> out = !hg status == ['M IPython\\extensions\\ipy_kitcfg.py', 'M IPython\\extensions\\ipy_rehashdir.py',... '? build\\lib\\IPython\\Debugger.py', '? build\\lib\\IPython\\extensions\\InterpreterExec.py', '? build\\lib\\IPython\\extensions\\InterpreterPasteInput.py',... (lines starting with ? are not under version control). [Q:/ipython]|35> junk = out.grep(r'^\?').fields(1) [Q:/ipython]|36> junk SList (.p,.n,.l,.s,.grep(),.fields() availab... 10: build\bdist.win32\winexe\temp\_ctypes.py 11: build\bdist.win32\winexe\temp\_hashlib.py 12: build\bdist.win32\winexe\temp\_socket.py Now we can just remove these files by doing ‘rm $junk.s’. The.s,.n,.p properties The.s property returns one string where lines are separated by single space (for convenient passing to system commands). The.n property return one string where the lines are separated by a newline (i.e. the original output of the function). If the items in string list are file names,.p can be used to get a list of “path” objects for convenient file manipulation. Shell Commands in IPython Any command that works at the command-line can be used in IPython by prefixing it with the ! character. For example, the ls, pwd, and echo commands can be run as follows: In : !ls myproject.txt In : !pwd /home/jake/projects/myproject In : !echo "printing from the shell" printing from the shell Passing Values to and from the Shell Shell commands can not only be called from IPython, but can also be made to interact with the IPython namespace. For example, you can save the output of any shell command to a Python list using the assignment operator: In : contents = !ls In : print(contents) ['myproject.txt'] In : directory = !pwd In : print(directory) ['/Users/jakevdp/notebooks/tmp/myproject'] Note that these results are not returned as lists, but as a special shell return type defined in IPython: In : type(directory) IPython.utils.text.SList This looks and acts a lot like a Python list, but has additional functionality, such as the grep and fields methods and the s, n, and p properties that allow you to search, filter, and display the results in convenient ways. For more information on these, you can use IPython's built-in help features. Communication in the other direction–passing Python variables into the shell–is possible using the {varname} syntax: In : message = "hello from Python" In : !echo {message} hello from Python The curly braces contain the variable name, which is replaced by the variable's contents in the shell command. Shell-Related Magic Commands If you play with IPython's shell commands for a while, you might notice that you cannot use !cd to navigate the filesystem: In : !pwd /home/jake/projects/myproject In : !cd.. In : !pwd /home/jake/projects/myproject The reason is that shell commands in the notebook are executed in a temporary subshell. If you'd like to change the working directory in a more enduring way, you can use the %cd magic command: In : %cd.. /home/jake/projects In fact, by default you can even use this without the % sign: In : cd myproject /home/jake/projects/myproject This is known as an automagic function, and this behavior can be toggled with the %automagic magic function. Besides %cd, other available shell-like magic functions are %cat, %cp, %env, %ls, %man, %mkdir, %more, %mv, %pwd, %rm, and %rmdir, any of which can be used without the % sign if automagic is on. This makes it so that you can almost treat the IPython prompt as if it's a normal shell: In : mkdir tmp In : ls myproject.txt tmp/ In : cp myproject.txt tmp/ In : ls tmp myproject.txt In : rm -r tmp This access to the shell from within the same terminal window as your Python session means that there is a lot less switching back and forth between interpreter and shell as you write your Python code. Errors and Debugging Most of the time when a Python script fails, it will raise an Exception. When the interpreter hits one of these exceptions, information about the cause of the error can be found in the traceback, which can be accessed from within Python. With the %xmode magic function, IPython allows you to control the amount of information printed when the exception is raised. Consider the following code: In : def func1(a, b): return a / b def func2(x): a = x b = x - 1 return func1(a, b) In : func2(1) --------------------------------------------------------------------------- ZeroDivisionError Traceback (most recent call last) in () ----> 1 func2(1) in func2(x) 5 a = x 6 b = x - 1 ----> 7 return func1(a, b) in func1(a, b) 1 def func1(a, b): ----> 2 return a / b 3 4 def func2(x): 5 a = x ZeroDivisionError: division by zero Calling func2 results in an error, and reading the printed trace lets us see exactly what happened. By default, this trace includes several lines showing the context of each step that led to the error. Using the %xmode magic function (short for Exception mode), we can change what information is printed. %xmode takes a single argument, the mode, and there are three possibilities: Plain, Context, and Verbose. The default is Context, and gives output like that just shown before. Plain is more compact and gives less information: In : %xmode Plain Exception reporting mode: Plain In : func2(1) Traceback (most recent call last): File "", line 1, in func2(1) File "", line 7, in func2 return func1(a, b) File "", line 2, in func1 return a / b ZeroDivisionError: division by zero The Verbose mode adds some extra information, including the arguments to any functions that are called: In : %xmode Verbose Exception reporting mode: Verbose In : func2(1) --------------------------------------------------------------------------- ZeroDivisionError Traceback (most recent call last) in () ----> 1 func2(1) global func2 = in func2(x=1) 5 a = x 6 b = x - 1 ----> 7 return func1(a, b) global func1 = a = 1 b = 0 in func1(a=1, b=0) 1 def func1(a, b): ----> 2 return a / b a = 1  b = 0 3 4 def func2(x): 5 a = x ZeroDivisionError: division by zero This extra information can help narrow-in on why the exception is being raised. So why not use the Verbose mode all the time? As code gets complicated, this kind of traceback can get extremely long. Depending on the context, sometimes the brevity of Default mode is easier to work with. Debugging: When Reading Tracebacks Is Not Enough The standard Python tool for interactive debugging is pdb, the Python debugger. This debugger lets the user step through the code line by line in order to see what might be causing a more difficult error. The IPython-enhanced version of this is ipdb, the IPython debugger. There are many ways to launch and use both these debuggers; we won't cover them fully here. Refer to the online documentation of these two utilities to learn more. In IPython, perhaps the most convenient interface to debugging is the %debug magic command. If you call it after hitting an exception, it will automatically open an interactive debugging prompt at the point of the exception. The ipdb prompt lets you explore the current state of the stack, explore the available variables, and even run Python commands! Let's look at the most recent exception, then do some basic tasks–print the values of a and b, and type quit to quit the debugging session: In : %debug > (2)func1() 1 def func1(a, b): ----> 2 return a / b 3 ipdb> print(a) 1 ipdb> print(b) 0 ipdb> quit The interactive debugger allows much more than this, though–we can even step up and down through the stack and explore the values of variables there: In : %debug > (2)func1() 1 def func1(a, b): ----> 2 return a / b 3 ipdb> up > (7)func2() 5 a = x 6 b = x - 1 ----> 7 return func1(a, b) ipdb> print(x) 1 ipdb> up > (1)() ----> 1 func2(1) ipdb> down > (7)func2() 5 a = x 6 b = x - 1 ----> 7 return func1(a, b) ipdb> quit This allows you to quickly find out not only what caused the error, but what function calls led up to the error. If you'd like the debugger to launch automatically whenever an exception is raised, you can use the %pdb magic function to turn on this automatic behavior: In : %xmode Plain %pdb on func2(1) Exception reporting mode: Plain Automatic pdb calling has been turned ON Traceback (most recent call last): File "", line 3, in func2(1) File "", line 7, in func2 return func1(a, b) File "", line 2, in func1 return a / b ZeroDivisionError: division by zero > (2)func1() 1 def func1(a, b): ----> 2 return a / b 3 ipdb> print(b) 0 ipdb> quit Finally, if you have a script that you'd like to run from the beginning in interactive mode, you can run it with the command %run -d, and use the next command to step through the lines of code interactively. Partial list of debugging commands There are many more available commands for interactive debugging than we've listed here; the following table contains a description of some of the more common and useful ones: Command Description list Show the current location in the file h(elp) Show a list of commands, or find help on a specific command q(uit) Quit the debugger and the program c(ontinue)Quit the debugger, continue in the program n(ext) Go to the next step of the program Repeat the previous command p(rint) Print variables s(tep) Step into a subroutine r(eturn) Return out of a subroutine Profiling and Timing CODE In the process of developing code and creating data processing pipelines, there are often trade-offs you can make between various implementations. Early in developing your algorithm, it can be counterproductive to worry about such things. As Donald Knuth famously quipped, "We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil." But once you have your code working, it can be useful to dig into its efficiency a bit. Sometimes it's useful to check the execution time of a given command or set of commands; other times it's useful to examine a multiline process and determine where the bottleneck lies in some complicated series of operations. IPython provides access to a wide array of functionality for this kind of timing and profiling of code. Here we'll discuss the following IPython magic commands: - `%time`: Time the execution of a single statement - `%timeit`: Time repeated execution of a single statement for more accuracy - `%prun`: Run code with the profiler - `%lprun`: Run code with the line-by-line profiler - `%memit`: Measure the memory use of a single statement - `%mprun`: Run code with the line-by-line memory profiler The last four commands are not bundled with IPython; to use them you'll need to get the `line_profiler` and `memory_profiler` extensions, which we will discuss in the following sections. ## Timing Code Snippets: %timeit and %time We saw the `%timeit` line magic and `%%timeit` cell magic in the introduction to magic functions in [IPython Magic Commands](01.03-Magic-Commands.ipynb); these can be used to time the repeated execution of snippets of code: %timeit sum(range(100)) Note that because this operation is so fast, `%timeit` automatically does a large number of repetitions. For slower commands, `%timeit` will automatically adjust and perform fewer repetitions: %%timeit total = 0 for i in range(1000): for j in range(1000): total += i * (-1) ** j Sometimes repeating an operation is not the best option. For example, if we have a list that we'd like to sort, we might be misled by a repeated operation; sorting a pre-sorted list is much faster than sorting an unsorted list, so the repetition will skew the result: import random L = [random.random() for i in range(100000)] %timeit L.sort() For this, the `%time` magic function may be a better choice. It also is a good choice for longer-running commands, when short, system-related delays are unlikely to affect the result. Let's time the sorting of an unsorted and a presorted list: import random L = [random.random() for i in range(100000)] print("sorting an unsorted list:") %time L.sort() print("sorting an already sorted list:") %time L.sort() Notice how much faster the presorted list is to sort, but notice also how much longer the timing takes with `%time` versus `%timeit`, even for the presorted list! This is a result of the fact that `%timeit` does some clever things under the hood to prevent system calls from interfering with the timing. For example, it prevents cleanup of unused Python objects (known as *garbage collection*) that might otherwise affect the timing. For this reason, `%timeit` results are usually noticeably faster than `%time` results. For `%time`, as with `%timeit`, using the `%%` cell magic syntax allows timing of multiline scripts: %%time total = 0 for i in range(1000): for j in range(1000): total += i * (-1) ** j For more information on `%time` and `%timeit`, as well as their available options, use the IPython help functionality (e.g., type `%time?` at the IPython prompt). ## Profiling Full Scripts: %prun A program is made up of many single statements, and sometimes timing these statements in context is more important than timing them on their own. Python contains a built-in code profiler (which you can read about in the Python documentation), but IPython offers a much more convenient way to use this profiler, in the form of the magic function `%prun`. By way of example, we'll define a simple function that does some calculations: def sum_of_lists(N): total = 0 for i in range(5): L = [j ^ (j >> i) for j in range(N)] total += sum(L) return total Now we can call `%prun` with a function call to see the profiled results: %prun sum_of_lists(1000000) The result is a table that indicates, in order of total time on each function call, where the execution is spending the most time. In this case, the bulk of the execution time is in the list comprehension inside `sum_of_lists`. From here, we could start thinking about what changes we might make to improve the performance of the algorithm. For more information on `%prun`, as well as its available options, use the IPython help functionality (i.e., type `%prun?` at the IPython prompt). ## Line-by-Line Profiling with %lprun The function-by-function profiling of `%prun` is useful, but sometimes it's more convenient to have a line-by-line profile report. This is not built into Python or IPython, but there is a `line_profiler` package available for installation that can do this. Start by using Python's packaging tool, `pip`, to install the `line_profiler` package: ``` $ pip install line_profiler ``` Next, you can use IPython to load the `line_profiler` IPython extension, offered as part of this package: