Fluent Python Programming PDF

2n ers C ov d Ed tho Py i n tio 3.1 Fluent...

2n ers C ov d Ed tho Py i n tio 3.1 Fluent n 0 Python Clear, Concise, and Effective Programming Luciano Ramalho  SECOND EDITION Fluent Python Clear, Concise, and Effective Programming Luciano Ramalho Beijing Boston Farnham Sebastopol Tokyo Fluent Python by Luciano Ramalho Copyright © 2022 Luciano Ramalho. All rights reserved. Printed in the United States of America. Published by O’Reilly Media, Inc., 1005 Gravenstein Highway North, Sebastopol, CA 95472. O’Reilly books may be purchased for educational, business, or sales promotional use. Online editions are also available for most titles (http://oreilly.com). For more information, contact our corporate/institu‐ tional sales department: 800-998-9938 or [email protected]. Acquisitions Editor: Amanda Quinn Indexer: Judith McConville Development Editor: Jeff Bleiel Interior Designer: David Futato Production Editor: Daniel Elfanbaum Cover Designer: Karen Montgomery Copyeditor: Sonia Saruba Illustrator: Kate Dullea Proofreader: Kim Cofer April 2022: Second Edition Revision History for the Second Edition 2022-03-31: First Release See http://oreilly.com/catalog/errata.csp?isbn=9781492056355 for release details. The O’Reilly logo is a registered trademark of O’Reilly Media, Inc. Fluent Python, the cover image, and related trade dress are trademarks of O’Reilly Media, Inc. The views expressed in this work are those of the author and do not represent the publisher’s views. While the publisher and the author have used good faith efforts to ensure that the information and instructions contained in this work are accurate, the publisher and the author disclaim all responsibility for errors or omissions, including without limitation responsibility for damages resulting from the use of or reliance on this work. Use of the information and instructions contained in this work is at your own risk. If any code samples or other technology this work contains or describes is subject to open source licenses or the intellectual property rights of others, it is your responsibility to ensure that your use thereof complies with such licenses and/or rights. 978-1-492-05635-5 [LSI] Para Marta, com todo o meu amor.  Table of Contents Preface...................................................................... xix Part I. Data Structures 1. The Python Data Model..................................................... 3 What’s New in This Chapter 4 A Pythonic Card Deck 5 How Special Methods Are Used 8 Emulating Numeric Types 9 String Representation 12 Boolean Value of a Custom Type 13 Collection API 14 Overview of Special Methods 15 Why len Is Not a Method 17 Chapter Summary 18 Further Reading 18 2. An Array of Sequences..................................................... 21 What’s New in This Chapter 22 Overview of Built-In Sequences 22 List Comprehensions and Generator Expressions 25 List Comprehensions and Readability 25 Listcomps Versus map and filter 27 Cartesian Products 27 Generator Expressions 29 Tuples Are Not Just Immutable Lists 30 Tuples as Records 30 v  Tuples as Immutable Lists 32 Comparing Tuple and List Methods 34 Unpacking Sequences and Iterables 35 Using * to Grab Excess Items 36 Unpacking with * in Function Calls and Sequence Literals 37 Nested Unpacking 37 Pattern Matching with Sequences 38 Pattern Matching Sequences in an Interpreter 43 Slicing 47 Why Slices and Ranges Exclude the Last Item 47 Slice Objects 48 Multidimensional Slicing and Ellipsis 49 Assigning to Slices 50 Using + and * with Sequences 50 Building Lists of Lists 51 Augmented Assignment with Sequences 53 A += Assignment Puzzler 54 list.sort Versus the sorted Built-In 56 When a List Is Not the Answer 59 Arrays 59 Memory Views 62 NumPy 64 Deques and Other Queues 67 Chapter Summary 70 Further Reading 71 3. Dictionaries and Sets...................................................... 77 What’s New in This Chapter 78 Modern dict Syntax 78 dict Comprehensions 79 Unpacking Mappings 80 Merging Mappings with | 80 Pattern Matching with Mappings 81 Standard API of Mapping Types 83 What Is Hashable 84 Overview of Common Mapping Methods 85 Inserting or Updating Mutable Values 87 Automatic Handling of Missing Keys 90 defaultdict: Another Take on Missing Keys 90 The __missing__ Method 91 Inconsistent Usage of __missing__ in the Standard Library 94 Variations of dict 95 vi | Table of Contents  collections.OrderedDict 95 collections.ChainMap 95 collections.Counter 96 shelve.Shelf 97 Subclassing UserDict Instead of dict 97 Immutable Mappings 99 Dictionary Views 101 Practical Consequences of How dict Works 102 Set Theory 103 Set Literals 105 Set Comprehensions 106 Practical Consequences of How Sets Work 107 Set Operations 107 Set Operations on dict Views 110 Chapter Summary 112 Further Reading 113 4. Unicode Text Versus Bytes................................................. 117 What’s New in This Chapter 118 Character Issues 118 Byte Essentials 120 Basic Encoders/Decoders 123 Understanding Encode/Decode Problems 125 Coping with UnicodeEncodeError 125 Coping with UnicodeDecodeError 126 SyntaxError When Loading Modules with Unexpected Encoding 128 How to Discover the Encoding of a Byte Sequence 128 BOM: A Useful Gremlin 129 Handling Text Files 131 Beware of Encoding Defaults 134 Normalizing Unicode for Reliable Comparisons 140 Case Folding 142 Utility Functions for Normalized Text Matching 143 Extreme “Normalization”: Taking Out Diacritics 144 Sorting Unicode Text 148 Sorting with the Unicode Collation Algorithm 150 The Unicode Database 150 Finding Characters by Name 151 Numeric Meaning of Characters 153 Dual-Mode str and bytes APIs 155 str Versus bytes in Regular Expressions 155 str Versus bytes in os Functions 156 Table of Contents | vii  Chapter Summary 157 Further Reading 158 5. Data Class Builders....................................................... 163 What’s New in This Chapter 164 Overview of Data Class Builders 164 Main Features 167 Classic Named Tuples 169 Typed Named Tuples 172 Type Hints 101 173 No Runtime Effect 173 Variable Annotation Syntax 174 The Meaning of Variable Annotations 175 More About @dataclass 179 Field Options 180 Post-init Processing 183 Typed Class Attributes 185 Initialization Variables That Are Not Fields 186 @dataclass Example: Dublin Core Resource Record 187 Data Class as a Code Smell 190 Data Class as Scaffolding 191 Data Class as Intermediate Representation 191 Pattern Matching Class Instances 192 Simple Class Patterns 192 Keyword Class Patterns 193 Positional Class Patterns 194 Chapter Summary 195 Further Reading 196 6. Object References, Mutability, and Recycling................................. 201 What’s New in This Chapter 202 Variables Are Not Boxes 202 Identity, Equality, and Aliases 204 Choosing Between == and is 206 The Relative Immutability of Tuples 207 Copies Are Shallow by Default 208 Deep and Shallow Copies of Arbitrary Objects 211 Function Parameters as References 213 Mutable Types as Parameter Defaults: Bad Idea 214 Defensive Programming with Mutable Parameters 216 del and Garbage Collection 219 Tricks Python Plays with Immutables 221 viii | Table of Contents  Chapter Summary 223 Further Reading 224 Part II. Functions as Objects 7. Functions as First-Class Objects............................................. 231 What’s New in This Chapter 232 Treating a Function Like an Object 232 Higher-Order Functions 234 Modern Replacements for map, filter, and reduce 235 Anonymous Functions 236 The Nine Flavors of Callable Objects 237 User-Defined Callable Types 239 From Positional to Keyword-Only Parameters 240 Positional-Only Parameters 242 Packages for Functional Programming 243 The operator Module 243 Freezing Arguments with functools.partial 247 Chapter Summary 249 Further Reading 250 8. Type Hints in Functions................................................... 253 What’s New in This Chapter 254 About Gradual Typing 254 Gradual Typing in Practice 255 Starting with Mypy 256 Making Mypy More Strict 257 A Default Parameter Value 258 Using None as a Default 260 Types Are Defined by Supported Operations 260 Types Usable in Annotations 266 The Any Type 266 Simple Types and Classes 269 Optional and Union Types 270 Generic Collections 271 Tuple Types 274 Generic Mappings 276 Abstract Base Classes 278 Iterable 280 Parameterized Generics and TypeVar 282 Static Protocols 286 Table of Contents | ix  Callable 291 NoReturn 294 Annotating Positional Only and Variadic Parameters 295 Imperfect Typing and Strong Testing 296 Chapter Summary 297 Further Reading 298 9. Decorators and Closures................................................... 303 What’s New in This Chapter 304 Decorators 101 304 When Python Executes Decorators 306 Registration Decorators 308 Variable Scope Rules 308 Closures 311 The nonlocal Declaration 315 Variable Lookup Logic 316 Implementing a Simple Decorator 317 How It Works 318 Decorators in the Standard Library 320 Memoization with functools.cache 320 Using lru_cache 323 Single Dispatch Generic Functions 324 Parameterized Decorators 329 A Parameterized Registration Decorator 329 The Parameterized Clock Decorator 332 A Class-Based Clock Decorator 335 Chapter Summary 336 Further Reading 336 10. Design Patterns with First-Class Functions................................... 341 What’s New in This Chapter 342 Case Study: Refactoring Strategy 342 Classic Strategy 342 Function-Oriented Strategy 347 Choosing the Best Strategy: Simple Approach 350 Finding Strategies in a Module 351 Decorator-Enhanced Strategy Pattern 353 The Command Pattern 355 Chapter Summary 357 Further Reading 358 x | Table of Contents Part III. Classes and Protocols 11. A Pythonic Object........................................................ 363 What’s New in This Chapter 364 Object Representations 364 Vector Class Redux 365 An Alternative Constructor 368 classmethod Versus staticmethod 369 Formatted Displays 370 A Hashable Vector2d 374 Supporting Positional Pattern Matching 377 Complete Listing of Vector2d, Version 3 378 Private and “Protected” Attributes in Python 382 Saving Memory with __slots__ 384 Simple Measure of __slot__ Savings 387 Summarizing the Issues with __slots__ 388 Overriding Class Attributes 389 Chapter Summary 391 Further Reading 392 12. Special Methods for Sequences............................................. 397 What’s New in This Chapter 398 Vector: A User-Defined Sequence Type 398 Vector Take #1: Vector2d Compatible 399 Protocols and Duck Typing 402 Vector Take #2: A Sliceable Sequence 403 How Slicing Works 404 A Slice-Aware __getitem__ 406 Vector Take #3: Dynamic Attribute Access 407 Vector Take #4: Hashing and a Faster == 411 Vector Take #5: Formatting 418 Chapter Summary 425 Further Reading 426 13. Interfaces, Protocols, and ABCs............................................. 431 The Typing Map 432 What’s New in This Chapter 433 Two Kinds of Protocols 434 Programming Ducks 435 Python Digs Sequences 436 Monkey Patching: Implementing a Protocol at Runtime 438 Defensive Programming and “Fail Fast” 440 Table of Contents | xi  Goose Typing 442 Subclassing an ABC 447 ABCs in the Standard Library 449 Defining and Using an ABC 451 ABC Syntax Details 457 Subclassing an ABC 458 A Virtual Subclass of an ABC 460 Usage of register in Practice 463 Structural Typing with ABCs 464 Static Protocols 466 The Typed double Function 466 Runtime Checkable Static Protocols 468 Limitations of Runtime Protocol Checks 471 Supporting a Static Protocol 472 Designing a Static Protocol 474 Best Practices for Protocol Design 476 Extending a Protocol 477 The numbers ABCs and Numeric Protocols 478 Chapter Summary 481 Further Reading 482 14. Inheritance: For Better or for Worse......................................... 487 What’s New in This Chapter 488 The super() Function 488 Subclassing Built-In Types Is Tricky 490 Multiple Inheritance and Method Resolution Order 494 Mixin Classes 500 Case-Insensitive Mappings 500 Multiple Inheritance in the Real World 502 ABCs Are Mixins Too 502 ThreadingMixIn and ForkingMixIn 503 Django Generic Views Mixins 504 Multiple Inheritance in Tkinter 507 Coping with Inheritance 510 Favor Object Composition over Class Inheritance 510 Understand Why Inheritance Is Used in Each Case 510 Make Interfaces Explicit with ABCs 511 Use Explicit Mixins for Code Reuse 511 Provide Aggregate Classes to Users 511 Subclass Only Classes Designed for Subclassing 512 Avoid Subclassing from Concrete Classes 513 Tkinter: The Good, the Bad, and the Ugly 513 xii | Table of Contents  Chapter Summary 514 Further Reading 515 15. More About Type Hints.................................................... 519 What’s New in This Chapter 519 Overloaded Signatures 520 Max Overload 521 Takeaways from Overloading max 525 TypedDict 526 Type Casting 534 Reading Type Hints at Runtime 537 Problems with Annotations at Runtime 538 Dealing with the Problem 540 Implementing a Generic Class 541 Basic Jargon for Generic Types 544 Variance 544 An Invariant Dispenser 545 A Covariant Dispenser 546 A Contravariant Trash Can 547 Variance Review 549 Implementing a Generic Static Protocol 552 Chapter Summary 554 Further Reading 555 16. Operator Overloading..................................................... 561 What’s New in This Chapter 562 Operator Overloading 101 562 Unary Operators 563 Overloading + for Vector Addition 566 Overloading * for Scalar Multiplication 572 Using @ as an Infix Operator 574 Wrapping-Up Arithmetic Operators 576 Rich Comparison Operators 577 Augmented Assignment Operators 580 Chapter Summary 585 Further Reading 587 Part IV. Control Flow 17. Iterators, Generators, and Classic Coroutines................................. 593 What’s New in This Chapter 594 Table of Contents | xiii  A Sequence of Words 594 Why Sequences Are Iterable: The iter Function 596 Using iter with a Callable 598 Iterables Versus Iterators 599 Sentence Classes with __iter__ 603 Sentence Take #2: A Classic Iterator 603 Don’t Make the Iterable an Iterator for Itself 605 Sentence Take #3: A Generator Function 606 How a Generator Works 607 Lazy Sentences 610 Sentence Take #4: Lazy Generator 610 Sentence Take #5: Lazy Generator Expression 611 When to Use Generator Expressions 613 An Arithmetic Progression Generator 615 Arithmetic Progression with itertools 618 Generator Functions in the Standard Library 619 Iterable Reducing Functions 630 Subgenerators with yield from 632 Reinventing chain 633 Traversing a Tree 634 Generic Iterable Types 639 Classic Coroutines 641 Example: Coroutine to Compute a Running Average 643 Returning a Value from a Coroutine 646 Generic Type Hints for Classic Coroutines 650 Chapter Summary 652 Further Reading 652 18. with, match, and else Blocks............................................... 657 What’s New in This Chapter 658 Context Managers and with Blocks 658 The contextlib Utilities 663 Using @contextmanager 664 Pattern Matching in lis.py: A Case Study 669 Scheme Syntax 669 Imports and Types 671 The Parser 671 The Environment 673 The REPL 675 The Evaluator 676 Procedure: A Class Implementing a Closure 685 Using OR-patterns 686 xiv | Table of Contents  Do This, Then That: else Blocks Beyond if 687 Chapter Summary 689 Further Reading 690 19. Concurrency Models in Python............................................. 695 What’s New in This Chapter 696 The Big Picture 696 A Bit of Jargon 697 Processes, Threads, and Python’s Infamous GIL 699 A Concurrent Hello World 701 Spinner with Threads 701 Spinner with Processes 704 Spinner with Coroutines 706 Supervisors Side-by-Side 711 The Real Impact of the GIL 713 Quick Quiz 713 A Homegrown Process Pool 716 Process-Based Solution 718 Understanding the Elapsed Times 718 Code for the Multicore Prime Checker 719 Experimenting with More or Fewer Processes 723 Thread-Based Nonsolution 724 Python in the Multicore World 725 System Administration 726 Data Science 727 Server-Side Web/Mobile Development 728 WSGI Application Servers 730 Distributed Task Queues 732 Chapter Summary 733 Further Reading 734 Concurrency with Threads and Processes 734 The GIL 736 Concurrency Beyond the Standard Library 736 Concurrency and Scalability Beyond Python 738 20. Concurrent Executors..................................................... 743 What’s New in This Chapter 743 Concurrent Web Downloads 744 A Sequential Download Script 746 Downloading with concurrent.futures 749 Where Are the Futures? 751 Launching Processes with concurrent.futures 754 Table of Contents | xv  Multicore Prime Checker Redux 755 Experimenting with Executor.map 758 Downloads with Progress Display and Error Handling 762 Error Handling in the flags2 Examples 766 Using futures.as_completed 769 Chapter Summary 772 Further Reading 772 21. Asynchronous Programming............................................... 775 What’s New in This Chapter 776 A Few Definitions 777 An asyncio Example: Probing Domains 778 Guido’s Trick to Read Asynchronous Code 780 New Concept: Awaitable 781 Downloading with asyncio and HTTPX 782 The Secret of Native Coroutines: Humble Generators 784 The All-or-Nothing Problem 785 Asynchronous Context Managers 786 Enhancing the asyncio Downloader 787 Using asyncio.as_completed and a Thread 788 Throttling Requests with a Semaphore 790 Making Multiple Requests for Each Download 794 Delegating Tasks to Executors 797 Writing asyncio Servers 799 A FastAPI Web Service 800 An asyncio TCP Server 804 Asynchronous Iteration and Asynchronous Iterables 811 Asynchronous Generator Functions 812 Async Comprehensions and Async Generator Expressions 818 async Beyond asyncio: Curio 821 Type Hinting Asynchronous Objects 824 How Async Works and How It Doesn’t 825 Running Circles Around Blocking Calls 825 The Myth of I/O-Bound Systems 826 Avoiding CPU-Bound Traps 826 Chapter Summary 827 Further Reading 828 xvi | Table of Contents Part V. Metaprogramming 22. Dynamic Attributes and Properties......................................... 835 What’s New in This Chapter 836 Data Wrangling with Dynamic Attributes 836 Exploring JSON-Like Data with Dynamic Attributes 838 The Invalid Attribute Name Problem 842 Flexible Object Creation with __new__ 843 Computed Properties 845 Step 1: Data-Driven Attribute Creation 846 Step 2: Property to Retrieve a Linked Record 848 Step 3: Property Overriding an Existing Attribute 852 Step 4: Bespoke Property Cache 853 Step 5: Caching Properties with functools 855 Using a Property for Attribute Validation 857 LineItem Take #1: Class for an Item in an Order 857 LineItem Take #2: A Validating Property 858 A Proper Look at Properties 860 Properties Override Instance Attributes 861 Property Documentation 864 Coding a Property Factory 865 Handling Attribute Deletion 868 Essential Attributes and Functions for Attribute Handling 869 Special Attributes that Affect Attribute Handling 870 Built-In Functions for Attribute Handling 870 Special Methods for Attribute Handling 871 Chapter Summary 873 Further Reading 873 23. Attribute Descriptors..................................................... 879 What’s New in This Chapter 880 Descriptor Example: Attribute Validation 880 LineItem Take #3: A Simple Descriptor 880 LineItem Take #4: Automatic Naming of Storage Attributes 887 LineItem Take #5: A New Descriptor Type 889 Overriding Versus Nonoverriding Descriptors 892 Overriding Descriptors 894 Overriding Descriptor Without __get__ 895 Nonoverriding Descriptor 896 Overwriting a Descriptor in the Class 897 Methods Are Descriptors 898 Descriptor Usage Tips 900 Table of Contents | xvii  Descriptor Docstring and Overriding Deletion 902 Chapter Summary 903 Further Reading 904 24. Class Metaprogramming.................................................. 907 What’s New in This Chapter 908 Classes as Objects 908 type: The Built-In Class Factory 909 A Class Factory Function 911 Introducing __init_subclass__ 914 Why __init_subclass__ Cannot Configure __slots__ 921 Enhancing Classes with a Class Decorator 922 What Happens When: Import Time Versus Runtime 925 Evaluation Time Experiments 926 Metaclasses 101 931 How a Metaclass Customizes a Class 933 A Nice Metaclass Example 934 Metaclass Evaluation Time Experiment 937 A Metaclass Solution for Checked 942 Metaclasses in the Real World 947 Modern Features Simplify or Replace Metaclasses 947 Metaclasses Are Stable Language Features 948 A Class Can Only Have One Metaclass 948 Metaclasses Should Be Implementation Details 949 A Metaclass Hack with __prepare__ 950 Wrapping Up 952 Chapter Summary 953 Further Reading 954 Afterword................................................................... 959 Index....................................................................... 963 xviii | Table of Contents  Preface Here’s the plan: when someone uses a feature you don’t understand, simply shoot them. This is easier than learning something new, and before too long the only living coders will be writing in an easily understood, tiny subset of Python 0.9.6.1 —Tim Peters, legendary core developer and author of The Zen of Python “Python is an easy to learn, powerful programming language.” Those are the first words of the official Python 3.10 tutorial. That is true, but there is a catch: because the language is easy to learn and put to use, many practicing Python programmers leverage only a fraction of its powerful features. An experienced programmer may start writing useful Python code in a matter of hours. As the first productive hours become weeks and months, a lot of developers go on writing Python code with a very strong accent carried from languages learned before. Even if Python is your first language, often in academia and in introductory books it is presented while carefully avoiding language-specific features. As a teacher introducing Python to programmers experienced in other languages, I see another problem that this book tries to address: we only miss stuff we know about. Coming from another language, anyone may guess that Python supports regu‐ lar expressions, and look that up in the docs. But if you’ve never seen tuple unpacking or descriptors before, you will probably not search for them, and you may end up not using those features just because they are specific to Python. This book is not an A-to-Z exhaustive reference of Python. Its emphasis is on the lan‐ guage features that are either unique to Python or not found in many other popular languages. This is also mostly a book about the core language and some of its libra‐ ries. I will rarely talk about packages that are not in the standard library, even though the Python package index now lists more than 60,000 libraries, and many of them are incredibly useful. 1 Message to the comp.lang.python Usenet group, Dec. 23, 2002: “Acrimony in c.l.p”. xix Who This Book Is For This book was written for practicing Python programmers who want to become pro‐ ficient in Python 3. I tested the examples in Python 3.10—most of them also in Python 3.9 and 3.8. When an example requires Python 3.10, it should be clearly marked. If you are not sure whether you know enough Python to follow along, review the top‐ ics of the official Python tutorial. Topics covered in the tutorial will not be explained here, except for some features that are new. Who This Book Is Not For If you are just learning Python, this book is going to be hard to follow. Not only that, if you read it too early in your Python journey, it may give you the impression that every Python script should leverage special methods and metaprogramming tricks. Premature abstraction is as bad as premature optimization. Five Books in One I recommend that everyone read Chapter 1, “The Python Data Model”. The core audience for this book should not have trouble jumping directly to any part in this book after Chapter 1, but often I assume you’ve read preceding chapters in each spe‐ cific part. Think of Parts I through V as books within the book. I tried to emphasize using what is available before discussing how to build your own. For example, in Part I, Chapter 2 covers sequence types that are ready to use, includ‐ ing some that don’t get a lot of attention, like collections.deque. Building user- defined sequences is only addressed in Part III, where we also see how to leverage the abstract base classes (ABCs) from collections.abc. Creating your own ABCs is dis‐ cussed even later in Part III, because I believe it’s important to be comfortable using an ABC before writing your own. This approach has a few advantages. First, knowing what is ready to use can save you from reinventing the wheel. We use existing collection classes more often than we implement our own, and we can give more attention to the advanced usage of avail‐ able tools by deferring the discussion on how to create new ones. We are also more likely to inherit from existing ABCs than to create a new ABC from scratch. And finally, I believe it is easier to understand the abstractions after you’ve seen them in action. The downside of this strategy is the forward references scattered throughout the chapters. I hope these will be easier to tolerate now that you know why I chose this path. xx | Preface How the Book Is Organized Here are the main topics in each part of the book: Part I, “Data Structures” Chapter 1 introduces the Python Data Model and explains why the special meth‐ ods (e.g., __repr__) are the key to the consistent behavior of objects of all types. Special methods are covered in more detail throughout the book. The remaining chapters in this part cover the use of collection types: sequences, mappings, and sets, as well as the str versus bytes split—the cause of much celebration among Python 3 users and much pain for Python 2 users migrating their codebases. Also covered are the high-level class builders in the standard library: named tuple fac‐ tories and the @dataclass decorator. Pattern matching—new in Python 3.10—is covered in sections in Chapters 2, 3, and 5, which discuss sequence patterns, mapping patterns, and class patterns. The last chapter in Part I is about the life cycle of objects: references, mutability, and garbage collection. Part II, “Functions as Objects” Here we talk about functions as first-class objects in the language: what that means, how it affects some popular design patterns, and how to implement func‐ tion decorators by leveraging closures. Also covered here is the general concept of callables in Python, function attributes, introspection, parameter annotations, and the new nonlocal declaration in Python 3. Chapter 8 introduces the major new topic of type hints in function signatures. Part III, “Classes and Protocols” Now the focus is on building classes “by hand”—as opposed to using the class builders covered in Chapter 5. Like any Object-Oriented (OO) language, Python has its particular set of features that may or may not be present in the language in which you and I learned class-based programming. The chapters explain how to build your own collections, abstract base classes (ABCs), and protocols, as well as how to cope with multiple inheritance, and how to implement operator overloading—when that makes sense. Chapter 15 continues the coverage of type hints. Part IV, “Control Flow” Covered in this part are the language constructs and libraries that go beyond tra‐ ditional control flow with conditionals, loops, and subroutines. We start with generators, then visit context managers and coroutines, including the challenging but powerful new yield from syntax. Chapter 18 includes a significant example using pattern matching in a simple but functional language interpreter. Chap‐ ter 19, “Concurrency Models in Python” is a new chapter presenting an overview of alternatives for concurrent and parallel processing in Python, their limitations, and how software architecture allows Python to operate at web scale. I rewrote Preface | xxi  the chapter about asynchronous programming to emphasize core language fea‐ tures—e.g., await, async dev, async for, and async with, and show how they are used with asyncio and other frameworks. Part V, “Metaprogramming” This part starts with a review of techniques for building classes with attributes created dynamically to handle semi-structured data, such as JSON datasets. Next, we cover the familiar properties mechanism, before diving into how object attribute access works at a lower level in Python using descriptors. The relation‐ ship among functions, methods, and descriptors is explained. Throughout Part V, the step-by-step implementation of a field validation library uncovers subtle issues that lead to the advanced tools of the final chapter: class decorators and metaclasses. Hands-On Approach Often we’ll use the interactive Python console to explore the language and libraries. I feel it is important to emphasize the power of this learning tool, particularly for those readers who’ve had more experience with static, compiled languages that don’t pro‐ vide a read-eval-print loop (REPL). One of the standard Python testing packages, doctest, works by simulating console sessions and verifying that the expressions evaluate to the responses shown. I used doctest to check most of the code in this book, including the console listings. You don’t need to use or even know about doctest to follow along: the key feature of doctests is that they look like transcripts of interactive Python console sessions, so you can easily try out the demonstrations yourself. Sometimes I will explain what we want to accomplish by showing a doctest before the code that makes it pass. Firmly establishing what is to be done before thinking about how to do it helps focus our coding effort. Writing tests first is the basis of test-driven development (TDD), and I’ve also found it helpful when teaching. If you are unfami‐ liar with doctest, take a look at its documentation and this book’s example code repository. I also wrote unit tests for some of the larger examples using pytest—which I find eas‐ ier to use and more powerful than the unittest module in the standard library. You’ll find that you can verify the correctness of most of the code in the book by typing python3 -m doctest example_script.py or pytest in the command shell of your OS. The pytest.ini configuration at the root of the example code repository ensures that doctests are collected and executed by the pytest command. xxii | Preface Soapbox: My Personal Perspective I have been using, teaching, and debating Python since 1998, and I enjoy studying and comparing programming languages, their design, and the theory behind them. At the end of some chapters, I have added “Soapbox” sidebars with my own perspec‐ tive about Python and other languages. Feel free to skip these if you are not into such discussions. Their content is completely optional. Companion Website: fluentpython.com Covering new features—like type hints, data classes, and pattern matching—made this second edition almost 30% larger than the first. To keep the book luggable, I moved some content to fluentpython.com. You will find links to articles I published there in several chapters. Some sample chapters are also in the companion website. The full text is available online at the O’Reilly Learning subscription service. The example code repository is on GitHub. Conventions Used in This Book The following typographical conventions are used in this book: Italic Indicates new terms, URLs, email addresses, filenames, and file extensions. Constant width Used for program listings, as well as within paragraphs to refer to program ele‐ ments such as variable or function names, databases, data types, environment variables, statements, and keywords. Note that when a line break falls within a constant_width term, a hyphen is not added—it could be misunderstood as part of the term. Constant width bold Shows commands or other text that should be typed literally by the user. Constant width italic Shows text that should be replaced with user-supplied values or by values deter‐ mined by context. Preface | xxiii  This element signifies a tip or suggestion. This element signifies a general note. This element indicates a warning or caution. Using Code Examples Every script and most code snippets that appear in the book are available in the Flu‐ ent Python code repository on GitHub at https://fpy.li/code. If you have a technical question or a problem using the code examples, please send email to [email protected]. This book is here to help you get your job done. In general, if example code is offered with this book, you may use it in your programs and documentation. You do not need to contact us for permission unless you’re reproducing a significant portion of the code. For example, writing a program that uses several chunks of code from this book does not require permission. Selling or distributing examples from O’Reilly books does require permission. Answering a question by citing this book and quoting example code does not require permission. Incorporating a significant amount of example code from this book into your product’s documentation does require permission. We appreciate, but generally do not require, attribution. An attribution usually includes the title, author, publisher, and ISBN, e.g., “Fluent Python, 2nd ed., by Luciano Ramalho (O’Reilly). Copyright 2022 Luciano Ramalho, 978-1-492-05635-5.” If you feel your use of code examples falls outside fair use or the permission given above, feel free to contact us at [email protected]. xxiv | Preface O’Reilly Online Learning For more than 40 years, O’Reilly Media has provided technol‐ ogy and business training, knowledge, and insight to help companies succeed. Our unique network of experts and innovators share their knowledge and expertise through books, articles, and our online learning platform. O’Reilly’s online learning platform gives you on-demand access to live training courses, in-depth learning paths, interactive coding environments, and a vast collection of text and video from O’Reilly and 200+ other publishers. For more information, visit http://oreilly.com. How to Contact Us Please address comments and questions concerning this book to the publisher: O’Reilly Media, Inc. 1005 Gravenstein Highway North Sebastopol, CA 95472 800-998-9938 (in the United States or Canada) 707-829-0515 (international or local) 707-829-0104 (fax) We have a web page for this book, where we list errata, examples, and any additional information. You can access this page at https://fpy.li/p-4. Email [email protected] to comment or ask technical questions about this book. For news and information about our books and courses, visit http://oreilly.com. Find us on Facebook: http://facebook.com/oreilly. Follow us on Twitter: https://twitter.com/oreillymedia. Watch us on YouTube: http://www.youtube.com/oreillymedia. Acknowledgments I did not expect updating a Python book five years later to be such a major undertak‐ ing, but it was. Marta Mello, my beloved wife, was always there when I needed her. My dear friend Leonardo Rochael helped me from the earliest writing to the final technical review, including consolidating and double-checking the feedback from the other tech reviewers, readers, and editors. I honestly don’t know if I’d have made it without your support, Marta and Leo. Thank you so much! Preface | xxv Jürgen Gmach, Caleb Hattingh, Jess Males, Leonardo Rochael, and Miroslav Šedivý were the outstanding technical review team for the second edition. They reviewed the whole book. Bill Behrman, Bruce Eckel, Renato Oliveira, and Rodrigo Bernardo Pimentel reviewed specific chapters. Their many suggestions from different perspec‐ tives made the book much better. Many readers sent corrections or made other contributions during the early release phase, including: Guilherme Alves, Christiano Anderson, Konstantin Baikov, K. Alex Birch, Michael Boesl, Lucas Brunialti, Sergio Cortez, Gino Crecco, Chukwuerika Dike, Juan Esteras, Federico Fissore, Will Frey, Tim Gates, Alexander Hagerman, Chen Hanxiao, Sam Hyeong, Simon Ilincev, Parag Kalra, Tim King, David Kwast, Tina Lapine, Wanpeng Li, Guto Maia, Scott Martindale, Mark Meyer, Andy McFar‐ land, Chad McIntire, Diego Rabatone Oliveira, Francesco Piccoli, Meredith Rawls, Michael Robinson, Federico Tula Rovaletti, Tushar Sadhwani, Arthur Constantino Scardua, Randal L. Schwartz, Avichai Sefati, Guannan Shen, William Simpson, Vivek Vashist, Jerry Zhang, Paul Zuradzki—and others who did not want to be named, sent corrections after I delivered the draft, or are omitted because I failed to record their names—sorry. During my research, I learned about typing, concurrency, pattern matching, and metaprogramming while interacting with Michael Albert, Pablo Aguilar, Kaleb Bar‐ rett, David Beazley, J. S. O. Bueno, Bruce Eckel, Martin Fowler, Ivan Levkivskyi, Alex Martelli, Peter Norvig, Sebastian Rittau, Guido van Rossum, Carol Willing, and Jelle Zijlstra. O’Reilly editors Jeff Bleiel, Jill Leonard, and Amelia Blevins made suggestions that improved the flow of the book in many places. Jeff Bleiel and production editor Danny Elfanbaum supported me throughout this long marathon. The insights and suggestions of every one of them made the book better and more accurate. Inevitably, there will still be bugs of my own creation in the final product. I apologize in advance. Finally, I want to extend my heartfelt thanks to my colleagues at Thoughtworks Brazil —and especially to my sponsor, Alexey Bôas—who supported this project in many ways, all the way. Of course, everyone who helped me understand Python and write the first edition now deserves double thanks. There would be no second edition without a successful first. xxvi | Preface Acknowledgments for the First Edition The Bauhaus chess set by Josef Hartwig is an example of excellent design: beautiful, simple, and clear. Guido van Rossum, son of an architect and brother of a master font designer, created a masterpiece of language design. I love teaching Python because it is beautiful, simple, and clear. Alex Martelli and Anna Ravenscroft were the first people to see the outline of this book and encouraged me to submit it to O’Reilly for publication. Their books taught me idiomatic Python and are models of clarity, accuracy, and depth in technical writ‐ ing. Alex’s 6,200+ Stack Overflow posts are a fountain of insights about the language and its proper use. Martelli and Ravenscroft were also technical reviewers of this book, along with Len‐ nart Regebro and Leonardo Rochael. Everyone in this outstanding technical review team has at least 15 years of Python experience, with many contributions to high- impact Python projects in close contact with other developers in the community. Together they sent me hundreds of corrections, suggestions, questions, and opinions, adding tremendous value to the book. Victor Stinner kindly reviewed Chapter 21, bringing his expertise as an asyncio maintainer to the technical review team. It was a great privilege and a pleasure to collaborate with them over these past several months. Editor Meghan Blanchette was an outstanding mentor, helping me improve the orga‐ nization and flow of the book, letting me know when it was boring, and keeping me from delaying even more. Brian MacDonald edited chapters in Part II while Meghan was away. I enjoyed working with them, and with everyone I’ve contacted at O’Reilly, including the Atlas development and support team (Atlas is the O’Reilly book pub‐ lishing platform, which I was fortunate to use to write this book). Mario Domenech Goulart provided numerous, detailed suggestions starting with the first early release. I also received valuable feedback from Dave Pawson, Elias Dor‐ neles, Leonardo Alexandre Ferreira Leite, Bruce Eckel, J. S. Bueno, Rafael Gonçalves, Alex Chiaranda, Guto Maia, Lucas Vido, and Lucas Brunialti. Over the years, a number of people urged me to become an author, but the most per‐ suasive were Rubens Prates, Aurelio Jargas, Rudá Moura, and Rubens Altimari. Mauricio Bussab opened many doors for me, including my first real shot at writing a book. Renzo Nuccitelli supported this writing project all the way, even if that meant a slow start for our partnership at python.pro.br. The wonderful Brazilian Python community is knowledgeable, generous, and fun. The Python Brasil group has thousands of people, and our national conferences bring together hundreds, but the most influential in my journey as a Pythonista were Leo‐ nardo Rochael, Adriano Petrich, Daniel Vainsencher, Rodrigo RBP Pimentel, Bruno Gola, Leonardo Santagada, Jean Ferri, Rodrigo Senra, J. S. Bueno, David Kwast, Luiz Preface | xxvii Irber, Osvaldo Santana, Fernando Masanori, Henrique Bastos, Gustavo Niemayer, Pedro Werneck, Gustavo Barbieri, Lalo Martins, Danilo Bellini, and Pedro Kroger. Dorneles Tremea was a great friend (incredibly generous with his time and knowl‐ edge), an amazing hacker, and the most inspiring leader of the Brazilian Python Association. He left us too early. My students over the years taught me a lot through their questions, insights, feed‐ back, and creative solutions to problems. Érico Andrei and Simples Consultoria made it possible for me to focus on being a Python teacher for the first time. Martijn Faassen was my Grok mentor and shared invaluable insights with me about Python and Neanderthals. His work and that of Paul Everitt, Chris McDonough, Tres Seaver, Jim Fulton, Shane Hathaway, Lennart Regebro, Alan Runyan, Alexander Limi, Martijn Pieters, Godefroid Chapelle, and others from the Zope, Plone, and Pyr‐ amid planets have been decisive in my career. Thanks to Zope and surfing the first web wave, I was able to start making a living with Python in 1998. José Octavio Cas‐ tro Neves was my partner in the first Python-centric software house in Brazil. I have too many gurus in the wider Python community to list them all, but besides those already mentioned, I am indebted to Steve Holden, Raymond Hettinger, A.M. Kuchling, David Beazley, Fredrik Lundh, Doug Hellmann, Nick Coghlan, Mark Pil‐ grim, Martijn Pieters, Bruce Eckel, Michele Simionato, Wesley Chun, Brandon Craig Rhodes, Philip Guo, Daniel Greenfeld, Audrey Roy, and Brett Slatkin for teaching me new and better ways to teach Python. Most of these pages were written in my home office and in two labs: CoffeeLab and Garoa Hacker Clube. CoffeeLab is the caffeine-geek headquarters in Vila Madalena, São Paulo, Brazil. Garoa Hacker Clube is a hackerspace open to all: a community lab where anyone can freely try out new ideas. The Garoa community provided inspiration, infrastructure, and slack. I think Aleph would enjoy this book. My mother, Maria Lucia, and my father, Jairo, always supported me in every way. I wish he was here to see the book; I am glad I can share it with her. My wife, Marta Mello, endured 15 months of a husband who was always working, but remained supportive and coached me through some critical moments in the project when I feared I might drop out of the marathon. Thank you all, for everything. xxviii | Preface  PART I Data Structures  CHAPTER 1 The Python Data Model Guido’s sense of the aesthetics of language design is amazing. I’ve met many fine lan‐ guage designers who could build theoretically beautiful languages that no one would ever use, but Guido is one of those rare people who can build a language that is just slightly less theoretically beautiful but thereby is a joy to write programs in. —Jim Hugunin, creator of Jython, cocreator of AspectJ, and architect of the.Net DLR1 One of the best qualities of Python is its consistency. After working with Python for a while, you are able to start making informed, correct guesses about features that are new to you. However, if you learned another object-oriented language before Python, you may find it strange to use len(collection) instead of collection.len(). This apparent oddity is the tip of an iceberg that, when properly understood, is the key to every‐ thing we call Pythonic. The iceberg is called the Python Data Model, and it is the API that we use to make our own objects play well with the most idiomatic language features. You can think of the data model as a description of Python as a framework. It formal‐ izes the interfaces of the building blocks of the language itself, such as sequences, functions, iterators, coroutines, classes, context managers, and so on. When using a framework, we spend a lot of time coding methods that are called by the framework. The same happens when we leverage the Python Data Model to build new classes. The Python interpreter invokes special methods to perform basic object operations, often triggered by special syntax. The special method names are always written with leading and trailing double underscores. For example, the syntax 1 “Story of Jython”, written as a foreword to Jython Essentials by Samuele Pedroni and Noel Rappin (O’Reilly). 3 obj[key] is supported by the __getitem__ special method. In order to evaluate my_collection[key], the interpreter calls my_collection.__getitem__(key). We implement special methods when we want our objects to support and interact with fundamental language constructs such as: Collections Attribute access Iteration (including asynchronous iteration using async for) Operator overloading Function and method invocation String representation and formatting Asynchronous programming using await Object creation and destruction Managed contexts using the with or async with statements Magic and Dunder The term magic method is slang for special method, but how do we talk about a specific method like __getitem__? I learned to say “dunder-getitem” from author and teacher Steve Holden. “Dun‐ der” is a shortcut for “double underscore before and after.” That’s why the special methods are also known as dunder methods. The “Lexical Analysis” chapter of The Python Language Reference warns that “Any use of __*__ names, in any context, that does not follow explicitly documented use, is subject to breakage without warning.” What’s New in This Chapter This chapter had few changes from the first edition because it is an introduction to the Python Data Model, which is quite stable. The most significant changes are: Special methods supporting asynchronous programming and other new features, added to the tables in “Overview of Special Methods” on page 15. Figure 1-2 showing the use of special methods in “Collection API” on page 14, including the collections.abc.Collection abstract base class introduced in Python 3.6. 4 | Chapter 1: The Python Data Model Also, here and throughout this second edition I adopted the f-string syntax intro‐ duced in Python 3.6, which is more readable and often more convenient than the older string formatting notations: the str.format() method and the % operator. One reason to still use my_fmt.format() is whe